StarRocks Does not read the hdfs correctly by hdfs-site.xml & core-site.xml - starrocks

I start a hive with spark thriftserver and set the hdfs as the storage.
And I copied the hdfs-site.xml and core-site.xml
core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://x.x.x.x:9000</value>
</property>
</configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
create a resource on starrocks
+-------+--------------+---------------------+---------------------------+
| Name | ResourceType | Key | Value |
+-------+--------------+---------------------+---------------------------+
| hive0 | hive | hive.metastore.uris | thrift://x.x.x.x:9083 |
+-------+--------------+---------------------+---------------------------+
create an external table
CREATE EXTERNAL TABLE `hive_table` (
`id` int(11) NULL COMMENT ""
) ENGINE=HIVE
COMMENT "HIVE"
PROPERTIES (
"database" = "rowdata",
"table" = "ht",
"resource" = "hive0",
"hive.metastore.uris" = "thrift://x.x.x.x:9083"
);
try to query the table got an error.
ERROR 1064 (HY000): Failed to get remote files, msg: com.starrocks.connector.exception.StarRocksConnectorException: Failed to get hive remote file's metadata on path: RemotePathKey{path='file:/opt/zw/spark-3.3.1-bin-hadoop3/sbin/spark-warehouse/rowdata.db/ht', isRecursive=false}. msg: File /opt/zw/spark-3.3.1-bin-hadoop3/sbin/spark-warehouse/rowdata.db/ht does not exist
It seems does not read the hdfs directly.
the starrocks version is 2.5.0.
Anyone know the reason?

I found out the reason. That's my hive configuration error. I found the hive meta is incorrect in the DBS table. the path is incorrect. After fixed that it works well now.

Related

Apache Flume agent not starting but not showing error

I'm attempting to run an Apache Flume agent from an AWS EC2 cluster but when I start the agent, it neither starts nor throws an obvious error.
I'm just starting with the simple example from Apache's documentation.
When I run:
ubuntu#ip-172-31-41-5:~/Flume$ ./bin/flume-ng agent --conf conf --conf-file example.conf --name a1 -Dflume.root.logger=DEBUG,console
The console output is the following:
Info: Sourcing environment configuration script /home/ubuntu/Flume/conf/flume-env.sh
Info: Including HBASE libraries found via (/home/ubuntu/hbase-2.4.4/bin/hbase) for HBASE access
Info: Including Hive libraries found via () for Hive access
+ exec /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Xmx20m -Dflume.root.logger=DEBUG,console -Dflume.root.logger=DEBUG,console -cp '/home/ubuntu/Flume/conf:/home/ubuntu/Flume/lib/*:/home/ubuntu/hbase-2.4.4/conf:/usr/lib/jvm/java-8-openjdk-amd64/lib/tools.jar:/home/ubuntu/hbase-2.4.4:/home/ubuntu/hbase-2.4.4/lib/shaded-clients/hbase-shaded-client-2.4.4.jar:/home/ubuntu/hbase-2.4.4/lib/client-facing-thirdparty/audience-annotations-0.5.0.jar:/home/ubuntu/hbase-2.4.4/lib/client-facing-thirdparty/commons-logging-1.2.jar:/home/ubuntu/hbase-2.4.4/lib/client-facing-thirdparty/htrace-core4-4.2.0-incubating.jar:/home/ubuntu/hbase-2.4.4/lib/client-facing-thirdparty/log4j-1.2.17.jar:/home/ubuntu/hbase-2.4.4/lib/client-facing-thirdparty/slf4j-api-1.7.30.jar:/home/ubuntu/hbase-2.4.4/conf:/lib/*' -Djava.library.path=:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib org.apache.flume.node.Application --conf-file example.conf --name a1 ./bin/flume-ng agent --conf-file example.conf --name a1
The agent doesn't throw an error but never gets further than this. I have also tried some variations including --conf-file conf/example.conf
Flume and Java appear to be installed correctly:
Flume
ubuntu#ip-172-31-41-5:~/Flume$ ./bin/flume-ng version
Source code repository: https://git.apache.org/repos/asf/flume.git
Revision: 1a15927e594fd0d05a59d804b90a9c31ec93f5e1
Compiled by rgoers on Sun Oct 16 14:44:15 MST 2022
From source with checksum bbbca682177262aac3a89defde369a37
Java
ubuntu#ip-172-31-41-5:~/Flume$ java -version
openjdk version "11.0.17" 2022-10-18
OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu222.04)
OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu222.04, mixed mode, sharing)
Example.conf
# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
Flume.env
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced
# during Flume startup.
# Enviroment variables can be set here.
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
# Give Flume more memory and pre-allocate, enable remote monitoring via JMX
# export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote"
# Let Flume write raw event data and configuration information to its log files for debugging
# purposes. Enabling these flags is not recommended in production,
# as it may result in logging sensitive user information or encryption secrets.
# export JAVA_OPTS="$JAVA_OPTS -Dorg.apache.flume.log.rawdata=true -Dorg.apache.flume.log.printconfig=true "
# Note that the Flume conf directory is always included in the classpath.
#FLUME_CLASSPATH=""
The only clue that I have is is in flume.log which shows the following error. I've even copied example.conf into the main Flume directory but it doesn't seem to make a difference.
03 Dec 2022 21:10:03,538 ERROR [main] (org.apache.flume.node.Application.main:506) - A fatal error occurred while running. Exception follows.org.apache.flume.conf.ConfigurationException: Unable to read file /home/ubuntu/Flume/example.confat org.apache.flume.node.FileConfigurationSource.<init>(FileConfigurationSource.java:52) ~[flume-ng-node-1.11.0.jar:1.11.0]at org.apache.flume.node.FileConfigurationSourceFactory.createConfigurationSource(FileConfigurationSourceFactory.java:40) ~[flume-ng-node-1.11.0.jar:1.11.0]at org.apache.flume.node.ConfigurationSourceFactory.getConfigurationSource(ConfigurationSourceFactory.java:39) ~[flume-ng-node-1.11.0.jar:1.11.0]at org.apache.flume.node.Application.main(Application.java:476) ~[flume-ng-node-1.11.0.jar:1.11.0]Caused by: java.nio.file.NoSuchFileException: /home/ubuntu/Flume/example.confat sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:1.8.0_352]at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_352]at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_352]at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:1.8.0_352]at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_352]at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_352]at java.nio.file.Files.readAllBytes(Files.java:3152) ~[?:1.8.0_352]at org.apache.flume.node.FileConfigurationSource.<init>(FileConfigurationSource.java:49) ~[flume-ng-node-1.11.0.jar:1.11.0]... 3 more

GLib-GIO-ERROR **: Settings schema 'com.github.mfru.vala-todo' is not installed

First off, I am new to Vala, Gtk and Meson, so I might be looking in the totally wrong directions.
I'm running on Pop_OS! 20.10.
I want to compile a small example Vala application with support for GSettings.
However, when compiling with ninja, installing with sudo ninja install and then executing I get the error mentioned in the title.
It seems that even though my gschema file is copied into /usr/share/glib-2.0/schemas, running glib-compile-schemas in my post-install script is not doing anything.
I could also confirm that by running glib-compile-schemas manually and then checking with dconf Editor whether my settings are available which they aren't.
data/gschema.xml
<?xml version="1.0" encoding="UTF-8"?>
<schemalist>
<schema path="/com/github/mfru/vala-todo" id="com.github.mfru.vala-todo">
<key name="pos-x" type="i">
<default>360</default>
<summary>Horizontal position</summary>
<description>The saved horizontal position of our window</description>
</key>
<key name="pos-y" type="i">
<default>360</default>
<summary>Vertical position</summary>
<description>The saved vertical position of our window</description>
</key>
<key name="window-width" type="i">
<default>600</default>
<summary>Window width</summary>
<description>The saved width of our window</description>
</key>
<key name="window-height" type="i">
<default>400</default>
<summary>Window height</summary>
<description>The saved height of our window</description>
</key>
</schema>
</schemalist>
data/meson.build
install_data(
'gschema.xml',
install_dir: join_paths (get_option ('datadir'), 'glib-2.0', 'schemas'),
rename: meson.project_name () + '.geschema.xml'
)
meson/post-install.py
#!/usr/bin/env python3
import os
import subprocess
schemadir = os.path.join(os.environ['MESON_INSTALL_PREFIX'], 'share', 'glib-2.0', 'schemas')
print("Schemadir is", schemadir)
if 'DESTDIR' not in os.environ:
print("Compiling the gsettings schemas...")
subprocess.call(['glib-compile-schemas', schemadir])
I also reconfigured the meson prefix with meson build --prefix=/usr --reconfigure as at first the Prefix pointed to /usr/local but all existing schemas on my system are under /usr/share
Finally, what I expect to happen is that my example application starts without crashing because of missing GSettings.
Resources I looked into:
https://mesonbuild.com/Configuring-a-build-directory.html
https://developer.gnome.org/gio/stable/glib-compile-schemas.html
Is there any way to create GSettings schema during installation in Vala?
You have a typo in your build.meson file.
It should be .gschema.xml instead of .geschema.xml.

How to mount ignite data to host system

I would like to run apache-ignite in docker and I am able to do that.
But the problem is whenever I spin the image and creates the tables in ignite it stays there as long as that container is running. If I restart the container or starts the ignite image again I did not get that data. I know whenever we spins the images it always creates the new container. In my case if I want to persist the data then I need to commit and push the container so that next start I will get it.
But is there any way, where I can store ignite data on host system and whenever I start the image it will read/write the data on that location (in short volume mounting).
Can anyone please share there experience or thought with example?
Thanks.
I am using this with docker-compose and below is my docker-compose.yml file.
version: "3.7"
services:
ignite:
image: apacheignite/ignite
environment:
- IGNITE_QUIET=false
volumes:
- "./ignite-main.xml:/opt/ignite/apache-ignite/config/default-config.xml"
ports:
- 11211:11211
- 47100:47100
- 47500:47500
- 49112:49112
If I run the docker-compose up command then I get the below error.
Recreating ignite-test_ignite_1 ... done
Attaching to ignite-test_ignite_1
ignite_1 | Ignite Command Line Startup, ver. 2.7.0#20181130-sha1:256ae401
ignite_1 | 2018 Copyright(C) Apache Software Foundation
ignite_1 |
ignite_1 | class org.apache.ignite.IgniteException: Failed to instantiate Spring XML application context [springUrl=file:/opt/ignite/apache-ignite/config/default-config.xml, err=Line 1 in XML document from URL [file:/opt/ignite/apache-ignite/config/default-config.xml] is invalid; nested exception is org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 31; cvc-elt.1: Cannot find the declaration of element 'property'.]
#Update Hello, After doing lots of RnD I able to solve this issue. Below are the configuration I made 1. docker-compose.yml
version: "3.5"
services:
ignite:
image: apacheignite/ignite
environment:
- IGNITE_QUIET=false
volumes:
- ignite-persistence-1:/opt/ignite/
- "./ignite_1.xml:/opt/ignite/apache-ignite/config/default-config.xml"
ports:
- 11211:11211
- 47100:47100
- 47500:47500
- 49112:49112
deploy:
replicas: 1
restart_policy:
condition: on-failure
delay: 30s
max_attempts: 10
window: 180s
volumes:
ignite-persistence-1:
ignite_1.xml for data persistence
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">
<bean id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
<!-- Enabling Apache Ignite Persistent Store. -->
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true" />
</bean>
</property>
</bean>
</property>
<property name="workDirectory" value="/opt/ignite/apache-ignite/data" />
<!-- Explicitly configure TCP discovery SPI to provide a list of initial nodes. -->
<property name="discoverySpi">
<bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
<!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
<property name="addresses">
<list>
<!-- In distributed environment, replace with actual host IP address. -->
<value>127.0.0.1:47500..47502</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
I kept docker-compose.yml and ignite_1.xml in same directory and opened the terminal from this directory and executed the below command.
docker-compose up
By using ignite-persistence-1:/opt/ignite/ I was able to persist the data even if I stop or down the docker-compose.
I hope this will help others as well.
Thanks.

Nuget package to insert project name in web.config

I've created a Nuget package which inserts a key/value pair called ApplicationName in the web.config file with a default value of Application Name.
Is there a way to get the name of the .Net MVC project that a user would be installing the package into the value of the key/value in a human readable format? i.e. Incorrect: ApplicationName Correct: Application Name
If it's not possible to get the project name, I suppose using some sort of command line option could work?
After a few days of pondering, here is the solution I came up with.
Create a web.config transform file to add the key/value pair to the AppSettings section.
Create an install.ps1 file, that grabs the project name, parses it and injects the new value for AppplicationName in the web.config.
Here's my web.config.install.xdt file:
<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
<appSettings xdt:Transform="InsertIfMissing">
<add key="ApplicationName" value="Application Name" xdt:Transform="InsertIfMissing" xdt:Locator="Match(key)" />
</appSettings>
</configuration>
Here's is my install.ps1 script:
# Runs every time a package is installed in a project
param($installPath, $toolsPath, $package, $project)
# $installPath is the path to the folder where the package is installed.
# $toolsPath is the path to the tools directory in the folder where the package is installed.
# $package is a reference to the package object.
# $project is a reference to the project the package was installed to.
$p = Get-Project
$project_readable_name = ($p.Name -creplace '([A-Z\W_]|\d+)(?<![a-z])',' $&').trim()
# Solution based on answer found on Stackoverflow: http://stackoverflow.com/questions/6901954/can-nuget-edit-a-config-file-or-only-add-to-it
$xml = New-Object xml
# Find the web.config
$config = $project.ProjectItems | where {$_.Name -eq "Web.config"}
if($config) {
# Find web.config's path on the file system
$localPath = $config.Properties | where {$_.Name -eq "LocalPath"}
# Load Web.config as XML
$xml.Load($localPath.Value)
# Select the ApplicationName node
$node = $xml.SelectSingleNode("configuration/appSettings/add[#key='ApplicationName']")
# Change the ApplicationName value
$node.SetAttribute("value", $project_readable_name)
# Save the Web.config file
$xml.Save($localPath.Value)
}
Hope this helps anyone else!

Jaxb binding file being ignored

I having a bit of trouble building JAXB bindings fora set of schemas and its driving me a bit batty. Here is the schema in question (it is just one of may schema in this particular build):
<schema xmlns="http://www.w3.org/2001/XMLSchema" xmlns:xml="http://www.w3.org/XML/1998/namespace" targetNamespace="http://www.w3.org/XML/1998/namespace">
<attribute name="lang" type="language">
<annotation>
<documentation>In due course, we should install the relevant ISO 2- and 3-letter
codes as the enumerated possible values . . .</documentation>
</annotation>
</attribute>
<attribute name="space" default="preserve">
<simpleType>
<restriction base="NCName">
<enumeration value="default"/>
<enumeration value="preserve"/>
</restriction>
</simpleType>
</attribute>
<attributeGroup name="specialAttrs">
<attribute ref="xml:lang"/>
<attribute ref="xml:space"/>
</attributeGroup>
xjc is claiming that the attributes declared above are already declared elsewhere:
parsing a schema...
[ERROR] 'lang' is already defined
line 26 of file:../../gml/3.1.1/smil/xml-mod.xsd
[ERROR] (related to above error) the first definition appears here
line 88 of http://www.w3.org/2001/03/xml.xsd
[ERROR] 'space' is already defined
line 34 of file:../../gml/3.1.1/smil/xml-mod.xsd
[ERROR] (related to above error) the first definition appears here
line 95 of http://www.w3.org/2001/03/xml.xsd
[ERROR] 'specialAttrs' is already defined
line 39 of file:../../gml/3.1.1/smil/xml-mod.xsd
[ERROR] (related to above error) the first definition appears here
line 111 of http://www.w3.org/2001/03/xml.xsd
Failed to parse a schema.
Doing a bit of research would lead me to believe that the problem is that I need a binding file to resolve the duplicate attributes....and I have one :
<jaxb:bindings version="1.0" xmlns:jaxb="http://java.sun.com/xml/ns/jaxb" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc" jaxb:extensionBindingPrefixes="xjc">
<jaxb:bindings schemaLocation="../../gml/3.1.1/smil/xml-mod.xsd" node="/xs:schema">
<jaxb:bindings node="//xs:attribute[#name='lang']">
<jaxb:property name="langAttribute"/>
</jaxb:bindings>
<jaxb:bindings node="//xs:attribute[#name='space']">
<jaxb:property name="spaceAttribute"/>
</jaxb:bindings>
<jaxb:bindings node="//xs:attributeGroup[#name='specialAttrs']">
<jaxb:property name="specialAttrsAttribute"/>
</jaxb:bindings>
</jaxb:bindings>
Yet the problem persists. The error message with or without the bindings file is the same. I know the file is being used , because if I mess up the xpath for the node selector I get an error message. It's like xjc knows the bindings file is there but ignores it.
Here's the command line that fails to build the bindings:
C:\tools\jaxb-ri-20110512\bin\xjc -episode ..\..\..\common.ogc -d ..\..\..\src -p com.ogc.bindings -b ..\..\..\bindings.xsd -catalog ..\..\..\ogc.cat -extension sosAll.xsd
I have tried this with both the jdk6 native jaxb (2.0) and jaxb 2.2.4 (I installed the jaxb-api.jar in the endorsed directory)
XJC (bundled with JDK), ignores binding file, if the path to the binding file or the XSD / WSDL is too long.
Issue observed and confirmed on Oracle 64 bit JDK 1.6.0.45, on Windows 7 ultimate and on Linux, XJC (bundled with JDK), ignores binding file, if the path to the binding file is too long.
I did NOT have the time to do detail research on the issue; I solved my problem by simply shortening the svn checkout path. It was the most bizarre issue - I could not reproduce the problem on my Windows 8 Laptop and on Linux, but on a Windows 7 workstation, build kept on failing. I identified this issue by confirming that other then path, there is NO difference between Windows 8 LAPTOP, where the build works, vs. Windows 7 Ultimate where the path is a bit long (6 characters -> /trunk).
Problem was solved without any code, configuration, xml schema or binding file change. Posting here so that others who may run into this, have a bit of easier time then me.
i believe this is an issue with namespace collision, you dont have a namespace defined for your schema elements and it seems you are using names for your attributes that are in conflict with the ones defined in xml's schema (xml.xsd)

Resources