I'm trying to create a pipeline task using beam java SDK and Google Dataflow to move data from Cloud SQL to Elastic search
I've created the following class main method:
public static void main(String[] args) throws Exception{
DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
options.setProject("staging");
options.setTempLocation("gs://csv_to_sql_staging/temp");
options.setRunner(DataflowRunner.class);
options.setGcpTempLocation("gs://csv_to_sql_staging/temp");
options.setUsePublicIps(false);
options.setJobName("tamer-new");
options.setSubnetwork("regions/us-central1/subnetworks/new-network");
final List<String> SCOPES = Arrays.asList(
"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/devstorage.full_control",
"https://www.googleapis.com/auth/userinfo.email",
"https://www.googleapis.com/auth/datastore",
"https://www.googleapis.com/auth/sqlservice.admin",
"https://www.googleapis.com/auth/pubsub");
options.setGcpCredential(ServiceAccountCredentials.fromStream(new ElasticSearchIO().getClass().getResourceAsStream("/staging-b648da5d2b9b.json")).createScoped(SCOPES)); options.setServiceAccount("data-flow#staging.iam.gserviceaccount.com");
Pipeline p = Pipeline.create(options);
p.begin();
PCollection < List < String >> rows = p.apply(JdbcIO. < List < String >> read().withQuery("select u.id, u.name from user_table").withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create("com.mysql.jdbc.Driver", "jdbc:mysql://google/nameDB_new?cloudSqlInstance=staging:europe-west1:sql-staging-instance&socketFactory=com.google.cloud.sql.mysql.SocketFactory&useUnicode=true&characterEncoding=UTF-8&user=user&password=password&useSSL=false")).withRowMapper(new RowMapper < List < String >> () {
#Override public List < String > mapRow(ResultSet resultSet) throws Exception {
List < String > addRow = new ArrayList < String > ();
for (int i = 1; i <= resultSet.getMetaData().getColumnCount(); i++) {
addRow.add(i - 1, String.valueOf(resultSet.getObject(i)));
}
//LOG.info(String.join(",", addRow));
return addRow;
}
})
.withCoder(ListCoder.of(StringUtf8Coder. < Object > of ()))
);
Write w = ElasticsearchIO.write().withConnectionConfiguration(
ElasticsearchIO.ConnectionConfiguration.create(new String[] {
"https://host:9243"
}, "user-temp", "String").withUsername("elastic").withPassword("password")
);
rows.apply(w.compose(new SerializableFunction() {
#Override public Object apply(Object input) {
// TODO Auto-generated method stub
return input;
}
}));
p.run().waitUntilFinish();
}
and below is the pom.xml file :
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.harmonica.dataflow</groupId>
<artifactId>com-harmonica-dataflow</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<maven-compiler-plugin.version>3.7.0</maven-compiler-plugin.version>
<exec-maven-plugin.version>1.6.0</exec-maven-plugin.version>
<slf4j.version>1.7.25</slf4j.version>
<beam.version>2.19.0</beam.version>
</properties>
<repositories>
<repository>
<id>ossrh.snapshots</id>
<name>Sonatype OSS Repository Hosting</name>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
<releases>
<enabled>false</enabled>
</releases>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>${maven-compiler-plugin.version}</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>${exec-maven-plugin.version}</version>
<configuration>
<cleanupDaemonThreads>false</cleanupDaemonThreads>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
<dependencies>
<!-- Beam Lib -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-elasticsearch</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-jdbc</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>8.0.19</version>
</dependency>
<dependency>
<groupId>com.google.cloud.sql</groupId>
<artifactId>mysql-socket-factory-connector-j-8</artifactId>
<version>1.0.15</version>
</dependency>
<!-- slf4j API frontend binding with JUL backend -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>${slf4j.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>${slf4j.version}</version>
</dependency>
</dependencies>
</project>
and when I execute this command:
man exec mvn compile exec:java -Dexec.mainClass=com.dataflow.ElasticSearchIO
The worker started successfully but then cant connect to the the Cloud SQL:
even thought I've done the flowing:
I've created a service account with owner access to the project and passed it to the runner options
I've created a VPC network with a name of new-network with a range of IP of 190.10.0.0/16 and assigned it to the pipeline options and then whitlisted this range in the cloud SQL
and however Im still getting this error:
Error message from worker: java.lang.RuntimeException:
org.apache.beam.sdk.util.UserCodeException: java.sql.SQLException:
Cannot create PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds
ago. The driver has not received any packets from the server.)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:194)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:165)
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:125)
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:352)
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748) Caused by:
org.apache.beam.sdk.util.UserCodeException: java.sql.SQLException:
Cannot create PoolableConnectionFactory (Communications link failure
The last packet sent successfully to the server was 0 milliseconds
ago. The driver has not received any packets from the server.)
org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:34)
org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn$DoFnInvoker.invokeSetup(Unknown
Source)
org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:80)
org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:62)
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:95)
org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:264)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:86)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:183)
... 14 more Caused by: java.sql.SQLException: Cannot create
PoolableConnectionFactory (Communications link failure The last packet
sent successfully to the server was 0 milliseconds ago. The driver has
not received any packets from the server.)
org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:735)
org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:605)
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:809)
org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn.setup(JdbcIO.java:881)
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException:
Communications link failure The last packet sent successfully to the
server was 0 milliseconds ago. The driver has not received any packets
from the server.
com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:174)
com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:456)
com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:53)
org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:355)
org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:116)
org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:731)
org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:605)
org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:809)
org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn.setup(JdbcIO.java:881)
org.apache.beam.sdk.io.jdbc.JdbcIO$ReadFn$DoFnInvoker.invokeSetup(Unknown
Source)
org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:80)
org.apache.beam.runners.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:62)
org.apache.beam.runners.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:95)
org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:264)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:86)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:183)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:165)
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:125)
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:352)
org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:305)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748) Caused by:
com.mysql.cj.exceptions.CJCommunicationsException: Communications link
failure The last packet sent successfully to the server was 0
milliseconds ago. The driver has not received any packets from the
server. sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
com.mysql.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:91)
com.mysql.cj.NativeSession.connect(NativeSession.java:144)
com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:956)
com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:826)
... 32 more Caused by: java.net.ConnectException: Connection timed out
(Connection timed out) java.net.PlainSocketImpl.socketConnect(Native
Method)
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
java.net.Socket.connect(Socket.java:589)
sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173)
com.google.cloud.sql.core.CoreSocketFactory.createSslSocket(CoreSocketFactory.java:233)
com.google.cloud.sql.core.CoreSocketFactory.connect(CoreSocketFactory.java:185)
com.google.cloud.sql.mysql.SocketFactory.connect(SocketFactory.java:48)
com.google.cloud.sql.mysql.SocketFactory.connect(SocketFactory.java:38)
com.mysql.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:65)
... 35 more
Plz any help will be highly appreciated!
Thanks in advance
You can use below piece of code to establish the connection:
Pipeline p = Pipeline.create(options);
//Increase pool size based on your records
ComboPooledDataSource dataSource = new ComboPooledDataSource();
dataSource.setDriverClass("com.mysql.jdbc.Driver");
dataSource.setJdbcUrl(
"jdbc:mysql://google/test?cloudSqlInstance=dataflowtest-:us-central1:sql-test&socketFactory=com.google.cloud.sql.mysql.SocketFactory");
dataSource.setUser("root");
dataSource.setPassword("root");
dataSource.setMaxPoolSize(10);
dataSource.setInitialPoolSize(6);
JdbcIO.DataSourceConfiguration config = JdbcIO.DataSourceConfiguration.create(dataSource);
// ADD rewriteBatchedStatements=true to improve write speed"
PCollection<KV<String, String>> sqlResult = p.apply(JdbcIO.<KV<String, String>>read()
.withDataSourceConfiguration(config)
.withQuery("select * from test_table").withCoder(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()))
.withRowMapper(new JdbcIO.RowMapper<KV<String, String>>() {
private static final long serialVersionUID = 1L;
public KV<String, String> mapRow(ResultSet resultSet) throws Exception {
return KV.of(resultSet.getString(1), resultSet.getString(2));
}
}));
Add below dependency in pom.xml
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-jdbc</artifactId>
<version>2.17.0</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.25</version>
</dependency>
<dependency>
<groupId>com.google.cloud.sql</groupId>
<artifactId>mysql-socket-factory</artifactId>
<version>1.0.0</version>
</dependency>
This should work..
If possible try the below code for sql connection:
connection = connectToCloudSql(map.get(LiteralConstant.URL.toString()),
map.get(LiteralConstant.USERNAME.toString()), map.get(LiteralConstant.PASSWORD.toString()));
Then use the below piece of code to get result from sql connection:
statement = connection.prepareCall("query");
statement.execute();
resultSet = statement.getResultSet();
ResultSetMetaData rsmd = resultSet.getMetaData();
int count = rsmd.getColumnCount();
if(!resultSet.next() || count < 1)
throw new ConnectionFailureException("Failed to connect to Cloud SQL");
for (int k = 1; k <= count; k++) {
row.set(rsmd.getColumnName(k), resultSet.getString(k));
}
Get the above result in PCollection
Note: Don't forget to enable Cloud sql api and Cloud sql admin api.
Maven dependency:
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.25</version>
</dependency>
<dependency>
<groupId>com.google.cloud.sql</groupId>
<artifactId>mysql-socket-factory</artifactId> <!-- mysql-socket-factory-connector-j-6 if using 6.x.x -->
<version>1.0.0</version>
</dependency>
This above piece of code worked in my case. Let me know in case this solution works for you.
Related
After starting the service discovery, When the proxy microservice is registered, the eureka is throwing the following error:
2020-06-11 21:07:19.778 ERROR 1040 --- [et_localhost-19] c.n.e.cluster.ReplicationTaskProcessor : Batch update failure with HTTP status code 404; discarding 1 replication tasks
2020-06-11 21:07:19.778 WARN 1040 --- [et_localhost-19] c.n.eureka.util.batcher.TaskExecutors : Discarding 1 tasks of TaskBatchingWorker-target_localhost-19 due to permanent error
2020-06-11 21:07:49.782 ERROR 1040 --- [et_localhost-19] c.n.e.cluster.ReplicationTaskProcessor : Batch update failure with HTTP status code 404; discarding 1 replication tasks
2020-06-11 21:07:49.782 WARN 1040 --- [et_localhost-19] c.n.eureka.util.batcher.TaskExecutors : Discarding 1 tasks of TaskBatchingWorker-target_localhost-19 due to permanent error
2020-06-11 21:08:11.629 INFO 1040 --- [a-EvictionTimer] c.n.e.registry.AbstractInstanceRegistry : Running the evict task with compensationTime 0ms
2020-06-11 21:08:19.796 ERROR 1040 --- [et_localhost-19] c.n.e.cluster.ReplicationTaskProcessor : Batch update failure with HTTP status code 404; discarding 1 replication tasks
2020-06-11 21:08:19.796 WARN 1040 --- [et_localhost-19] c.n.eureka.util.batcher.TaskExecutors : Discarding 1 tasks of TaskBatchingWorker-target_localhost-19 due to permanent error
Error Log
The dependencies are as follows:
<properties>
<java.version>1.8</java.version>
<spring-cloud.version>Hoxton.SR3</spring-cloud.version>
<springfox-swagger2-version>2.9.2</springfox-swagger2-version>
<springfox-swagger2-ui-version>2.9.2</springfox-swagger2-ui-version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-zuul</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-config</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
bootstrap.yml
spring:
application:
name: tng-crs-eureka
server:
port: 8761
servlet:
context-path: /eureka
eureka:
client:
registerWithEureka: false
fetchRegistry: false
Any help is highly appreciated. I have tried by disabling the batch update tasks also. But, it didn't worked.
Finally, The mistake that I have made was figured out. I have set the context-path of /eureka. And that is the reason for all this trouble. Removing the context path of eureka resolves the issue for my case.
I exported DataFlow application to Runnable JAR file from Eclipse.
I get below exception when I run the command Desktop$java -jar mariadevconn.jar
Why is the current working directory of command line added to the Google Storage path? Thank you for your help.
org.apache.beam.runners.dataflow.options.DataflowPipelineOptions$StagingLocationFactory create
INFO: No stagingLocation provided, falling back to gcpTempLocation
java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:52)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:142)
at com.masterconcept.cloud.dataflow.connector.MariaDBConnector.main(MariaDBConnector.java:98)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more
Caused by: java.lang.IllegalArgumentException: Expected a valid 'gs://' path but was given '/Users/henry/Desktop/gs:/hellodataflowfromhenry/temp/staging/'
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.getGcsPath(GcsPathValidator.java:101)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPath(GcsPathValidator.java:75)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:60)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:237)
... 9 more
Caused by: java.lang.IllegalArgumentException: Invalid GCS URI: /Users/henry/Desktop/gs:/hellodataflowfromhenry/temp/staging/
at org.apache.beam.sdks.java.extensions.google.cloud.platform.core.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:191)
at org.apache.beam.sdk.util.gcsfs.GcsPath.fromUri(GcsPath.java:116)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.getGcsPath(GcsPathValidator.java:99)
... 12 more
Part of the code:
DataflowPipelineOptions options = PipelineOptionsFactory.fromArgs(argv).withValidation().as(DataflowPipelineOptions.class);
options.setGcpTempLocation("gs://hellodataflowfromhenry/temp");
options.setRunner(DataflowRunner.class);
....
pom.xml:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.2.0</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
<version>2.2.0</version>
</dependency>
work directory:
Desktop$pwd
/Users/henry/Desktop
I am using Neo4J 3.2.0 community edition and I was able to connect and establish connection properly using traditional java driver and running Cypher queries.
However, I wanted to create rich data model, so I imported the following dependencies:
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-ogm-core</artifactId>
<version>2.1.3</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-ogm-bolt-driver</artifactId>
<version>2.1.3</version>
</dependency>
Post that, I wrote the following code to establish the connection:
Configuration configuration = new Configuration();
configuration.driverConfiguration().setDriverClassName("org.neo4j.ogm.drivers.bolt.driver.BoltDriver")
.setURI("bolt://localhost:7687");
SessionFactory sessionFactory = new SessionFactory(configuration, "org.neo4j.example.domain");
Session session = sessionFactory.openSession();
But irrespective of weather my Neo4j server is turned on or not, it establish connection, and the logs dont show any errors, just the following three lines for the logging system:
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Any help would be appreciated.
Adding the following dependency fixes it:
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.25</version>
</dependency>
I am following the simple-weather tutorial in Maven by Example. When I execute the program, I get below exception. Does this need any specific classpath setting?
POM
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.example.maven.weather</groupId>
<artifactId>simple-weather</artifactId>
<version>1.0</version>
<packaging>jar</packaging>
<name>simple-weather</name>
<url>http://maven.apache.org</url>
<licenses>
<license>
<name>Apache 2</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
<comments>A business-friendly OSS license</comments>
</license>
</licenses>
<organization>
<name>Sonatype</name>
<url>http://www.sonatype.com</url>
</organization>
<developers>
<developer>
<id>jason</id>
<name>Jason Van Zyl</name>
<email>jason#maven.org</email>
<url>http://www.sonatype.com</url>
<organization>Sonatype</organization>
<organizationUrl>http://www.sonatype.com</organizationUrl>
<roles>
<role>developer</role>
</roles>
<timezone>-6</timezone>
</developer>
</developers>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.14</version>
</dependency>
<dependency>
<groupId>dom4j</groupId>
<artifactId>dom4j</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>jaxen</groupId>
<artifactId>jaxen</artifactId>
<version>1.1.1</version>
</dependency>
<dependency>
<groupId>velocity</groupId>
<artifactId>velocity</artifactId>
<version>1.5</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
Main.java
package com.example.maven.weather;
import java.io.InputStream;
import org.apache.log4j.PropertyConfigurator;
public class Main {
public static void main(String[] args) throws Exception {
// Configure Log4J
PropertyConfigurator.configure(Main.class.getClassLoader()
.getResource("log4j.properties"));
// Read the Zip Code from the Command-line (if none supplied, use 60202)
String zipcode = "60202";
try {
zipcode = args[0];
} catch( Exception e ) {}
// Start the program
new Main(zipcode).start();
}
private String zip;
public Main(String zip) {
this.zip = zip;
}
public void start() throws Exception {
// Retrieve Data
InputStream dataIn = new YahooRetriever().retrieve( zip );
// Parse Data
Weather weather = new YahooParser().parse( dataIn );
// Format (Print) Data
System.out.print( new WeatherFormatter().format( weather ) );
}
}
Run:
mvn exec:java -Dexec.mainClass=com.exa
mple.maven.weather.Main
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building simple-weather 1.0
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> exec-maven-plugin:1.2.1:java (default-cli) # simple-weather >>>
[INFO]
[INFO] <<< exec-maven-plugin:1.2.1:java (default-cli) # simple-weather <<<
[INFO]
[INFO] --- exec-maven-plugin:1.2.1:java (default-cli) # simple-weather ---
[WARNING]
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurato
r.java:433)
at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.
java:336)
at com.example.maven.weather.Main.main(Main.java:12)
... 6 more
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.902s
[INFO] Finished at: Thu May 30 14:23:06 IST 2013
[INFO] Final Memory: 4M/15M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java (d
efault-cli) on project simple-weather: An exception occured while executing the
Java class. null: InvocationTargetException: NullPointerException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit
ch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please rea
d the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE
xception
It seems like your application can't find log4j.properties
Do you use standard maven project structure? Where is log4j.properties located?
'resources' folder should be in main folder (src/main/resources) instead of src/resources .
I have a simple EJB 3.1 project which uses Glassfish 3.1.2 as AS and Maven 3 for dependency management. In the pom.xml of ejb project I set the generateClient option to true. I successfully deployed my ear to the server and then I tried to create a simple standalone client. Here is the pom.xml of the client:
<project>
<modelVersion>4.0.0</modelVersion>
<groupId>duan</groupId>
<artifactId>ejb31-app-client</artifactId>
<version>1.0-SNAPSHOT</version>
<name>ejb31-app-client</name>
<description>my app client</description>
<dependencies>
<dependency>
<groupId>org.glassfish.main.appclient.client</groupId>
<artifactId>gf-client-module</artifactId>
<version>3.1.2</version>
<type>pom</type>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tools</artifactId>
<groupId>com.sun</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>duan</groupId>
<artifactId>ejb31</artifactId>
<version>1.0-SNAPSHOT</version>
<type>ejb-client</type>
</dependency>
</dependencies>
where ejb31 is my ejb project for which I client is generated.
The Java class used in my application client is:
public class Test {
private static HelloWorldBeanRemote helloWorldBean;
public static void main(String[] args) {
String jndiPath = "java:global/ejb31-ear-1.0/ejb31-ejb/HelloWorldBean";
try {
Context ctx = new InitialContext();
System.out.println("Looking up bean at: " + jndiPath);
helloWorldBean = (HelloWorldBeanRemote) ctx.lookup(jndiPath);
System.out.println("Found helloWorldBean:" + helloWorldBean);
System.out.println("Calling sayHello():");
String greeting = helloWorldBean.sayHello();
System.out.println("HelloWorldBean said:" + greeting);
} catch (NamingException e) {
System.err.println("Could not find HelloWorldBeanRemote!");
System.err.println("JNDI path used for lookup:" + jndiPath);
e.printStackTrace();
}
}
}
The application client is compiled without problem by Maven. If I run this from Eclipse I get this exception:
Caused by: java.lang.ClassNotFoundException: ro.duan.ejb.HelloWorldBeanRemote
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at com.sun.ejb.EJBUtils.getBusinessIntfClassLoader(EJBUtils.java:688)
at com.sun.ejb.EJBUtils.loadGeneratedRemoteBusinessClasses(EJBUtils.java:463)
at com.sun.ejb.EJBUtils.lookupRemote30BusinessObject(EJBUtils.java:414)
... 7 more
But then if I manually add ejb-client.jar to the build path it works like a charm. So my conclusion is that somehow ejb-client.jar is available at compile time, but not at runtime. Any thoughts on how to solve this?
Remove the following line:
<scope>compile</scope>
Read more about dependency scope at:
http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Scope