Added the dataflow dependency to the project. The project builds , but on start up ( using jetty ) I get a runtime exception
<dependency>
<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
<version>1.9.0</version>
</dependency>
caused by: java.lang.ClassNotFoundException: com.google.auth.http.HttpTransportFactory
at org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass(SelfFirstStrategy.java:50)
at org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass(ClassRealm.java:259)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:235)
at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass(ClassRealm.java:227)
at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:487)
at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:428)
if i remove the dependency. The start up has not issues.
Any idea why the dataflow dependency is causing startup error.
Added the exclusion for the conflicting dependency, and it works.
<dependency>
<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
<version>1.9.0</version>
<exclusions>
<exclusion>
<groupId>com.google.auth</groupId>
<artifactId>google-auth-library-oauth2-http</artifactId>
</exclusion>
</exclusions>
</dependency>
Related
When adding the below dependency to extension/guacamole-auth-jdbc-base the application's startup fails with the error ClassNotFound for error. There must be some dependency conflict but not able to understand which dependency is causing the issue.
Have tried using azure-sdk-bom to remove azure dependency version but was of no effort.
Have also tried getting the mvn dependency tree but most of the library result in been conflict with/without the azure dependency mentioned. Please help to understand how the postgres jar is build
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
<version>1.6.0</version>
</dependency>
<dependency>
<groupId>com.microsoft.graph</groupId>
<artifactId>microsoft-graph</artifactId>
<version>5.38.0</version>
</dependency>
Caused by: java.lang.ClassNotFoundException: org.apache.guacamole.auth.postgresql.PostgreSQLAuthenticationProvider
Was able to resolve the issue. The azure SDK was adding a transitive dependency in the guacamole-auth-jdbc-postgres module. Adding exclusion list for the azure sdk in childe module pom (guacamole-auth-jdbc-postgres) solved the issue.
<!-- Guacamole JDBC Authentication -->
<dependency>
<groupId>org.apache.guacamole</groupId>
<artifactId>guacamole-auth-jdbc-base</artifactId>
<version>1.3.0</version>
<exclusions>
<exclusion>
<groupId>com.azure</groupId>
<artifactId>azure-identity</artifactId>
</exclusion>
<exclusion>
<groupId>com.microsoft.graph</groupId>
<artifactId>microsoft-graph</artifactId>
</exclusion>
</exclusions>
</dependency>
I have a java service running in Openshift which involves connecting to ECS S3, read Parquet files and writes to SQL Server Database. This is a Rest service.
I have some of the following in my POM
org.springframework.boot version 2.3.5
Spring-cloud.version Hoxton.SR12
Java 1.8
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version> 3.3.1</version>
<dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version> 3.3.1</version>
<dependency>
This is working with the versions above.
However Aqua Code Scan reported Critical Vulnerabilities and to fix the those, I changed the versions as suggested by the scan tool as below.
org.springframework.boot version 2.5.12
Spring-cloud.version 2020.0.5
Java 1.8
<!-- This is added new -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<!--<version>2.9.10.4</version> -->
<version>2.12.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-aws</artifactId>
<version> 3.3.1</version>
<exclusions>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client-runtime</artifactId>
</exclusion>
<exclusion>
<groupId>org.apache.htrace</groupId>
<artifactId>htrace-core4</artifactId>
</exclusion>
<exclusion>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-bundle</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- Same kind of exclusions for hadoop-common -->
Upon changing, build succeeds, but I ended up getting the following during Deployment in Openshit
2022-05-13 13:33:20.5244200 [main] [] ERROR o.s.boot.SpringApplication - Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'configurationPropertiesBeans' defined in class path resource [org/springframework/cloud/autoconfigure/ConfigurationPropertiesRebinderAutoConfiguration.class]: Post-processing of merged bean definition failed; nested exception is java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.context.properties.ConfigurationPropertiesBeans] from ClassLoader [org.springframework.boot.loader.LaunchedURLClassLoader#42110406]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:597)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)
at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:270)
at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:762)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:567)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:775)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:448)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:339)
at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:143)
at org.springframework.cloud.bootstrap.BootstrapApplicationListener.bootstrapServiceContext(BootstrapApplicationListener.java:206)
at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:117)
at org.springframework.cloud.bootstrap.BootstrapApplicationListener.onApplicationEvent(BootstrapApplicationListener.java:74)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131)
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:82)
at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:63)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:117)
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:111)
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:62)
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:375)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:333)
at com.rbc.wmi.bor0.data.ingestion.S3IngestionApplication.main(S3IngestionApplication.java:15)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: java.lang.IllegalStateException: Failed to introspect Class [org.springframework.cloud.context.properties.ConfigurationPropertiesBeans] from ClassLoader [org.springframework.boot.loader.LaunchedURLClassLoader#42110406]
at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:485)
at org.springframework.util.ReflectionUtils.doWithLocalMethods(ReflectionUtils.java:321)
at org.springframework.orm.jpa.support.PersistenceAnnotationBeanPostProcessor.buildPersistenceMetadata(PersistenceAnnotationBeanPostProcessor.java:417)
Could anyone help please?
I got an error message saying Dataflow 2.5 (Java) is the last supported release and I should use Beam. Is there a migration guide? I can find Dataflow 1.x to 2.x but not Dataflow to Beam.
For example, DataflowPipelineOptions doesn't seem to be installed if you use the maven archetype suggested in the Beam documentation.
Specifically:
import org.apache.beam.runners.dataflow.options.DataflowPipelineOptions
is not found when I use the pom.xml generated using:
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter \
-DarchetypeVersion=2.13.0 \
-DgroupId=com.myexample \
-DartifactId=newpackage \
-Dversion="1.1" \
-DinteractiveMode=false
even after adding:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.13.0</version>
<scope>runtime</scope>
</dependency>
to the generated pom.xml.
You need a few additional Google Cloud dependencies in your pom.xml in order to run your Beam pipeline on Dataflow. Things worked for me after I added:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>${beam.version}</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>${beam.version}</version>
<exclusions>
<exclusion>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-extensions-protobuf</artifactId>
<version>${beam.version}</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-jdbc</artifactId>
<version>${beam.version}</version>
</dependency>
In addition, you may need to add a few more parameters to your startup script. I had to add:
gcpTempLocation=gs://$BUCKET/tmp
This blog post may be helpful. A user here described their migration.
I believe the package renaming (com.google.cloud.dataflow to org.apache.beam) and new class/method signatures is already done if you're on Dataflow 2.x SDKs.
So I think in this case the migration should be straightforward. Please try removing the Dataflow SDK and introducing org.apache.beam on the lastest version. It may work without modification. You could also try using org.apache.beam on 2.5 first. Then upgrading to 2.13, and see if that goes smoothly as well.
Recently, I faced with some build errors in my project which is using JetBrains/xodus v1.2.3. In project's pom.xml I have such dependecy
<dependency>
<groupId>org.jetbrains.xodus</groupId>
<artifactId>xodus-environment</artifactId>
<version>1.2.3</version>
</dependency>
But during build process I've faced with such error:
[ERROR] Version conflict for artifact org.jetbrains:annotations, selected 13.0, dependencies require 15.0
So I've checked dependency tree in IDEA. I was wondered to see such dependency tree
As you can see library org.jetbrains.xodus:xodus-environment has several dependencies on org.jetbrains:annotations first one is 13.0 and another one is 15.0.
So I solved problem with such configuration
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
<version>1.2.51</version>
<exclusions>
<exclusion>
<groupId>org.jetbrains</groupId>
<artifactId>annotations</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.xodus</groupId>
<artifactId>xodus-environment</artifactId>
<version>1.2.3</version>
<exclusions>
<exclusion>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
</exclusion>
</exclusions>
</dependency>
But I think there is better way to solve this problem or maybe it is bug in xodus library
When running a Dataflow unittest that uses DataflowAssert I get the exception
java.lang.RuntimeException: DataflowAssert requires that JUnit and Hamcrest be linked in.
What dependencies do I need to add to my pom file to fix this exception?
Add the following dependencies to your pom file to link in the required libraries
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.11</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest-all</artifactId>
<version>1.3</version>
<scope>test</scope>
</dependency>