Apache Beam running Pipeline from DoFn throws NoSuchMethodError - google-cloud-dataflow

I am running an Application on Google AppEngine which generates Dataflow-Templates and starts them. In one of those pipelines, inside a DoFn, the process generates another pipeline and waits for it to finish, before it continues its work. Until a few days ago, that was not a problem. But now, I get a NoSuchMethodError when pipeline.run() is called.
The stacktrace:
java.lang.NoSuchMethodError: org.apache.beam.sdk.common.runner.v1.RunnerApi$FunctionSpec$Builder.setPayload(Lcom/google/protobuf/ByteString;)Lorg/apache/beam/sdk/common/runner/v1/RunnerApi$FunctionSpec$Builder;
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.WindowingStrategyTranslation.toProto(WindowingStrategyTranslation.java:224)
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.WindowingStrategyTranslation.toProto(WindowingStrategyTranslation.java:299)
at org.apache.beam.runners.dataflow.repackaged.org.apache.beam.runners.core.construction.WindowingStrategyTranslation.toProto(WindowingStrategyTranslation.java:285)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.serializeWindowingStrategy(DataflowPipelineTranslator.java:129)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.access$1500(DataflowPipelineTranslator.java:114)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$5.groupByKeyHelper(DataflowPipelineTranslator.java:806)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$5.translate(DataflowPipelineTranslator.java:784)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$5.translate(DataflowPipelineTranslator.java:781)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.visitPrimitiveTransform(DataflowPipelineTranslator.java:442)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:663)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:655)
at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:446)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator$Translator.translate(DataflowPipelineTranslator.java:386)
at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:173)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:537)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:170)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at com.mycompany.projectsign.dataflow.dofn.downloads.something.RunDownloadPipeline.processElement(RunDownloadPipeline.java:150)
The referenced line of my code is the pipeline.run()
When I look at the maven-dependencies in Eclipse, the right dependencies (and versions) are added in the project and the RunnerApi.FunctionSpec.Builder.setPayload(com.google.protobuf.ByteString) method exists, too. I continued enforcing the versions with dependencyManagement:
<dependencyManagement>
<dependencies>
....
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.2.0</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-core</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
<version>1.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-common-runner-api</artifactId>
<version>2.2.0</version>
</dependency>
.....
</dependencies>
</dependencyManagement>
When running in AppEngine or creating the pipeline from localhost (running directly on Google Dataflow) the templates can be created and started without a problem, but running the created Pipeline inside the DoFn throws the same error.
However, when I run the Pipeline as a local pipeline directly on my computer (with DirectRunner), the pipeline runs without a problem and creates the other pipelines on GoogleDataflow.
I updated the beam-version to 2.2.0.
If some Googler is reading this, this is one of the failing JobIds: 2017-12-11_07_01_17-3122752092943950314
What might be the reason for the NoSuchMethodError? Could it be a conflicting dependency or something else?
Any help is highly appreciated :-)

Related

Getting ClassNotFoundException in Grails 2.5.6

Migrating from 1.3.7 to 2.5.6
java.lang.ClassNotFoundException: org.codehaus.groovy.grails.project.compiler.GrailsProjectCompiler how can we fix this?
Trying to build project using goal grails:clean grails:war -Dmaven.skip.test=true
Grails Version : 2.5.6
Java version 1.7
As stated by the comments, you're in for quite a ride to get this up and running.
This particular error is probably because you are missing the new dependencies needed for Grails. This is what you need if you have a maven project which I assume you have while getting this problem:
<dependency>
<groupId>org.grails</groupId>
<artifactId>grails-dependencies</artifactId>
<version>${grails.version}</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>org.grails</groupId>
<artifactId>grails-test</artifactId>
<version>${grails.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.grails</groupId>
<artifactId>grails-plugin-testing</artifactId>
<version>${grails.version}</version>
<scope>test</scope>
</dependency>
You also have to remove all of the old dependencies to the 1.3 grails core. In 1.3 it was a series of dependencies if I don't remember incorrectly.
If it is not a maven project you should get this for free by specifying the inerits("global") in your BuildConfig.groovy:
grails.project.dependency.resolution = {
// inherit Grails' default dependencies
inherits("global") {
// specify dependency exclusions here; for example, uncomment this to
}
...
That said, this will not be the last of your problems while upgrading. I have successfully managed to upgrade some small projects but it has taken quite an effort. For larger projects, I would seriously consider if it is worth it.
My best advice if you decide to go through with it is to generate a new empty 2.5.6 project and compare settings and dependencies to your legacy project.

apache beam 2.2 pipeline.apply no such method exception

public static void main(String[] args) {
//Pipeline p = Pipeline.create(PipelineOptionsFactory.fromArgs(args).withValidation().create());
DataflowPipelineOptions options = PipelineOptionsFactory.as(DataflowPipelineOptions.class);
options.setRunner(DataflowRunner.class);
options.setStagingLocation("gs://bucketname/stageapache");
options.setTempLocation("gs://bucketname/stageapachetemp");
options.setProject("projectid");
Pipeline p=Pipeline.create(options);
p.apply(TextIO.read().from("gs://bucketname/filename.csv"));
//p.apply(FileIO.match().filepattern("gs://bucketname/f.csv"));
p.run();
}
pom.xml
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-core</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>2.0.0</version>
</dependency>
<!-- https://mvnrepository.com/artifact/org.apache.beam/beam-runners-google-cloud-dataflow-java -->
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.0.0</version>
</dependency>
Error
Dec 08, 2017 5:09:35 PM org.apache.beam.runners.dataflow.DataflowRunner fromOptions
INFO: PipelineOptions.filesToStage was not specified. Defaulting to files from the classpath: will stage 85 files. Enable logging at DEBUG level to see which files will be staged.
Exception in thread "main" java.lang.NoSuchMethodError: org.apache.beam.sdk.values.PCollection.createPrimitiveOutputInternal(Lorg/apache/beam/sdk/Pipeline;Lorg/apache/beam/sdk/values/WindowingStrategy;Lorg/apache/beam/sdk/values/PCollection$IsBounded;)Lorg/apache/beam/sdk/values/PCollection;
at org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$ParDoSingle.expand(PrimitiveParDoSingleFactory.java:68)
at org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$ParDoSingle.expand(PrimitiveParDoSingleFactory.java:58)
at org.apache.beam.sdk.Pipeline.applyReplacement(Pipeline.java:550)
at org.apache.beam.sdk.Pipeline.replace(Pipeline.java:280)
at org.apache.beam.sdk.Pipeline.replaceAll(Pipeline.java:201)
at org.apache.beam.runners.dataflow.DataflowRunner.replaceTransforms(DataflowRunner.java:688)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:498)
at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:153)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:303)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:289)
at com.pearson.apachebeam.StarterPipeline.main(StarterPipeline.java:60)
In the above code if add FileIO/TextIO line i am getting the above error, with out adding that line if i run it is creating job since there is no operations in it is failing. i am stuck up at this in my development i migrated to apache beam 2.2 to get control of the file we read from storage
Help will be appreciated
Thanks
The issue is that your pom.xml is depending on different components of the Beam SDK at different versions: beam-sdks-java-core at 2.2.0, but beam-sdks-java-io-google-cloud-platform and beam-runners-google-cloud-dataflow-java at 2.0.0. They need to be at the same version.

Startup Error: java.lang.IncompatibleClassChangeError: org/apache/struts2/convention/DefaultClassFinder$InfoBuildingVisitor

I have a Struts2 application, originally XML-based, but now for the first time I introduced an Annotation-based action, and the application broke on startup.
Given the following Action mapping, on startup of the application, I get the error
java.lang.InstantiationError: com.opensymphony.xwork2.util.finder.ClassFinder
at org.apache.struts2.convention.PackageBasedActionConfigBuilder.findActions(PackageBasedActionConfigBuilder.java:390)
at org.apache.struts2.convention.PackageBasedActionConfigBuilder.buildActionConfigs(PackageBasedActionConfigBuilder.java:347)
at org.apache.struts2.convention.ClasspathPackageProvider.loadPackages(ClasspathPackageProvider.java:53)
at com.opensymphony.xwork2.config.impl.DefaultConfiguration.reloadContainer(DefaultConfiguration.java:199)
at com.opensymphony.xwork2.config.ConfigurationManager.getConfiguration(ConfigurationManager.java:67)
at org.apache.struts2.dispatcher.Dispatcher.getContainer(Dispatcher.java:906)
at org.apache.struts2.dispatcher.Dispatcher.init_PreloadConfiguration(Dispatcher.java:445)
at org.apache.struts2.dispatcher.Dispatcher.init(Dispatcher.java:486)
at org.apache.struts2.dispatcher.InitOperations.initDispatcher(InitOperations.java:75)
at org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.init(StrutsPrepareAndExecuteFilter.java:63)
at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:279)
There was a similar question here, but no one responded,
Issue with Struts2 Filter Exception
Libraries Used:
struts2-convention-plugin-2.3.14.3.jar
struts2-core-2.5.10.1.jar
struts2-json-plugin-2.5.10.1.jar
struts2-spring-plugin-2.5.10.1.jar
struts2-tiles-plugin-2.5.10.1.jar
xwork-core-2.1.6.jar
Update: I thought the Convention-Plugin-JAR 2.3.14.3 was incompatible with the other 2.5.10.1 ones, so I downloaded: struts2-convention-plugin-2.5.10.1.jar
Now on startup getting
java.lang.IncompatibleClassChangeError: org/apache/struts2/convention/DefaultClassFinder$InfoBuildingVisitor
at org.apache.struts2.convention.DefaultClassFinder.readClassDef(DefaultClassFinder.java:459) ~[struts2-convention-plugin-2.5.10.1.jar:2.5.10.1]
at org.apache.struts2.convention.DefaultClassFinder.<init>(DefaultClassFinder.java:90) [struts2-convention-plugin-2.5.10.1.jar:2.5.10.1]
at org.apache.struts2.convention.PackageBasedActionConfigBuilder.buildClassFinder(PackageBasedActionConfigBuilder.java:397) [struts2-convention-plugin-2.5.10.1.jar:2.5.10.1]
at org.apache.struts2.convention.PackageBasedActionConfigBuilder.findActions(PackageBasedActionConfigBuilder.java:379) [struts2-convention-plugin-2.5.10.1.jar:2.5.10.1]
at org.apache.struts2.convention.PackageBasedActionConfigBuilder.buildActionConfigs(PackageBasedActionConfigBuilder.java:335) [struts2-convention-plugin-2.5.10.1.jar:2.5.10.1]
at org.apache.struts2.convention.ClasspathPackageProvider.loadPackages(ClasspathPackageProvider.java:53) [struts2-convention-plugin-2.5.10.1.jar:2.5.10.1]
at com.opensymphony.xwork2.config.impl.DefaultConfiguration.reloadContainer(DefaultConfiguration.java:199) [struts2-core-2.5.10.1.jar:2.5.10.1]
SOLUTION The solution was to add ASM 5.x JARs which are a dependency of the Struts-Convention JAR for annotations. We were using ASM 3.3. I added the ASM-Core, -Common, -Tree Version 5.1 JARs.
struts2-convention plugin 2.5.10.1 has compile dependencies
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.4</version>
</dependency>
<dependency>
<groupId>org.ow2.asm</groupId>
<artifactId>asm</artifactId>
<version>5.1</version>
</dependency>
<dependency>
<groupId>org.ow2.asm</groupId>
<artifactId>asm-commons</artifactId>
<version>5.1</version>
</dependency>
Note: if you don't need asm and asm-commons dependencies you can downgrade to Struts 2.5.5.
Struts2 uses ASM 5x for the Convention plugin. Check you classpath and make sure you have appropriate version of ASM jars.

Arquillian + jacoco IT coverage

I configured 2 projects to use last jacoco version 0.7.8 and last Arquillian jacoco extension (1.0.09Alpha) it works like a charm (for jenkins and sonar 6.2)! but i have a bigger project, when i launch only Arquillian IT test my war archive is created and have all classes and so tests OK, when i run the same tests with IT code coverage, no class are included in the arquillian archive and have this error :
org.jboss.shrinkwrap.api.exporter.ArchiveExportException: Failed to write asset to output: /WEB-INF/...
Caused by: java.lang.RuntimeException: Could not instrument Asset org.jboss.shrinkwrap.api.asset.ClassLoaderAsset
Same configuration as other project BOM Arquillian 1.1.12Final arquillian suite 1.1.2 container 2.0.2 testng.....
any help ?
finally it was lib error indeed library asm-debug-all version was omitted because other library (apache-tika-parsers) already imported an older version (in pom.xml)... make an exclude in pom.xml fix the issue, we can check dependencies hierarchy in eclipse for example.
jacoco-arquillian extension use asm to instrument code...
<dependency>
<groupId>org.apache.tika</groupId>
<artifactId>tika-parsers</artifactId>
<version>1.9</version>
<scope>${defaultScope}</scope>
<exclusions>
<exclusion>
<groupId>org.bouncycastle</groupId>
<artifactId>bcprov-jdk15</artifactId>
</exclusion>
<exclusion>
<groupId>org.bouncycastle</groupId>
<artifactId>bcmail-jdk15</artifactId>
</exclusion>
<exclusion>
<groupId>org.ow2.asm</groupId>
<artifactId>asm-debug-all</artifactId>
</exclusion>
</exclusions>
</dependency>

Dataflow fails with java.lang.NoSuchMethodError: io.grpc.protobuf.ProtoUtils.marshaller(Lcom/google/protobuf/Message;)

I'm trying to get a Dataflow job to run on Google Cloud. It always fails with:
java.lang.NoSuchMethodError: io.grpc.protobuf.ProtoUtils.marshaller(Lcom/google/protobuf/Message;)Lio/grpc/MethodDescriptor$Marshaller
It's a maven project, here are my dependencies:
<dependencies>
<dependency>
<groupId>com.google.cloud.dataflow</groupId>
<artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
<version>1.8.0</version>
</dependency>
<dependency>
<groupId>com.google.cloud</groupId>
<artifactId>google-cloud-pubsub</artifactId>
<version>0.4.0</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>3.0.0</version>
</dependency>
</dependencies>
I've tried a bunch of different dependency versions. What am I missing?
It has the same result whether I run via exec:java or via a shade jar.
Full stack trace:
(e8dbd0c1b8b8a22): java.lang.NoSuchMethodError:
io.grpc.protobuf.ProtoUtils.marshaller(Lcom/google/protobuf/Message;)Lio/grpc/MethodDescriptor$Marshaller;
at com.google.iam.v1.IAMPolicyGrpc.(IAMPolicyGrpc.java:56) at
com.google.cloud.pubsub.spi.v1.PublisherSettings$Builder.(PublisherSettings.java:487)
at
com.google.cloud.pubsub.spi.v1.PublisherSettings$Builder.createDefault(PublisherSettings.java:508)
at
com.google.cloud.pubsub.spi.v1.PublisherSettings$Builder.access$000(PublisherSettings.java:402)
at
com.google.cloud.pubsub.spi.v1.PublisherSettings.defaultBuilder(PublisherSettings.java:224)
at
com.google.cloud.pubsub.spi.DefaultPubSubRpc.(DefaultPubSubRpc.java:138)
at
com.google.cloud.pubsub.PubSubOptions$DefaultPubSubRpcFactory.create(PubSubOptions.java:60)
at
com.google.cloud.pubsub.PubSubOptions$DefaultPubSubRpcFactory.create(PubSubOptions.java:54)
at com.google.cloud.ServiceOptions.rpc(ServiceOptions.java:399) at
com.google.cloud.pubsub.PubSubImpl.(PubSubImpl.java:115) at
com.google.cloud.pubsub.PubSubOptions$DefaultPubSubFactory.create(PubSubOptions.java:43)
at
com.google.cloud.pubsub.PubSubOptions$DefaultPubSubFactory.create(PubSubOptions.java:38)
at com.google.cloud.ServiceOptions.service(ServiceOptions.java:391) at
com.google.lindsaysmith.titan.DataflowBulkLoadNodes$SendPubSub.sendPubsubMessage(DataflowBulkLoadNodes.java:41)
at
com.google.lindsaysmith.titan.DataflowBulkLoadNodes$SendPubSub.processElement(DataflowBulkLoadNodes.java:32)
at
com.google.cloud.dataflow.sdk.util.SimpleDoFnRunner.invokeProcessElement(SimpleDoFnRunner.java:49)
at
com.google.cloud.dataflow.sdk.util.DoFnRunnerBase.processElement(DoFnRunnerBase.java:139)
at
com.google.cloud.dataflow.sdk.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:190)
at
com.google.cloud.dataflow.sdk.runners.worker.ForwardingParDoFn.processElement(ForwardingParDoFn.java:42)
at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerLoggingParDoFn.processElement(DataflowWorkerLoggingParDoFn.java:47)
at
com.google.cloud.dataflow.sdk.util.common.worker.ParDoOperation.process(ParDoOperation.java:55)
at
The usual answer in this situation is that you really must use exactly the version of gRPC and Protocol Buffers declared in the dependencies of the Dataflow SDK. This includes all transitive dependencies, so you may have to suppress gRPC or protobuf dependencies of other libraries so they do not intefere.
You can see the versions here (gRPC) and here (protobuf). I'm leaving them out of this answer so it does not get out of date.

Resources