I am getting below error while running dataflow job. I am trying to update my existing beam version to 2.11.0 but I am getting below error at run time.
java.lang.IncompatibleClassChangeError: Class
org.apache.beam.model.pipeline.v1.RunnerApi$StandardPTransforms$Primitives
does not implement the requested interface
com.google.protobuf.ProtocolMessageEnum at
org.apache.beam.runners.core.construction.BeamUrns.getUrn(BeamUrns.java:27)
at
org.apache.beam.runners.core.construction.PTransformTranslation.(PTransformTranslation.java:58)
at
org.apache.beam.runners.core.construction.UnconsumedReads$1.visitValue(UnconsumedReads.java:49)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:666)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:649)
at
org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
at
org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
at
org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
at
org.apache.beam.runners.core.construction.UnconsumedReads.ensureAllReadsConsumed(UnconsumedReads.java:40)
at
org.apache.beam.runners.dataflow.DataflowRunner.replaceTransforms(DataflowRunner.java:868)
at
org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:660)
at
org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:173)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313) at
org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
This usually means that the version of com.google.protobuf:protobuf-java that Beam was built with does not match the version at runtime. Does your pipeline code also depend on protocol buffers?
UPDATE: I have filed https://issues.apache.org/jira/browse/BEAM-6839 to track this. It is not expected.
I don't have enough rep to leave a comment, but I ran into this issue and later figured out that my problem was that I had different beam versions in my pom.xml. Some had 2.19 and some had 2.20.
I would do a quick search of your versions in the pom or gradle file to make sure they are all the same.
This may be caused by incompatible dependencies. I successfully upgraded beam from 2.2.0 to 2.20.0 by upgrading the dependencies at the same time.
beam.version: 2.20.0
guava.version: 29.0-jre
bigquery.version: v2-rev20191211-1.30.9
google-api-client.version: 1.30.9
google-http-client.version: 1.34.0
pubsub.version: v1-rev20200312-1.30.9
Related
Issue Summary:
Hi,
I am using avro version 1.11.0 for parsing an avro file and decoding it. We have a custom requirement, so i am not able to use ReadFromAvro. When trying this with dataflow there arises a dependency issues as avro-python3 with version 1.82 is already available. The issue is of class TimestampMillisSchema which is not present in avro-python3. It fails stating Attribute TimestampMillisSchema not found in avro.schema. I then tried passing a requirements file with avro==1.11.0 but now the dataflow was not able to start giving error "Error syncing pod" which seems to be because of dependencies conflicts.
To Solve the issue , we set an experiment flag (--experiments=no_use_multiple_sdk_containers ) which ran fine.
I want to know a better solution of my issue and also does the above flag will effect the pipeline performance.
Please try with the dataflow run command:
--prebuild_sdk_container_engine=cloud_build --experiments=use_runner_v2
this would use cloud build to build the container with your extra dependencies and then would use it within the dataflow run.
I am deploying Apache-druid image using docker-compose. I am getting following error,
Error: Could not find or load main class io.druid.cli.Main.
I have set CLASSPATH properly. Files are in CLASSPATH.
Any Pointers? Thanks for your help.
Thanks,
Shubhada
I fixed this issue. I was using different version of Druid. Which has differnnt package hierarchy of the class Main. Changed it to correct one.
If others run into similar issue, here's some more context. From Druid version 0.13.0 onwards, the config scripts need to use org.apache.druid.cli.Main. So if you are working on a prior version, and run into this error, change io.druid.cli.Main to org.apache.druid.cli.Main in your scripts. Better option is to upgrade cleanly to a newer version.
Apache OfBiz is not installing correctly, and fails to compile in the command prompt.
After creating the system variable JAVA_HOME to C:\Program Files\Java\jdk1.8.0_40, and editing "Path" to be C:\Program Files (x86)\Java\jre7\bin;C:\apache-ant-1.9.4\bin, I downloaded OfBiz 13.07.01 to my C:\ folder and unzipped it there. In the command prompt, I typed the following:
C:\Users\CalS>cd C:\apache-ofbiz-13.07.01
C:\apache-ofbiz-13.07.01>ant load-seed
Then, after about 50 seconds, I get this:
BUILD FAILED
C:\apache-ofbiz-13.07.01\build.xml:229: the following error occurred while executing this line:
C:\apache-ofbiz-13.07.01\build.xml:248: the following error occurred while executing this line:
C:\apache-ofbiz-13.07.01\build.xml:39: the following error occurred while executing this line:
C:\apache-ofbiz-13.07.01\build.xml:91: compile failed; see the compiler error output for details.
Please note it has been years since I dealt with DOS, so I do not know how to access the error output.
This is after I get a few dozen errors like:
[javac16] class file for org.ofbiz.widget.ContentWorkerInterfaice not found
and
[javac16] warning: [options] bootstrap class path not set in conjunctions with -source 1.6
Under 'classes'.
Misc. I have tried 'ant run-install' and 'load-demo' commands without avail. I've followed step-by-step tutorials, but very likely missed something. Please let me know what I can do to fix this and run this program successfully. Thanks!
Please have a look at the following Apache Jira Tickets for OFBiz where your problem is addressed and was fixed, so that OFBiz could be build with java 1.8.
The build errors occur due to missing fileset entries in the build.xml for some applications (party, workeffort, product, order, ebay, and pos), see: OFBIZ-5835
A fix is available in related ticket: OFBIZ-6079
There was another bug in the current release branches (checked 14.12.01, 12.04.06, 13.07.02) that I fixed last week. The fix is already committed to the branches.
See: OFBIZ-6252
You have to compile/run with the same Java version.
Seems you have some inconsistencies: JAVA_HOME ist 1.8, Path is set to jre 7 and the warning states it is using an 1.6 compiler.
With the 13.07. Release, using Java 1.7 or 1.8 is recommended and supported.
Alright, so it looks like Apache OFBiz and Java JDK 1.8.XX don't get along. I found help on another forum that confirmed the discrepancy in compatibility between OFBiz 13.07.01/Apache ant 1.9.4 and JDK 1.8.XX. This will cause the compiling of the Apache Ant to fail (which seems to run off of JDK 1.6).
I remedied the problem by downloading the archived JDK 1.7.0_67 from Oracle, re-mapping the System Variables accordingly, and re-initializing the Command Prompt.
It works now! Thank you all for your contributions.
Though solved, let me add something important. JDK version is not always an issue in such errors. Ofbiz v13.X.X works well on JDK 1.7 and above. The error shown is a peculiar issue with Ofbiz v13.07.01 dist.
As Martin pointed out, one need to add widget jars in the classpath of order, party, product & workeffort. Add the below line
<fileset dir="../../framework/widget/build/lib" includes="*.jar"/>
in build.xml of order, party, product & workeffort under applications directory.
I've been using Grails with Terracotta DSO by adding ehcache.xml and setting up the Terracotta Server. It works quite well, but I've come along the message that Terracotta doesn't support their Server (OpenSource or DSO however named) any more.
So I've been wondering if anybody tried to replace Terracotta Server with BigMemory, does that work? And if so, which dependencies do I need and where do I get those from?
I'm now using these dependencies:
runtime 'net.sf.ehcache:ehcache-core:2.6.2'
runtime 'net.sf.ehcache:ehcache-terracotta:2.6.2'
runtime 'org.terracotta:terracotta-toolkit-1.6-runtime:5.2.0'
Thanks for feedback.
BigMemoryMax is a kit (tarball) that includes the latest version of the Terracotta Server.
The latest version is BigMemoryMax 4.1.0 and you can download it from : http://terracotta.org/downloads/bigmemorymax.
You would then need to update your dependencies to have something similar to this :
runtime 'net.sf.ehcache:ehcache:2.8.0'
runtime 'org.terracotta:terracotta-toolkit-runtime:4.1.0'
I have been using IntelliJ for almost a year ad I have always been really happy with it. However, yesterday I set it up on my new laptop (Ubuntu 11.04), and haven't seen the plugins module since.. :-(
Ran grails clean, tried to change the project structure/settings to include $HOME/.grails/1.3.x/projects/projectName/plugins, but still nothing. My understanding is that it should pick up the plugins automatically, am I right?
For the record, I am using Grails 1.3.4, IntelliJ IDEA Ultimate 9.0.4.
I'd strongly recommend switching to the latest Intellij version (10.5.1 as of now). Support for Grails has been improved a lot since 9.x. If you don't want to upgrade, check the following areas:
are all used classpath variables set correctly?
are you referencing the correct Grails version?
I'm using IntelliJ 11.1.3 with Grails 2.1.1.
General Issue:
The IDE Build/Make Project/Run Unit-tests sometimes fails to resolve classes referenced as dependencies within Plugins and produces 'no class...' errors. Normal grails targets (compile, run-app, test-app) work without issue!
Workaround:
Restarting IntelliJ 'magically' corrected my Plugin related 'no class...' errors.
What didn't work:
grails resolve-dependencies (makes sense because grails run-app worked fine!)
grails clean (again makes sense this is just purging the output)
Seems like the IDE Build/Make Project/Run Unit-tests uses a stale classpath in some circumstances. Unfortunately I don't have a repeatable test case but noticed that modifying BuildConfig and doing grails refresh-dependencies or compile or run-app doesn't make the IDE update it's list of grailsPlugins!
I've had IDEA do this once in a while to me as well. Even in the latest version (though I do agree you should upgrade, but 9->10 isn't free). For me, I just had to kill IDEA and restarted it.
IMPORTANT UPDATE! It will be fixed in 11.1.2! YEEAAAH!
In my case plugins not resolving because of custom system property 'grails.work.dir'. If your project using default 'grails.work.dir' than OK otherwise plugins won't be resolved. Tested on Idea 10.5.4, 11.1.1 and windows 7.