Generating Self executing JAR using Dataflow - google-cloud-dataflow

Question

As per the direction in the link, tried to build the JAR file for scheduling dataflow job in Airflow. Added relevant dependency in the dependency section and given main class name in the build section in the POM file. When I execute the dataflow job using below maven command then getting below error.
Could some one guide me to solve this issue.
Maven Command
- mvn package
- java -jar target/sample-1.0.0.jar
Error Message:
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/beam/sdk/options/PipelineOptions
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Unknown Source)
at java.lang.Class.privateGetMethodRecursive(Unknown Source)
at java.lang.Class.getMethod0(Unknown Source)
at java.lang.Class.getMethod(Unknown Source)
at sun.launcher.LauncherHelper.validateMainClass(Unknown Source)
at sun.launcher.LauncherHelper.checkAndLoadMain(Unknown Source)
Caused by: java.lang.ClassNotFoundException: org.apache.beam.sdk.options.PipelineOptions
My Pipeline File
public interface DefaultOptions extends PipelineOptions,GcpOptions,ApplicationNameOptions,DataflowPipelineDebugOptions,DataflowPipelineWorkerPoolOptions,
BigQueryOptions,GcsOptions,StreamingOptions,CloudDebuggerOptions,DataflowProfilingOptions,PubsubOptions {
My Pipeline Initialization in class file
PipelineOptionsFactory.register(DefaultOptions.class);
DefaultOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().as(DefaultOptions.class);
options.setRunner(DataflowRunner.class);
options.setProject(options.getDataFlowProjectName());
options.setTempLocation(options.getDataFlowProjectTempLocation());
options.setGcpTempLocation(options.getDataFlowProjectTempLocation());
Pipeline p = Pipeline.create(options);
POM File:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.11.0</version>
</dependency>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.1.1</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>SampleWindowJar</mainClass>
</manifest>
</archive>
</configuration>
</plugin>

Answer 1

java.lang.NoClassDefFoundError is throwing since you didn't copy the dependencies before packaging it to jar.While running the jar jvm is trying to load class org.apache.beam.sdk.options.PipelineOptions from the dependencies. To fix that you need to add one more plugin maven-dependency-plugin
Sample plugin section in pom.xml will be like below
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>prepare-package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>
${project.build.directory}/lib
</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>${maven-jar-plugin.version}</version>
<configuration>
<archive>
<manifest>
<addClasspath>true</addClasspath>
<classpathPrefix>lib/</classpathPrefix>
<mainClass>YOUR_MAIN_CLASS_NAME_INCLUDING_PACKAGE</mainClass>
</manifest>
</archive>
</configuration>
</plugin>
</plugins>
In the below link they didn't mentioned about the copying the jar. May be because of that you missed that plugin.
Self executing JAR
After adding the correct plugins and dependencies run to build jar
mvn package
Use below command to run in dataflow
java -jar target/<YOUR_PROJECT_ARTIFACT_ID>-<YOUR_PROJECT_VERSION>.jar \
--runner=DataflowRunner \
--project=<YOUR_GCP_PROJECT_ID> \
--region=<GCP_REGION> \
--tempLocation=gs://<YOUR_GCS_BUCKET>/temp/
To run and debug locally use DirectRunner
java -jar target/<YOUR_PROJECT_ARTIFACT_ID>-<YOUR_PROJECT_VERSION>.jar \
--runner=DirectRunner
Also pass your program arguments(eg: pipeline options)

Answer 2

The Cloud Dataflow dependency is not the only dependency that you'll need. You need also all the Beam SDK-related dependencies. The easiest way to obtain them is with a Maven archetype, which Beam provides:
https://beam.apache.org/get-started/quickstart-java/
mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-starter \
-DarchetypeVersion=2.11.0 \
-DgroupId=org.yourorg \
-DartifactId=my-beam-pipeline \
-Dversion="0.1" \
-Dpackage=org.yourorg.pipelines \
-DinteractiveMode=false
cd my-beam-pipeline
You'll see a pom.xml file, and a directory with your starter pipeline that you can develop.
Once you have your archetype created, that'll include all the basic SDK dependencies, and you can add the Dataflow Runner as well:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.11.0</version>
</dependency>