Apache Beam: Exceptions after upgrade to 2.5.0 - google-cloud-dataflow

Question

Java version: 8
Runner: Dataflow
Everything worked before I upgrade to beam 2.5.0. Exception:
Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)
at org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create(Pipeline.java:150)
at com.geotab.bigdata.streaming.mapserver.backfill.MapServerBatchBeamApplication.main(MapServerBatchBeamApplication.java:115)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)
... 4 more
Caused by: java.lang.NoSuchMethodError: com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(Ljava/lang/String;)Lcom/google/api/client/googleapis/services/AbstractGoogleClient$Builder;
at com.google.api.services.storage.Storage$Builder.setBatchPath(Storage.java:9307)
at com.google.api.services.storage.Storage$Builder.<init>(Storage.java:9286)
at org.apache.beam.sdk.util.Transport.newStorageClient(Transport.java:95)
at org.apache.beam.sdk.util.GcsUtil$GcsUtilFactory.create(GcsUtil.java:96)
at org.apache.beam.sdk.util.GcsUtil$GcsUtilFactory.create(GcsUtil.java:84)
at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:592)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:533)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155)
at com.sun.proxy.$Proxy25.getGcsUtil(Unknown Source)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:88)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:61)
my gradle config:
dependencies {
compile 'org.apache.beam:beam-runners-direct-java:2.5.0'
compile 'org.apache.beam:beam-runners-reference-parent:2.4.0'
compile 'org.apache.beam:beam-runners-reference-java:2.5.0'
compile ('org.apache.beam:beam-runners-google-cloud-dataflow-java:2.5.0') {
exclude group: 'com.google.guava'
}
compile ('com.google.api-client:google-api-client:1.22.0') {
force = true //version 1.23.0 causes HTTP response 404 not found exception in Beam
//Is this still the case? Which version should I use with beam 2.5.0?
exclude group: 'com.google.guava'
}
compile 'com.google.guava:guava:23.0'
compile 'com.typesafe:config:1.3.2'
compile 'org.apache.httpcomponents:httpclient:4.5.5'
}
Side questions are:
Which google-api-client version is compatible with beam 2.5?
Which com.google.guava:guava version is compatible for beam and google-api-client?

Answer 1

Cloud Dataflow SDK 2.5.0 for Java has the following compile and runtime dependencies:
+-----------------------+-------------------+----------+
| GroupId | ArtifactId | Version |
+-----------------------+-------------------+----------+
| com.google.api-client | google-api-client | 1.23.0 |
| com.google.guava | guava | 20.0 |
+-----------------------+-------------------+----------+

Answer 2

The error indicates that AbstractGoogleClient.Builder is missing. Checking the documentation for the google-api-java-client versions, this was present in 1.19 and 1.20, but from 1.21 onward is not to be found.
According to this GitHub issue report, updating to at least 1.23 solves those kinds of problems. Note also this comment from Nilson Pontello: "You must make a clean build after changing from 1.22.0 to 1.23.0 otherwise old libs will still be used, specially if you are generating war files using maven."