Failed to provisioning worker for TestDataflowRunner in asia-southeast1 zone - google-cloud-dataflow

Question

I have a question I'd like to ask about TestDataflowRunner.
I created a beam pipeline (Java) that runs using DataflowRunner. I deployed the worker in zone asia-southeast1 belong with the network. The pipeline runs normally as expected in DataflowRunner. Therefore, I also want to create #ValidatesRunner tests using TestDataflowRunner. I've run the test using the same service account and the same network. The execution graph also looks loaded fine, but failed to provisioning the worker.
Following is the command that I use to run the tests.
task validatesRunnerTests(type: Test) {
group = "Verification"
description = "Run tests that require a Dataflow runner to validate that pipelines/transforms work correctly"
systemProperty "beamTestPipelineOptions", JsonOutput.toJson([
"--runner=TestDataflowRunner",
"--project=$projectId",
"--region=us-central1",
"--workerZone=$zone",
"--usePublicIps=false",
"--network=$network",
"--subnetwork=$subnetwork",
"--tempRoot=$stagingBucket",
"--serviceAccount=$serviceAccount",
])
useJUnit {
includeCategories 'org.apache.beam.sdk.testing.ValidatesRunner'
}
}
The service account has included following roles.
roles/dataflow.admin
roles/dataflow.worker
I've only got following errors log, but I can't found any error logs in Stackdriver VM Instance.
2020-08-12 17:16:27.061 ICT Startup of the worker pool in zone asia-southeast1-a failed to bring up any of the desired 1 workers. The project quota may have been exceeded or access control policies may be preventing the operation; review the Stackdriver Logging "GCE VM Instance" log for diagnostics.
2020-08-12 17:16:27.095 ICT Workflow failed. Causes: Internal Issue (8c283568ab7f3c3c): 82159483:17
Does anyone know the problem and can help me?
Thank you

Answer 1

The issue seems to be getting the job to launch with private IPs only. I launched a Dataflow job using the TestDataflowRunner, disabling public IPs.
I first cloned the WordCount repo [1], and made three changes to WordCount.java.
Added the following libs:
import org.apache.beam.sdk.testing.TestPipeline;
import org.apache.beam.sdk.testing.TestPipelineOptions;
Changed:
public interface WordCountOptions extends PipelineOptions {
to
public interface WordCountOptions extends TestPipelineOptions {
and
Pipeline p = Pipeline.create(options);
to
Pipeline p = TestPipeline.create(options);
I used Maven, and ran the following command:
mvn clean compile exec:java \
-Dexec.mainClass=org.apache.beam.examples.WordCount \
-Dexec.args=" \
--project=$PROJECT_ID\
--tempRoot=gs://mcbuckety/tempf
--output=gs://$BUCKET_NAME/testrundf \
--runner=TestDataflowRunner
--usePublicIps=false
--subnetwork=regions/us-central1/subnetworks/private"
-Pdataflow-runner
The code compiled and launched a Dataflow job. I went to the GCE page to check that the VM did not have Public IPs, and that was indeed the case. I know the code is not strictly testing; I am not asserting anything, but I got the pipeline to launch.
[1] https://beam.apache.org/get-started/quickstart-java/#get-the-wordcount-code