We are using Terraform to deploy the resouce "google_dataflow_job" on Google Cloud.
We did a successfully deployement few weeks ago (Dataflow API is already enabled since years).
We now get this error when we are executing a "terraform plan"
Error: Error when reading or editing Dataflow job 2020-10-07_06_20_01-18099947630909311965: googleapi: Error 400: (10a8bef84dbdde13): There is no cloudservices robot account for your project. Please ensure that the Dataflow API is enabled for your project., failedPrecondition
We have theses accounts:
Does anyone know how to add this cloudservices robot account?
I got the same error when running terraforms and it seems to be solved when recreating the job.
I did the following steps:
Cancel the job from the console
Remove the job from terraform state
rerun terraform in order to create the dataflow job again
Related
Im working in GitLab and Jenkins Integrations. Its working fine in scheduling process.
I'm trying CI from GitLab with Jenkins. All the blogs and tutorials explained with JenkinsCI But that option is not available in GitLab->Integrations. GitLab-Integration please refer the image.
I have tried without Jenkins CI using webhook getting the error.
enter image description here
Error: Webhook executed successfully But returning Http 500
I have tried my personal account in that I can see the jenkins under Integration able to add setting then moved to Active Integration
I am trying to integrate jenkins with Gitlab for CI such that if push event is happens then build will be triggered. I am using two things to connect. One is Gitlab acess token: I have created project specific access token and added in Jenkins global configuration. Second is SSH primary key: I have added this in my job configuration. Now when I test the webhook I get below error. Can any leads help me to fix it?
Hook executed successfully but returned HTTP 500 <head resURL="/static/861bc558" data-rooturl=""
I will attach full error if needed
Our company policy requires the policy contraint "compute.requireShieldedVm" to be enabled. However, when running a Cloud Dataflow job, it is failing to create a worker with the error :
Constraint constraints/compute.requireShieldedVm violated for project projects/********. The boot disk's 'initialize_params.source_image' field specifies a non-Shielded image: projects/dataflow-service-producer-prod/global/images/dataflow-dataflow-owned-resource-20200216-22-rc00. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information."
Is there any way when running a Dataflow job to request that a ShieldedVm be used for the worker compute?
It is not possible to provide a custom image as there is no such parameter that one can provide during job submission as can be seen here Job Submission Parameters
Alternatively, if you are running a Python based dataflow job you can setup the environment through setup files. An example of which can be found here Dataflow - Custom Python Package Environment
Got an error message from dataflow. I tried to cancel it manually but didn't work. Error:
2017-12-13 (03:44:56) Workflow failed. Causes: The Dataflow appears to be stuck. Please reach out to t...: Workflow failed. Causes: The Dataflow appears to be stuck. Please reach out to the Dataflow team at http://stackoverflow.com/questions/tagged/google-cloud-dataflow.
Can someone help on this job? Thanks!
Had a similar problem. Maybe this is the reason: When the default VPC network has been renamed or changed on a project, it is necessary to inform the new network name through WorkerOptions.network (--network on the CLI) when running a job remotely (DataflowRunner), or else the job gets stuck with no clue/clear log error message with what happened.
I am struggling with this, and initially thought it could be the result of switching the pipeline data source from Cloud Datastore to Firebase Firestore, which required a new project. But I've since found the same error in separate pipelines. All pipelines run successfully on the local DirectRunner and the permissions appear to be the same as the old project.
It looks like none of the VMs are booting and the pipeline never scales above 0 workers. "The Dataflow appears to be stuck" is the only error message I could find and there is nothing in StackDriver. Tried every dependency management variation I could find in the docs but it doesn't seem to be the problem.
My last Dataflow job-id is 2017-10-11_11_12_01-15165703816317931044.
Tried elevating the access roles of all services accounts and still no luck.
Without any logging information, it's hard to pinpoint. But this can happen if you have changed the permissions or roles of the Dataflow service account or the Compute Engine service account so that the service account does not have enough permissions to get the images for the Dataflow workers.