My Dataflow job fails when it tries to access a secret:
"Exception in thread "main" com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission 'secretmanager.versions.access' denied for resource 'projects/REDACTED/secrets/REDACTED/versions/latest' (or it may not exist)."
I launch the job using gcloud dataflow flex-template run. I am able to view the secret in the console. The same code works when I run it on my laptop. As I understand it, when I submit a job with the above command, it runs under a service account that may have different permissions. How do I determine which service account the job runs under?
Since Dataflow creates workers, they create instances. You can check this on Logging
Open GCP console
Open Logging -> Logs Explorer (make sure you are not using the "Legacy Logs Viewer")
At the query builder type in protoPayload.serviceName="compute.googleapis.com"
Click Run Query
Expand the entry for v1.compute_instances.create or any other resources used by compute.googleapis.com
You should be able to see the service account used for creating the instance. This service account (boxed in red) is used anything related to the running the dataflow job.
Take note that I tested this using the official dataflow quick start.
By default the worker nodes of dataflow run with the compute engine default service account (YOUR_PROJECT_NUMBER-compute#developer.gserviceaccount.com) lacking of the "Secret Manager Secret Accessor" rights.
Either you need to add those rights to the service account or you have to specify the service account in the pipeline options:
gcloud dataflow flex-template run ... --parameters service_account_email="your-service-account-name#YOUR_PROJECT_NUMBER.iam.gserviceaccount.com"
Related
I am running a cloudbuild.yaml job in Google Cloud Platform that builds, pushes and tags a Docker Image and then it creates a Compute Engine instance to run that image via gcr.io/cloud-builders/gcloud.create-with-container. I also specify a service account to be used in this step:
- id: "Create Compute Engine instance"
name: gcr.io/cloud-builders/gcloud
args: [
'compute',
'instances',
'create-with-container',
'${INSTANCE_NAME}',
'--container-image',
'eu.gcr.io/${PROJECT_ID}/${PROJECT_ID}-${REPO_NAME}',
'--zone',
'${ZONE}',
'--service-account',
'${SERVICE_ACCOUNT},
'--machine-type',
'n2-standard-4'
]
However I am getting an error:
Already have image (with digest): gcr.io/cloud-builders/gcloud
ERROR: (gcloud.compute.instances.create-with-container) Could not fetch resource:
- Required 'compute.instances.create' permission for 'projects/...'
The service account in use does have the permissions for that as it has been assigned "role": "roles/compute.instanceAdmin.v1", which includes compute.instances.* as per documentation.
Anyone has experienced this or a similar situation and can give a hint on how to proceed? Am I missing something obvious? I have tried using other service accounts, including the project default compute account and get the same error. One thing to note is I do not specify a service account for Docker steps (gcr.io/cloud-builders/docker).
Make sure that you are not misinterpreting service accounts.
There is a special service account used by Cloud Build.
There is also the service account to "be used" by the VM/instance you are creating.
The "compute.instances.create" permission should be granted to the special Cloud Build account, not to the account for the instance.
The Cloud Build account has a name like 123123123#cloudbuild.gserviceaccount.com.
In the Cloud Console go to Cloud Build -> Settings -> Service Accounts
and check if correct permissions are granted.
I have a parent project that has an artifact registry configured for docker.
A child project has a cloud run service that needs to pull its image from the parent.
The child project also has a service account that is authorized to access the repository via an IAM role roles/artifactregistry.writer.
When I try to start my service I get an error message:
Google Cloud Run Service Agent must have permission to read the image,
europe-west1-docker.pkg.dev/test-parent-project/docker-webank-private/node:custom-1.
Ensure that the provided container image URL is correct and that the
above account has permission to access the image. If you just enabled
the Cloud Run API, the permissions might take a few minutes to
propagate. Note that the image is from project [test-parent-project], which
is not the same as this project [test-child-project]. Permission must be
granted to the Google Cloud Run Service Agent from this project.
I have tested manually connecting with docker login and using the service account's private key and the docker pull command works perfectly from my PC.
cat $GOOGLE_APPLICATION_CREDENTIALS | docker login -u _json_key --password-stdin https://europe-west1-docker.pkg.dev
> Login succeeded
docker pull europe-west1-docker.pkg.dev/bfb-cicd-inno0/docker-webank-private/node:custom-1
> OK
The service account is also attached to the cloud run service:
You have 2 types of service account used in Cloud Run:
The Google Cloud Run API service account
The Runtime service account.
In your explanation, and your screenshot, you talk about the runtime service account, the identity that will be used by the service when it runs and call Google Cloud API.
BUT before running, the service must be deployed. This time, it's a Google Cloud Run internal process that run to pull the container, create a revision and do all the required internal stuff. To do that job, a service account also exist, it's named "service agent".
In the IAM console, you can find it: the format is the following
service-<PROJECT_NUMBER>#serverless-robot-prod.iam.gserviceaccount.com
Don't forget to tick the checkbox in the upper right corner to include the Google Managed service account
If you want that this deployment service account be able to pull image in another project, grant on it the correct permission, not on the runtime service account.
I'm quite new to github actions and gcloud. I have trouble to get my github-CI/CD-Pipeline running because I can't push any docker image to the google Cloud Registry due to access restrictions.
What have I done so far:
I have a Quarkus app hosted on github
I used github actions to build the Maven project and the docker image
I created a project in google Cloud and added a service account which I use for the github action. The login seems to work:
Run google-github-actions/setup-gcloud#master
/usr/bin/tar xz --warning=no-unknown-keyword -C /home/runner/work/_temp/ac85f67a-89fa-4eb4-8d30-3f6379124ec2 -f /home/runner/work/_temp/de491940-a4b1-4a15-bf0a-95d563e68362
/opt/hostedtoolcache/gcloud/342.0.0/x64/bin/gcloud --quiet config set project ***
Updated property [core/project].
Successfully set default project
/opt/hostedtoolcache/gcloud/342.0.0/x64/bin/gcloud --quiet auth activate-service-account github-actions#***.iam.gserviceaccount.com --key-file -
Activated service account credentials for: [github-actions#***.iam.gserviceaccount.com]
If I now try to push the docker image I get the following (expected) error message:
Run docker push "$GCR_HOSTNAME/$PROJECT_ID/$IMAGE:$IMAGE_TAG"
The push refers to repository [eu.gcr.io/***/***]
715ac1ae8693: Preparing
435cfe5f5775: Preparing
313d03d71d4d: Preparing
c5c8d86ccee1: Preparing
1b0f2238925b: Preparing
144a43b910e8: Preparing
4a2bc86056a8: Preparing
144a43b910e8: Waiting
4a2bc86056a8: Waiting
denied: Token exchange failed for project '***'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control
Error: Process completed with exit code 1.
Next, I opened the Google Cloud Console and created a custom role (IAM & Admin -> Roles -> Create Role) which has the necessary permissions.
Then, I had trouble to assign my new custom role to the service account (IAM & Admin -> Service Accounts -> Manage Access -> Add member). I used the email address of the service account as "New members", but I could not choose the custom role I just created. What am I missing here?
I read somewhere that I can also add service accounts as member (IAM & Admin -> IAM -> Add). Again I used the email address of the service account as "New Members". This time I could choose my custom role. What's the difference to the first approach?
Anyways, if a I try to run the github action again, now I get the following error:
Run docker push "$GCR_HOSTNAME/$PROJECT_ID/$IMAGE:$IMAGE_TAG"
The push refers to repository [eu.gcr.io/***/***]
c4f14c9d3b6e: Preparing
fe78d438e8e2: Preparing
843fcae4a8f4: Preparing
dcf8cc80cedb: Preparing
45e8815b101d: Preparing
144a43b910e8: Preparing
4a2bc86056a8: Preparing
144a43b910e8: Waiting
4a2bc86056a8: Waiting
denied: Access denied.
Error: Process completed with exit code 1.
The error message is different, so I guess the permission for the service account somehow worked, but still I can't succeed. Which steps did I miss?
Any help is highly appreciated. Thanks a lot!
One way to debug this is to create a key for the service account on your local host, configure your script|gcloud to use the service account as its credentials and then try the push manually.
One immediate problem may be that you're not authenticating against Google Container Registry (GCR). GCR implements Docker's registry API and you'll need to use one of the mechanisms to authenticate before you can interact with the registry.
Notes:
I think you don't need to create a custom role. You have 2 options. Either (preferred) create an account specifically for the CI/CD job and grant it the minimum set of roles needed including storage.buckets.get. I think you can start with roles/storage.admin (link) and perhaps refine later.
You can grant roles e.g. roles/storage.admin to a Project in which case the permission applies to all Cloud Storage resources or to a specific Bucket in which case the permission applies only to the bucket and its objects.
Service Accounts have a dual role in GCP. As an identity and as a resource (that can be used by other identities). It can be confusing.
I'm trying via Jenkins to push an image to the container repository. It was working at first, but now, I got "access denied"
docker -- push gcr.io/xxxxxxx-yyyyy-138623/myApp:master.1
The push refers to a repository [gcr.io/xxxxxxx-yyyyy-138623/myApp]
bdc3ba7fdb96: Preparing
5632c278a6dc: Waiting
denied: Access denied.
the Jenkinsfile look like :
sh("gcloud docker --authorize-only")
sh("docker -- push gcr.io/xxxxxxx-yyyyy-138623/hotelpro4u:master.1")
Remarks:
Jenkins is running in Google Cloud
If I try in Google Shell or from my computer, it's working
I followed this tutorial : https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes
I'm stuck while 12 hours.... I need help
That error means that the GKE node is not authorized to push to the GCS bucket that is backing your repository.
This could be because:
The cluster does not have the correct scopes to authenticate to GCS. Did you create the cluster w/ --scopes storage-rw?
The service account that the cluster is running as does not have permissions on the bucket. Check the IAM & Admin section on your project to make sure that the service account has the necessary role.
Building on #cj-cullen's answer above, you have two options:
Destroy the node pool and then, from the CLI, recreate it with the missing https://www.googleapis.com/auth/projecthosting,storage-rw scope. The GKE console does not have the capability to change the default scopes when creating a node pool from the console.
Stop each instance in your cluster. In the console, click the edit button for the instance. You should now be able to add the appropriate https://www.googleapis.com/auth/projecthosting,storage-rw scope.
I'm using the ArtifactDeployer plugin to deploy the build job artifacts to a remote location (Windows share SMB).
However Jenkins never manages to succeed. Throwing errors like:
[ArtifactDeployer] - Starting deployment from the post-action ...
[ArtifactDeployer] - [ERROR] - Failed to deploy. Can't create the directory ... Build step
[ArtifactDeployer] - Deploy artifacts from workspace to remote directories' changed build result to FAILURE
Local deployment works fine.
The Jenkins machine OS is Windows 7 32-bit Prof.
Jenkins is running as a service using a local system account.
I tried using another account, my user account but the service failed to start (Windows error 1069: the service did not start due to a logon failure).
The network service account did run but than Jenkins throws errors it can't access the .NET framework.
When manually trying the remote copy, this works fine. I can create directories and write to it. On the same machine of course.
I tried two different remote reference in Jenkins:
1) \\targetdirectory
2) I:\ - by mapping a drive letter to the remote dir in windows
No success...
Any tips or suggestions? Thanks!
Update 15/02/2012:
Still no solution or workaround for this issue.
It's not only the plugin, I hit also this issue using "Execute Windows batch command".
I found a bug report that I want to share.
Solution
I found a solution. You have to grant access persmission to the computer in a domain instead of the user of that machine. Seems very logic if you look back to it.
A 2nd solution is to run the service using a domain user account. Above I made a mistake by using the local user .\user in stead of DOMAIN\user.
If you don't have a domain, the following will work for sure. This should work even if you have a domain.
Background Info:
You need your mapped drive to be mapped for the same account that the service is using AND be available at the right time. Normally mapped drives are mapped only for the logged in user, at the time that they log in. Service user contexts don't get "logged in" per se -- for example, if I map a drive as MyUser and the service runs as MyUser, the drive won't be available until I actually log in by typing in my password. However, we can use a script to map the drive at startup (instead of login) for a particular user. Jenkins normally runs as Local System Account, so if you don't want to change that, you'll need to run the script below as the SYSTEM user. You can instead create a specific user for Jenkins to run as, if you don't want to grant this mapped drive to all services/processes that run as SYSTEM, and run both the service and the script below as that user (this is probably more secure).
Solution Steps:
In ArtifactDeployer you want to deploy to a mapped network drive. In my case this is S:.
There is no special setup for permissions on the remote share. (In my case, a Windows Server 2008 share with a username and password that is used for mapping the drive.)
Write a batch file MapDrives.bat in a place that your chosen user (default: SYSTEM) has access to, with the following in it:
net use S: "\\server_name\share_name" /persistent:yes password_here /USER:username_here
Note that I am mapping to S: in that line.
Via Task Scheduler, create a task that runs as the same user as the service (default: SYSTEM), triggers At Startup, and as it's action, runs the batch file MapDrives.bat.
Reboot and it should work!
Citations:
After diving through many pages and many tests, ultimately, the best suggestions were found here, and led me to the above solution.
https://stackoverflow.com/a/4763324/150794
Make sure your 'local system account' has access rights to the remote directory (including write access). Then use the notation
\\targetdirectory
Mapping drive letters to remote directories only applies to the user account you are currently working with. The drive letter mapping will not be available to any other account.