cloud build does not recognize build directory argument - google-cloud-run

I am trying to build a Cloud Run job with a trigger from Cloud Build and secrets from Secret Manager. I managed to get the trigger that I use to build my Dockerfile to run, but the build itself fails with the following error:
BUILD
Starting Step #0 - "build image"
Step #0 - "build image": Already have image (with digest): gcr.io/cloud-builders/docker
Step #0 - "build image": "docker build" requires exactly 1 argument.
Step #0 - "build image": See 'docker build --help'.
Step #0 - "build image":
Step #0 - "build image": Usage: docker build [OPTIONS] PATH | URL | -
Step #0 - "build image":
Step #0 - "build image": Build an image from a Dockerfile
Finished Step #0 - "build image"
ERROR
ERROR: build step 0 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1
What I have already tried:
Verified that there is a build directory in the command;
Rearranged the order of build arguments just in case;
I also tried breakout syntax (with '|' as one of the arguments), but it did not work out - the image was not built at all.
UPDATED: I tried running the build without --build-args and it started actually building! Looks like a bug.
Here is my cloudbuild.yaml:
steps:
- id: "build image"
name: "gcr.io/cloud-builders/docker"
entrypoint: 'bash'
args:
['-c', 'docker build --build-arg CONTAINER_PRIVATE_KEY=$$PRIVATE_KEY --build-arg CONTAINER_PUBLIC_KEY=$$PUBLIC_KEY -t gcr.io/${PROJECT_ID}/${_JOB_NAME} .']
secretEnv: [ 'PRIVATE_KEY', 'PUBLIC_KEY' ]
- id: "push image"
name: "gcr.io/cloud-builders/docker"
args: [ "push", "gcr.io/${PROJECT_ID}/${_JOB_NAME}" ]
- id: "deploy to cloud run"
name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args:
[
'beta', 'run', '${_JOB_NAME}',
'--image', 'gcr.io/${PROJECT_ID}/${_JOB_NAME}',
'--region', '${_REGION}',
'--set-env-vars', "BUCKET=${_BUCKET}",
'--set-env-vars', "MNT_DIR=${_MNT_DIR}"
]
images:
- "gcr.io/${PROJECT_ID}/${_JOB_NAME}"
availableSecrets:
secretManager:
- versionName: "projects/${_PROJECT_ID_NUMBER}/secrets/${_CONTAINER_PRIVATE_KEY_SECRET_NAME}/versions/latest"
env: "PRIVATE_KEY"
- versionName: "projects/${_PROJECT_ID_NUMBER}/secrets/${_CONTAINER_PUBLIC_KEY_SECRET_NAME}/versions/latest"
env: "PUBLIC_KEY"

So, after extensive testing and trying out various options I have managed to figure out what was causing the issue, below is the correct argument string (it goes in the args):
["-c", "docker build --build-arg 'CONTAINER_PRIVATE_KEY=$$PRIVATE_KEY' --build-arg 'CONTAINER_PUBLIC_KEY=$$PUBLIC_KEY' -t gcr.io/${PROJECT_ID}/${_JOB_NAME} ."]
The problem was lack of single quotes around build-args' values. Basically, in this context a build-arg value is a single string, not a key-value pair

Related

Cloud Builds Failulre , unable to find logs to see what is going on

i am kicking off a dataflow flex template using a cloud build. In my cloud build file i am attempting to do 3 things
build an image
publish it
run a flex template job using that image
this is my yaml file
substitutions:
_IMAGE: my_logic:latest4
_JOB_NAME: 'pipelinerunner'
_TEMP_LOCATION: ''
_REGION: us-central1
_FMPKEY: ''
_PYTHON_VERSION: '3.8'
# checkout this link https://github.com/davidcavazos/python-docs-samples/blob/master/dataflow/gpu-workers/cloudbuild.yaml
steps:
- name: gcr.io/cloud-builders/docker
args:
[ 'build'
, '--build-arg=python_version=$_PYTHON_VERSION'
, '--tag=gcr.io/$PROJECT_ID/$_IMAGE'
, '.'
]
# Push the image to Container Registry.
- name: gcr.io/cloud-builders/docker2
args: [ 'push', 'gcr.io/$PROJECT_ID/$_IMAGE' ]
- name: gcr.io/$PROJECT_ID/$_IMAGE
entrypoint: python
args:
- /dataflow/template/main.py
- --runner=DataflowRunner
- --project=$PROJECT_ID
- --region=$_REGION
- --job_name=$_JOB_NAME
- --temp_location=$_TEMP_LOCATION
- --sdk_container_image=gcr.io/$PROJECT_ID/$_IMAGE
- --disk_size_gb=50
- --year=2018
- --quarter=QTR1
- --fmpkey=$_FMPKEY
- --setup_file=/dataflow/template/setup.py
options:
logging: CLOUD_LOGGING_ONLY
# Use the Compute Engine default service account to launch the job.
serviceAccount: projects/$PROJECT_ID/serviceAccounts/$PROJECT_NUMBER-compute#developer.gserviceaccount.com
And this is the command i am launching
gcloud beta builds submit \
--config run.yaml \
--substitutions _REGION=$REGION \
--substitutions _FMPKEY=$FMPKEY \
--no-source
The error message i am getting is this
Logs are available at [https://console.cloud.google.com/cloud-build/builds/0f5953cc-7802-4e53-b7c4-7e79c6f0d0c7?project=111111111].
ERROR: (gcloud.beta.builds.submit) build 0f5953cc-7802-4e53-b7c4-7e79c6f0d0c7 completed with status "FAILURE
but i cannot access the logs from the URL mentioned above
I cannot see the logs, so i am unable to see what is wrong, but i stongly suspect somethign in my run.yaml is not quite right
Note: before this, i was building the image myself by launching this command
gcloud builds submit --project=$PROJECT_ID --tag $TEMPLATE_IMAGE .
and my run.yaml just contained 1 step, the last one, and everything worked fine
But i am trying to see if i can do everything in the yaml file
Could anyone advise on what might be incorrect? I dont have much experience with yaml files for cloud build
thanks and regards
Marco
I guess the pipeline does not work because (in the second step) the container: gcr.io/cloud-builders/docker2 does not exist (check https://gcr.io/cloud-builders/ - there is a docker container, but not a docker2one).
This second step pushes the final container to the registry and, it is a dependence of the third step, which will fail too.
You can build the container and push it to the container registry in just one step:
steps:
- name: 'gcr.io/cloud-builders/docker'
args: ['build', '-t', 'gcr.io/$PROJECT_ID/$IMAGE_NAME', '<path_to_docker-file>']
images: ['gcr.io/$PROJECT_ID/$IMAGE_NAME']
Ok, sorted, the problem was the way i was launching the build command
this is the original
gcloud beta builds submit \
--config run.yaml \
--substitutions _REGION=$REGION \
--substitutions _FMPKEY=$FMPKEY \
--no-source
apparently when i removed the --no-source all worked fine.
I think i copied and pasted the command without really understanding it
regards

My cloudbuild.yaml is failing. Please review my cloudbuild.yaml

I am trying to set a react app to a kubernetes cluster. All my kubernetes files resides in k8s/ folder. In k8s/ folder I have a deployment.yaml and service.yaml file.
The below is my cloudbuild.yaml file which resides in the root folder. This part gcr.io/cloud-builders/kubectl Stage 3 is failing. I get the below error
build step 2 "gcr.io/cloud-builders/kubectl" failed: step exited with non-zero status: 1
steps:
# Build the image - Stage 1
- name: 'gcr.io/cloud-builders/docker'
args: ['build','-t','gcr.io/${_PROJECT}/${_CONTAINERNAME}:${_VERSION}','.']
timeout: 1500s
# Push the image - Stage 2
- name: 'gcr.io/cloud-builders/docker'
args: ['push','gcr.io/${_PROJECT}/${_CONTAINERNAME}:${_VERSION}']
# Deploy changes to kubernetes config files - Stage 3
- name: "gcr.io/cloud-builders/kubectl"
args: ["apply", "-f", "k8s/"]
env:
- 'CLOUDSDK_COMPUTE_ZONE=${_ZONE}'
- 'CLOUDSDK_CONTAINER_CLUSTER=${_GKE_CLUSTER}'
# These are variable substitutions
substitutions:
#GCP Specific configuration. Please DON'T change anything
_PROJECT: my-projects-121212
_ZONE: us-central1-c
_GKE_CLUSTER: cluster-1
#Repository Specific configuration. DevOps can change this settings
_DEPLOYMENTNAME: react
_CONTAINERNAME: react
_REPO_NAME: react-app
# Developers ONLY change
_VERSION: v1.0
options:
substitution_option: 'ALLOW_LOOSE'
machineType: 'N1_HIGHCPU_8'
timeout: 2500s
In step 3, there are double quotes name: "gcr.io/cloud-builders/kubectl"
If you replace them with single quotes, the issue should be fixed.

Invalid ENTRYPOINT when deploying Docker image inside Google Cloud

I´m getting this error when I run 'gcloud builds submit --config cloudbuild.yaml' from gcloud-cli.
Step #1: Deploying...
Step #1: Setting IAM Policy.....................................done
Step #1: Creating Revision.....................................................failed
Step #1: Deployment failed
Step #1: ERROR: (gcloud.run.deploy) Cloud Run error: Invalid argument error. Invalid ENTRYPOINT. [name: "gcr.io/customerapi-275705/quickstart-image#sha256:0d1965181fa4c2811c3fcbd63d68de5b4c348ee5b62615594946dea48fee9735"
Step #1: error: "Command \"/quickstart.sh\": invalid mode \"-rw-rw-rw-\" for /quickstart.sh"
Step #1: ].
Finished Step #1
The file is supposed to have '+x' (read/execute) permissions set by chmod. The Windows equivalent would be '/grant User:F'.
Step #1: error: "Command \"/quickstart.sh\": invalid mode \"-rw-rw-rw-\" for /quickstart.sh"
-rw-rw-rw seems about right to me. What am I missing?
This is in my Dockerfile
FROM alpine
COPY quickstart.sh /
CMD ["\/quickstart.sh"]
And this is my cloudbuild.yaml
steps:
- name: 'gcr.io/cloud-builders/docker'
args: [ 'build', '-t', 'gcr.io/$PROJECT_ID/quickstart-image', '.' ]
- name: 'gcr.io/cloud-builders/gcloud'
args:
- 'run'
- 'deploy'
- 'myservice'
- '--image'
- 'gcr.io/$PROJECT_ID/quickstart-image'
- '--region'
- 'europe-north1'
- '--platform'
- 'managed'
- '--allow-unauthenticated'
images:
- 'gcr.io/$PROJECT_ID/quickstart-image'
I believe that in the Cloud Build environment sandbox your quickstart.sh does not have execution permissions, which you can check adding this step to your Cloud Build cloudbuild.yaml config file:
- name: 'ubuntu'
entrypoint: 'bash'
args:
- '-c'
- |
ls -lart
I am not sure if the Cloud Build sandbox will allow you to give execution permissions to a bash script but you might try to do it by adding another step with chmod +x quickstart.sh.
I was having the same problem a few hours ago, I fixed it by adding an exec form ENTRYPOINT to the end of the Dockerfile.
I tried the shell form ENTRYPOINT, but it didn't work, presumably because of the following:
The shell form prevents any CMD or run command line arguments from being used, but has the disadvantage that your ENTRYPOINT will be started as a subcommand of /bin/sh -c, which does not pass signals. This means that the executable will not be the container’s PID 1 - and will not receive Unix signals - so your executable will not receive a SIGTERM from docker stop <container>.
GCP probably needs to pass some command line arguments.
source

Kubernetes - How to setup parallels clusters (Prod, Dev) sharing the same repositories/pipeline

I am currently facing a situation where I need to deploy a small cluster (only 4 pods for now) which will be containing 4 different microservices. This cluster has to be duplicated so I can have one PRODUCTION cluster and one DEVELOPMENT cluster.
Even if it's not hard from my point of view (Creating a cluster and then uploading docker images to pods with parameters in order to use the right resources connection strings), I am stuck at the CD/CI part..
From a CloudBuild trigger, how to push the Docker image to the right "cluster's pod", I have absolutely no idea AT ALL how to achieve it...
There is my cloudbuild.yaml
steps:
#step 1 - Getting the previous (current) image
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args: [
'-c',
'docker pull gcr.io/{PROJECT_ID}/{SERVICE_NAME}:latest || exit 0'
]
#step 2 - Build the image and push it to gcr.io
- name: 'gcr.io/cloud-builders/docker'
args: [
'build',
'-t',
'gcr.io/{PROJECT_ID}/{SERVICE_NAME}',
'-t',
'gcr.io/{PROJECT_ID}/{SERVICE_NAME}:latest',
'.'
]
#step 3 - Deploy our container to our cluster
- name: 'gcr.io/cloud-builders/kubectl'
args: ['apply', '-f', 'service.yaml', '--force']
env:
- 'CLOUDSDK_COMPUTE_ZONE={CLUSTER_ZONE}'
- 'CLOUDSDK_CONTAINER_CLUSTER={CLUSTER_NAME}'
#step 4 - Set the image
- name: 'gcr.io/cloud-builders/kubectl'
args: [
'set',
'image',
'deployment',
'{SERVICE_NAME}',
'{SERVICE_NAME}=gcr.io/{PROJECT_ID}/{SERVICE_NAME}'
]
env:
- 'CLOUDSDK_COMPUTE_ZONE={CLUSTER_ZONE}'
- 'CLOUDSDK_CONTAINER_CLUSTER={CLUSTER_NAME}'
# push images to Google Container Registry with tags
images: [
'gcr.io/{PROJECT_ID}/{SERVICE_NAME}',
'gcr.io/{PROJECT_ID}/{SERVICE_NAME}:latest'
]
Can anyone help me out? I don't really know in which direction to go to..
Did you know helm chart? It is designed for different environment deployment.
With different values.yaml file, you can quickly deploy to different environment with same source code base.
For example, you can name the values.yaml file with environment.
values-dev.yaml
values-sit.yaml
values-prod.yaml
the only differences are some varialbes, such as environment (dev/sit/prod), and namespaces.
so when you run the deployment, it will be:
env=${ENVIRONMENT}
helm install -f values-${env}.yaml myredis ./redis
So my question is:
How are you triggering these builds? Manually? GitHub Trigger? HTTP Trigger using the REST API?
so you're almost there for the building/pushing part, you would need to use substitution variables https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values
If you would be triggering the builds manually, you would edit the build trigger and change the sub variable for what you want it to be.
GitHub Trigger -- this is a little more complex as you might want to do releases or branches.
HTTP Trigger, same as manual, in your request you change the sub variable.
So here's part of one of our repo build files, as you will see there are different sub. variables we use, sometimes we want to build the image AND deploy to cluster, other times we just want to build or deploy.
steps:
# pull docker image
- name: 'gcr.io/cloud-builders/docker'
id: pull-docker-image
entrypoint: 'bash'
args:
- '-c'
- |
docker pull $${_TAG_DOCKER_IMAGE} || exit 0
# build docker image
- name: 'gcr.io/cloud-builders/docker'
id: build-docker-image
entrypoint: 'bash'
args:
- '-c'
- |
if [[ "${_BUILD_IMAGE}" == "true" ]]; then
docker build -t ${_DOCKER_IMAGE_TAG} --cache-from $${_TAG_DOCKER_IMAGE} .;
else
echo "skipping ... BUILD_IMAGE=${_BUILD_IMAGE}";
fi
# push docker image
- name: 'gcr.io/cloud-builders/docker'
id: push-docker-image
entrypoint: 'bash'
args:
- '-c'
- |
if [[ "${_BUILD_IMAGE}" == "true" ]]; then
docker push ${_DOCKER_IMAGE_TAG};
else
echo "skipping ... BUILD_IMAGE=${_BUILD_IMAGE}";
fi
# tag docker image
- name: 'gcr.io/cloud-builders/gcloud'
id: tag-docker-image
entrypoint: 'bash'
args:
- '-c'
- |
if [[ "${_BUILD_IMAGE}" == "true" ]]; then
gcloud container images add-tag ${_DOCKER_IMAGE_TAG} $${_TAG_DOCKER_IMAGE} -q;
else
echo "skipping ... BUILD_IMAGE=${_BUILD_IMAGE}";
fi
# update service image on environment
- name: 'gcr.io/cloud-builders/kubectl'
id: update service deployment image
entrypoint: 'bash'
args:
- '-c'
- |
if [[ "${_UPDATE_CLUSTER}" == "true" ]]; then
/builder/kubectl.bash set image deployment $REPO_NAME master=${_DOCKER_IMAGE_TAG} --namespace=${_DEFAULT_NAMESPACE};
else
echo "skipping ... UPDATE_CLUSTER=${_UPDATE_CLUSTER}";
fi
env:
- 'CLOUDSDK_COMPUTE_ZONE=${_CLOUDSDK_COMPUTE_ZONE}'
- 'CLOUDSDK_CONTAINER_CLUSTER=${_CLOUDSDK_CONTAINER_CLUSTER}'
# subs are needed because of our different ENVs
# _DOCKER_IMAGE_TAG = ['gcr.io/$PROJECT_ID/$REPO_NAME:gcb-${_COMPANY_ENV}-$SHORT_SHA', 'other']
# _COMPANY_ENV = ['dev', 'staging', 'prod']
# _DEFAULT_NAMESPACE = ['default'] or ['custom1', 'custom2']
# _CLOUDSDK_CONTAINER_CLUSTER = ['dev', 'prod']
# _CLOUDSDK_COMPUTE_ZONE = ['us-central1-a']
# _BUILD_IMAGE = ['true', 'false']
# _UPDATE_CLUSTER = ['true', 'false']
substitutions:
_DOCKER_IMAGE_TAG: $DOCKER_IMAGE_TAG
_COMPANY_ENV: dev
_DEFAULT_NAMESPACE: default
_CLOUDSDK_CONTAINER_CLUSTER: dev
_CLOUDSDK_COMPUTE_ZONE: us-central1-a
_BUILD_IMAGE: 'true'
_UPDATE_CLUSTER: 'true'
options:
substitution_option: 'ALLOW_LOOSE'
env:
- _TAG_DOCKER_IMAGE=gcr.io/$PROJECT_ID/$REPO_NAME:${_COMPANY_ENV}-latest
- DOCKER_IMAGE_TAG=gcr.io/$PROJECT_ID/$REPO_NAME:gcb-${_COMPANY_ENV}-$SHORT_SHA
tags:
- '${_COMPANY_ENV}'
- 'build-${_BUILD_IMAGE}'
- 'update-${_UPDATE_CLUSTER}'
we have two workflows --
github trigger builds and deploys under the 'dev' environment.
we trigger via REST API https://cloud.google.com/cloud-build/docs/api/reference/rest/v1/projects.builds/create (we replace the variables via the request.json) -- this method also works using the gcloud builds --substitutions CLI.
Hope that answers your question!
The short answer for this is to apply GitOps deployment practices in your workflow.
All your Kubernetes YAMLs or Helmcharts are in a single git repository which is used by a GitOps operator.
In your CI pipeline, you only have to build and push docker images.
The GitOps operator intelligently fetches images versions and make changes and a commit to the target Kubernetes YAML in the repository.
Every change in the GitOps repository is applied to the cluster.
See https://fluxcd.io/

GitHub CI error parsing HTTP 404 response body when pushing the image

Coming from this issue
I am using GitHub Actions for Gradle project with this given steps:
name: Java CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v1
- name: Set up JDK 13
uses: actions/setup-java#v1
with:
java-version: 13
- run: ./gradlew bootJar
- name: Login to Github regestry
run: docker login docker.pkg.github.com -u xxxxx -p xxxxx
- name: Build the Docker image
run: docker build . -t docker.pkg.github.com/sulimanlab/realtime-chat/realtimechat-snapshot-0.$GITHUB_REF
- name: Push the image to github
run: docker push docker.pkg.github.com/sulimanlab/realtime-chat/realtimechat-snapshot-0.$GITHUB_REF
At the last step I get this error:
The push refers to repository
[docker.pkg.github.com/sulimanlab/realtime-chat/realtimechat-snapshot-0.refs/heads/master]
3aad04996f8f: Preparing
77cae8ab23bf: Preparing
error parsing HTTP 404 response body: invalid character 'p' after top-level value:
"404 page not found\n"
actually I was using the wrong environment variable to tag my images.
I used $GITHUB_REF what I should use $GITHUB_SHA

Resources