I intend to pass my npm token to gcp cloud build,
so that I can use it in a multistage build, to install private npm packages.
I have the following abridged Dockerfile:
FROM ubuntu:14.04 AS build
ARG NPM_TOKEN
RUN echo "NPM_TOKEN:: ${NPM_TOKEN}"
and the following abridged cloudbuild.yaml:
---
steps:
- name: gcr.io/cloud-builders/gcloud
entrypoint: 'bash'
args: [ '-c', 'gcloud secrets versions access latest --secret=my-npm-token > npm-token.txt' ]
- name: gcr.io/cloud-builders/docker
args:
- build
- "-t"
- gcr.io/my-project/my-program
- "."
- "--build-arg NPM_TOKEN= < npm-token.txt"
- "--no-cache"
I based my cloudbuild.yaml on the documentation, but it seems like I am not able to put two and two together, as the expression: "--build-arg NPM_TOKEN= < npm-token.txt" does not work.
I have tested the DockerFile, when I directly pass in the npm token, and it works. I simply have trouble passing in a token from gcloud secrets as a build argument to docker.
Help is greatly appreciated!
Your goal is to get the secret file contents into the build argument. Therefore you have to read the file content using either NPM_TOKEN="$(cat npm-token.txt)"or NPM_TOKEN="$(< npm-token.txt)".
name: gcr.io/cloud-builders/docker
entrypoint: 'bash'
args: [ '-c', 'docker build -t gcr.io/my-project/my-program . --build-arg NPM_TOKEN="$(cat npm-token.txt)" --no-cache' ]
Note: The gcr.io/cloud-builders/docker however use exec entrypoint form. Therefore you set entrypoint to bash.
Also note that you save the secret to the build workspace (/workspace/..). This also allows you to copy the secret as a file into your container.
FROM ubuntu:14.04 AS build
ARG NPM_TOKEN
COPY npm-token.txt .
RUN echo "NPM_TOKEN:: $(cat npm-token.txt)"
I won't write your second step like you did, but like this:
- name: gcr.io/cloud-builders/docker
entrypoint: "bash"
args:
- "-c"
- |
build -t gcr.io/my-project/my-program . --build-arg NPM_TOKEN=$(cat npm-token.txt) --no-cache
Related
Cloud build do the following:
Build image from dockerfile (see dockerfile below)
Push image to container registry
Update service in Cloud Run
My issue is the following:
As I'm running my tests on build time, I need my MONGODB_URI secret on build time, but I've read that using --build-arg is not safe to expose secrets.
I could run npm install and npm run test in Cloud Build container, but it would make build time longer as I'll have to run npm install two times.
Is there a way I can run only once npm install without having to expose secrets ?
Dockerfile
FROM node:16
COPY . ./
WORKDIR /
RUN npm install
ARG env
ARG mongodb_uri
ENV ENVIRONMENT=$env
ENV DB_REMOTE_PROD=$mongodb_uri
RUN npm run test
RUN npm run build
CMD ["node", "./build/index.js"]
And my cloudbuild.yaml config:
steps:
- name: gcr.io/cloud-builders/docker
args:
- '-c'
- >-
docker build --no-cache -t
$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA
--build-arg ENVIRONMENT=staging --build-arg mongodb_uri=$$MONGODB_URI -f
Dockerfile .
id: Build
entrypoint: bash
secretEnv:
- MONGODB_URI
- name: gcr.io/cloud-builders/docker
args:
- push
- '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
id: Push
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk:slim'
args:
- '-c'
- >-
gcloud run services update $_SERVICE_NAME --platform=managed
--image=$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA
--labels=managed-by=gcp-cloud-build-deploy-cloud-run,commit-sha=$COMMIT_SHA,gcb-build-id=$BUILD_ID,gcb-trigger-id=$_TRIGGER_ID
--region=$_DEPLOY_REGION --update-env-vars=ENVIRONMENT=staging
--update-env-vars=DB_REMOTE_PROD=$$MONGODB_URI --quiet
id: Deploy
entrypoint: bash
secretEnv:
- MONGODB_URI
images:
- '$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA'
options:
substitutionOption: ALLOW_LOOSE
logging: CLOUD_LOGGING_ONLY
substitutions:
_TRIGGER_ID: 44b16efe-0219-41af-b32b-9b98438728c3
_GCR_HOSTNAME: eu.gcr.io
_PLATFORM: managed
_SERVICE_NAME: app-staging
_DEPLOY_REGION: europe-southwest1
availableSecrets:
secretManager:
- versionName: projects/PROJECT_ID/secrets/mongodb_app_staging/versions/1
env: MONGODB_URI
With my method, the secret is exposed not only in the container since I pass it as build-args, but it is also exposed to all users with access to cloud run...
Your container build step is not so bad. You provide your secrets to your container, but it is not visible. It's only reference.
You can hardly do better. A solution could be to perform the secret access directly inside the Dockerfile, but I'm not sure that the "security gain" worth the Dockerfile increased complexity. (If you want more details, let me know, I will be able to share sample)
About Cloud Run, you are totally true: exposing secret in plain text in environment variable is totally not acceptable.
For that, you can use the Cloud Run - Secret Manager integration. Instead of doing this
gcloud run services update $_SERVICE_NAME --platform=managed
--image=$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA
--labels=managed-by=gcp-cloud-build-deploy-cloud-run,commit-sha=$COMMIT_SHA,gcb-build-id=$BUILD_ID,gcb-trigger-id=$_TRIGGER_ID
--region=$_DEPLOY_REGION --update-env-vars=ENVIRONMENT=staging
--update-env-vars=DB_REMOTE_PROD=$$MONGODB_URI --quiet
You can do that (and you now longer need to set the secret as env var of your Cloud Build step)
gcloud run services update $_SERVICE_NAME --platform=managed
--image=$_GCR_HOSTNAME/$PROJECT_ID/$REPO_NAME/$_SERVICE_NAME:$COMMIT_SHA
--labels=managed-by=gcp-cloud-build-deploy-cloud-run,commit-sha=$COMMIT_SHA,gcb-build-id=$BUILD_ID,gcb-trigger-id=$_TRIGGER_ID
--region=$_DEPLOY_REGION --update-env-vars=ENVIRONMENT=staging
--update-secrets=DB_REMOTE_PROD=mongodb_app_staging:1 --quiet
Background
in my gitlab-ci file I am trying to build a docker image, however even though I have docker:dind as a service, it is failing.
.gitlab-ci
---
stages:
- build
- docker
build:
stage: build
image: fl4m3ph03n1x/my-app:1.0
variables:
MIX_ENV: prod
script:
- mix deps.get
- mix deps.compile
- mix compile
artifacts:
paths:
- .hex/
- _build/
- deps/
- mix.lock
build_image:
stage: docker
image: fl4m3ph03n1x/my-app:1.0
variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: ""
DOCKER_HOST: tcp://docker:2375/
services:
- docker:dind
script:
- echo ${CI_JOB_TOKEN} | docker login --password-stdin -u ${CI_REGISTRY_USER} ${CI_REGISTRY}
- docker build . -t ${CI_REGISTRY_IMAGE}:latest
- docker push ${CI_REGISTRY_IMAGE}:latest
The problematic stage is docker.
As you can see I am trying to:
login into docker
build an image from gitlab's registry
push that image
Error
However, I am getting the following error:
$ echo ${CI_JOB_TOKEN} | docker login --password-stdin -u
${CI_REGISTRY_USER} ${CI_REGISTRY} /bin/bash: line 110: docker:
command not found
Which is confusing, because docker:dind is supposed to actually prevent this from happening:
https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#enable-registry-mirror-for-dockerdind-service
Question
So clearly I am missing something here. What am I doing wrong?
EDIT
This is my Dockerfile
FROM elixir:1.10
# Install Hex + Rebar
RUN mix do local.hex --force, local.rebar --force
COPY . /
WORKDIR /
ENV MIX_ENV=prod
RUN mix do deps.get --only $MIX_ENV, deps.compile
RUN mix release
EXPOSE 8080
ENV PORT=8080
ENV SHELL=/bin/bash
CMD ["_build/prod/rel/my_app/bin/my_app", "start"]
image is used to specify the image in which to run the script. You want to run the script in a docker image, to build your image.
The image keyword is the name of the Docker image the Docker executor runs to perform the CI tasks.
https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#define-image-and-services-from-gitlab-ciyml
After all, isn't your application image CI_REGISTRY_IMAGE in this? You don't want to build the image in itself.
- docker build . -t ${CI_REGISTRY_IMAGE}:latest
- docker push ${CI_REGISTRY_IMAGE}:latest
I have a command like this:
docker build -f ${DOCKERFILE} ${BUILD_ARGS} -t ${IMAGE} ${BUILD_CONTEXT}
When I pass these environment variables it works:
IMAGE_SUBNAME: 'frontend'
DOCKERFILE: ./frontend/Dockerfile
BUILD_CONTEXT: ./frontend
But when I try to pass these, it throw me error (see below):
IMAGE_SUBNAME: 'frontend'
DOCKERFILE: ./frontend/Dockerfile
BUILD_CONTEXT: ./frontend
BUILD_ARGS: ENV=${CI_COMMIT_REF_SLUG} // CI_COMMIT_REF_SLUG equals to "dev"
"docker build" requires exactly 1 argument.
I have already read a lot of articles, but I can't find my mistake. Hope to get help.
In my action.yml I defined an input:
name: 'test action'
author: Param Thakkar
description: 'test'
inputs:
test_var:
description: 'A test variable'
required: true
runs:
using: 'docker'
image: 'Dockerfile'
And in my workflow I passed the test_var:
name: CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Test the GH action
uses: paramt/github-actions-playground#master
with:
test_var: "this is just a test"
So there should be an environment variable that's created when the workflow runs, right? But when I run this short python script:
import os
print(os.getenv('TEST_VAR'))
print("It works!")
exit(0)
It prints:
None
It works!
I think that I have to pass the ENV variable through my Dockerfile... Right now my Dockerfile looks like this:
FROM python:latest
# Add files to the image
ADD entrypoint.py /entrypoint.py
ADD requirements.txt /requirements.txt
# Save ENV var in a temp file
RUN $TEST_VAR > /temp_var
# Install dependencies and make script executable
RUN pip install -r requirements.txt
RUN chmod +x entrypoint.py
RUN echo "temp var: "
RUN cat /temp_var
# Run script with the ENV var
ENTRYPOINT export TEST_VAR="$TEST_VAR"; /entrypoint.py
But the variable isn't echoed and isn't passed to the pythons script either.. am I missing something? When I tried to set my $TEMP_VAR to a random piece of string, it is sent through to the Python script. Is this a mistake on my behalf or is the GitHub action not working as intended?
Here's the link to the test repo
I think you are trying to read the wrong environment variable name. GitHub Actions adds INPUT_ to the name of the input variable. So try the following:
print(os.getenv('INPUT_TEST_VAR'))
From the documentation:
When you specify an input to an action in a workflow file or use a
default input value, GitHub creates an environment variable for the
input with the name INPUT_. The environment variable
created converts input names to uppercase letters and replaces spaces
with _ characters.
For example, if a workflow defined the numOctocats and octocatEyeColor
inputs, the action code could read the values of the inputs using the
INPUT_NUMOCTOCATS and INPUT_OCTOCATEYECOLOR environment variables.
https://help.github.com/en/articles/metadata-syntax-for-github-actions#inputs
A bit late but for the next one, you can also use the env field :
name: CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Test the GH action
uses: paramt/github-actions-playground#master
env:
test_var: "this is just a test"
which will be included during the creation of your docker and pass without the prefix INPUT_
Keep env vars secret by specifying them in Settings -> Secrets in the repo, and then calling them in the workflow:
For example, consider a workflow that runs an R script followed by a Python script. First, in .github/workflows/my_job.yml notice the MY_VAR variable, which points to a stored secret with ${{ secrets.MY_VAR}}. The rest is standard code (run on cron, specify Ubuntu OS and Docker image, define workflow steps).
on:
schedule:
- cron: '0 17 * * *'
jobs:
my_job:
name: my job
env:
MY_VAR: ${{ secrets.MY_VAR }}
runs-on: ubuntu-18.04
container:
image: docker.io/my_username/my_image:my_tag
steps:
- name: checkout_repo
uses: actions/checkout#v2
- name: run some code
run: bash ./src/run.sh
Next, in the scripts that compose your workflow, you can access the env var specified in the workflow file above as you would locally.
For example, in the repo, let's assume src/run.sh calls an R script followed by a Python script.
In R access the env var and store as an object:
my_var <- Sys.getenv("MY_VAR")
.
.
.
In Python access the env var and store as an object:
import os
my_var = os.getenv("MY_VAR")
.
.
.
See the docs here.
In my case, none of the answers worked. Here's how I fixed it.
---
name: Build and Push Docker Image to AWS ECR
on:
push:
branches: [ master ]
env:
FOO: '${{ secrets.FOO }}'
jobs:
build-and-push:
name: Build Project and Push to AWS ECR
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout#v2
...
- name: Build and Push to AWS ECR
id: build-image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
run: |
docker build --build-arg FOO=$FOO -t $ECR_REGISTRY/crew-charge-app:latest .
docker push $ECR_REGISTRY/crew-charge-app:latest
I first had to get the FOO variable from github secrets using ${{secrets.FOO}} then pass it onto the docker file using docker build --build-arg FOO=$FOO --build-arg BAR=$BAR -t .
Then inside the docker file I had to declare both as an ARG and ENV to be available at all times.
FROM node:14
ARG FOO=${FOO}
ENV FOO=${FOO}
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
COPY package.json /usr/src/app
RUN yarn install
COPY . /usr/src/app
RUN FOO=$FOO yarn build
EXPOSE 80
CMD ["yarn", "start" ]
Important part was to RUN FOO=$FOO yarn build because setting the ENV alone doesn't pass it onto the container.
- name: 'gcr.io/cloud-builders/mvn'
args: ['clean',
'package',
'-Ddockerfile.skip',
'-DskipTests'
]
- name: 'gcr.io/cloud-builders/mvn'
args: ['dockerfile:build',
'-Ddockerfile.skip',
'-DskipTests'
]
when I run these two commands on top locally, I do have the target folder with docker folder and the image-name file in it
On this step it fails:
..
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- -c
- |
docker push $(cat /workspace/target/docker/image-name)
cat: /workspace/target/docker/image-name: No such file or directory
I tried target/docker, app/target/docker
In My Dockerfile:
...
WORKDIR /app
...
ADD target/${JAR_FILE} app.jar
...
Question: how to see target folder, how to make
docker push $(cat /workspace/target/docker/image-name) work?
similar to answer here: Cloud Build fails to build the the simple build step with maven
how to see target folder
there isn't currently a way to check the remote workspace for the target folder, but you can debug with cloud-build-local and write the workspace locally.
https://cloud.google.com/cloud-build/docs/build-debug-locally
https://github.com/GoogleCloudPlatform/cloud-build-local
make sure that target/ or .jar files are not being ignored by gcloudignore or gitignore
https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore
https://github.com/GoogleCloudPlatform/cloud-builders/issues/236
I also wonder if the docker step is not picking up on what is being produced by the dockerfile plugin, does dockerfile:push work?
steps:
- name: 'gcr.io/cloud-builders/mvn'
args: ['dockerfile:build']
- name: 'gcr.io/cloud-builders/mvn'
args: ['dockerfile:push']