I want to automate my build process and need to pass an environment variable to run some of the commands in the Dockerfile. I was wondering if there was any way to do this in Dockerhub. I know docker cloud has something like this, but I was wondering whether the functionality was there in Dockerhub since there is the --build-args argument in the cli for normal building.
Set up Automated builds
Docker Hub (https://hub.docker.com) can automatically build images from source code in an external repository and automatically push the built image to your Docker repositories which will be hosted under your Docker Hub repositories account Eg: https://cloud.docker.com/u/binbash/repository/list
When you set up automated builds (also called autobuilds), you create a list of branches and tags that you want to build into Docker images. When you push code to a source code branch (currently only GitHub / Bitbucket are supported) for one of those listed image tags, the push uses a webhook to trigger a new build, which produces a Docker image. The built image is then pushed to the Docker Hub registry.
For detailed implementation steps please refer to https://docs.docker.com/docker-hub/builds/
Environment variables for builds
You can set the values for environment variables (actually they are mapped to build ARG values - docker build --build-arg - to be exclusively used at build-time - https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg).
NOT to be confused with the environment values, ENV VARS, used by your service at runtime (docker run --env MYVAR1=foo - https://docs.docker.com/v17.12/edge/engine/reference/commandline/run/#set-environment-variables--e-env-env-file)
These Environment Variables configured from the Docker Hub UI are used in your build processes when you configure an automated build. Add your build environment variables by clicking the plus sign next to the Build environment variables section, and then entering a variable name and the value.
When you set variable values from the Docker Hub UI, they can be used by the commands you set in hooks files (THIS IS VERY IMPORTANT and will be extended below), but they are stored so that only users who have admin access to the Docker Hub repository can see their values. This means you can use them to safely store access tokens or other information that should remain secret.
Build hook examples (to implement Docker Hub UI Env vars)
Adding variables from the auto-build’s web UI makes them available inside the hooks. In the hook, you’ll have to use that value to set a custom build arg using --build-arg. Finally, you have to use this custom build arg inside your Dockerfile to manually set an environment variable using ENV command or export.
Example:
Say your want an environment variable TERRAFORM_VERSION='0.12.0-beta2' in your build environment
Step 1.
Add this in the auto-build’s web UI for ‘build environment variables’
Step 2.
Create a custom build hook i.e create a folder called hooks in the same directory as your Dockerfile. Inside the hooks folder, create a file called build. This creates the custom build hook. Docker will use this to build your image. Contents of build:
#!/bin/bash
docker build -t $IMAGE_NAME --build-arg TERRAFORM_VERSION=$TERRAFORM_VERSION .
NOTE: Here $TERRAFORM_VERSION is coming from the web UI.
Step3:
In your Dockerfile
ARG TERRAFORM_VERSION
ENV TERRAFORM_VERSION $TERRAFORM_VERSION
NOTE: Here $TERRAFORM_VERSION is coming from the custom build args in your bash script file named build.
Complete example: https://github.com/binbashar/public-docker-images/tree/master/terraform-resources
That's it! It should work now. Probably renaming ‘build environment variables’ to ‘custom hook environment variables’ in Docker Hub will ease the understanding of this concept in the official documentation (https://docs.docker.com/docker-hub/builds/advanced/).
Extra Points!
There are a number of key environment arguments set upon launching a build script, all of which you can use in your hooks and which can all be useful in making custom build-args.
SOURCE_BRANCH: the name of the branch or the tag that is currently being tested.
SOURCE_COMMIT: the SHA1 hash of the commit being tested.
COMMIT_MSG: the message from the commit being tested and built.
DOCKER_REPO: the name of the Docker repository being built.
DOCKERFILE_PATH: the Dockerfile currently being built.
DOCKER_TAG: the Docker repository tag being built.
IMAGE_NAME: the name and tag of the Docker repository being built. (This variable is a combination of DOCKER_REPO:DOCKER_TAG.)
An example:
Step 1. Create the Dockerfile:
ARG NODE_VERSION
FROM node:$NODE_VERSION
Step 2. Create the hooks/build file:
#!/bin/bash
NODE_VERSION=$(echo $DOCKER_TAG | cut -d "-" -f2)
if [ $DOCKER_TAG == "latest" ]
then
docker build . --build-arg NODE_VERSION=${DOCKER_TAG} -t ${IMAGE_NAME}
else
docker build . --build-arg NODE_VERSION=${NODE_VERSION} -t ${IMAGE_NAME}
fi
Source: github.com/SamuelA
Related
My local development environment is behind a corporate proxy, with our own Docker registry, etc. However, we deploy on public infrastructure, meaning we can't access the corporate registries, and so have to pull from a public one (eg DockerHub).
Is there any way (eg via environment variables) for me to configure Docker to pull from a private registry when developing locally, and from a public registry when it goes through our CI/CD pipeline?
For example, let's say we're deploying a Node.JS application - locally, I would want the FROM node:16 line to get interpreted as FROM corporate.proxy/node:16.
There are a couple methods that would probably work - having two separate Dockerfiles, eg Dockerfile.dev and Dockerfile.prod, or wrapping it in some sort of script that will take care of making the change. I'm looking for a way to do it via Docker's configuration, if it's possible at all.
You can use ARG instruction to change the FROM line in Dockerfile.
ARG IMG
FROM ${IMG}
Then you can build image like this:
docker build --build-arg IMG=node:16 .
or
docker build --build-arg IMG=corporate.proxy/node:16 .
From Dockerfile reference document:
ARG is the only instruction that may precede FROM in the Dockerfile
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
I am using ubuntu 20.04
while following book -> https://www.oreilly.com/library/view/kubeflow-for-machine/9781492050117/
on page 17, it says the following (only relevant parts) which I don't understand....
You will want to store container images called a
container registry. The container registry will be accessed by your Kubeflow cluster.
I am going to use docker hub as container registry. Next
we'll assume that you've set your container registry via an environment variable
$CONTAINER_REGISTRY, in your shell" NOTE: If you use registry that isn't on Google Cloud
Platform, you will need to configure Kubeflow pipelines container builder to have access to
your registry by following the Kaniko configuration guide -> https://oreil.ly/88Ep-
First, I do not understand how to set container registry through environment variable, am I supposed to give it a link??
Second, I've gone into Kaniko config guide and did everything as told -> creating config.json with "auth":"mypassword for dockerhub". After that In the book it says:
To make sure your docker installation is properly configured, you can write one line Dc and
push it to your registry."
Example 2.7 Specify the new container is built on top of Kubeflow's
container
FROM gcr.io/kubeflow-images-public/tensorflow-2.1.0-notebook-cpu:1.0.0
Example 2.8 Build new container and push to registry for use
IMAGE="${CONTAINER_REGISTRY}/kubeflow/test:v1"
docker build -t "${IMAGE}" -f Dockerfile . docker push "${IMAGE}"
I've created Dockerfile with code from Example2.7 inside it, then ran code from Example 2.8 however not working.
Make sure that:
You set the environment variable using export CONTAINER_REGISTRY=docker.io/your_username in your terminal (or in your ~/.bash_profile and run source ~/.bash_profile).
Your .docker/config.json does not have your password in plain text but in base64, for example the output of echo -n 'username:password' | base64
The docker build and the docker push are two separate commands, in your example they're seen as one command, unlike the book.
I am using Jenkins to build and publish the docker image. However I need to use two docker file in one application just for my use case.
Requirement is can I build and published two different docker image from two different Docker file from one source, or i need to use custom file name like argument *-f* in docker command so that I can build two pipeline, in Jenkins?
By default Jenkins picks the file with *Dockerfile*
ex: docker build -t dockerfile -f Dockerfile-custom-name .
def projectImage = docker.build("imageName:tag", "-f Dockerfile-custom-name .")
This way you can even add --build-arg or any other argument before the -f. It is important to have the -f and . at the end of the second parameter.
Here's the documentation for it:
Builds test-image from the Dockerfile found at ./dockerfiles/test/Dockerfile.
Edit (answer to comments):
Yes, my first solution proposal works in a Jenkins pipeline.
This should also work:
sh '''docker build -t docker1-tag:latest us.gcr.io/project-name/docker1-tag:latest -f "Dockerfile1" .'''
The only downside of this one, is that you don't have the image in a variable, so you cannot use the run, withRun, push and other methods.
In your specific case, I would build the image with the name/tag of us.gcr.io/project-name/docker1-tag:latest, so I can call the push method and the image will be pushed to the registry. To have the second tag, a separate sh '''docker tag us.gcr.io/project-name/docker1-tag:latest docker1-tag:latest''' call would suffice.
What I'd like to do is this:
FROM mcr.microsoft.com/dotnet/core/aspnet:3.0-alpine
# more stuff
But, this needs to run on an isolated clean build machine which does not have internet access, so I need to route it through a local mirror server (e.g. Artifactory or Nexus or another similar thing)
If the docker image were hosted on docker hub (e.g. FROM ubuntu) then the --registry-mirrors docker configuration option would solve it for us.
BUT because microsoft have decided not to use docker hub, the registry mirror thing isn't working either.
After much effort, I've set up a a custom domain mirror for mcr.microsoft.com and so now I can do this:
FROM microsoft-docker-mirror.local.domain/dotnet/core/aspnet:3.0-alpine
# more stuff
That works. But we have remote workers who may not be on the local office LAN and can't see my local mirror server. What I now want to do is vary it depending on environment. E.g.
ENV MICROSOFT_DOCKER_REPO mcr.microsoft.com
FROM ${MICROSOFT_DOCKER_REPO}/dotnet/core/aspnet:3.0-alpine
My isolated build machine would set the MICROSOFT_DOCKER_REPO environment variable, and everyone else's machines would use the default mcr.microsoft.com
Anyway. Adding the ENV line to the dockerfile results in docker throwing this error:
Error response from daemon: No build stage in current context
There seem to be lots of references to the fact that the FROM line must be the first line in the file, before even any comments...
How can I reference an environment variable in my FROM statement? Or alternatively, how can I make registry mirrors work for things that aren't docker hub?
Thanks!
"ARG is the only instruction that may precede FROM in the Dockerfile" - Dockerfile reference.
ARG MICROSOFT_DOCKER_REPO=mcr.microsoft.com
FROM ${MICROSOFT_DOCKER_REPO}/dotnet/core/aspnet:3.0-alpine
Build with a non-default MICROSOFT_DOCKER_REPO using the --build-arg: docker build --rm --build-arg MICROSOFT_DOCKER_REPO=repo.example.com -t so:58196638 .
Note: you can pass the --build-arg from the hosts environment i.e. MICROSOFT_DOCKER_REPO=repo.example.com docker build --rm --build-arg MICROSOFT_DOCKER_REPO=${MICROSOFT_DOCKER_REPO} -t so:58196638 .
I've got a jenkins server monitoring a git repo and building a docker image on code change. The .git directory is ignored as part of the build, but I want to associate the git commit hash with the image so that I know exactly what version of the code was used to make it and check whether the image is up to date.
The obvious solution is to tag the image with something like "application-name-branch-name:commit-hash", but for many develop branches I only want to keep the last good build, and adding more tags will make cleaning up old builds harder (rather than using the jenkins build number as the image is built, then retagging to :latest and untagging the build number)
The other possibility is labels, but while this looked promising initially, they proved more complicated in practice..
The only way I can see to apply a label directly to an image is in the Dockerfile, which cannot use the build environment variables, so I'd need to use some kind of templating to produce a custom Dockerfile.
The other way to apply a label is to start up a container from the image with some simple command (e.g. bash) and passing in the labels as docker run arguments. The container can then be committed as the new image. This has the unfortunate side effect of making the image's default command whatever was used with the labelling container (so bash in this case) rather than whatever was in the original Dockerfile. For my application I cannot use the actual command, as it will start changing the application state.
None of these seem particularly ideal - has anyone else found a better way of doing this?
Support for this was added in docker v1.9.0, so updating your docker installation to that version would fix your problem if that is OK with you.
Usage is described in the pull-request below:
https://github.com/docker/docker/pull/15182
As an example, take the following Dockerfile file:
FROM busybox
ARG GIT_COMMIT=unknown
LABEL git-commit=$GIT_COMMIT
and build it into an image named test as anyone would do naïvely:
docker build -t test .
Then inspect the test image to check what value ended up for the git-commit label:
docker inspect -f '{{index .ContainerConfig.Labels "git-commit"}}' test
unkown
Now, build the image again, but this time using the --build-arg option:
docker build -t test --build-arg GIT_COMMIT=0123456789abcdef .
Then inspect the test image to check what value ended up for the git-commit label:
docker inspect -f '{{index .ContainerConfig.Labels "git-commit"}}' test
0123456789abcdef
References:
Docker build command documentation for the --build-arg option
Dockerfile reference for the ARG directive
Dockerfile reference for the LABEL directive
You can specify a label on the command line when creating your image. So you would write something like
docker build -t myproject --label "myproject.version=githash" .
instead of hard-coding the version you can also get it directly from git:
docker build -t myproject --label "myproject.version=`git describe`" .
To read out the label from your images you can use docker inspect with a format string:
docker inspect -f '{{index .Config.Labels "myproject.version"}}' myproject
If you are using docker-compose, you could add the following to the build section:
labels:
git-commit-hash: ${COMMIT_HASH}
where COMMIT_HASH is your environment variable, which holds commit hash.