My local development environment is behind a corporate proxy, with our own Docker registry, etc. However, we deploy on public infrastructure, meaning we can't access the corporate registries, and so have to pull from a public one (eg DockerHub).
Is there any way (eg via environment variables) for me to configure Docker to pull from a private registry when developing locally, and from a public registry when it goes through our CI/CD pipeline?
For example, let's say we're deploying a Node.JS application - locally, I would want the FROM node:16 line to get interpreted as FROM corporate.proxy/node:16.
There are a couple methods that would probably work - having two separate Dockerfiles, eg Dockerfile.dev and Dockerfile.prod, or wrapping it in some sort of script that will take care of making the change. I'm looking for a way to do it via Docker's configuration, if it's possible at all.
You can use ARG instruction to change the FROM line in Dockerfile.
ARG IMG
FROM ${IMG}
Then you can build image like this:
docker build --build-arg IMG=node:16 .
or
docker build --build-arg IMG=corporate.proxy/node:16 .
From Dockerfile reference document:
ARG is the only instruction that may precede FROM in the Dockerfile
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
Related
I have a Git repo with a simple Dockerfile. First row goes like this:
FROM python:3.7
My company has an internal registry with the base images. Because of this, the DevOps guys want me to change the Dockerfile to:
FROM registry.company.com:5000/python:3.7
I don't want this infrastructure detail baked in my code. URLs may change, I may want to build this image in another environment, etc. If possible, I would rather indicate the server in the pipeline, but the documentation regarding docker build has no parameter for this.
Is there a way to avoid editing the Dockerfile in this situation?
You would use a build arg for this:
ARG registry=docker.io/library
FROM ${registry}/python:3.7
Then for the build process:
docker build --build-arg registry=registry.company.com:5000 ...
Use docker.io for the registry name for the default Docker Hub, and library is the repository for official docker images, both of which you normally don't see when using the short format. Note that I usually include the library part in the local mirror so that official docker images and other repos that are mirrored can all use the same registry variable:
ARG registry=docker.io
FROM ${registry}/library/python:3.7
That means your local registry would need to have registry.company.com:5000/library/python:3.7.
To force users to specify the registry as part of the build, then don't provide a default value to the arg (or you could default the value of registry to something internal if that's preferred):
ARG registry
FROM ${registry}/python:3.7
You can work around the situation by manually pulling and re-tagging the image. docker build (and docker run) won't try to pull an image that already appears to be present locally, but that also means there's no verification that it actually matches what Docker Hub has. That means you can pull the image from your mirror, then docker tag it to look like a Docker Hub image:
docker pull registry.company.com:5000/python:3.7
docker tag registry.company.com:5000/python:3.7 python:3.7
TL;DR Must you put your private Docker URI in your Dockerfile FROM command if the parent image is in a private registry?
I thought this should be an easily answerable question, but cannot find a good set of Google keywords...
Detail:
I have three repos, all built with separate CI calls; the order of these CI executions is correct for this DAG (i.e. parent image will be available when a child needs it):
Repo 1 holds a Dockerfile that constructs a base image with dependencies. It is slow moving
Repo 2 & 3 hold applications that are built in a Dockerfile that pulls FROM the image built in Repo 1. These change frequently.
As I understand it, if you don't specify a repo URI in your FROM command, docker assumes you are pulling from DockerHub. Is this correct?
If these images are stored in a private registry, is it true that the private registry must be explicitly included in the child Dockerfiles? Is there another way?
Docker looks for the image name, on your local repository, if it does not find the image there, it would pull from the docker hub.
At first thought, it feels intuitive that we should be able to configure a private repository as the default, and then use it just as we would use docker hub.
However, this seems to be a topic of lengthy discussion, you can follow the discussion here.
Unfortunately, at the time of writing this, to build from a private repo, you will need to specify the complete URI in your Dockerfile:
FROM <aws_account_id>.dkr.ecr.<aws region>.amazonaws.com/<private image name>:latest
You will need to configure your Docker daemon to authenticate and pull from your private ECR repository:
aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com
and then go for
docker build .
Alternatively, you can use arguments, to construct your ECR URI. This will keep things clean and parameterised in your Dockerfile while being explicit that you are using a private repo.
Eg:
In your Dockerfile
ARG PRIVATE_REPO
FROM ${PRIVATE_REPO}your_image_name
And build the docker image with:
docker build . --build-arg PRIVATE_REPO=aws_account_id.dkr.ecr.region.amazonaws.com/
Your basic assumptions are correct: if your base image is in a private registry, you generally must include the registry name in downstream Dockerfiles' FROM lines, docker run commands, Compose and Kubernetes image: lines, and so on. If the registry part of the image name is absent, it defaults to docker.io.
# Expands to docker.io/library/alpine:latest
FROM alpine
This only actually matters if the image isn't already present on the local machine, in which case Docker will automatically pull it. If it's already present, Docker doesn't worry about where it came from and just uses it as-is. If you have a manual multi-stage build, but don't actually push any of the intermediate stages anywhere, it doesn't actually matter if you tag it for any particular repository.
# "step 2" Dockerfile
# Build with: `docker build -t step2 -f step2.Dockerfile .`
# Starts from image in internal repository
FROM registry.example.com/img/step1
# Application Dockerfile
# Depends on "step 2" image being built first
# Since we expect step2:latest image to be present locally, won't pull
FROM step2
What I'd like to do is this:
FROM mcr.microsoft.com/dotnet/core/aspnet:3.0-alpine
# more stuff
But, this needs to run on an isolated clean build machine which does not have internet access, so I need to route it through a local mirror server (e.g. Artifactory or Nexus or another similar thing)
If the docker image were hosted on docker hub (e.g. FROM ubuntu) then the --registry-mirrors docker configuration option would solve it for us.
BUT because microsoft have decided not to use docker hub, the registry mirror thing isn't working either.
After much effort, I've set up a a custom domain mirror for mcr.microsoft.com and so now I can do this:
FROM microsoft-docker-mirror.local.domain/dotnet/core/aspnet:3.0-alpine
# more stuff
That works. But we have remote workers who may not be on the local office LAN and can't see my local mirror server. What I now want to do is vary it depending on environment. E.g.
ENV MICROSOFT_DOCKER_REPO mcr.microsoft.com
FROM ${MICROSOFT_DOCKER_REPO}/dotnet/core/aspnet:3.0-alpine
My isolated build machine would set the MICROSOFT_DOCKER_REPO environment variable, and everyone else's machines would use the default mcr.microsoft.com
Anyway. Adding the ENV line to the dockerfile results in docker throwing this error:
Error response from daemon: No build stage in current context
There seem to be lots of references to the fact that the FROM line must be the first line in the file, before even any comments...
How can I reference an environment variable in my FROM statement? Or alternatively, how can I make registry mirrors work for things that aren't docker hub?
Thanks!
"ARG is the only instruction that may precede FROM in the Dockerfile" - Dockerfile reference.
ARG MICROSOFT_DOCKER_REPO=mcr.microsoft.com
FROM ${MICROSOFT_DOCKER_REPO}/dotnet/core/aspnet:3.0-alpine
Build with a non-default MICROSOFT_DOCKER_REPO using the --build-arg: docker build --rm --build-arg MICROSOFT_DOCKER_REPO=repo.example.com -t so:58196638 .
Note: you can pass the --build-arg from the hosts environment i.e. MICROSOFT_DOCKER_REPO=repo.example.com docker build --rm --build-arg MICROSOFT_DOCKER_REPO=${MICROSOFT_DOCKER_REPO} -t so:58196638 .
I want to automate my build process and need to pass an environment variable to run some of the commands in the Dockerfile. I was wondering if there was any way to do this in Dockerhub. I know docker cloud has something like this, but I was wondering whether the functionality was there in Dockerhub since there is the --build-args argument in the cli for normal building.
Set up Automated builds
Docker Hub (https://hub.docker.com) can automatically build images from source code in an external repository and automatically push the built image to your Docker repositories which will be hosted under your Docker Hub repositories account Eg: https://cloud.docker.com/u/binbash/repository/list
When you set up automated builds (also called autobuilds), you create a list of branches and tags that you want to build into Docker images. When you push code to a source code branch (currently only GitHub / Bitbucket are supported) for one of those listed image tags, the push uses a webhook to trigger a new build, which produces a Docker image. The built image is then pushed to the Docker Hub registry.
For detailed implementation steps please refer to https://docs.docker.com/docker-hub/builds/
Environment variables for builds
You can set the values for environment variables (actually they are mapped to build ARG values - docker build --build-arg - to be exclusively used at build-time - https://docs.docker.com/engine/reference/commandline/build/#set-build-time-variables---build-arg).
NOT to be confused with the environment values, ENV VARS, used by your service at runtime (docker run --env MYVAR1=foo - https://docs.docker.com/v17.12/edge/engine/reference/commandline/run/#set-environment-variables--e-env-env-file)
These Environment Variables configured from the Docker Hub UI are used in your build processes when you configure an automated build. Add your build environment variables by clicking the plus sign next to the Build environment variables section, and then entering a variable name and the value.
When you set variable values from the Docker Hub UI, they can be used by the commands you set in hooks files (THIS IS VERY IMPORTANT and will be extended below), but they are stored so that only users who have admin access to the Docker Hub repository can see their values. This means you can use them to safely store access tokens or other information that should remain secret.
Build hook examples (to implement Docker Hub UI Env vars)
Adding variables from the auto-build’s web UI makes them available inside the hooks. In the hook, you’ll have to use that value to set a custom build arg using --build-arg. Finally, you have to use this custom build arg inside your Dockerfile to manually set an environment variable using ENV command or export.
Example:
Say your want an environment variable TERRAFORM_VERSION='0.12.0-beta2' in your build environment
Step 1.
Add this in the auto-build’s web UI for ‘build environment variables’
Step 2.
Create a custom build hook i.e create a folder called hooks in the same directory as your Dockerfile. Inside the hooks folder, create a file called build. This creates the custom build hook. Docker will use this to build your image. Contents of build:
#!/bin/bash
docker build -t $IMAGE_NAME --build-arg TERRAFORM_VERSION=$TERRAFORM_VERSION .
NOTE: Here $TERRAFORM_VERSION is coming from the web UI.
Step3:
In your Dockerfile
ARG TERRAFORM_VERSION
ENV TERRAFORM_VERSION $TERRAFORM_VERSION
NOTE: Here $TERRAFORM_VERSION is coming from the custom build args in your bash script file named build.
Complete example: https://github.com/binbashar/public-docker-images/tree/master/terraform-resources
That's it! It should work now. Probably renaming ‘build environment variables’ to ‘custom hook environment variables’ in Docker Hub will ease the understanding of this concept in the official documentation (https://docs.docker.com/docker-hub/builds/advanced/).
Extra Points!
There are a number of key environment arguments set upon launching a build script, all of which you can use in your hooks and which can all be useful in making custom build-args.
SOURCE_BRANCH: the name of the branch or the tag that is currently being tested.
SOURCE_COMMIT: the SHA1 hash of the commit being tested.
COMMIT_MSG: the message from the commit being tested and built.
DOCKER_REPO: the name of the Docker repository being built.
DOCKERFILE_PATH: the Dockerfile currently being built.
DOCKER_TAG: the Docker repository tag being built.
IMAGE_NAME: the name and tag of the Docker repository being built. (This variable is a combination of DOCKER_REPO:DOCKER_TAG.)
An example:
Step 1. Create the Dockerfile:
ARG NODE_VERSION
FROM node:$NODE_VERSION
Step 2. Create the hooks/build file:
#!/bin/bash
NODE_VERSION=$(echo $DOCKER_TAG | cut -d "-" -f2)
if [ $DOCKER_TAG == "latest" ]
then
docker build . --build-arg NODE_VERSION=${DOCKER_TAG} -t ${IMAGE_NAME}
else
docker build . --build-arg NODE_VERSION=${NODE_VERSION} -t ${IMAGE_NAME}
fi
Source: github.com/SamuelA
Is there any way of pulling images from a private registry during a docker build instead of docker hub?
I deployed a private registry and I would like to be able to avoid naming its specific ip:port in the Dockerfile's FROM instruction. I was expecting a docker build option or a docker environment variable to change the default registry.
The image name should include the FQDN of the registry host.
So if you want to FROM <some private image> you must specifiy it as FROM registry_host:5000/foo/bar
In the future this won't be a requirement, but unfortunately for now it is.
I was facing the same issue in 2019. I solved this using arguments (ARG).
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
Arguments allow you to set optional parameters (with defaults) that can be used in your FROM line.
Dockerfile-project-dev
ARG REPO_LOCATION=privaterepo.company.net/
ARG BASE_VERSION=latest
FROM ${REPO_LOCATION}project/base:${BASE_VERSION}
...
For my use-case I normally want to pull from the private repo, but if I'm working on the Dockerfiles I may want to be able to build from an image on my own machine, without having to modify the FROM line in my Dockerfile. To tell Docker to search my local machine for the image at build time I would do this:
docker build -t project/dev:latest -f ./Dockerfile-project-dev --build-arg REPO_LOCATION='' .
The docker folks generally want to ensure that if you run docker pull foo/bar you'll get the same thing (i.e., the foo/bar image from Docker Hub) regardless of your local environment.
This means that there are no options available to have Docker use anything else without an explicit hostname/port.