I have a build time Argument with which I would like to build my Docker file; Since that argument is confidential in nature (github key of a private repo) I don't want that to end up in a docker image. This quotes is from official docker documentation regarding the build-time arguments.
Warning: It is not recommended to use build-time variables for passing
secrets like github keys, user credentials etc. Build-time variable
values are visible to any user of the image with the docker history
command.
Anybody know what is the recommended way to achieve the same?
With docker 18.09+, that will be: docker build --secret id=mysecret,src=/secret/file (using buildkit).
See PR 1288, announced in this tweet.
--secret is now guarded by API version 1.39.
Example:
printf "hello secret" > ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - . <<EOF
# syntax = tonistiigi/dockerfile:secrets20180808
FROM busybox
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
EOF
I'd rely on context of the Dockerfile for that. Basically, have something else (i.e. Jenkins, sub-repos) that's trusted with your Github key pull down all the necessary repos to relative locations that give your Dockerfile the context it needs. Nothing in the Docker build process itself should be managing secrets.
I can be more specific, if you specify more about your use-case. If it's just a single repo you need, you can just stick the Dockerfile in the root of that repo and rely on something else to provide credentials for cloning the repo down.
This is the point of docker secret
see for example
https://blog.docker.com/2017/02/docker-secrets-management/
Related
I wonder if it is possible to make Docker automatically mount volumes during build or run container phase. With podman it is easy, using /usr/share/containers/mounts.conf, but I need to use Docker CE.
If it is not, may I somehow use host RHEL subscription during Docker build phase? I need to use RHEL UBI image and I have to use companys Satellite
A container image build in docker is designed to be self contained and portable. It shouldn't matter whether you run the build on your host or a CI server in the cloud. To do that, they rely on the build context and args to the build command, rather than other settings on the host, where possible.
buildah seems to have taken a different approach with their tooling, allowing you to use components from the host in your build, giving you more flexibility, but also fragility.
That's a long way of saying the "feature" doesn't exist in docker, and if it gets created, I doubt it would look like what you're describing. Instead, with buildkit, they allow you to inject secrets from the build command line, which are mounted into the steps where they are required. An example of this is available in the buildkit docs:
# syntax = docker/dockerfile:1.3
FROM python:3
RUN pip install awscli
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials aws s3 cp s3://... ...
And to build that Dockerfile, you would pass the secret as a CLI arg:
$ docker build --secret id=aws,src=$HOME/.aws/credentials .
TL;DR Must you put your private Docker URI in your Dockerfile FROM command if the parent image is in a private registry?
I thought this should be an easily answerable question, but cannot find a good set of Google keywords...
Detail:
I have three repos, all built with separate CI calls; the order of these CI executions is correct for this DAG (i.e. parent image will be available when a child needs it):
Repo 1 holds a Dockerfile that constructs a base image with dependencies. It is slow moving
Repo 2 & 3 hold applications that are built in a Dockerfile that pulls FROM the image built in Repo 1. These change frequently.
As I understand it, if you don't specify a repo URI in your FROM command, docker assumes you are pulling from DockerHub. Is this correct?
If these images are stored in a private registry, is it true that the private registry must be explicitly included in the child Dockerfiles? Is there another way?
Docker looks for the image name, on your local repository, if it does not find the image there, it would pull from the docker hub.
At first thought, it feels intuitive that we should be able to configure a private repository as the default, and then use it just as we would use docker hub.
However, this seems to be a topic of lengthy discussion, you can follow the discussion here.
Unfortunately, at the time of writing this, to build from a private repo, you will need to specify the complete URI in your Dockerfile:
FROM <aws_account_id>.dkr.ecr.<aws region>.amazonaws.com/<private image name>:latest
You will need to configure your Docker daemon to authenticate and pull from your private ECR repository:
aws ecr get-login-password --region region | docker login --username AWS --password-stdin aws_account_id.dkr.ecr.region.amazonaws.com
and then go for
docker build .
Alternatively, you can use arguments, to construct your ECR URI. This will keep things clean and parameterised in your Dockerfile while being explicit that you are using a private repo.
Eg:
In your Dockerfile
ARG PRIVATE_REPO
FROM ${PRIVATE_REPO}your_image_name
And build the docker image with:
docker build . --build-arg PRIVATE_REPO=aws_account_id.dkr.ecr.region.amazonaws.com/
Your basic assumptions are correct: if your base image is in a private registry, you generally must include the registry name in downstream Dockerfiles' FROM lines, docker run commands, Compose and Kubernetes image: lines, and so on. If the registry part of the image name is absent, it defaults to docker.io.
# Expands to docker.io/library/alpine:latest
FROM alpine
This only actually matters if the image isn't already present on the local machine, in which case Docker will automatically pull it. If it's already present, Docker doesn't worry about where it came from and just uses it as-is. If you have a manual multi-stage build, but don't actually push any of the intermediate stages anywhere, it doesn't actually matter if you tag it for any particular repository.
# "step 2" Dockerfile
# Build with: `docker build -t step2 -f step2.Dockerfile .`
# Starts from image in internal repository
FROM registry.example.com/img/step1
# Application Dockerfile
# Depends on "step 2" image being built first
# Since we expect step2:latest image to be present locally, won't pull
FROM step2
What I'd like to do is this:
FROM mcr.microsoft.com/dotnet/core/aspnet:3.0-alpine
# more stuff
But, this needs to run on an isolated clean build machine which does not have internet access, so I need to route it through a local mirror server (e.g. Artifactory or Nexus or another similar thing)
If the docker image were hosted on docker hub (e.g. FROM ubuntu) then the --registry-mirrors docker configuration option would solve it for us.
BUT because microsoft have decided not to use docker hub, the registry mirror thing isn't working either.
After much effort, I've set up a a custom domain mirror for mcr.microsoft.com and so now I can do this:
FROM microsoft-docker-mirror.local.domain/dotnet/core/aspnet:3.0-alpine
# more stuff
That works. But we have remote workers who may not be on the local office LAN and can't see my local mirror server. What I now want to do is vary it depending on environment. E.g.
ENV MICROSOFT_DOCKER_REPO mcr.microsoft.com
FROM ${MICROSOFT_DOCKER_REPO}/dotnet/core/aspnet:3.0-alpine
My isolated build machine would set the MICROSOFT_DOCKER_REPO environment variable, and everyone else's machines would use the default mcr.microsoft.com
Anyway. Adding the ENV line to the dockerfile results in docker throwing this error:
Error response from daemon: No build stage in current context
There seem to be lots of references to the fact that the FROM line must be the first line in the file, before even any comments...
How can I reference an environment variable in my FROM statement? Or alternatively, how can I make registry mirrors work for things that aren't docker hub?
Thanks!
"ARG is the only instruction that may precede FROM in the Dockerfile" - Dockerfile reference.
ARG MICROSOFT_DOCKER_REPO=mcr.microsoft.com
FROM ${MICROSOFT_DOCKER_REPO}/dotnet/core/aspnet:3.0-alpine
Build with a non-default MICROSOFT_DOCKER_REPO using the --build-arg: docker build --rm --build-arg MICROSOFT_DOCKER_REPO=repo.example.com -t so:58196638 .
Note: you can pass the --build-arg from the hosts environment i.e. MICROSOFT_DOCKER_REPO=repo.example.com docker build --rm --build-arg MICROSOFT_DOCKER_REPO=${MICROSOFT_DOCKER_REPO} -t so:58196638 .
I recently switched to multi-stage docker builds, and it doesn't appear that there's any caching on intermediate builds. I'm not sure if this is a docker limitation, something which just isn't available or whether I'm doing something wrong.
I am pulling down the final build and doing a --cache-from at the start of the new build, but it always runs the full build.
This appears to be a limitation of docker itself and is described under this issue - https://github.com/moby/moby/issues/34715
The workaround is to:
Build the intermediate stages with a --target
Push the intermediate images to the registry
Build the final image with a --target and use multiple --cache-from paths, listing all the intermediate images and the final image
Push the final image to the registry
For subsequent builds, pull the intermediate + final images down from the registry first
Since the previous answer was posted, there is now a solution using the BuildKit backend: https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources
This involves passing the argument --build-arg BUILDKIT_INLINE_CACHE=1 to your docker build command. You will also need to ensure BuildKit is being used by setting the environment variable DOCKER_BUILDKIT=1 (on Linux; I think BuildKit might be the default backend on Windows when using recent versions of Docker Desktop). A complete command line solution for CI might look something like:
export DOCKER_BUILDKIT=1
# Use cache from remote repository, tag as latest, keep cache metadata
docker build -t yourname/yourapp:latest \
--cache-from yourname/yourapp:latest \
--build-arg BUILDKIT_INLINE_CACHE=1 .
# Push new build up to remote repository replacing latest
docker push yourname/yourapp:latest
Some of the other commenters are asking about docker-compose. It works for this too, although you need to additionally specify the environment variable COMPOSE_DOCKER_CLI_BUILD=1 to ensure docker-compose uses the docker CLI (with BuildKit thanks to DOCKER_BUILDKIT=1) and then you can set BUILDKIT_INLINE_CACHE: 1 in the args: section of the build: section of your YAML file to ensure the required --build-arg is set.
I'd like to add another important point to the answer
--build-arg BUILDKIT_INLINE_CACHE=1 caches only the last layer, and works in case nothing changed.
So, to enable the caching of layers for the whole build, this argument should be replaced by --cache-to type=inline,mode=max. See the documentation
Is there any way of pulling images from a private registry during a docker build instead of docker hub?
I deployed a private registry and I would like to be able to avoid naming its specific ip:port in the Dockerfile's FROM instruction. I was expecting a docker build option or a docker environment variable to change the default registry.
The image name should include the FQDN of the registry host.
So if you want to FROM <some private image> you must specifiy it as FROM registry_host:5000/foo/bar
In the future this won't be a requirement, but unfortunately for now it is.
I was facing the same issue in 2019. I solved this using arguments (ARG).
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
Arguments allow you to set optional parameters (with defaults) that can be used in your FROM line.
Dockerfile-project-dev
ARG REPO_LOCATION=privaterepo.company.net/
ARG BASE_VERSION=latest
FROM ${REPO_LOCATION}project/base:${BASE_VERSION}
...
For my use-case I normally want to pull from the private repo, but if I'm working on the Dockerfiles I may want to be able to build from an image on my own machine, without having to modify the FROM line in my Dockerfile. To tell Docker to search my local machine for the image at build time I would do this:
docker build -t project/dev:latest -f ./Dockerfile-project-dev --build-arg REPO_LOCATION='' .
The docker folks generally want to ensure that if you run docker pull foo/bar you'll get the same thing (i.e., the foo/bar image from Docker Hub) regardless of your local environment.
This means that there are no options available to have Docker use anything else without an explicit hostname/port.