Extract comman command from DockerFiles - docker

We have a couple of DockerFiles that runs the same command
COPY --from=/source /dest
Since we will change the /source from time to time, is it possible to extract this into some common util file and use it everywhere?

You could try to pass it as an ARG to docker build.
docker build --build-arg mysourcedir=/source Dockerfile
and in the Dockerfile
# use /source as default if no arg is supplied
ARG mysourcedir=/source
...
COPY --from=$mysourcedir /dest
...
Or if you use docker compose then you could put this as an environment variable into a .env file and pass it with the --env-file flag. Details can be found in the documentation.

Related

How to pass GitLab CI file variable to docker container?

This question is a little like this question. How to pass GitLab CI file variable to Dockerfile and docker container?
I used the most voted answer however I failed.
The below is what I want to do.
I set up the gitlab ci variable called PIP_CONFIG with file type.
and the value is like
[global]
timeout = 60
in .gitlab-ci.yml
...
- docker build -t $IMAGe
--build-arg PIP_CONFIG=$PIP_CONFIG
src/.
...
in Dockerfile
FROM python:3
ARG PIP_CONFIG
RUN mkdir ~/.pip
RUN echo $PIP_CONFIG > ~/.pip/pip.conf
...
...
RUN pip install -r requirements.txt
And then I got an error
Configuration file could not be loaded.
File contains no section headers.
file: '/root/.pip/pip.conf', line: 1
'/builds/xxx/xxx/project_name.tmp/PIPCONFIG\n' <<<< this line
...
It seems like it only wrote the path of temp file rather than the content of file.
I also try use COPY $PIP_CONFIG ~/.pip/pip.conf but it said the path is not exsist in /builds/xxx/xxx/project_name.tmp/PIPCONFIG.
Could someone tell me what should I do correctly? Thanks.
PS: The reason why I do not write the config directly in repository and jus use COPY from repo is that there is some sensitive token in pip config.
After some try, I understand that just use type 'variable' in the gitlab ci setting.
And then pass the value with quote("$VARABLE") for maybe you have the space or break line in your value.
like this
in .gitlab-ci.yml
...
- docker build -t $IMAGe
--build-arg PIP_CONFIG="$PIP_CONFIG"
src/.
...
Remember add quote for Dockerfile, too.
FROM python:3
ARG PIP_CONFIG
RUN mkdir ~/.pip
RUN echo "$PIP_CONFIG" > ~/.pip/pip.conf
...
...
RUN pip install -r requirements.txt
And then, I can do what I want, write a pip config to image from gitlab ci variable.
Hope can help others.

Docker ARG tricky behavior

Docker build tricky behavior:
Dockerfile
ARG php_version="7.2"
ARG flavor="stretch"
FROM php:${php_version}-fpm-${flavor}
#ARG php_version="7.2"
ENV php_v $php_version
CMD echo $php_v
If you build it and run:
docker build -t args:1.0 .
docker run -it --name testargs args:1.0
Output is empty string instead of the expected "7.2"
But if the third ARG line is uncommented we get "7.2"
Why does it behave this way?
Each FROM in a Dockerfile represents a new build stage and an ARG declared before the FROM is not available for use in the newer build stages starting with another FROM.
To quote the relevant section of the doc:
An ARG declared before a FROM is outside of a build stage, so it can’t
be used in any instruction after a FROM.

Why can't I load environment variables in dockerfile? [duplicate]

I'm building a container for a ruby app. My app's configuration is contained within environment variables (loaded inside the app with dotenv).
One of those configuration variables is the public ip of the app, which is used internally to make links.
I need to add a dnsmasq entry pointing this ip to 127.0.0.1 inside the container, so it can fetch the app's links as if it were not containerized.
I'm therefore trying to set an ENV in my Dockerfile which would pass an environment variable to the container.
I tried a few things.
ENV REQUEST_DOMAIN $REQUEST_DOMAIN
ENV REQUEST_DOMAIN `REQUEST_DOMAIN`
Everything passes the "REQUEST_DOMAIN" string instead of the value of the environment variable though.
Is there a way to pass environment variables values from the host machine to the container?
You should use the ARG directive in your Dockerfile which is meant for this purpose.
The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag.
So your Dockerfile will have this line:
ARG request_domain
or if you'd prefer a default value:
ARG request_domain=127.0.0.1
Now you can reference this variable inside your Dockerfile:
ENV request_domain=$request_domain
then you will build your container like so:
$ docker build --build-arg request_domain=mydomain Dockerfile
Note 1: Your image will not build if you have referenced an ARG in your Dockerfile but excluded it in --build-arg.
Note 2: If a user specifies a build argument that was not defined in the Dockerfile, the build outputs a warning:
[Warning] One or more build-args [foo] were not consumed.
So you can do:
cat Dockerfile | envsubst | docker build -t my-target -
Then have a Dockerfile with something like:
ENV MY_ENV_VAR $MY_ENV_VAR
I guess there might be a problem with some special characters, but this works for most cases at least.
This is for those looking to pass env variable from docker-compose using .env file to dockerfile during build and then pass those args as env variable to container.
Typical docker-compose file
services:
web:
build:
context: ./api
dockerfile: Dockerfile
args:
- SECRET_KEY=$SECRET_KEY
- DATABASE_URL=$DATABASE_URL
- AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
Pass the env variable present in .env file to args in build command.
Typical .env file
SECRET_KEY=blahblah
DATABASE_URL=dburl
Now when you run docker-compose up -d command, docker-compose file takes values from .env file then pass it to docker-compose file. Now Dockerfile of web containes all those varibales through args during build. Now typical dockerfile of web,
FROM python:3.6-alpine
ARG SECRET_KEY
ARG DATABASE_URL
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_BUCKET
ARG AWS_REGION
ARG CLOUDFRONT_DOMAIN
ENV CELERY_BROKER_URL redis://redis:6379/0
ENV CELERY_RESULT_BACKEND redis://redis:6379/0
ENV C_FORCE_ROOT true
ENV SECRET_KEY ${SECRET_KEY?secretkeynotset}
ENV DATABASE_URL ${DATABASE_URL?envdberror}
Now we recieved those secret_key and db url as arg in dokcerfile. Now let's use those in ENV as ENV SECRET_KEY ${SECRET_KEY?secretkeynotset}. Now even docker container has those variables in it's environment.
Remember not to use ARG $SECRET_KEY(which I did). It should be ARG SECRET_KEY
An alternative using envsubst without losing the ability to use commands like COPY or ADD, and without using intermediate files would be to use Bash's Process Substitution:
docker build -f <(envsubst < Dockerfile) -t my-target .
Load environment variables from a file you create at runtime.
export MYVAR="my_var_outside"
cat > build/env.sh <<EOF
MYVAR=${MYVAR}
EOF
... then in the Dockerfile
ADD build /build
RUN /build/test.sh
where test.sh loads MYVAR from env.sh
#!/bin/bash
. /build/env.sh
echo $MYVAR > /tmp/testfile
If you just want to find and replace all environment variables ($ExampleEnvVar) in a Dockerfile then build it this would work:
envsubst < /path/to/Dockerfile | docker build -t myDockerImage . -f -
When using build-arg...
docker build --build-arg CODE_VERSION=1.2 Dockerfile
...consider that the variable is not availabe after FROM:
ARG CODE_VERSION=latest
FROM base:${CODE_VERSION}
An ARG declared before a FROM is outside of a build stage, so it can’t be used in any instruction after a FROM.
Generally ARGs should be placed after FROM if not required during FROM:
FROM base:xy
ARG ABC=123
To use the default value of an ARG declared before the first FROM use an ARG instruction without a value inside of a build stage:
ARG VERSION=latest
FROM busybox:$VERSION
ARG VERSION
RUN echo $VERSION > image_version
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
add -e key for passing environment variables to container.
example:
$ MYSQLHOSTIP=$(sudo docker inspect -format="{{ .NetworkSettings.IPAddress }}" $MYSQL_CONRAINER_ID)
$ sudo docker run -e DBIP=$MYSQLHOSTIP -i -t myimage /bin/bash
root#87f235949a13:/# echo $DBIP
172.17.0.2

docker copy issues and set host env variable

I am new to docker.
I would like to understand the following questions. I have been searching but I can't find the answers to my questions.
Why do I always get a wrong path when I tried to copy the file?
Does that mean I can only copy the files into the docker image from the same directory where I have my dockerfile? Is there a way to COPY files from other directories on the host?
Is there a way to passing in host's environment variables directly in the Dockerfile without using "ARG" and --build-arg flag?
Below is what I currently have
file structure is like this:
/home/user1/docker
|__ Dockerfile
In the Dockerfile:
From
ARG BLD_DIR=/tmp
RUN mkdir /workdir
WORKDIR /workdir
COPY ${BLD_DIR}/a.file /workdir
I ran
root#localhost> echo $BLD_DIR
/tmp/build <-- BLD_DIR is a custom variable; meaning it's different on each dev env
docker build --build-arg BLD_DIR=${BLD_DIR} -t docker-test:1.0 -f Dockerfile
Always got error like
COPY failed: stat
/var/lib/docker/tmp/docker-builder035089075/tmp/build/a.file: no such file
or directory
In a Dockerfile, you can only copy files that are available in the current Docker build context.
By default, all files in the directory where you run your docker build command are copied to the Docker context folder.
So, when you use ADD or COPY commands, all your paths are in fact relative to build folder, as the documentation states:
Multiple resources may be specified but the paths of files and directories will be interpreted as relative to the source of the context of the build.
This is voluntary because building an image using docker build should not depend on auxiliary files on your system: the same Docker image should not be different if built on 2 different machines.
However, you can have a directory structure like such:
/home/user1/
|___file1
|___docker/
|___|___ Dockerfile
If you run docker build -t test -f docker/Dockerfile . in the /home/user1 folder, your build context will be /home/user1, so you can COPY file1 in your Dockerfile.
For the very same reason, you cannot use environment variables directly in a Dockerfile. The idea is that your docker build command should "pack" all the information needed to generate the same image on 2 different systems.
However, you can hack your way around it using docker-compose, like explaned here: Pass host environment variables to dockerfile.

Docker build-arg and copy

Trying to copy a folders content, it works when i hard code the path like:
COPY ./my-folder /path/to/location
But need to be able to change this path so i tried using a build argument like this:
COPY ${folderVariable} /path/to/location
and then build with
--build-arg folderVariable=./my-folder
But it copies everything in the same folder as "my-folder", when i only want the contents of "my-folder"
You need to define it with ARG in Dockerfile before using:
FROM alpine:3.3
ARG folderVariable=./my-folder # Optional default value to be `./my-folder`
COPY ${folderVariable} /opt/my-folder
And build it like:
docker build --build-arg folderVariable=./folder-copy -t test .
More details please refer to: https://docs.docker.com/engine/reference/builder/#arg
Expansion still does not work for the COPY --from=$var ... case.
But you can create intermediate image as an alias, like this:
ARG docsBranch=4.5
ARG docsFullPath=registry.myCompany.pro/group/project-docs/docs:$docsBranch
# Lifehack
FROM $docsFullPath as docs
FROM node:10.21.0-buster-slim
WORKDIR /app
# Now we can use docs instead of $docsFullPath
COPY --from=docs /app/html/en ./documentation/en

Resources