I'm trying to use docker and docker-compose to build a set of containers with some customized parameters using environment variables - for example, I want to mount a directory at a specific location in the container. This specific location is stored in an environment variable and created in the Dockerfile using a command like:
RUN mkdir $custom_location
I keep the parameters in an .env file which populates the docker-compose file.
Once the containers are running, using the printenv command I can see that the transfer of the env variables has worked and I can manually run the command
mkdir $custom_location but the command hasn't been successful during the dockerfile build process.
A summarized version of the Dockerfile looks like this:
FROM python:3.6.5 as my_base
ENV CUSTOM_PATH=$CUSTOM_PATH
RUN mkdir $CUSTOM_PATH
A summarized version of the docker-compose.yml looks like this:
service:
environment:
- CUSTOM_PATH=${CUSTOM_PATH}
build: ./Docker
The error I'm getting looks like this:
Step 20/20 : RUN mkdir $CUSTOM_PATH ---> Running in 437aa3f09dbb
mkdir: missing operand Try 'mkdir --help' for more information. The
command '/bin/sh -c mkdir $CUSTOM_PATH' returned a non-zero code: 1
Why would this be?
The ENV variables are successfully in the running container. Using printenv I can see them.
You may successfully use ARG with ENV in your Dockerfile. See the docs for this:
You can use an ARG or an ENV instruction to specify variables that are available to the RUN instruction. Environment variables defined using the ENV instruction always override an ARG instruction of the same name. Consider this Dockerfile with an ENV and ARG instruction.
FROM ubuntu
ARG CONT_IMG_VER
ENV CONT_IMG_VER ${CONT_IMG_VER:-v1.0.0}
RUN echo $CONT_IMG_VER
So, in your case it should be
FROM python:3.6.5 as my_base
ARG CUSTOM_PATH=<some default path if you wish>
ENV CUSTOM_PATH=$CUSTOM_PATH
RUN mkdir $CUSTOM_PATH
After that you may build the image with docker build --build-arg CUSTOM_PATH=/var/log/whatever . and it should work.
Related
My directory structure looks like this.
|
|
--- Dockerfile
| --- .env
Content of .env file looks like this.
VERSION=1.2.0
DATE=2022-05-10
I want to access VERSION and DATE as environment variable both during build time and run time. So ENV is the one I should use. I know that.
How exactly can I do that ?
I tried using RUN command in Dockerfile like
RUN export $(cat .env)
But, it can only be accessed during runtime and not build time.
So, how can this be achieved with ENV ?
I can do it manually like
ENV VERSION 1.2.0
ENV DATE 2022-05-10
But, it is inefficient when I have many environment variables.
P.S. I cannot use docker-compose because the image is going to be used by kubernetes pods, so.
You could firstly export these variables as environmetal variables to your bash
source .env
Then use --build-arg flag to pass them to your docker build
docker image build --build-arg VERSION=$VERSION --build-arg DATE=$DATE .
Next in your dockerfile
ARG VERSION
ARG DATE
ENV version=$VERSION
ENV date=$DATE
As a result, for example you can access the variables in your build phase as VERSION and in your container as version
What I've read from others says that there is no "docker build --env-file...." option/apparatus. As such, this situation makes a good argument for shifting more of the content of the dockerfile to a shell script that the dockerfile simply copies and runs, as you can source the .env file that way.
greetings.sh
#!/bin/sh
source variables.env
echo Foo $copyFileTest
variables.env
export copyFileTest="Bar"
dockerfile
FROM alpine:latest
COPY variables.env .
RUN source variables.env && echo Foo $copyFileTest #outputs: Foo Bar
COPY greetings.sh .
RUN chmod +x /greetings.sh
RUN /greetings.sh #outputs: Foo Bar
RUN echo $copyFileTest #does not work, outputs nothing
You can specify the env_file in the docker-compose.dev.yml file as follows:
# docker-compose.dev.yml
services:
app:
...
env_file:
- .env.development
and you have to have a .env.development file containing all the environment variables you need to pass to the docker image.
e.g.:
# .env.development
REACT_APP_API_URL="https://some-api-url.com/api/graphql"
Hello I have a problem with manually running deployment.
I use GitlabCI, dockerfile and kubernetes.
FROM python:3.8
RUN mkdir /app
COPY . /app/
WORKDIR /app
RUN pip install -r requirements.txt
CMD ["python", "main.py", "${flag1}", "${ARG1}", "${flag2}, "${ARG2}"]
i need to run my app with command like this "python3 main.py -g argument1 -c argument2", and every run I need using other arguments. Im using this:
Then my pipeline run bash script who check if variable "${ARG1}" is empty, if is empty, then unset "${FLAG1}". The next step is deploy to kubernetes using standard deployment using gitlabCI.
My idea Is bad because those environment variables aren't passing to Dockerfile. Anybody have some idea? Can't use Dockers build-args because they are don't support "CMD" step.
You are using the array-syntax for the command (CMD), therefore there is no shell that could expand the variables, but the data is directly used for the exec system call.
If you want the variables to be expaned, use
CMD python main.py ${flag1} ${ARG1} ${flag2} ${ARG2}
or replace the command completely in kubernetes pod/replica/deployment definition, optionally with variables replaced.
Additional note: The CMD is executed at runtime of the container, not at build time.
I'm building a container for a ruby app. My app's configuration is contained within environment variables (loaded inside the app with dotenv).
One of those configuration variables is the public ip of the app, which is used internally to make links.
I need to add a dnsmasq entry pointing this ip to 127.0.0.1 inside the container, so it can fetch the app's links as if it were not containerized.
I'm therefore trying to set an ENV in my Dockerfile which would pass an environment variable to the container.
I tried a few things.
ENV REQUEST_DOMAIN $REQUEST_DOMAIN
ENV REQUEST_DOMAIN `REQUEST_DOMAIN`
Everything passes the "REQUEST_DOMAIN" string instead of the value of the environment variable though.
Is there a way to pass environment variables values from the host machine to the container?
You should use the ARG directive in your Dockerfile which is meant for this purpose.
The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg <varname>=<value> flag.
So your Dockerfile will have this line:
ARG request_domain
or if you'd prefer a default value:
ARG request_domain=127.0.0.1
Now you can reference this variable inside your Dockerfile:
ENV request_domain=$request_domain
then you will build your container like so:
$ docker build --build-arg request_domain=mydomain Dockerfile
Note 1: Your image will not build if you have referenced an ARG in your Dockerfile but excluded it in --build-arg.
Note 2: If a user specifies a build argument that was not defined in the Dockerfile, the build outputs a warning:
[Warning] One or more build-args [foo] were not consumed.
So you can do:
cat Dockerfile | envsubst | docker build -t my-target -
Then have a Dockerfile with something like:
ENV MY_ENV_VAR $MY_ENV_VAR
I guess there might be a problem with some special characters, but this works for most cases at least.
This is for those looking to pass env variable from docker-compose using .env file to dockerfile during build and then pass those args as env variable to container.
Typical docker-compose file
services:
web:
build:
context: ./api
dockerfile: Dockerfile
args:
- SECRET_KEY=$SECRET_KEY
- DATABASE_URL=$DATABASE_URL
- AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID
Pass the env variable present in .env file to args in build command.
Typical .env file
SECRET_KEY=blahblah
DATABASE_URL=dburl
Now when you run docker-compose up -d command, docker-compose file takes values from .env file then pass it to docker-compose file. Now Dockerfile of web containes all those varibales through args during build. Now typical dockerfile of web,
FROM python:3.6-alpine
ARG SECRET_KEY
ARG DATABASE_URL
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG AWS_BUCKET
ARG AWS_REGION
ARG CLOUDFRONT_DOMAIN
ENV CELERY_BROKER_URL redis://redis:6379/0
ENV CELERY_RESULT_BACKEND redis://redis:6379/0
ENV C_FORCE_ROOT true
ENV SECRET_KEY ${SECRET_KEY?secretkeynotset}
ENV DATABASE_URL ${DATABASE_URL?envdberror}
Now we recieved those secret_key and db url as arg in dokcerfile. Now let's use those in ENV as ENV SECRET_KEY ${SECRET_KEY?secretkeynotset}. Now even docker container has those variables in it's environment.
Remember not to use ARG $SECRET_KEY(which I did). It should be ARG SECRET_KEY
An alternative using envsubst without losing the ability to use commands like COPY or ADD, and without using intermediate files would be to use Bash's Process Substitution:
docker build -f <(envsubst < Dockerfile) -t my-target .
Load environment variables from a file you create at runtime.
export MYVAR="my_var_outside"
cat > build/env.sh <<EOF
MYVAR=${MYVAR}
EOF
... then in the Dockerfile
ADD build /build
RUN /build/test.sh
where test.sh loads MYVAR from env.sh
#!/bin/bash
. /build/env.sh
echo $MYVAR > /tmp/testfile
If you just want to find and replace all environment variables ($ExampleEnvVar) in a Dockerfile then build it this would work:
envsubst < /path/to/Dockerfile | docker build -t myDockerImage . -f -
When using build-arg...
docker build --build-arg CODE_VERSION=1.2 Dockerfile
...consider that the variable is not availabe after FROM:
ARG CODE_VERSION=latest
FROM base:${CODE_VERSION}
An ARG declared before a FROM is outside of a build stage, so it can’t be used in any instruction after a FROM.
Generally ARGs should be placed after FROM if not required during FROM:
FROM base:xy
ARG ABC=123
To use the default value of an ARG declared before the first FROM use an ARG instruction without a value inside of a build stage:
ARG VERSION=latest
FROM busybox:$VERSION
ARG VERSION
RUN echo $VERSION > image_version
https://docs.docker.com/engine/reference/builder/#understand-how-arg-and-from-interact
add -e key for passing environment variables to container.
example:
$ MYSQLHOSTIP=$(sudo docker inspect -format="{{ .NetworkSettings.IPAddress }}" $MYSQL_CONRAINER_ID)
$ sudo docker run -e DBIP=$MYSQLHOSTIP -i -t myimage /bin/bash
root#87f235949a13:/# echo $DBIP
172.17.0.2
I have a dockerfile with these lines:
ARG ENVIRONMENT
ENV ENVIRONMENT $ENVIRONMENT
RUN npm run ng build --configuration=${ENVIRONMENT}
I cant get the "RUN npm run ng build --configuration= to pass the value of $ENVIRONMENT to the npm command.
What is the syntax for this?
Per the Dockerfile ARG docs,
The ARG instruction defines a variable that users can pass at build-time to the builder with the docker build command using the --build-arg = flag.
in order to accept an argument as part of the build, we use --build-arg.
Dockerfile ENV docs:
The ENV instruction sets the environment variable to the value .
We also need to include an ENV statement because the CMD will be executed after the build is complete, and the ARG will not be available.
FROM busybox
ARG ENVIRONMENT
ENV ENVIRONMENT $ENVIRONMENT
CMD echo $ENVIRONMENT
will cause an environment variable to be set in the image, so that it is available during a docker run command.
docker build -t test --build-arg ENVIRONMENT=awesome_environment .
docker run -it test
This will echo awesome_environment.
Try changing your RUN command do this:
RUN npm run ng build --configuration=$ENVIRONMENT
This should work. Check here
Thanks.
I am working on a task that involves building a docker image with centOs as its base using a Dockerfile . One of the steps inside the dockerfile needs http_proxy and https_proxy ENV variables to be set in order to work behind the proxy.
As this Dockerfile will be used by multiple teams having different proxies, I want to avoid having to edit the Dockerfile for each team. Instead I am looking for a solution which allows me to pass ENV variables at build time, e.g.,
sudo docker build -e http_proxy=somevalue .
I'm not sure if there is already an option that provides this. Am I missing something?
Containers can be built using build arguments (in Docker 1.9+) which work like environment variables.
Here is the method:
FROM php:7.0-fpm
ARG APP_ENV=local
ENV APP_ENV=${APP_ENV}
RUN cd /usr/local/etc/php && ln -sf php.ini-${APP_ENV} php.ini
and then build a production container:
docker build --build-arg APP_ENV=prod .
For your particular problem:
FROM debian
ENV http_proxy=${http_proxy}
and then run:
docker build --build-arg http_proxy=10.11.24.31 .
Note that if you build your containers with docker-compose, you can specify these build-args in the docker-compose.yml file, but not on the command-line. However, you can use variable substitution in the docker-compose.yml file, which uses environment variables.
So I had to hunt this down by trial and error as many people explain that you can pass ARG -> ENV but it doesn't always work as it highly matters whether the ARG is defined before or after the FROM tag.
The below example should explain this clearly. My main problem originally was that all of my ARGS were defined prior to FROM which resulted all the ENV to be undefined always.
# ARGS PRIOR TO FROM TAG ARE AVAIL ONLY TO FROM for dynamic a FROM tag
ARG NODE_VERSION
FROM node:${NODE_VERSION}-alpine
# ARGS POST FROM can bond/link args to env to make the containers environment dynamic
ARG NPM_AUTH_TOKEN
ARG EMAIL
ARG NPM_REPO
ENV NPM_AUTH_TOKEN=${NPM_AUTH_TOKEN}
ENV EMAIL=${EMAIL}
ENV NPM_REPO=${NPM_REPO}
# for good measure, what do we really have
RUN echo NPM_AUTH_TOKEN: $NPM_AUTH_TOKEN && \
echo EMAIL: $EMAIL && \
echo NPM_REPO: $NPM_REPO && \
echo $HI_5
# remember to change HI_5 every build to break `docker build`'s cache if you want to debug the stdout
..... # rest of whatever you want RUN, CMD, ENTRYPOINT etc..
I faced the same situation.
According to Sin30's answer pretty solution is using shell,
CMD ["sh", "-c", "cd /usr/local/etc/php && ln -sf php.ini-$APP_ENV php.ini"]