I want to know if what I'm doing is considered best practice, or is there a better way:
What I want
I want to have a Docker image, which will have some environment variables predefined when run as a container. I want to do that by running some shell script that will export those variables.
What I'm doing
My dockerfile looks like this:
Dockerfile code..
..
..
RUN useradd -m develop
RUN echo ". /env.sh" >> /home/develop/.bashrc
USER develop
Is that a good way?
Using the Dockerfile ENV directive would be much more usable than putting environment variables into a file.
ENV SOME_VARIABLE=some-value
# Do not use .bashrc at all
# Do not `RUN .` or `RUN source`
Most ways to use Docker don't involve running shell dotfiles like .bashrc. Adding settings there won't usually have any effect. In a Dockerfile, any environment variable settings in a RUN instruction will get lost at the end of that line, including files you read in using the shell . command (or the equivalent but non-standard source).
For example, given the Dockerfile you show, a docker run invocation like this never invokes a shell at all and never reads the .bashrc file:
docker run the-image env \
| grep A_VARIABLE_FROM_THE_BASHRC
There are some workarounds to this (my answer to How to source a script with environment variables in a docker build process? describes a startup-time entrypoint wrapper) but the two best ways are to (a) restructure your application to need fewer environment variables and have sensible defaults if they're not set, and (b) use ENV instead of a file of environment-variable settings.
** Check edits
Yes will probably work, if in the right order.
# Add user
RUN useradd -m develop
#switch to user
USER develop
#run script as user.
#RUN echo "./env.sh" >> /home/develop/.bashrc
RUN /bin/bash -c "source ./env.sh"
Although the duplicated RUN useradd is not necessary at all
Related
Objective
I have an env variable script file that looks like:
#!/bin/sh
export FOO="public"
export BAR="private"
I would like to source the env variables to be available when a docker image is being built. I am aware that I can use ARG and ENV with build args, but I have too many Env Variables, and I am afraid that will be a lengthy list.
It's worth mentioning that I only need the env variables to install a specific step in my docker file (will highlight in the Dockerfile below), and do not necessarily want them to be available in the built image after that.
What I have tried so far
I have tried having a script (envs.sh) that export env vars like:
#!/bin/sh
export DOG="woof"
export CAT="meow"
My Docker file looks like:
FROM fishtownanalytics/dbt:0.18.1
# Define working directory
# Load ENV Vars
COPY envs.sh envs.sh
CMD ["sh", "envs.sh"]
# Install packages required
CMD ["sh", "-c", "envs.sh"]
RUN dbt deps # I need to env variables to be available for this step
# Exposing DBT Port
EXPOSE 8081
But that did not seem to work. How can I export env variables as a script to the docker file?
In the general case, you can't set environment variables in a RUN command: each RUN command runs a new shell in a new container, and any environment variables you set there will get lost at the end of that RUN step.
However, you say you only need the variables at one specific step in your Dockerfile. In that special case, you can run the setup script and the actual command in the same RUN step:
FROM fishtownanalytics/dbt:0.18.1
COPY envs.sh envs.sh
RUN . ./envs.sh \
&& dbt deps
# Anything that envs.sh `export`ed is lost _after_ the RUN step
(CMD is irrelevant here: it only provides the default command that gets run when you launch a container from the built image, and doesn't have any effect on RUN steps. It also looks like the image declares an ENTRYPOINT so that you can only run dbt subcommands as CMD, not normal shell commands. I also use the standard . to read in a script file instead of source, since not every container has a shell that provides that non-standard extension.)
Your CMD call runs a new shell (sh) that defines those variables and then dies, leaving the current process unchanged. If you want those environment variables to apply to the current process, you could source it:
CMD ["source", "envs.sh"]
I'm completely new in Docker. I have the following idea in mind: I need to provide single image that will be able based on runtime arguments like profile/stage and python is included or not perform different scripts.
These scripts are used lot's of params that can be override from outside. I searched over the similar issues but I didn't find anything similar.
I have the following idea in mind but it seems quite difficult to support and ugly I hope someone can provide better solution:
The image content is raw:
FROM openjdk:8
#ARG py_ver=2.7
#RUN if [-z "$py_ver" ] ; then echo python version not provided ; else echo python version is $py_ver ; fi
#FROM python:${py_ver}
# set the working directory in the container
WORKDIR /models
# copy the dependencies file to the working directory
COPY training.sh execution.sh requirements/ ./
#RUN pip install --no-cache-dir -r requirements.txt
ENV profile="training"
ENV pythonEnabled=false
RUN if [ "$profile" = "training" ]; then \
command="java=/usr/bin/java training.sh"; \
else \
command="java=/usr/bin/java execution.sh"; \
fi
ENTRYPOINT ["${command}"]
I suppose I have several issues: 1) I need to have 1 image but based on runtime parameters I need to choose appropriate run script; 2) I have to pass a lot of args to training and execution scripts (app. 6-7 params). It's a bit difficult to do with "-e"
3) My image can download all python versions and use in runtime specified in args version.
I revised docker-compose but it helps if you need to manage several services. It's not my case. I have single service with different setup params and preparation flow. Could someone suggest better approach than having spaghetti if-else conditions for selected in runtime python version and profile?
It might help to look at this question in two parts. First, how can you control what runtime you're using; and second, how can you control what happens when the container runs?
A Docker image typically contains a single application, but if there's a substantial code base and several ways to invoke it, you can package that all together. In Python land, a Flask application and an associated Celery worker might be bundled together, for example. Regardless, the image still contains a single language interpreter and the associated libraries: you wouldn't build an image with three versions of Python and four versions of supporting libraries.
For things that control the single language interpreter and library stack that get built into an image, ARG as you've shown it is the correct way to do it:
ARG py_ver=3.9
RUN apt-get install python${py_ver}
ARG requirements=requirements.txt
RUN pip install -r ${requirements}
If you need to build the same image for multiple language versions, you can build it using a shell loop, or similar automation:
for py_ver in 3.6 3.7 3.8 3.9; do
docker build --build-arg py_ver="$py_ver" -t "myapp:python${py_ver}" .
done
docker run ... myapp:python3.9
As far as what gets run when you launch the container, you have a couple of choices. You can provide an alternate command when you start the container, and the easiest thing to do is to discard the entire "profile" section at the end of the Dockerfile and just provide that:
docker run ... myapp:python3.9 \
training.sh
You mention that a couple of the invocations are more involved. You can wrap these in shell scripts
#!/bin/sh
java -Dfoo=bar -Dbaz.quux.meep=peem ... \
-jar myapp.jar \
arg1 arg2 arg3
and then COPY them into your image into one of the usual executable paths
COPY training-full.sh /usr/local/bin
and then you can just run that script as the main container command
docker run ... myapp:python3.9 training-full.sh
You can, with some care, use ENTRYPOINT here. The important detail is that the CMD gets passed to the ENTRYPOINT as command-line arguments, and in your Dockerfile the ENTRYPOINT generally must have JSON-array syntax. You could in principle use this to create artificial "commands":
#!/bin/sh
case "$1" of
training)
shift
exec training.sh foo bar "$#"
;;
execution)
shift
exec execution.sh "$#"
;;
*)
exec "$#"
;;
esac
Then you can launch the container in a couple of ways
docker run --rm myapp:python3.9 training
docker run --rm myapp:python3.9 execution 'hello world'
docker run --rm myapp:python3.9 ls -l /
docker run --rm -it myapp:python3.9 bash
I developed a few ROS packages and I want to put the packages in a docker container because installing all the ROS packages all the time is tedious. Therefore I created a dockerfile that uses a base ROS image, installed all the necessary dependencies, copied my workspace, built the workspace in the docker container and sourced everything afterward. You can find the docker file here:
FROM ros:kinetic-ros-base
RUN apt-get update && apt-get install locales
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
RUN apt-get update && apt-get install -y \
&& rm -rf /var/likb/apt/lists/*
COPY . /catkin_ws/src/
WORKDIR /catkin_ws
RUN /bin/bash -c '. /opt/ros/kinetic/setup.bash; catkin_make'
RUN /bin/bash -c '. /opt/ros/kinetic/setup.bash; source devel/setup.bash'
CMD ["roslaunch", "master_launch sim_perception.launch"]
The problem is: When I run the docker container wit the "run" command, docker doesn't seem to know that I sourced my new ROS workspace and therefore it cannot launch automatically my launch script. If I run the docker container as bash script with "run -it bash" I can source my workspace again and then roslaunch my .launch file.
So can someone tell me how to write my dockerfile correctly so I launch my .launch file automatically when I run the container? Thanks!
From Docker Docs
Each RUN instruction is run independently and won't effect next instruction so when you run last Line no PATH are saved from ROS.
You need Source .bashrc or every environment you need using source first.
You can wrap everything you want (source command and roslaunch command) inside a sh file then just run that file at the end
If you review the convention of ros_entrypoint.sh you can see how best to source the workspace you would like in the docker. We're all so busy learning how to make docker and ros do the real things, it's easy to skip over some of the nuance of this interplay. This sucked forever for me; hope this is helpful for you.
I looked forever and found what seemed like only bad advice, and in the absence of an explicit standard or clear guidance I've settled into what seems like a sane approach that also allows you to control what launches at runtime with environment variables. I now consider this as the right solution for my needs.
In the Dockerfile for the image you want to set the start/launch behavior;
towards the end; you should use ADD line to insert your own ros_entrypoint.sh (example included); Set it as the ENTRYPOINT and then a CMD to run by default run something when the docker start.
note: you'll (obviously?) need to run the docker build process for these changes to be effective
Dockerfile looks like this:
all your other dockerfile ^^
.....
# towards the end
COPY ./ros_entrypoint.sh /
ENTRYPOINT ["/ros_entrypoint.sh"]
CMD ["bash"]
Example ros_entryppoint.sh:
#!/bin/bash
set -e
# setup ros environment
if [ -z "${SETUP}" ]; then
# basic ros environment
source "/opt/ros/$ROS_DISTRO/setup.bash"
else
#from environment variable; should be a absolute path to the appropriate workspaces's setup.bash
source $SETUP
fi
exec "$#"
Used in this way the docker will automatically source either the basic ros bits... or if you provide another workspace's setup.bash path in the $SETUP environment variable, it will be used in the container.
So a few ways to work with this:
From the command line prior to running docker
export SETUP=/absolute/path/to/the/setup.bash
docker run -it your-docker-image
From the command line (inline)
docker run --env SETUP=/absolute/path/to/the/setup.bash your-docker-image
From docker-compose
service-name:
network_mode: host
environment:
- SETUP=/absolute/path/to/the_workspace/devel/setup.bash #or whatever
command: roslaunch package_name launchfile_that_needed_to_be_sourced.launch
#command: /bin/bash # wake up and do something else
Say I have a Dockerfile:
.
.
RUN echo 'source /root/script.sh' >> /etc/bash.bashrc
(The script adds some env variables)
If I:
1) Do this:
docker run -it -v /home/user/script.sh:/root/script.sh image
It takes me to shell where if I call "env" I see the variable set by the script
But if I:
2) Do this:
docker run -it -v /home/user/script.sh:/root/script.sh image env
It prints out env and exits and my variable is missing
What am I missing? I need the variable to exists even if I specify a command/script like "env" at the end of the docker run command
When you run a command like
docker run ... image command
Docker directly runs the command you give; it doesn’t launch any kind of shell, and there’s no opportunity for a .bashrc or similar file to be read.
I’d suggest two things here:
If your program does need environment variables set in some form, set them directly using Dockerfile ENV directives. Don’t try to edit .bashrc or /etc/profile or any other shell dotfile; they won’t reliably get run.
As much as you can install things in places so that you don’t need to change environment variables. For instance, Python supports a “virtual environment” concept that allows an isolated library environment, which requires changing $PATH and similar things; but Docker provides the same isolation on its own, so just install things into the “global” package space.
If you really can’t manage either of these things, then you can write an entrypoint script that sets environment variables and then launches the container’s command. This might look like
#!/bin/sh
. /root/script.sh
exec "$#"
And then you could include this in your Dockerfile like
...
COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
CMD ["/app/myapp"]
(If you need to use docker exec to get a debugging shell in the container, that won’t be a child process of the entrypoint and won’t get its environment variables.)
I have two Dockerfiles Dockerfile.A & Dockerfile.B where Dockerfile.B inherits using the FROM keyword from Dockerfile.A. In Dockerfile.A I set an environment variable that I would like to use in Dockerfile.B (PATH). Is this possible, and how would I go about doing it?
So far I have tried the following in Dockerfile.A:
RUN export PATH=/my/new/dir:$PATH
ENV PATH=/my/new/dir:$PATH
RUN echo "PATH=/my/new/dir:$PATH" >/etc/profile
And in Dockerfile.B, respectively:
Just use tools in the path to see if they were available (they were not)
ENV PATH
RUN source /etc/profile
I realized that every RUN command is executed in it's own environment, and that is probably why the ENV keyword exists, to make it possible to treat environments independently of the RUN commands. But I am not sure what that means for my case.
So how can I do this?
Works as expected for me.
Dockerfile.A
FROM alpine:3.6
ENV TEST=VALUE
Build it.
docker build -t imageA .
Dockerfile.B
FROM imageA
CMD echo $TEST
Build it.
$ docker build -t imageB .
Run it
$ docker run -it imageB
VALUE