I have been using docker since some time now. I have encountered a situation wherein I need to execute instructions present in the Dockerfile based on some condition. For example here is the snippet of Dockerfile
FROM centos:centos7
MAINTAINER Akshay <akshay#dm.com>
# Update and install required binaries
RUN yum update -y \
&& yum install -y which wget openssh-server sudo java-1.8.0-openjdk \
&& yum clean all
#install Maven
RUN curl -Lf http://archive.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz -o /tmp/apache-maven-3.3.9.tar.gz
RUN tar -xzf /tmp/apache-maven-3.3.9.tar.gz -C /opt \
&& rm /tmp/apache-maven-3.3.9.tar.gz
ENV M2_HOME "/opt/apache-maven-3.3.9"
ENV PATH ${PATH}:${M2_HOME}/bin:
# Install Ant
ENV ANT_VERSION 1.9.4
RUN cd && \
wget -q http://archive.apache.org/dist/ant/binaries/apache-ant-${ANT_VERSION}-bin.tar.gz && \
tar -xzf apache-ant-${ANT_VERSION}-bin.tar.gz && \
mv apache-ant-${ANT_VERSION} /opt/ant && \
rm apache-ant-${ANT_VERSION}-bin.tar.gz
ENV ANT_HOME /opt/ant
ENV PATH ${PATH}:/opt/ant/bin
......
So as you can see in my docker file that I have installation instructions for both maven and ant. But now I've to install either one of them based on a condition. I know that I can use ARG instruction in the Dockerfile to fetch the argument during the build time, but the problem is I couldn't find any docs on how to enclose them in an if/else block.
I have read few other stackoverflow posts too regarding this, but in those I see that they have asked to use conditional statements inside of an instruction eg RUN if $BUILDVAR -eq "SO"; then export SOMEVAR=hello; else export SOMEVAR=world; fiLink, or writing a separate script file like this
But as you can see, my case is different. I can't make us of this since I would have a bunch of other instructions too which would be depended on that argument. I have to do something like this
ARG BUILD_TOOL
if [ "${BUILD_TOOL}" = "MAVEN" ]; then
--install maven
elif [ "${BUILD_TOOL}" = "ANT" ]; then
--install ant
fi
.... other instructions ....
if [ "${BUILD_TOOL}" = "MAVEN" ]; then
--some other dependent commands
elif [ "${BUILD_TOOL}" = "ANT" ]; then
--some other dependent commands
fi
If you don't want to use all those RUN if statements, you can instead create a bash script with the setup procedure and call it from the Dockerfile. For example:
FROM centos:centos7
MAINTAINER Someone <someone#email.com>
ARG BUILD_TOOL
COPY setup.sh /setup.sh
RUN ./setup.sh
RUN rm /setup.sh
And the setup.sh file (don't forget to make it executable):
if [ "${BUILD_TOOL}" = "MAVEN" ]; then
echo "Step 1 of MAVEN setup";
echo "(...)";
echo "Done MAVEN setup";
elif [ "${BUILD_TOOL}" = "ANT" ]; then
echo "Step 1 of ANT setup";
echo "(...)";
echo "Done ANT setup";
fi
You can then build it using docker build --build-arg BUILD_TOOL=MAVEN . (or ANT).
Note that I used a shell script here, but if you have other interpreters available (ex: python or ruby), you can also use them to write the setup script.
Related
I have a base Docker image:
FROM ubuntu:21.04
WORKDIR /app
RUN apt-get update && apt-get install -y wget bzip2 \
&& wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba \
&& touch /root/.bashrc \
&& ./bin/micromamba shell init -s bash -p /opt/conda \
&& cp /root/.bashrc /opt/conda/bashrc \
&& apt-get clean autoremove --yes \
&& rm -rf /var/lib/{apt,dpkg,cache,log}
SHELL ["bash", "-l" ,"-c"]
and derive from it another one:
ARG BASE
FROM $BASE
RUN source /opt/conda/bashrc && micromamba activate \
&& micromamba create --file environment.yaml -p /env
While building the second image I get the following error: micromamba: command not found for the RUN section.
If I run 1st base image manually I can launch micromamba, it is running correctly
I can run temporary image which were created for 2nd image building, micromamba is available via CLI, running correctly.
If I inherit from debian:buster, or alpine, for example, it is building perfectly.
What a problem with the Ubuntu? Why it cannot see micromamba during 2nd Docker image building?
PS using scaffold for building, so it can understand correctly, where is $BASE and what is it.
The ubuntu:21.04 image comes with a /root/.bashrc file that begins with:
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
[ -z "$PS1" ] && return
When the second Dockerfile executes RUN source /opt/conda/bashrc, PS1 is not set and thus the remainder of the bashrc file does not execute. The remainder of the bashrc file is where micromamba initialization occurs, including the setup of the micromamba bash function that is used to activate a micromamba environment.
The debian:buster image has a smaller /root/.bashrc that does not have a line similar to [ -z "$PS1" ] && return and therefore the micromamba function gets loaded.
The alpine image does not come with a /root/.bashrc so it also does not contain the code to exit the file early.
If you want to use the ubuntu:21.04 image, you could modify you first Dockerfile like this:
FROM ubuntu:21.04
WORKDIR /app
RUN apt-get update && apt-get install -y wget bzip2 \
&& wget -qO- https://micromamba.snakepit.net/api/micromamba/linux-64/latest | tar -xvj bin/micromamba \
&& touch /root/.bashrc \
&& ./bin/micromamba shell init -s bash -p /opt/conda \
&& grep -v '[ -z "\$PS1" ] && return' /root/.bashrc > /opt/conda/bashrc # this line has been modified \
&& apt-get clean autoremove --yes \
&& rm -rf /var/lib/{apt,dpkg,cache,log}
SHELL ["bash", "-l" ,"-c"]
This will strip out the one line that causes the early termination.
Alternatively, you could make use of the existing mambaorg/micromamba docker image. The mambaorg/micromamba:latest is based on debian:slim, but mambaorg/micromamba:jammy will get you a ubuntu-based image (disclosure: I maintain this image).
I use this Dockefile copied from here
#we are using ubuntu base image
FROM ubuntu:18.04
# installing requirements to get and extract prebuilt binaries
RUN apt-get update && apt-get install -y \
xz-utils \
curl \
&& rm -rf /var/lib/apt/lists/*
#Getting prebuilt binary from llvm
RUN curl -SL https://github.com/llvm/llvm-project/releases/download/llvmorg-10.0.0/clang+llvm-10.0.0-x86_64-linux-gnu-ubuntu-18.04.tar.xz \
| tar -xJC . && \
mv clang+llvm-10.0.0-x86_64-linux-gnu-ubuntu-18.04 clang_10 && \
echo ‘export PATH=/clang_10/bin:$PATH’ >> ~/.bashrc && \
echo ‘export LD_LIBRARY_PATH=/clang_10/lib:$LD_LIBRARY_PATH’ >> ~/.bashrc
#start the container from bash
CMD [ “/bin/bash” ]
Then I run
docker build -t clang_10 .
It works. With the clang_10 image well created.
But then I run
docker run -it clang_10
Nothing happens! The console does not switch to a Linux prompt as expected. Where do things go wrong?
The problem is the types of quotes: Use " instead of “.
Change the last line from
CMD [ “/bin/bash” ]
to
CMD [ "/bin/bash" ]
change the CMD to CMD [ "/bin/bash" ] or ["/bin/sh"]
I'm trying to run a cron job in a selenium container in Docker Desktop for windows. Because I think I'm running into several problems, it's hard for me to figure out which details matter so I'll try to be as thorough as possible.
Environment:
Docker Desktop for Windows (to avoid line ending problems I make the cron string in the Dockerfile)
Selenium-Chrome (one thing to note is that most things are run under seluser instead of root here. I say that because some of the other solutions don't work because of this)
Problem:
I cannot run python in my cron job
Related stack overflow links I've checked:
There are a lot but this is the main one.
For example, this snippet logs to the log file shown appropriately:
FROM selenium/standalone-chrome
COPY . /home/seluser/
# # install selenium
RUN echo "**** install packages ****" && \
sudo apt-get update && \
sudo apt-get install -y cron && \
echo "**** cleanup ****" && \
sudo apt-get clean && \
sudo rm -rf \
/tmp/* \
/var/lib/apt/lists/* \
/var/tmp/*
# Create the log file to be able to run tail
RUN touch /home/seluser/cron.log
# Setup cron job
RUN echo "* * * * * echo "Hello, World!" >> /home/seluser/cron.log" | sudo crontab
# Run the command on container startup
CMD sudo cron && tail -f /home/seluser/cron.log
But this doesn't:
FROM selenium/standalone-chrome
COPY . /home/seluser/
# # install selenium
RUN echo "**** install packages ****" && \
sudo apt-get update && \
sudo apt-get install -y cron && \
echo "**** cleanup ****" && \
sudo apt-get clean && \
sudo rm -rf \
/tmp/* \
/var/lib/apt/lists/* \
/var/tmp/*
# Create the log file to be able to run tail
RUN touch /home/seluser/cron.log
# Setup cron job
RUN echo "* * * * * /usr/bin/python3 -c print("Hello world") >> /home/seluser/cron.log" | sudo crontab
# Run the command on container startup
CMD sudo cron && tail -f /home/seluser/cron.log
In your RUN command, you have both double quotes surrounding the entire command and also double quotes in the embedded Python script. The shell consumes these pairs of quotes, and so there are no quotes in the Python fragment at all.
You can work around this by using single quotes for either pair of quotes, or by backslash-escaping the double quotes within the double-quoted string. All of these will work:
RUN echo "... print(\"Hello world\") ..." | crontab
RUN echo "... print('Hello world') ..." | crontab
# NB: will not expand environment variables inside the string
RUN echo '... print("Hello world") ...' | crontab
The first form with echo appears to work because echo "Hello world", echo Hello world, and echo "Hello" "world" all print out the same string; it doesn't matter whether it's one or two parameters, and echo(1) itself never sees the double quotes. In the Python case, though, you need to pass the double quotes into the interpreter.
If you're facing challenges like this, often the best approach is to break out the embedded script into a separate file.
#!/usr/bin/env python3
# greet.py
if __name__ == '__main__':
print('Hello world')
Then your cron job can just run the script.
RUN echo '* * * * * /home/seluser/greet.py >> /home/seluser/cron.log' | crontab
I've omitted sudo in my examples. In a Dockerfile you are generally running as root already, or you can specify USER root to switch to root. sudo is tricky to use in scripts and tends to lead to unnecessary and insecure configuration in Docker.
A final Dockerfile for this might look like:
FROM selenium/standalone-chrome
# Install OS packages _before_ copying any application code in.
# This avoids an expensive reinstallation on rebuild.
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive \
apt-get install --assume-yes --no-install-recommends \
cron
# Now copy your own code in. (Rebuilds will start here, if anything
# in the source tree has changed.)
COPY . /home/seluser
# Set up the cron environment.
RUN echo '* * * * * /home/seluser/greet.py >> /home/seluser/cron.log' | crontab \
&& touch /home/seluser/cron.log
# Declare the main container command. (Actually run the cron daemon;
# don't make the container process be tail(1) with cron being an
# unmonitored side effect.)
CMD ["cron", "-f"]
I build my project by Dockerfile. The project need to installation of Openvino. Openvino needs to set some environment variables dynamically by a script that depends on architecture. The sciprt is: script to set environment variables
As I learn, Dockerfile can't set enviroment variables to image from a script.
How do I follow way to solve the problem?
I need to set the variables because later I continue install opencv that looks the enviroment variables.
What I think that if I put the script to ~/.bashrc file to set variables when connect to bash, if I have any trick to start bash for a second, it could solve my problem.
Secondly I think that build openvino image, create container from that, connect it and initiliaze variables by running script manually in container. After that, convert the container to image. Create new Dockerfile and continue building steps by using this images for ongoing steps.
Openvino Dockerfile exp and line that run the script
My Dockerfile:
FROM ubuntu:18.04
ARG DOWNLOAD_LINK=http://registrationcenter-download.intel.com/akdlm/irc_nas/16612/l_openvino_toolkit_p_2020.2.120.tgz
ENV INSTALLDIR /opt/intel/openvino
# openvino download
RUN curl -LOJ "${DOWNLOAD_LINK}"
# opencv download
RUN wget -O opencv.zip https://github.com/opencv/opencv/archive/4.3.0.zip && \
wget -O opencv_contrib.zip https://github.com/opencv/opencv_contrib/archive/4.3.0.zip
RUN apt-get -y install sudo
# openvino installation
RUN tar -xvzf ./*.tgz && \
cd l_openvino_toolkit_p_2020.2.120 && \
sed -i 's/decline/accept/g' silent.cfg && \
./install.sh -s silent.cfg && \
# rm -rf /tmp/* && \
sudo -E $INSTALLDIR/install_dependencies/install_openvino_dependencies.sh
WORKDIR /home/sa
RUN /bin/bash -c "source /opt/intel/openvino/bin/setupvars.sh" && \
echo "source /opt/intel/openvino/bin/setupvars.sh" >> /home/sa/.bashrc && \
echo "source /opt/intel/openvino/bin/setupvars.sh" >> ~/.bashrc && \
$INSTALLDIR/deployment_tools/model_optimizer/install_prerequisites/install_prerequisites.sh && \
$INSTALLDIR/deployment_tools/demo/demo_squeezenet_download_convert_run.sh
RUN bash
# opencv installation
RUN unzip opencv.zip && \
unzip opencv_contrib.zip && \
# rm opencv.zip opencv_contrib.zip && \
mv opencv-4.3.0 opencv && \
mv opencv_contrib-4.3.0 opencv_contrib && \
cd ./opencv && \
mkdir build && \
cd build && \
cmake -D CMAKE_BUILD_TYPE=RELEASE -D WITH_INF_ENGINE=ON -D ENABLE_CXX11=ON -D CMAKE_INSTALL_PREFIX=/usr/local -D INSTALL_PYTHON_EXAMPLES=OFF -D INSTALL_C_EXAMPLES=OFF -D ENABLE_PRECOMPILED_HEADERS=OFF -D OPENCV_ENABLE_NONFREE=ON -D OPENCV_EXTRA_MODULES_PATH=/home/sa/opencv_contrib/modules -D PYTHON_EXECUTABLE=/usr/bin/python3 -D WIDTH_GTK=ON -D BUILD_TESTS=OFF -D BUILD_DOCS=OFF -D WITH_GSTREAMER=OFF -D WITH_FFMPEG=ON -D BUILD_EXAMPLES=OFF .. && \
make && \
make install && \
ldconfig
You need to cause the shell to load that file in every RUN command where you use it, and also at container startup time.
For startup time, you can use an entrypoint wrapper script:
#!/bin/sh
# Load the script of environment variables
. /opt/intel/openvino/bin/setupvars.sh
# Run the main container command
exec "$#"
Then in the Dockerfile, you need to include the environment variable script in RUN commands, and make this script be the image's ENTRYPOINT.
RUN . /opt/intel/openvino/bin/setupvars.sh && \
/opt/intel/openvino/deployment_tools/model_optimizer/install_prerequisites/install_prerequisites.sh && \
/opt/intel/openvino/deployment_tools/demo/demo_squeezenet_download_convert_run.sh
RUN ... && \
. /opt/intel/openvino/bin/setupvars.sh && \
cmake ... && \
make && \
...
COPY entrypoint.sh .
ENTRYPOINT ["./entrypoint.sh"]
CMD same as the command you set in the original image
If you docker exec debugging shells in the container, they won't see these environment variables and you'll need to manually re-read the environment variable script. If you use docker inspect to look at low-level details of the container, it also won't show the environment variables.
It looks like that script just sets a couple of environment variables (especially $LD_LIBRARY_PATH and $PYTHONPATH), if to somewhat long-winded values, and you could just set these with ENV statements in the Dockerfile.
If you look at the docker build output, there are lines like ---> 0123456789ab after each build step; those are valid image IDs that you can docker run. You could run
docker run --rm 0123456789ab \
env \
| sort > env-a
docker run --rm 0123456789ab \
sh -c '. /opt/intel/openvino/bin/setupvars.sh && env' \
| sort > env-b
This will give you two local files with the environment variables with and without running this setup script. Find the differences (say, with comm(1)), put ENV before each line, and add that to your Dockerfile.
You can't really use .bashrc in Docker. Many common paths don't invoke its startup files: in the language of that documentation, neither a Dockerfile RUN command nor a docker run instruction is an "interactive shell" so those don't read dot files, and usually docker run ... command doesn't invoke a shell at all.
You also don't need sudo (you are already running as root, and an interactive password prompt will fail); RUN sh -c is redundant (Docker inserts it on its own); and source isn't a standard shell command (prefer the standard ., which will work even on Alpine-based images that don't have shell extensions).
I need to install command line tools like jq, curl etc in the docker image created by maven jib plugin. How can I achieve this? Any help would be greatly appreciated. Thanks.
As explained in the other answer, using a base image customized with pre-installed tools that rarely change is a good solution.
Alternatively, you may put curl using Jib's <extraDirectories> feature, which enables adding arbitrary files to the target image. Check the Maven and Gradle docs for more details. As explained in the docs, you will also need to configure <permissions> to set executable bits to curl.
If you prefer, you could even set up your Maven or Gradle builds to download curl and unpack it. Here's an example Jib setup (showing both Maven and Gradle) from the Jib repository.
Adding a reference Dockerfile and you can build your own base image by creating your Dockerfile and then build it.
FROM openjdk:8-jdk-alpine
RUN apk add --no-cache curl tar bash procps
# Downloading and installing Maven
ARG MAVEN_VERSION=3.6.1
ARG USER_HOME_DIR="/root"
ARG SHA=b4880fb7a3d81edd190a029440cdf17f308621af68475a4fe976296e71ff4a4b546dd6d8a58aaafba334d309cc11e638c52808a4b0e818fc0fd544226d952544
ARG BASE_URL=https://apache.osuosl.org/maven/maven-3/${MAVEN_VERSION}/binaries
RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
&& echo "Downlaoding maven" \
&& curl -fsSL -o /tmp/apache-maven.tar.gz ${BASE_URL}/apache-maven-${MAVEN_VERSION}-bin.tar.gz \
\
&& echo "Checking download hash" \
&& echo "${SHA} /tmp/apache-maven.tar.gz" | sha512sum -c - \
\
&& echo "Unziping maven" \
&& tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven --strip-components=1 \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/apache-maven.tar.gz \
&& ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
ENV MAVEN_HOME /usr/share/maven
ENV MAVEN_CONFIG "$USER_HOME_DIR/.m2"
# Downloading and installing Gradle
# 1- Define a constant with the version of gradle you want to install
ARG GRADLE_VERSION=4.0.1
# 2- Define the URL where gradle can be downloaded from
ARG GRADLE_BASE_URL=https://services.gradle.org/distributions
# 3- Define the SHA key to validate the gradle download
# obtained from here https://gradle.org/release-checksums/
ARG GRADLE_SHA=d717e46200d1359893f891dab047fdab98784143ac76861b53c50dbd03b44fd4
# 4- Create the directories, download gradle, validate the download, install it, remove downloaded file and set links
RUN mkdir -p /usr/share/gradle /usr/share/gradle/ref \
&& echo "Downlaoding gradle hash" \
&& curl -fsSL -o /tmp/gradle.zip ${GRADLE_BASE_URL}/gradle-${GRADLE_VERSION}-bin.zip \
\
&& echo "Checking download hash" \
&& echo "${GRADLE_SHA} /tmp/gradle.zip" | sha256sum -c - \
\
&& echo "Unziping gradle" \
&& unzip -d /usr/share/gradle /tmp/gradle.zip \
\
&& echo "Cleaning and setting links" \
&& rm -f /tmp/gradle.zip \
&& ln -s /usr/share/gradle/gradle-${GRADLE_VERSION} /usr/bin/gradle
# 5- Define environmental variables required by gradle
ENV GRADLE_VERSION 4.0.1
ENV GRADLE_HOME /usr/bin/gradle
ENV GRADLE_USER_HOME /cache
ENV PATH $PATH:$GRADLE_HOME/bin
VOLUME $GRADLE_USER_HOME
CMD [""]
Ref:- https://docs.docker.com/engine/reference/builder/
Once your custom image is ready, push it to Registry and then reference it in jib in following manner.
mvn compile jib:build \
-Djib.from.image=customImage