I'm trying to write a Jenkins pipeline to build a Dockerfile based off a multi-line string parameter.
I use a Multi-line string parameter to allow a user to post the contents of their dockerfile, eg:
FROM ubuntu
#Pre-reqs
RUN apt-get update \
&& apt-get -y upgrade
Part of my process currently is to take this parameter and echo it to a Dockerfile:
sh 'echo ${dockerContents} > Dockerfile'
However, this creates a Dockerfile with the output all one one line:
FROM ubuntu #Pre-reqs RUN apt-get update \ && apt-get -y upgrade
I know a hacky workaround would be to request new line characters in the multi-line parameter entry but there must be a better way.
Any suggestions appreciated.
P.S - there is an open issue around this in Jenkins already (https://issues.jenkins.io/browse/JENKINS-13916?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel&showAll=true)
Related
Is it possible to use the same shell in all RUN commands when building a docker image? As opposed to each RUN command running on its own shell.
Use case: at some point, I need to source some file containing environment variables that are used later on. I cannot do this, because the commands run in different shells:
RUN source something.sh
RUN ./install.sh
RUN ... more commands
Instead I have to do:
RUN source something.sh && \
./install.sh && \
... more commands
Which I'm trying to avoid since it hurts readability, it's error prone and does not allow inserting comments in between commands.
Any ideas?
Thanks!
It's not possible to have separate RUN statement run in the same shell.
If you don't like the look of concatenated commands, you could write a shell script and RUN that.
You'll have to get it into the container by using a COPY statement.
Or you can use wget or curl to fetch it and pipe it into a shell. That requires that wget or curl is present in the container, so you might have to install them first.
If you use curl and Debian, it could look like this
RUN apt update && \
apt install -y curl && \
curl -sL https://github.com/link/to/my/install-script.sh | bash
If you COPY it in, it'd look like this
COPY install-script.sh .
RUN ./install-script.sh
I created a custom image with the following Dockerfile:
FROM apache/airflow:2.1.1-python3.8
USER root
RUN apt-get update \
&& apt-get -y install gcc gnupg2 \
&& curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add - \
&& curl https://packages.microsoft.com/config/debian/10/prod.list > /etc/apt/sources.list.d/mssql-release.list
RUN apt-get update \
&& ACCEPT_EULA=Y apt-get -y install msodbcsql17 \
&& ACCEPT_EULA=Y apt-get -y install mssql-tools
RUN echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& echo 'export PATH="$PATH:/opt/mssql-tools/bin"' >> ~/.bashrc \
&& source ~/.bashrc
RUN apt-get -y install unixodbc-dev \
&& apt-get -y install python-pip \
&& pip install pyodbc
RUN echo -e “AIRFLOW_UID=$(id -u) \nAIRFLOW_GID=0” > .env
USER airflow
The image creates successfully, but when I try to run it, I get this error:
"airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above."
I have tried supplying a group ID with the --user, but I can't figure it out.
How can I start this custom Airflow Docker image?
Thanks!
First of all this line is wrong:
RUN echo -e “AIRFLOW_UID=$(id -u) \nAIRFLOW_GID=0” > .env
If you are running it with Docker Compose (I presume you took it from https://airflow.apache.org/docs/apache-airflow/stable/start/docker.html), this is something you should run on "Host" machine, not in the image. Remove that line, it has no effect.
Secondly - it really depends what "command" you run. The "GROUP_OR_COMMAND" message you got is the output of "airflow" command. You have not copied the whole output of your command but this is a message you get when you try to run airflow without telling it what to do. When you run the image you will run by default the airflow command which has a number of subcommands that can be executed. So the "see help above" message tells you the very thing you should do - look at the help and see what subcommand you wanted to run (and possibly run it).
docker run -it apache/airflow:2.1.2
usage: airflow [-h] GROUP_OR_COMMAND ...
positional arguments:
GROUP_OR_COMMAND
Groups:
celery Celery components
config View configuration
connections Manage connections
dags Manage DAGs
db Database operations
jobs Manage jobs
kubernetes Tools to help run the KubernetesExecutor
pools Manage pools
providers Display providers
roles Manage roles
tasks Manage tasks
users Manage users
variables Manage variables
Commands:
cheat-sheet Display cheat sheet
info Show information about current Airflow and environment
kerberos Start a kerberos ticket renewer
plugins Dump information about loaded plugins
rotate-fernet-key
Rotate encrypted connection credentials and variables
scheduler Start a scheduler instance
sync-perm Update permissions for existing roles and optionally DAGs
version Show the version
webserver Start a Airflow webserver instance
optional arguments:
-h, --help show this help message and exit
airflow command error: the following arguments are required: GROUP_OR_COMMAND, see help above.
when you extend the official image, it will pass the parametor to "airflow" command which causing this problem. Check this out: https://airflow.apache.org/docs/docker-stack/entrypoint.html#entrypoint-commands
I am experimenting with docker's buildx and noticed that everything seems to be straight forward except for one thing. My Dockerfile needs to pull certain packages depending on the architecture.
For example, here's a piece of the Dockerfile:
FROM XYZ
# Set environment variable for non-interactive install
ARG DEBIAN_FRONTEND=noninteractive
# Run basic commands to update the image and install basic stuff.
RUN apt update && \
apt dist-upgrade -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" && \
apt autoremove -y && \
apt clean -y && \
...
# Install amazon-ssm-agent
mkdir /tmp/ssm && \
curl https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/debian_amd64/amazon-ssm-agent.deb -o /tmp/ssm/amazon-ssm-agent.deb && \
As you can see from above, the command is set to pull down the Amazon SSM agent using a hard-coded link.
What's the best way to approach this? Should I just modify this Dockerfile to create a bunch of if conditions?
Docker automatically defines a set of ARGs for you when you're using the BuildKit backend (which is now the default). You need to declare that ARG, and then (within the RUN command) you can use an environment variable $TARGETOS to refer to the target operating system (the documentation suggests linux or windows).
FROM ...
# Must be explicitly declared, and after FROM
ARG TARGETOS
# Then it can be used like a normal environment variable
RUN curl https://s3.amazonaws.com/ec2-downloads-$TARGETOS/...
There is a similar $TARGETPLATFORM if you need to build either x86 or ARM images, but its syntax doesn't necessarily match what's in this URL. If $TARGETPLATFORM is either amd64 or arm, you may need to reconstruct the Debian architecture string. You can set a shell variable within a single RUN command and it will last until the end of that command, but no longer.
ARG TARGETPLATFORM
RUN DEBARCH="$TARGETPLATFORM"; \
if [ "$DEBARCH" = "arm" ]; then DEBARCH=arm64; fi; \
curl .../debian-$DEBARCH/...
Consider the following Dockerfile:
FROM alpine:edge
EXPOSE \
# web portal
8080 \
# backdoor
8081
Built like so:
docker build .
We observe such output:
Sending build context to Docker daemon 17.1TB
Step 1/2 : FROM alpine:edge
---> 7463224280b0
Step 2/2 : EXPOSE 8080 8081
---> Using cache
---> 7953f8df04d9
[WARNING]: Empty continuation line found in:
EXPOSE 8080 8081
[WARNING]: Empty continuation lines will become errors in a future release.
Successfully built 7953f8df04d9
So, given that it'll soon become illegal to put comments in the middle of a multi-line section: what's the new recommended way to comment multi-line commands?
This is particularly important for RUN commands, since we are encouraged to reduce image layers by &&ing commands together.
Not sure exactly when this was introduced, but I'm currently experiencing this in version:
🍔 docker --version
Docker version 17.07.0-ce, build 8784753
I'm using Docker's edge release stream, so maybe this will not yet look familiar if you are using Docker stable.
17.07.0-ce started to warn on empty continuation lines. However, it incorrectly treated comment-only lines as empty. This is fixed in moby#35004, and being included in the 17.10.0-ce.
On top of what others have said above (the error might be related to comments inside continuation blocks and/or windows cr/lf characters = use dos2unix), this message can also show up when your last command ends with a backslash \ character. For example, if you have this:
RUN apt-get update \
&& apt-get upgrade \
&& apt-get -y install build-essential curl gnupg libfontconfig ca-certificates bzip2 \
&& curl -sL https://deb.nodesource.com/setup_16.x | bash - \
&& apt-get -y install nodejs \
&& apt-get clean \
&& rm -rf /tmp/* /var/lib/apt/lists/* \
Notice the last \ at the end. This will get you the same error:
docker [WARNING]: Empty continuation line found in:
So, just remove that last \ and you're all set.
You could break the RUN commands out on to separate lines, and then use the experimental (at time of writing*) --squash command.
* note that it's been suggested that multi-stage builds might make --squash redundant. That is actively being discussed here, with a proposal open here.
If, like me, you came here with the same error but no comments in your Dockerfile's RUN item, you have either mixed or DOS line endings. Run dos2unix on your Dockerfile and that'll fix it.
Edit: Solved- typo
I have a Dockerfile that successfully creates a virtualenv using virtualenvwrapper (along with setting up a heap of "standard" settings/packages in our normal environment). I am using the resulting image as a "base image" for further use. All good so far. However, the following Dockerfile (based of the first image, "base_image_14.04") falls down at the last line:
FROM base_image_14.04
USER root
RUN DEBIAN_FRONTEND=noninteractive \
apt-get update && apt-get install -y \
libproj0 libproj-dev \
libgeos-c1v5 libgeos-dev \
libjpeg62 libjpeg-dev \
zlib1g zlib1g-dev \
libfreetype6 libfreetype6-dev \
libgdal20 libgdal-dev \
&& rm -rf /var/lib/apt/lists
USER webdev
RUN ["/bin/bash", "-ic", "mkproject maproxy"]
EXPOSE 80
WORKDIR $PROJECT_HOME/mapproxy
ADD ./requirements.txt .
RUN ["/bin/bash", "-ic", "workon mapproxy && pip install -r requirements.txt"]
The "mkproject mapproxy" works fine. If I comment out the last line it builds successfully and I can spin up the container and run "workon mapproxy" manually, not a problem. But when I try and build with the last line, it gives a workon error:
ERROR: Environment 'mapproxy' does not exist. Create it with 'mkvirtualenv mapproxy'.
workon is being called, but for some reason it can't find the mapproxy virtualenv.
WORKON_HOME & PROJECT_HOME both exist (defined in the parent image) and point to the correct locations (and are used successfully by "mkproject mapproxy").
So why is workon returning an error when the mapproxy virtualenv exists? The same error happens when I isolate that last line into a third Dockerfile building on the second.
Solved: It was a simple typo. mkproject maproxy instead of mapproxy. :sigh:
I am trying to build a docker image and am running into similar problems.
First question was why use a virtual env in docker? The main reason in a nutshell is to minimize effort to migrate an existing and working approach into a docker container. I will eventually use docker-compose, but I wanted to start by getting my feet wet with it all in a single docker container.
In my first attempt I installed almost everything with apt-get, including uwsgi. I installed my app "globally" with pip3. The app has command line functionality and a separate flask web app, hence the need for uwsgi. The command line functionality works, but when I make a request of the flask app uwsgi / python has a problem with locale: Fatal Python error: Py_Initialize: Unable to get the locale encoding and ImportError: No module named 'encodings
I have stripped away all my app specific additions to narrow down the problem. This is the Dockerfile I'm using:
# Docker image definition for testing
FROM ubuntu:xenial
# Create a user
RUN useradd -G sudo -ms /bin/bash tester
RUN echo 'tester:password' | chpasswd
WORKDIR /home/tester
# Skipping apt-get update to save some build time. Some are kept
# to insure they are the same as on host setup.
RUN apt-get install -y python3 python3-dev python3-pip \
virtualenv virtualenvwrapper sudo nano && \
apt-get clean -qy
# After above, can we use those installed in rest of Dockerfile?
# Yes, but not always, such as with virtualenvwrapper. What about
# virtualenv? How do you "source" the script? Doesn't appear to be
# installed, as bash complains "source needs a single parameter"
ENV VIRTUALENVWRAPPER_PYTHON /usr/bin/python3
ENV VIRTUALENVWRAPPER_VIRTUALENV /usr/bin/virtualenv
RUN ["/bin/bash", "-c", "source", "/usr/share/virtualenvwrapper/virtualenvwrapper.sh"]
# Create a virtualenv so uwsgi can find locale
# RUN mkdir /home/tester/.virtualenv && virtualenv -p`which python3` /home/bts_tools/.virtualenv/bts_tools
RUN mkvirtualenv -p`which python3` bts_tools && \
workon bts_tools && \
pip3 --disable-pip-version-check install --upgrade bts_tools
USER tester
ENTRYPOINT ["/bin/bash"]
CMD ["--login"]
The build fails on the line I try to source the virtualenvwrapper script. Bash complains source needs an argument - the file to be sourced. So I comment out the RUN lines and it builds without error. When I run the resulting container I see all the additions to the ENV that virtualenvwrapper makes (you can see all of them by executing the "set" command without any args), and the script to be sourced is there too.
So my question is why doesn't docker find them? How does the docker build process work if the results of any previous RUNs or ENVs aren't applied for subsequent use in the Dockerfile? I know some things are applied and work, for example if you apt-get nginx you can refer to /etc/nginx or alter things under that folder. You can create a user and set it's password or cd into its home folder for example. If I move the WORKDIR before the RUN useradd -G I see a warning from useradd the home folder already exists. I tried to use the "time" program to time how long it takes to do various things in the Dockerfile and docker complains it can't find 'time'.
So what exactly is going on? I have spent the last 3 days trying to figure this out. It just shouldn't be this difficult. What am I missing?
Parts of the bts_tools flask app worked when I wasn't using virtual envs. Most of the app didn't work, and the issue was this locale problem. Since everything works on the host outside of docker, and after trying to alter the PATH, PYTHONHOME, PYTHONPATH in my uwsgi start script to overcome the dreaded "locale encoding" fatal error, I decided to try to replicate the host setup as closely as possible since that didn't have the locale issue. When I have had that problem before I could run dpkg-reconfigure python3 or fix with changes to PATH or ENV settings. If you google the problem you'll see many people have difficulties with python & locale. It's almost enough reason to avoid using python!
I posted this elsewhere about locale issue, if it helps.