mkdir .ssh in a Dockerfile, folder is not there? - docker

I want my Dockerfile to mkdir .ssh/
But it does not, why not?
FROM jenkinsci/jnlp-slave
MAINTAINER Johnny5 isAlive <johnny5#hotmail.com>
USER root
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get update
RUN apt-get install unzip git curl vim -y
USER jenkins
RUN mkdir -p /home/jenkins/.ssh && touch /home/jenkins/.ssh/aFile
...building...
Looks fine?
Step 12 : RUN mkdir -p /home/jenkins/.ssh && touch /home/jenkins/.ssh/aFile
---> Running in ca19a679580d
---> 5980df7db482
Removing intermediate container ca19a679580d
Successfully built 5980df7db482
Running and looking around, the .ssh/ folder and aFile inside are not there ...
$ docker run -it -u 0 --entrypoint /bin/bash 5980df7db482
root#4aa40a18baf2:~# pwd
/home/jenkins
root#4aa40a18baf2:~# ls -al
total 24
drwxr-xr-x 3 jenkins jenkins 4096 Oct 17 23:17 .
drwxr-xr-x 4 root root 4096 Sep 14 08:50 ..
-rw-r--r-- 1 jenkins jenkins 220 Nov 12 2014 .bash_logout
-rw-r--r-- 1 jenkins jenkins 3515 Nov 12 2014 .bashrc
-rw-r--r-- 1 jenkins jenkins 675 Nov 12 2014 .profile
drwxr-xr-x 2 jenkins jenkins 4096 Sep 14 08:50 .tmp
root#4aa40a18baf2:~#

If I pull the parent image, jenkinsci/jnlp-slave, and inspect it with docker inspect jenkinsci/jnlp-slave, I can see that it already has a volume defined at /home/jenkins:
[
{
...
"ContainerConfig": {
...
"Volumes": {
"/home/jenkins": {}
},
...
}
]
This means that during each build step, any changes you make to that location won't be committed to your new layer.
Here's a simplified version of your Dockerfile to highlight what's going on:
FROM jenkinsci/jnlp-slave
RUN mkdir -p /home/jenkins/.ssh
Now, let's build with the following command: docker build --no-cache --rm=false -t jns .:
Sending build context to Docker daemon 2.56 kB
Step 1 : FROM jenkinsci/jnlp-slave
---> d7731d944ad7
Step 2 : RUN mkdir -p /home/jenkins/.ssh
---> Running in 520a8e2f7cae
---> 962189878d5e
Successfully built 962189878d5e
The --no-cache option makes the command easier to work with on repeat invocations. The --rm=false will cause the builder to not remove the containers created for each step.
In this case, the builder ran the Step 2 in 520a8e2f7cae on my system. I can now do a docker inspect 520a8e2f7cae and see the actual container used for this step. Specifically, I'm curious about the mounts location:
[
{
...
"Mounts": [
{
"Name": "e34fd82bd190f21dbd63b5cf70167a16674cd00d95fdc6159314c25c6d08e10e",
"Source": "/var/lib/docker/volumes/e34fd82bd190f21dbd63b5cf70167a16674cd00d95fdc6159314c25c6d08e10e/_data",
"Destination": "/home/jenkins",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
...
}
]
I see that there's an anonymous volume with id e34fd82bd190f21dbd63b5cf70167a16674cd00d95fdc6159314c25c6d08e10e for /home/jenkins.
I can inspect the contents of that volume like this:
$ docker run --rm -v e34fd82bd190f21dbd63b5cf70167a16674cd00d95fdc6159314c25c6d08e10e:/volume alpine ls -lah /volume
total 28
drwxr-xr-x 4 10000 10000 4.0K Oct 18 02:49 .
drwxr-xr-x 25 root root 4.0K Oct 18 02:55 ..
-rw-r--r-- 1 10000 10000 220 Nov 12 2014 .bash_logout
-rw-r--r-- 1 10000 10000 3.4K Nov 12 2014 .bashrc
-rw-r--r-- 1 10000 10000 675 Nov 12 2014 .profile
drwxr-xr-x 2 10000 10000 4.0K Oct 18 02:49 .ssh
drwxr-xr-x 2 10000 10000 4.0K Sep 14 08:50 .tmp
The .ssh directory created in the RUN step is in this volume. Since volumes aren't part of the container's write layer, it won't get committed. I can confirm this by doing a docker diff on this container:
docker diff 520a8e2f7cae
There is no output, showing no changes to the container's filesystem, which is why it doesn't come forward into this layer of the image.
The other contents at this location are files in the parent image that were committed before the VOLUME directive that made /home/jenkins into a volume.

Related

Bug or misuse : docker layer content not available in final image

I am having a hard time trying to build a docker image for some python project, but at some point a RUN command resulting content is not available in the next RUN.
I mean I have a RUN command that generate a directory with some content and that directory is not available for the next RUN command.
Here is the Dockerfile snippet :
RUN poetry config virtualenvs.create true && \
poetry config virtualenvs.in-project true && \
poetry install -n --no-root --no-dev && \
ls -la
RUN ls -la
The first ls -la shows a .venv directory (see below) while it is not present anymore in the second one.
And here is the build output:
Step 9/14 : RUN poetry config virtualenvs.create true && poetry config virtualenvs.in-project true && poetry install -n --no-root --no-dev && ls -la
---> Running in 3a93a4994133
...
drwxrwxr-x 4 root odoo 4096 Sep 29 20:08 .
drwxr-xr-x 1 root root 4096 Sep 16 02:35 ..
drwxr-xr-x 4 odoo odoo 4096 Sep 29 20:08 .venv
-rw-rw-r-- 1 odoo odoo 15824 Sep 2 23:18 poetry.lock
-rw-rw-r-- 1 odoo odoo 2057 Sep 2 23:50 pyproject.toml
drwxr-xr-x 3 odoo odoo 4096 Sep 29 20:08 src
Removing intermediate container 3a93a4994133
---> 694c4e54863c
Step 10/14 : RUN ls -la
---> Running in 5d0ebcf8f1e9
total 32
drwxrwxr-x 3 root odoo 4096 Sep 29 20:08 .
drwxr-xr-x 1 root root 4096 Sep 16 02:35 ..
-rw-rw-r-- 1 odoo odoo 15824 Sep 2 23:18 poetry.lock
-rw-rw-r-- 1 odoo odoo 2057 Sep 2 23:50 pyproject.toml
drwxr-xr-x 3 odoo odoo 4096 Sep 29 20:08 src
Removing intermediate container 5d0ebcf8f1e9
---> 750255eac6bb
If someone can provide any lead, it would be really appreciated.
Thanks

Why does docker image content differ from the container created from it?

Following is the Dockerfile for the image,
FROM jenkins/jenkins:lts-jdk11
USER jenkins
RUN jenkins-plugin-cli --plugins "blueocean:1.25.2 http_request" && ls -la /var/jenkins_home
When this is built using docker build -t ireshmm/jenkins:lts-jdk11 ., following is the output,
Sending build context to Docker daemon 3.072kB
Step 1/3 : FROM jenkins/jenkins:lts-jdk11
---> 9aee0d53624f
Step 2/3 : USER jenkins
---> Using cache
---> 49d657d24299
Step 3/3 : RUN jenkins-plugin-cli --plugins "blueocean:1.25.2 http_request" && ls -la /var/jenkins_home
---> Running in b459c4c48e3e
Done
total 20
drwxr-xr-x 3 jenkins jenkins 4096 Jan 22 16:49 .
drwxr-xr-x 1 root root 4096 Jan 12 15:46 ..
drwxr-xr-x 3 jenkins jenkins 4096 Jan 22 16:49 .cache
-rw-rw-r-- 1 jenkins root 7152 Jan 12 15:42 tini_pub.gpg
Removing intermediate container b459c4c48e3e
---> 5fd5ba428f1a
Successfully built 5fd5ba428f1a
Successfully tagged ireshmm/jenkins:lts-jdk11
When create a container and list files docker run -it --rm ireshmm/jenkins:lts-jdk11 ls -la /var/jenkins_home, following is the output:
total 40
drwxr-xr-x 3 jenkins jenkins 4096 Jan 22 16:51 .
drwxr-xr-x 1 root root 4096 Jan 12 15:46 ..
-rw-r--r-- 1 jenkins jenkins 4683 Jan 22 16:51 copy_reference_file.log
drwxr-xr-x 2 jenkins jenkins 16384 Jan 22 16:51 plugins
-rw-rw-r-- 1 jenkins root 7152 Jan 12 15:42 tini_pub.gpg
Question: Why do the contents of /var/jenkins_home differ while building the image and the inside the container created from it given that no command is run after listing the files while building image? How can that happen?
The jenkins/jenkins:lts-jdk11 has an ENTRYPOINT that runs /usr/local/bin/jenkins.sh, which among other things creates the copy_reference_file.log file:
$ grep -i copy_reference /usr/local/bin/jenkins.sh
: "${COPY_REFERENCE_FILE_LOG:="${JENKINS_HOME}/copy_reference_file.log"}"
touch "${COPY_REFERENCE_FILE_LOG}" || { echo "Can not write to ${COPY_REFERENCE_FILE_LOG}. Wrong volume permissions?"; exit 1; }
echo "--- Copying files at $(date)" >> "$COPY_REFERENCE_FILE_LOG"
find "${REF}" \( -type f -o -type l \) -exec bash -c '. /usr/local/bin/jenkins-support; for arg; do copy_reference_file "$arg"; done' _ {} +
The ENTRYPOINT scripts runs whenever you start a container from that image (before any command you've provided on the command line).

Docker: file permissions with --volume bind mount

I'm following the guidelines from: https://denibertovic.com/posts/handling-permissions-with-docker-volumes/ to setup a --volume bind mount in my container and creating a user in the guest container with the same UID as my host user - the theory being that my container user should be able to access the mount. It's not working for me and I'm looking for some pointers to try next.
More background details:
My Dockerfile starts from an alpine base and adds python dev packages. It copies across an entrypoint.sh script per guidelines from denibertovic. It then jumps to the entrpoint.sh script.
FROM alpine
RUN apk update
RUN apk add bash
RUN apk add python3
RUN apk add python3-dev
RUN apk add su-exec
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
The entrpoint.sh script adds a user to the container with the UID passed in as an environment variable.
#!/bin/bash
# Add local user
# Either use the LOCAL_USER_ID if passed in at runtime or
# fallback
USER_ID=${LOCAL_USER_ID:-9001}
echo "Starting with UID : $USER_ID"
adduser -s /bin/bash -u $USER_ID -H -D user
export HOME=/home/user
su-exec user "$#"
The container builds no problem.
I then run it with the following command line:
sudo docker run -it -e LOCAL_USER_ID=`id -u` -v `realpath ../..`:/ws django-runtime /bin/bash
You'll see that I'm passing in my host UID to be mapped to the container user's UID and I'm asking for a volume bind mount from my local working directory to the /ws mountpoint in the container.
From the bash shell inside the container I can see that /ws is owned by the 'user' UID matching my own 'id'. However, when I go to list the contents of /ws I get a Permission Denied error as follows:
[dleclair#localhost runtime]$ sudo docker run -it -e LOCAL_USER_ID=`id -u` -v `realpath ../..`:/ws django-runtime /bin/bash
[sudo] password for dleclair:
Starting with UID : 1000
bash-5.0$ id
uid=1000(user) gid=1000(user) groups=1000(user)
bash-5.0$ ls -la .
total 0
drwxr-xr-x 1 root root 27 Feb 8 09:15 .
drwxr-xr-x 1 root root 27 Feb 8 09:15 ..
-rwxr-xr-x 1 root root 0 Feb 8 09:15 .dockerenv
drwxr-xr-x 1 root root 18 Feb 8 07:44 bin
drwxr-xr-x 5 root root 360 Feb 8 09:15 dev
drwxr-xr-x 1 root root 91 Feb 8 09:15 etc
drwxr-xr-x 2 root root 6 Jan 16 21:52 home
drwxr-xr-x 1 root root 17 Jan 16 21:52 lib
drwxr-xr-x 5 root root 44 Jan 16 21:52 media
drwxr-xr-x 2 root root 6 Jan 16 21:52 mnt
drwxr-xr-x 2 root root 6 Jan 16 21:52 opt
dr-xr-xr-x 119 root root 0 Feb 8 09:15 proc
drwx------ 2 root root 6 Jan 16 21:52 root
drwxr-xr-x 1 root root 21 Feb 8 07:44 run
drwxr-xr-x 1 root root 21 Feb 8 08:22 sbin
drwxr-xr-x 2 root root 6 Jan 16 21:52 srv
dr-xr-xr-x 13 root root 0 Feb 8 01:58 sys
drwxrwxrwt 2 root root 6 Jan 16 21:52 tmp
drwxr-xr-x 1 root root 19 Feb 8 07:44 usr
drwxr-xr-x 1 root root 19 Jan 16 21:52 var
drwxrwxr-x 5 user user 111 Feb 8 02:15 ws
bash-5.0$
bash-5.0$
bash-5.0$ cd /ws
bash-5.0$ ls -la
ls: can't open '.': Permission denied
total 0
bash-5.0$
Appreciate any pointers anyone can offer. Thanks!
After more searching I found the answer to my problem here: Permission denied on accessing host directory in Docker and here: http://www.projectatomic.io/blog/2015/06/using-volumes-with-docker-can-cause-problems-with-selinux/.
In short, the problem was with the SELinux default labels for the volume mount blocking access to the mounted files. The solution was to add a ':Z' trailer to the -v command line argument to force docker to set the appropriate flags against the mounted files to allow access.
The command line therefore became:
sudo docker run -it -e LOCAL_USER_ID=`id -u` -v `realpath ../..`:/ws:Z django-runtime /bin/bash
Worked like a charm.

How to save data from a docker container on a local host?

I run R-Studio in a container on GitLab. R-Studio build a lot of csv and pdf files. When I run
docker run --rm -it registry.gitlab.com/user/paperboy /bin/bash
I can find in the folder /home/output/csv and /home/output/pdf the files. I will save all this files in a /output/csv and /output/pdf files on a host, in my case on GitLab. The question is how to save data outside the docker Container?
Here is my Dockerfile.
FROM rocker/r-base:latest
RUN apt-get update \
&& apt-get install -yq --no-install-recommends groff \
&& rm -rf /var/lib/apt/lists/*
# Create directories
RUN mkdir -p /home/output/ /home/output/csv/ /home/output/pdf/ /home/script/
WORKDIR /home/script
# Install R-packages
COPY /src/install_packages.R /home/script/install_packages.R
RUN Rscript /home/script/install_packages.R
# Copy data
COPY /src/pairs.csv /home/script/pairs.csv
COPY /src/master.R /home/script/master.R
COPY /src/paperboy.ms /home/script/paperboy.ms
# Run the script
RUN ["Rscript", "master.R"]
$ docker run -d
-v $(pwd)/output/:/home/output
-v $(pwd)/output/csv/:/home/output/csv
-v $(pwd)/output/pdf/:/home/output/pdf
$CONTAINER_IMAGE/$DOCKER_IMAGE
5d11eb7e3d93e8b98b6381f1970c25be426ff67abef5e378b715263f174849c9
This is a part from the .gitlab-ci.yml
run:
stage: run
script:
- git remote set-url origin https://$GIT_CI_USER:$GIT_CI_PASS#gitlab.com/$CI_PROJECT_PATH.git
- git config --global user.name ""
- git config --global user.email ""
- git checkout
- docker login registry.gitlab.com --username gitlab+deploy-token-aaaa --password bbbb
- docker pull $CONTAINER_IMAGE/$DOCKER_IMAGE
- docker image ls
- docker run -t -d
-v $(pwd)/output/:/home/output
-v $(pwd)/user/paperboy/output/csv/:/home/output/csv
-v $(pwd)/user/paperboy/output/pdf/:/home/output/pdf
$CONTAINER_IMAGE/$DOCKER_IMAGE
- rm -rf "%CACHE_PATH%/%CI_PIPELINE_ID%"
- pwd
- ls -la
- ls -laR output
- git status
only:
- master
The csv and pdf folder are empty.
$ ls -laR output
output:
total 32
drwxr-xr-x 4 root root 4096 Oct 8 11:37 .
drwxrwxrwx 5 root root 4096 Oct 8 11:37 ..
drwxr-xr-x 2 root root 4096 Oct 8 11:37 csv
drwxr-xr-x 2 root root 4096 Oct 8 11:37 pdf
output/csv:
total 16
drwxr-xr-x 2 root root 4096 Oct 8 11:37 .
drwxr-xr-x 4 root root 4096 Oct 8 11:37 ..
output/pdf:
total 16
drwxr-xr-x 2 root root 4096 Oct 8 11:37 .
drwxr-xr-x 4 root root 4096 Oct 8 11:37 ..

Docker: created files disappear between layers

Running Docker version 17.06.0-ce, build 02c1d87, I have a dockerfile that looks like this:
FROM maven:3.5.2-jdk-8-alpine as builder
RUN chmod -R 777 /root/.m2 &&\
mkdir -p /root/.m2/repository/com/foo/bar &&\
echo "Text" > /root/.m2/repository/com/foo/bar/baz.txt &&\
ls -R -a -l /root/.m2/repository/com/foo
RUN ls -R -a -l /root/.m2/repository/com/foo
The first RUN command successfully creates a file, but the second command can't find it:
Step 1/46 : FROM maven:3.5.2-jdk-8-alpine as builder
---> 293423a981a7
Step 2/46 : RUN chmod -R 777 /root/.m2 && mkdir -p /root/.m2/repository/com/foo/bar && echo "Text" > /root/.m2/repository/com/foo/bar/baz.txt && ls -R -a -l /root/.m2/repository/com/foo
---> Running in a1c0fd142856
/root/.m2/repository/com/foo:
total 12
drwxr-xr-x 3 root root 4096 Nov 30 13:32 .
drwxr-xr-x 3 root root 4096 Nov 30 13:32 ..
drwxr-xr-x 2 root root 4096 Nov 30 13:32 bar
/root/.m2/repository/com/foo/bar:
total 12
drwxr-xr-x 2 root root 4096 Nov 30 13:32 .
drwxr-xr-x 3 root root 4096 Nov 30 13:32 ..
-rw-r--r-- 1 root root 5 Nov 30 13:32 baz.txt
---> b997ccbfd5b0
Step 3/46 : RUN ls -R -a -l /root/.m2/repository/com/foo
---> Running in 603671c87ecc
ls: /root/.m2/repository/com/foo: No such file or directory
The command '/bin/sh -c ls -R -a -l /root/.m2/repository/com/foo' returned a non-zero code: 1
What's going on? (NB. this is a toy example, but there is a real issue in that JARs installed into the Maven repository seem to disappear between layers.)
The upstream maven image defines this directory as a volume. Once an image does this, you cannot reliably make changes to that directory in the image.
From their Dockerfile:
ARG USER_HOME_DIR="/root"
...
VOLUME "$USER_HOME_DIR/.m2"
The Dockerfile documentation describes this behavior:
Changing the volume from within the Dockerfile: If any build steps change the data within the volume after it has been declared, those changes will be discarded.
Your options are to:
Use another directory for your build
Request that the upstream image removes this VOLUME definition
Build your own image without this definition (it's fairly easy to fork their repo and do your own build)
For more details, you can see an old blog post by me about this behavior and the problems it creates.

Resources