Why does docker image content differ from the container created from it? - docker

Following is the Dockerfile for the image,
FROM jenkins/jenkins:lts-jdk11
USER jenkins
RUN jenkins-plugin-cli --plugins "blueocean:1.25.2 http_request" && ls -la /var/jenkins_home
When this is built using docker build -t ireshmm/jenkins:lts-jdk11 ., following is the output,
Sending build context to Docker daemon 3.072kB
Step 1/3 : FROM jenkins/jenkins:lts-jdk11
---> 9aee0d53624f
Step 2/3 : USER jenkins
---> Using cache
---> 49d657d24299
Step 3/3 : RUN jenkins-plugin-cli --plugins "blueocean:1.25.2 http_request" && ls -la /var/jenkins_home
---> Running in b459c4c48e3e
Done
total 20
drwxr-xr-x 3 jenkins jenkins 4096 Jan 22 16:49 .
drwxr-xr-x 1 root root 4096 Jan 12 15:46 ..
drwxr-xr-x 3 jenkins jenkins 4096 Jan 22 16:49 .cache
-rw-rw-r-- 1 jenkins root 7152 Jan 12 15:42 tini_pub.gpg
Removing intermediate container b459c4c48e3e
---> 5fd5ba428f1a
Successfully built 5fd5ba428f1a
Successfully tagged ireshmm/jenkins:lts-jdk11
When create a container and list files docker run -it --rm ireshmm/jenkins:lts-jdk11 ls -la /var/jenkins_home, following is the output:
total 40
drwxr-xr-x 3 jenkins jenkins 4096 Jan 22 16:51 .
drwxr-xr-x 1 root root 4096 Jan 12 15:46 ..
-rw-r--r-- 1 jenkins jenkins 4683 Jan 22 16:51 copy_reference_file.log
drwxr-xr-x 2 jenkins jenkins 16384 Jan 22 16:51 plugins
-rw-rw-r-- 1 jenkins root 7152 Jan 12 15:42 tini_pub.gpg
Question: Why do the contents of /var/jenkins_home differ while building the image and the inside the container created from it given that no command is run after listing the files while building image? How can that happen?

The jenkins/jenkins:lts-jdk11 has an ENTRYPOINT that runs /usr/local/bin/jenkins.sh, which among other things creates the copy_reference_file.log file:
$ grep -i copy_reference /usr/local/bin/jenkins.sh
: "${COPY_REFERENCE_FILE_LOG:="${JENKINS_HOME}/copy_reference_file.log"}"
touch "${COPY_REFERENCE_FILE_LOG}" || { echo "Can not write to ${COPY_REFERENCE_FILE_LOG}. Wrong volume permissions?"; exit 1; }
echo "--- Copying files at $(date)" >> "$COPY_REFERENCE_FILE_LOG"
find "${REF}" \( -type f -o -type l \) -exec bash -c '. /usr/local/bin/jenkins-support; for arg; do copy_reference_file "$arg"; done' _ {} +
The ENTRYPOINT scripts runs whenever you start a container from that image (before any command you've provided on the command line).

Related

Bug or misuse : docker layer content not available in final image

I am having a hard time trying to build a docker image for some python project, but at some point a RUN command resulting content is not available in the next RUN.
I mean I have a RUN command that generate a directory with some content and that directory is not available for the next RUN command.
Here is the Dockerfile snippet :
RUN poetry config virtualenvs.create true && \
poetry config virtualenvs.in-project true && \
poetry install -n --no-root --no-dev && \
ls -la
RUN ls -la
The first ls -la shows a .venv directory (see below) while it is not present anymore in the second one.
And here is the build output:
Step 9/14 : RUN poetry config virtualenvs.create true && poetry config virtualenvs.in-project true && poetry install -n --no-root --no-dev && ls -la
---> Running in 3a93a4994133
...
drwxrwxr-x 4 root odoo 4096 Sep 29 20:08 .
drwxr-xr-x 1 root root 4096 Sep 16 02:35 ..
drwxr-xr-x 4 odoo odoo 4096 Sep 29 20:08 .venv
-rw-rw-r-- 1 odoo odoo 15824 Sep 2 23:18 poetry.lock
-rw-rw-r-- 1 odoo odoo 2057 Sep 2 23:50 pyproject.toml
drwxr-xr-x 3 odoo odoo 4096 Sep 29 20:08 src
Removing intermediate container 3a93a4994133
---> 694c4e54863c
Step 10/14 : RUN ls -la
---> Running in 5d0ebcf8f1e9
total 32
drwxrwxr-x 3 root odoo 4096 Sep 29 20:08 .
drwxr-xr-x 1 root root 4096 Sep 16 02:35 ..
-rw-rw-r-- 1 odoo odoo 15824 Sep 2 23:18 poetry.lock
-rw-rw-r-- 1 odoo odoo 2057 Sep 2 23:50 pyproject.toml
drwxr-xr-x 3 odoo odoo 4096 Sep 29 20:08 src
Removing intermediate container 5d0ebcf8f1e9
---> 750255eac6bb
If someone can provide any lead, it would be really appreciated.
Thanks

Using the same Docker image file permissions differ from machine to machine

I have a problem, that I cannot grasp at all. I'm running my Jenkins pipeline in a Docker container on the master node. Now I added another node and want to run the pipeline there as well.
However, using the same image I get different file permissions in the container:
### master
> docker image ls node:10.20.1-stretch
REPOSITORY TAG IMAGE ID CREATED SIZE
node 10.20.1-stretch c5f1efe092a0 13 days ago 912MB
> docker run --rm -ti -u 1000:1000 node:10.20.1-stretch ls -la /home/node
total 20
drwxr-xr-x 2 1000 1000 4096 May 15 20:31 .
drwxr-xr-x 3 0 0 4096 May 15 20:31 ..
-rw-r--r-- 1 1000 1000 220 May 15 2017 .bash_logout
-rw-r--r-- 1 1000 1000 3526 May 15 2017 .bashrc
-rw-r--r-- 1 1000 1000 675 May 15 2017 .profile
### node 1
> docker image ls node:10.20.1-stretch
REPOSITORY TAG IMAGE ID CREATED SIZE
node 10.20.1-stretch c5f1efe092a0 13 days ago 912MB
> docker run --rm -ti -u 1000:1000 node:10.20.1-stretch ls -la /home/node
total 20
drwxr-xr-x 2 0 0 4096 May 26 05:42 .
drwxr-xr-x 1 0 0 4096 May 26 05:42 ..
-rw-r--r-- 1 0 0 220 May 26 05:42 .bash_logout
-rw-r--r-- 1 0 0 3526 May 26 05:42 .bashrc
-rw-r--r-- 1 0 0 675 May 26 05:42 .profile
I observed a similar behavior for the /tmp directory, which has chmod 1777 on master and 1755 on node 1.
# master
> docker -v
Docker version 19.03.9, build 9d988398e7
> dockerd -v
Docker version 19.03.9, build 9d988398e7
# node 1
> docker -v
Docker version 19.03.10, build 9424aeaee9
> dockerd -v
Docker version 19.03.10, build 9424aeaee9
I assume the wrong behavior is on node 1, as the /home/node directory and all of its children are owned by root:root there, but the same directory is owned by node:node on the master. However, I already upgraded the Docker version on node 1 from 19.03.8 to 19.03.10 and nothing changed.
It there anything I don't understand about Docker containers? I have been working with them for a while, but never observed such a behavior.
I have change the storage driver from overlay2 to aufs. Now I have the correct permissions.

Docker: created files disappear between layers

Running Docker version 17.06.0-ce, build 02c1d87, I have a dockerfile that looks like this:
FROM maven:3.5.2-jdk-8-alpine as builder
RUN chmod -R 777 /root/.m2 &&\
mkdir -p /root/.m2/repository/com/foo/bar &&\
echo "Text" > /root/.m2/repository/com/foo/bar/baz.txt &&\
ls -R -a -l /root/.m2/repository/com/foo
RUN ls -R -a -l /root/.m2/repository/com/foo
The first RUN command successfully creates a file, but the second command can't find it:
Step 1/46 : FROM maven:3.5.2-jdk-8-alpine as builder
---> 293423a981a7
Step 2/46 : RUN chmod -R 777 /root/.m2 && mkdir -p /root/.m2/repository/com/foo/bar && echo "Text" > /root/.m2/repository/com/foo/bar/baz.txt && ls -R -a -l /root/.m2/repository/com/foo
---> Running in a1c0fd142856
/root/.m2/repository/com/foo:
total 12
drwxr-xr-x 3 root root 4096 Nov 30 13:32 .
drwxr-xr-x 3 root root 4096 Nov 30 13:32 ..
drwxr-xr-x 2 root root 4096 Nov 30 13:32 bar
/root/.m2/repository/com/foo/bar:
total 12
drwxr-xr-x 2 root root 4096 Nov 30 13:32 .
drwxr-xr-x 3 root root 4096 Nov 30 13:32 ..
-rw-r--r-- 1 root root 5 Nov 30 13:32 baz.txt
---> b997ccbfd5b0
Step 3/46 : RUN ls -R -a -l /root/.m2/repository/com/foo
---> Running in 603671c87ecc
ls: /root/.m2/repository/com/foo: No such file or directory
The command '/bin/sh -c ls -R -a -l /root/.m2/repository/com/foo' returned a non-zero code: 1
What's going on? (NB. this is a toy example, but there is a real issue in that JARs installed into the Maven repository seem to disappear between layers.)
The upstream maven image defines this directory as a volume. Once an image does this, you cannot reliably make changes to that directory in the image.
From their Dockerfile:
ARG USER_HOME_DIR="/root"
...
VOLUME "$USER_HOME_DIR/.m2"
The Dockerfile documentation describes this behavior:
Changing the volume from within the Dockerfile: If any build steps change the data within the volume after it has been declared, those changes will be discarded.
Your options are to:
Use another directory for your build
Request that the upstream image removes this VOLUME definition
Build your own image without this definition (it's fairly easy to fork their repo and do your own build)
For more details, you can see an old blog post by me about this behavior and the problems it creates.

mkdir .ssh in a Dockerfile, folder is not there?

I want my Dockerfile to mkdir .ssh/
But it does not, why not?
FROM jenkinsci/jnlp-slave
MAINTAINER Johnny5 isAlive <johnny5#hotmail.com>
USER root
RUN rm /bin/sh && ln -s /bin/bash /bin/sh
RUN apt-get update
RUN apt-get install unzip git curl vim -y
USER jenkins
RUN mkdir -p /home/jenkins/.ssh && touch /home/jenkins/.ssh/aFile
...building...
Looks fine?
Step 12 : RUN mkdir -p /home/jenkins/.ssh && touch /home/jenkins/.ssh/aFile
---> Running in ca19a679580d
---> 5980df7db482
Removing intermediate container ca19a679580d
Successfully built 5980df7db482
Running and looking around, the .ssh/ folder and aFile inside are not there ...
$ docker run -it -u 0 --entrypoint /bin/bash 5980df7db482
root#4aa40a18baf2:~# pwd
/home/jenkins
root#4aa40a18baf2:~# ls -al
total 24
drwxr-xr-x 3 jenkins jenkins 4096 Oct 17 23:17 .
drwxr-xr-x 4 root root 4096 Sep 14 08:50 ..
-rw-r--r-- 1 jenkins jenkins 220 Nov 12 2014 .bash_logout
-rw-r--r-- 1 jenkins jenkins 3515 Nov 12 2014 .bashrc
-rw-r--r-- 1 jenkins jenkins 675 Nov 12 2014 .profile
drwxr-xr-x 2 jenkins jenkins 4096 Sep 14 08:50 .tmp
root#4aa40a18baf2:~#
If I pull the parent image, jenkinsci/jnlp-slave, and inspect it with docker inspect jenkinsci/jnlp-slave, I can see that it already has a volume defined at /home/jenkins:
[
{
...
"ContainerConfig": {
...
"Volumes": {
"/home/jenkins": {}
},
...
}
]
This means that during each build step, any changes you make to that location won't be committed to your new layer.
Here's a simplified version of your Dockerfile to highlight what's going on:
FROM jenkinsci/jnlp-slave
RUN mkdir -p /home/jenkins/.ssh
Now, let's build with the following command: docker build --no-cache --rm=false -t jns .:
Sending build context to Docker daemon 2.56 kB
Step 1 : FROM jenkinsci/jnlp-slave
---> d7731d944ad7
Step 2 : RUN mkdir -p /home/jenkins/.ssh
---> Running in 520a8e2f7cae
---> 962189878d5e
Successfully built 962189878d5e
The --no-cache option makes the command easier to work with on repeat invocations. The --rm=false will cause the builder to not remove the containers created for each step.
In this case, the builder ran the Step 2 in 520a8e2f7cae on my system. I can now do a docker inspect 520a8e2f7cae and see the actual container used for this step. Specifically, I'm curious about the mounts location:
[
{
...
"Mounts": [
{
"Name": "e34fd82bd190f21dbd63b5cf70167a16674cd00d95fdc6159314c25c6d08e10e",
"Source": "/var/lib/docker/volumes/e34fd82bd190f21dbd63b5cf70167a16674cd00d95fdc6159314c25c6d08e10e/_data",
"Destination": "/home/jenkins",
"Driver": "local",
"Mode": "",
"RW": true,
"Propagation": ""
}
],
...
}
]
I see that there's an anonymous volume with id e34fd82bd190f21dbd63b5cf70167a16674cd00d95fdc6159314c25c6d08e10e for /home/jenkins.
I can inspect the contents of that volume like this:
$ docker run --rm -v e34fd82bd190f21dbd63b5cf70167a16674cd00d95fdc6159314c25c6d08e10e:/volume alpine ls -lah /volume
total 28
drwxr-xr-x 4 10000 10000 4.0K Oct 18 02:49 .
drwxr-xr-x 25 root root 4.0K Oct 18 02:55 ..
-rw-r--r-- 1 10000 10000 220 Nov 12 2014 .bash_logout
-rw-r--r-- 1 10000 10000 3.4K Nov 12 2014 .bashrc
-rw-r--r-- 1 10000 10000 675 Nov 12 2014 .profile
drwxr-xr-x 2 10000 10000 4.0K Oct 18 02:49 .ssh
drwxr-xr-x 2 10000 10000 4.0K Sep 14 08:50 .tmp
The .ssh directory created in the RUN step is in this volume. Since volumes aren't part of the container's write layer, it won't get committed. I can confirm this by doing a docker diff on this container:
docker diff 520a8e2f7cae
There is no output, showing no changes to the container's filesystem, which is why it doesn't come forward into this layer of the image.
The other contents at this location are files in the parent image that were committed before the VOLUME directive that made /home/jenkins into a volume.

How to remove data from docker volume?

I prepared Docker image containing libraries needed for building my other project. I want to have a directory /myLibs with libraries from different projects, e.g:
/myLibs:
projectA
projectB
projectC
Dockerfile:
FROM my-base:1.0
VOLUME /myLibs
COPY projectA/bin/*.so* /myLibs/projectA/bin/
CMD /bin/bash
Built:
docker build -t my-libs:1.0 .
Then I want to update libs in this image every time when I rebuild projectA. So I prepared Dockerfile:
FROM my-libs:1.0 # I changed parent image, because /myLibs/projectB and /myLibs/projectC should remain
VOLUME /myLibs
RUN ls -al /myLibs && rm -rf /myLibs/projectA && ls -al /myLibs
RUN ls -al /myLibs
COPY projectA/bin/*.so* /myLibs/projectA/bin/
CMD /bin/bash
As a result I have old projectA libs in my volume:
Step 4 : RUN ls -al /myLibs && rm -rf /myLibs/projectA && ls -al /myLibs
---> Running in 1e3e25084e69
total 12
drwxr-xr-x 3 root root 4096 Jul 16 13:52 .
drwxr-xr-x 75 root root 4096 Jul 16 13:52 ..
drwxr-xr-x 4 root root 4096 Jul 16 13:51 projectA
total 8
drwxr-xr-x 2 root root 4096 Jul 16 13:52 .
drwxr-xr-x 75 root root 4096 Jul 16 13:52 ..
---> d5973da5965c
Removing intermediate container 1e3e25084e69
Step 5 : RUN ls -al /myLibs
---> Running in 1d93575b50c2
total 12
drwxr-xr-x 3 root root 4096 Jul 16 13:52 .
drwxr-xr-x 75 root root 4096 Jul 16 13:52 ..
drwxr-xr-x 4 root root 4096 Jul 16 13:51 projectA
---> 6d2a48a5b67b
How can I remove files from volume?
If you want to change the files on rebuild, you probably don't want to do it in the volume. The volume is generally for data you want to persist. Remember the volume mounting will occur after the container builds, so what's probably happening is the volume with the old data is mounting over any changes you are making in the image (re)build.
What are you using /myLibs for? If they are read-only files you want to set up in the build, you might be better off not using a volume and make them part of the image. If you want to modify them, it's probably better to manage that after the build - there is no real reason to rebuild the image if you are just changing files in a networked volume.

Resources