Dockerfile
FROM nginx:1.7
# Deliberately declaring VOLUME before RUN
VOLUME /var/tmp
RUN touch /var/tmp/test.txt
Observation
Now I believe I understand the implications of declaring the VOLUME statement before
creating test.txt - the volume /var/tmp exposed at runtime will be based on the intermediary container prior to the test.txt file is created so it will be empty (hope this observation is correct)
So as expected the following docker run does not show test.txt:
docker run kz/test-volume:0.1
But then I tried supplying the volume at runtime as below:
docker run -v /var/tmp kz/test-volume:0.1
Question
The outcome was the same. So what does this mean? does the -v /var/tmp in the docker run command map to the empty /var/tmp dir exposed by the VOLUME command in the Dockerfile and not the /var/tmp directory with the test.txt in the latest image?
Feeling a tad bit confused.
There is no image which contains /var/tmp/test.txt. The effect of declaring the volume before creating the file is that the RUN instruction runs in a temporary container which has its own volume. Volumes bypass the Union File System so when the build saves that intermediate container, the contents of the volume are not saved, so they don't get persisted in the image layer.
Every container you create from that image will have its own volume, the -v option doesn't change that, unless you use it to map the volume to a host path.
Using your Dockerfile, you can see that by inspecting two containers. The first without the -v option:
> docker run -d temp
c3c4f7de411f166b3a67397ff1221552fe5b94c46bc100725a50a57231da427b
> docker inspect -f '{{ .Mounts }}' c3c
[
{67267d2eeb57373f76a9dd50c25f744b7d99d0a75647bf06aa0d17b70807cf71/var/lib/docker/volumes/67267d2eeb57373f76a9dd50c25f744b7d99d0a75647bf06aa0d17b70807cf71/_data /var/cache/nginx local true }
{91490ea73e3a9d42df9f00e32a91fe571d2143f54248071959765d5d55c23d46/var/lib/docker/volumes/91490ea73e3a9d42df9f00e32a91fe571d2143f54248071959765d5d55c23d46/_data /var/tmp local true }
]
Here there are two volumes, mounted from /var/lib/docker on the host. One is from the nginx base image, and one from your image. With the explicit -v:
> docker run -d -v /var/tmp temp
6fa1a8713b2d6638675a3d048669943419bc7a3924ed98371771100bcfde3954
> docker inspect -f '{{ .Mounts }}' 6fa
[
{9adf6954ed3e826f23a914cbbd768753e6dec1f176eed10e03c2d5503287d101/var/lib/docker/volumes/9adf6954ed3e826f23a914cbbd768753e6dec1f176eed10e03c2d5503287d101/_data /var/cache/nginx local true }
{7dd8be71ce88017ffbeef249837bdd1c96c071b802a2f43b18fd406983e1076a/var/lib/docker/volumes/7dd8be71ce88017ffbeef249837bdd1c96c071b802a2f43b18fd406983e1076a/_data /var/tmp local true }
]
Same outcome, but different host paths, because each container has its own volume.
Related
When using -v switch the files from container should be copied to localhost volume right? But it seems like the directory jenkins_home isn't created at all.
If I create the jenkins_home directory manually and then mount it, the directory is still empty.
I want to preserve the jenkins configs so I could re-run image later.
docker run -p 8080:8080 -p 50000:50000 -d -v jenkins_home:/var/jenkins_home jenkins/jenkins:latest
If you docker run -v jenkins_home:... where the first half of the -v option has no slashes in it at all, that syntax creates a named Docker volume; it isn't a bind mount.
If you docker run -v "$PWD/jenkins_home:..." then that host directory is mounted over the corresponding container directory. At startup time, nothing is ever copied into the host directory; if the host directory is empty, that empty directory gets mounted into the container, hiding everything that was in the image.
If you use the docker run -v named-volume:... syntax, and the named volume is empty, then in this case only, and only the very first time the container is run, the contents of the image are copied into the named volume. This doesn't work for bind mounts, and it doesn't work if there is already data in the volume (perhaps from a previous docker run). It also does not work in other container environments such as Kubernetes. I do not recommend relying on this behavior.
Probably the easiest way to make this work is to launch a one-off container to export the contents of the image, and then use bind-mount syntax:
cd jenkins_home
docker run \
--rm \ # clean up this container when done
-w /var/jenkins_home \ # set the current container directory
jenkins/jenkins \ # the image to run
tar cf - . \ # write a tar file to stdout
| tar xf - # and unpack it on the host
# Now launch the container as normal
docker run -d -p ... -v "$PWD:/var/jenkins_home" jenkins/jenkins
Figured it out.
Turned out that by default it creates the volume in /var/lib/docker/volumes/jenkins_home/ instead of in the current directory.
Also I had tried docker volume create jenkins_home before running the docker image to mount. So not sure if it was the -v jenkins_home:/var/jenkins_home or if it was docker create volume that created the directory in /var/lib/docker/volumes/.
FROM ubuntu:15.04
RUN mkdir -p /app/tina
RUN touch /app/tina/foo.txt
RUN echo "testing tina" > /app/tina/foo.txt
VOLUME /app/tina
CMD sh
As per Docker guide
This Dockerfile results in an image that causes docker run to create a
new mount point at /app/tina and copy the foo.txt file into the newly
created volume
but when I do
docker run --rm -it -v /tmp/foo:/app/tina imagename sh
ls /app/tina/
I can't find foo.txt inside it.
From https://docs.docker.com/engine/reference/builder/#volume
The VOLUME instruction creates a mount point with the specified name and marks it as holding externally mounted volumes from native
host or other containers.
You are using /tmp/foo which is a directory, not a volume. Try:
docker volume create my-vol
docker run --rm -it -v my-vol:/app/tina imagename ls /app/tina/
The problem is that attaching an external directory as a volume using -v actually performs a bind mount: /tmp/foo directory is mounted to the /app/tina directory of the container.
In Linux, when you mount something, all files which were previously seen in the mount point (/app/tina in your case) become invisible. So, when you mount /tmp/foo (empty directory) to /app/tina (which contains foo.txt), the foo.txt file becomes invisible and you see the contents of /tmp/foo in the /app/tina directory, i.e. nothing.
You may ensure that you will see foo.txt in /app/tina when you will unmount tmp/foo from it:
root#84d8cfad500a:/# ls /app/tina
root#84d8cfad500a:/# umount /app/tina
root#84d8cfad500a:/# ls /app/tina
foo.txt
However, this would work only in the privileged (docker run --privileged) container (otherwise you will not be able to unmount /app/tina).
Your files are hidden. This is simply how mounts work. If I were to plug in a flash drive and mount it to ~/someDirectory, then anything in ~/someDirectory would be masked by the files available in the new mount. The volumes feature in docker works the same way.
You can avoid this behavior if you create entrypoint.sh and put these lines into entrypoint
RUN mkdir -p /app/tina
RUN touch /app/tina/foo.txt
RUN echo "testing tina" > /app/tina/foo.txt
when you create container (not image) docker creates volume and after it creates foo.txt and puts "testing tina" to the file.
Of course, don't forget to mention entrypoint in Dockerfile
When running Docker, you can mount files and directories using the --volume option. E.g.:
docker run --volume /remote ./local myimage
I'm running a docker image that defines VOLUMESs in the Dockerfile. I need to access a config file that happens to be inside one of the defined volumes. I'd like to have that file "synced" on the host so that I can edit it. I know I could run docker exec ..., but I hope to circumvent that overhead for only editing one file. I found out that the volumes created by the VOLUMES line are stored in /var/lib/docker/volumes/<HASH>/_data.
Using docker inspect I was able to find the directory that is mounted:
docker inspect gitlab-runner | grep -B 1 '"Destination": "/etc/gitlab-runner"' | head -n 1 | cut -d '"' -f 4
Output:
/var/lib/docker/volumes/9c233c085c36380c6c33035222c16e5d061368c5060cc81dda2a9a713a2b2b3b/_data
So the question is:
Is there a way to re-mount volumes defined in an image? OR to somehow get the directory easier than my oneliner above?
EDIT after comments by zeppelin I've tried rebinding the volume with no success:
$ mkdir etc
$ docker run -d --name test1 gitlab/gitlab-runner
$ docker run -d --name test2 -v ~/etc:/etc/gitlab-runner gitlab/gitlab-runner
$ docker exec test1 ls /etc/gitlab-runner/
certs
config.toml
$ docker exec test2 ls /etc/gitlab-runner/
# empty. no files
$ ls etc
# also empty
docker inspect shows correctly that the volume is bound to ~/etc, but the files inside the container at /etc/gitlab-runner/ seem lost.
$ docker run -d --name test1 gitlab/gitlab-runner
$ docker run -d --name test2 -v ~/etc:/etc/gitlab-runner gitlab/gitlab-runner
You've got two different volume types there. One I call an anonymous volume (a very long uuid visible when you run docker volume ls). The second is a host volume or bind mount that maps a directory on the host directly into the container. So each container you spun up is looking at different places.
Anonymous volumes and named volumes (docker run -d -v mydata:/etc/gitlab-runner gitlab/gitlab-runner) get initialized to the contents of the image at that directory location. This initialization only happens when the volume is empty and is mounted into a new container. Host volumes, as you've seen, only get the contents of the host filesystem, even if it's empty at that location.
With that background, the short answer to your question is no, you cannot mount a file inside the container back out to your host. But you can copy the file out with several methods, assuming you don't overlay the source of the file with a host volume mount. With a running container, there's the docker cp command. Personally, I like:
docker run --rm -v ~/etc:/target gitlab/gitlab-runner \
cp -av /etc/gitlab-runner/. /target/.
If you have a named volume with data you want to copy in or out, you can use any image with the tools you need to do the copy:
docker run --rm -v mydata:/source -v ~/etc:/target busybox \
cp -av /source/. /target/.
Try to avoid modifying data inside a container from the host directly, much nicer is when you wrap your task into another container that you then start with "--volumes-from" option when possible in your case.
Not sure I understood your problem, anyway, as for the documentation you mention,
The VOLUME instruction creates a mount point with the specified name
and marks it as holding externally mounted volumes from native host or
other containers. [...] The docker run command initializes the newly
created volume with any data that exists at the specified location
within the base image.
So, following the example Dockerfile , after having built the image
docker build -t mytest .
and having the container running
docker run -d -ti --name mytestcontainer mytest /bin/bash
you can access it from the container itself, e.g.
docker exec -ti mytestcontainer ls -l /myvol/greeting
docker exec -ti mytestcontainer cat /myvol/greeting
Hope it helps.
when I run nodered with
docker run -v D:/mydir:/data
the content of /data is copied in my volume at first run, thats what I've expected.
If I make
docker run -v D:/mydir:/usr/src/node-red/node_modules nodered
Then the volume is empty
I was expecting to get the content of node_modules being copied in the volume at start time... what am I missing ?
I can illustrate that a little bit more :
docker run --rm -v d:/VM:/data nodered/node-red-docker ls /data
--> list files
docker run --rm ls /usr/src/node-red/node_modules
--> list content of node_modules
docker run --rm -v d:/VM:/usr/src/node-red/node_modules nodered/node-red-docker ls /usr/src/node-red/node_modules
--> is empty !
You're mounting host directories as volumes, so there isn't any copying going on - the mount path inside the container is being mapped to the path on the host, so you're seeing the contents of the host directory.
Volumes sit outside the Union File System when you mount them, so you don't get an overlay which merges the contents of the image and the contents of the host directory. Instead you're effectively bypassing the contents of the image for that volume, and repointing it to your host.
Samples:
touch /docker/nodered-modules/sample.txt
docker run --rm -v /docker/nodered-modules:/usr/src/node-red/node_modules nodered/node-red-docker ls /usr/src/node-red/node_modules
sample.txt
touch /docker/nodered-data/sample.txt
docker run --rm -v /docker/nodered-data:/data nodered/node-red-docker ls /data
sample.txt
The reason you're seeing a difference is because the /data volume is defined in the Dockerfile and empty in the image, so you see the contents of your host directory as expected. The modules directory isn't empty in the image, but you're repointing it to an empty directory on your host.
Docker does not support copying data from the base image into host directories that are mounted as container volumes.
I was using Docker in the old way, with a volume container:
docker run -d --name jenkins-data jenkins:tag echo "data-only container for Jenkins"
But now I changed to the new way by creating a named volume:
docker volume create --name my-jenkins-volume
I bound this new volume to a new Jenkins container.
The only thing I've left is a folder in which I have the /var/jenkins_home of my previous jenkins container. (by using docker cp)
Now I want to fill my new named volume with the content of that folder.
Can I just copy the content of that folder to /var/lib/jenkins/volume/my-jenkins-volume/_data?
You can certainly copy data directly into /var/lib/docker/volumes/my-jenkins-volume/_data, but by doing this you are:
Relying on physical access to the docker host. This technique won't work if you're interacting with a remote docker api.
Relying on a particular aspect of the volume implementation would could change in the future, breaking any processes you have that rely on it.
I think you are better off relying on things you can accomplish using the docker api, via the command line client. The easiest solution is probably just to use a helper container, something like:
docker run -v my-jenkins-volume:/data --name helper busybox true
docker cp . helper:/data
docker rm helper
You don't need to start some container to add data to already existing named volume, just create a container and copy data there:
docker container create --name temp -v my-jenkins-volume:/data busybox
docker cp . temp:/data
docker rm temp
You can reduce the accepted answer to one line using, e.g.
docker run --rm -v `pwd`:/src -v my-jenkins-volume:/data busybox cp -r /src /data
Here are steps for copying contents of ~/data to docker volume named my-vol
Step 1. Attach the volume to a "temporary" container. For that run in terminal this command :
docker run --rm -it --name alpine --mount type=volume,source=my-vol,target=/data alpine
Step 2. Copy contents of ~/data into my-vol . For that run this commands in new terminal window :
cd ~/data
docker cp . alpine:/data
This will copy contents of ~/data into my-vol volume. After copy exit the temporary container.
You can add this BASH function to your .bashrc to copy files to a existing Docker volume without running a container
# Usage: copy-to-docker-volume SRC_PATH DEST_VOLUME_NAME [DEST_PATH]
copy-to-docker-volume() {
SRC_PATH=$1
DEST_VOLUME_NAME=$2
DEST_PATH="${3:-}"
# create smallest Docker image possible
echo -e 'FROM scratch\nLABEL empty=""' | docker build -t empty -
# create temporary container to be able to mount volume
CONTAINER_ID=$(docker container create -v my-volume:/data empty cmd)
# copy files to volume
docker cp "${SRC_PATH}" "${CONTAINER_ID}":"/data/${DEST_PATH}"
# remove temporary container
docker rm "${CONTAINER_ID}"
}
Example
# create volume as destination
docker volume create my-volume
# create directory to copy
mkdir my-dir
echo "hello file1" > my-dir/my-file-1
# copy directory to volume
copy-to-docker-volume my-dir my-volume
# list directory on volume
docker run --rm -it -v my-volume:/data busybox ls -la /data/my-dir
# show file content on volume
docker run --rm -it -v my-volume:/data busybox cat /data/my-dir/my-file-1
# create another file to copy
echo "hello file2" > my-file-2
# copy file to directory on volume
copy-to-docker-volume my-file-2 my-volume my-dir
# list (updated) directory on volume
docker run --rm -it -v my-volume:/data busybox ls -la /data/my-dir
# check volume content
docker run --rm -it -v my-volume:/data busybox cat /data/my-dir/my-file-2
If you don't want to create a docker and you can access as privileged user to , simply do (on Linux systems):
docker volume create my_named_volume
sudo cp -p . /var/lib/docker/volumes/my_named_volume/_data/
Furthermore, it also allows you to access data in docker runtime or also with docker containers stopped.
If you don't want to create a temp helper container on windows docker desktop (backed by wsl2) then
copy the files to below location
\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\my-volume\_data
here my-volume is the name of your named volume. browse the above path from address bar in your file explorer. This is a internal network created by wsl in windows.
Note: it might be better to use docker API like mentioned by larsks, but I have not faced any issues on windows.
Similarly on linux files can be copied to
/var/lib/docker/volumes/my-volume/_data/