Can I create a "template volume" in docker? - docker

I have multiple docker containers, and all of them must have a common content in a subdirectory. This common content is pretty much standard, so I would enjoy being able to package it somehow.
One option would be to create a volume, put the files in the volume, and then bind the containers to the volume, however from what I understand of docker volumes, the result would be that the volume is shared among containers. Any changes a container does to the volume's content, will appear in the other container. I don't want this.
Note that keeping it readonly is not an option. I want the data to be read-write, I simply don't want it to be shared among containers, and at the same time I don't want to keep them in the image.
Is this possible in Docker?

As long as you don't re-use the same volume for other containers, you can use a docker image as template, and use it to "propagate" the volume data;
1. Create a directory for all your template files;
mkdir data
# just creating some dummy files here to illustrate the concept
touch data/foo data/bar data/baz data/bla data/bla2
2. Create a Dockerfile for building the template image
This image contains default data to be used for the containers to used
We're using a tiny image ("hello-world") as it requires a command to
allow a container to be created from it
FROM hello-world
COPY . /data/
3. Build the template image
docker build -t template-data .
4. Create a new volume, and propagate the data
Then, you can create volume, create a container from the image, and
attach the volume to it. The first time the volume is used and is still
empty, the files are copied from the container to the volume.
After the volume is created, and propagated, we don't really need the
container anymore (the data is copied to the volume), so we're passing the
--rm flag as well, so that the container (not the volume, because it's a
"named" volume) is removed directly after it exits
# create an empty volume
docker volume create --name data-volume1
# start the container (which copies the data), and remove the container
docker run -it --rm -v data-volume1:/data template-data
5. Use the volume for your application
Then start your application container, and attach the volume (which now
contains the template data) to it.
For this example, I just start an alpine container and show the contents
of the volume, but normally this would be your application;
docker run --rm -v data-volume1:/somewhere alpine ls -l /somewhere
And you can see the data is there;
docker run --rm -v data-volume1:/somewhere alpine ls -l /somewhere
total 0
-rw-r--r-- 1 root root 0 Jun 2 20:14 bar
-rw-r--r-- 1 root root 0 Jun 2 20:14 baz
-rw-r--r-- 1 root root 0 Jun 2 20:14 bla
-rw-r--r-- 1 root root 0 Jun 2 20:14 bla2
-rw-r--r-- 1 root root 0 Jun 2 20:14 foo
You can do this multiple times, but you need to create a new volume
for each project/application, otherwise they share the same volume,
so are working on the same data;
docker volume create --name data-volume2
docker volume create --name data-volume3
docker volume create --name data-volume4
docker run -it --rm -v data-volume2:/data template-data
docker run -it --rm -v data-volume3:/data template-data
docker run -it --rm -v data-volume4:/data template-data
docker run --rm -v data-volume2:/somewhere alpine ls -l /somewhere
docker run --rm -v data-volume3:/somewhere alpine ls -l /somewhere
docker run --rm -v data-volume4:/somewhere alpine ls -l /somewhere
Hope this helps!

Related

Docker container file getting lost after I stop the container [duplicate]

This question already has answers here:
I lose my data when the container exits
(11 answers)
Closed 3 years ago.
I pulled Ubuntu image using docker pull.
I connect to the container using docker exec and then create a file and then exit.
Again, when I execute docker exec file is lost.
How to maintain the file in that container, I have tried dockerfile and tagging docker images, it works.
But, is there any other way to maintain the files in docker container for a longer time?
One option is to commit your changes. After you've added the file, and while the container is still running, you should run:
docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]
Another option, maybe you'll want to use a volume, but that depends on your logic and needs.
The best way to persist content in containers its with Docker Volumes:
╭─exadra37#exadra37-Vostro-470 ~/Developer/DevNull/stackoverflow
╰─➤ sudo docker run --rm -it -v $PWD:/data ubuntu
root#00af7ccf1d3b:/# echo "Persits data with Docker Volumes" > /data/docker-volumes.txt
root#00af7ccf1d3b:/# cat /data/docker-volumes.txt
Persits data with Docker Volumes
root#00af7ccf1d3b:/# exit
╭─exadra37#exadra37-Vostro-470 ~/Developer/DevNull/stackoverflow
╰─➤ ls -al
total 12
drwxr-xr-x 2 exadra37 exadra37 4096 Nov 25 15:34 .
drwxr-xr-x 8 exadra37 exadra37 4096 Nov 25 15:33 ..
-rw-r--r-- 1 root root 33 Nov 25 15:34 docker-volumes.txt
╭─exadra37#exadra37-Vostro-470 ~/Developer/DevNull/stackoverflow
╰─➤ cat docker-volumes.txt
Persits data with Docker Volumes
The docker command explained:
sudo docker run --rm -it -v $PWD:/data alpine
I used the flag -v to map the current dir $PWD to the /data dir inside the container
inside the container:
I wrote some content to it
I read that same content
I exited the container
On the host:
I used ls -al to confirm that the file was persisted to my computer.
I confirmed could access that same file in my computer filesystem.

Docker: trying to understand VOLUME in Dockerfile

First create a volume sample_vol
docker volume create sample_vol
My Dockerfile
FROM archlinux/base
RUN touch /root/testing [**edited** find note at RUN below]
# VOLUME sample_vol:/root [**edited** this will not work, because VOLUME will not accpet named volumes. So this will not mount at /root, it will mount at sample_vol:/root which does not exist]
VOLUME "/root" or VOLUME ["/root"] [**edited** this will create a local mount volume only till the time the container is running. I tried to use named volumes like VOLUME ["name:/root"] but didnt work ]
# RUN touch /root/testing [**edited** this will not work because volume when mounted will only copy files till it got declared]
build the image
docker build -t archlinux/sample_vol .
checking whether testing file is created in sample_vol
docker run --rm -it -v=sample_vol:/tmp/myvolume archlinux/base ls /tmp/myvolume
It does not show any file testing created
while
$ docker run --rm -it --name sample_vol archlinux/sample_vol ls /root/testing
It shows the file testing is created in the /root/ of build image
So why sample_vol is not mounted at /root and testing is created inside it.
Update: Reason i found can be due to
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#volume
Changing the volume from within the Dockerfile: If any build steps
change the data within the volume after it has been declared, those
changes will be discarded.
You are misunderstanding docker-volume.
Docker-Image are more about build time.
Docker-Volume is useful only in runtime.
Try running following commands to get an idea:
docker run --rm -it -v=sample_vol:/tmp/myvolume archlinux/base touch /tmp/myvolume/1.txt
docker run --rm -it -v=sample_vol:/tmp/myvolume archlinux/base touch /tmp/myvolume/2.txt
docker run --rm -it -v=sample_vol:/tmp/myvolume archlinux/base touch /tmp/myvolume/3.txt
docker run --rm -it -v=sample_vol:/tmp/myvolume archlinux/base ls -altr /tmp/myvolume/
1st container create a file 1.txt in docker volume mounted at /tmp/myvolume and then container gets deleted after this operation.
2nd container create a file 2.txt in docker volume mounted at /tmp/myvolume and then container gets deleted after this operation.
3rd container create a file 3.txt in docker volume mounted at /tmp/myvolume and then container gets deleted after this operation.
4th container list files in docker volume mounted at /tmp/myvolume and then container gets deleted after this operation.
Docker volume is to store persistent data outside of the lifecycle of container.That means when you remove container , you still have data outside of the container living inside volume.
So next time if you create a container and attach that docker volume - you will automatically get all the data with new container.
Consider an example of database image where you want to have data in volume so that when you change the container to the higher version - you will get the old data in the new database.

Docker compose reusing volumes

I'm trying to create a new Docker image that no longer uses volumes from a running container that does use images. The volumes were created using docker-compose file, not Dockerfile. The problem is, when I launch a new container via new docker-compose.yml file it still has the volumes mapped. I still need to keep these volumes and the original containers/images that use them. Also, if possible I would like to continue to use the same docker image, just add a new version, or :latest. Here's the steps I used:
New version of an existing image:
docker commit <image id> existingImage:new-version
Create a new image from current running container:
docker commit <Image ID> newimage
Create new docker-compose.yml with no volumes defined and run docker-compose with a different project name
docker-compose -p <new project name>
Running without docker-compose, just use docker run:
docker run -d -p 8093:80 <img>:<version>
Any time I run any combination of these the volumes are still mapped from the original image. So my question is, how to I create a container from an image that once had mapped volumes but I no longer want to use the volumes?
Edit:
Additional things I've tried:
Stop container, remove container, restart docker, run docker compose again. No luck.
Edit 2:
Decided to start over on the image. Using a base image, launched a container with an updated docker compose file that uses the now unrelated image. Run docker-compose -f up -d -> STILL has these same volumes mapped even though the image does not (and never has) any volumes mapped, and the current docker-compose.yml file does not map files. It looks like docker-compose caches what volumes are mapped for projects.
After searching for caching options in docker-compose, I came across this article: How to get docker-compose to always re-create containers from fresh images?
which seems to solve the problem of caching images but not containers caching volumes
According to another SO post what I am trying to do is not possible. For future reference, one cannot attach volumes to an image, and then later decide to remove them. A new image must be created without the volumes instead. Reference:
How to remove configure volumes in docker images
To remove volumes along with the containers used by docker-compose, use docker-compose down -v.
To start containers with docker-compose, leave your existing volumes intact, but not use those volumes, you should change your project name. You can use docker-compose -p new_project_name up -d for that.
Edit: here's an example showing how docker-compose does not reuse named volumes between different projects, but it does reuse and persist the volume unless you do a down -v:
$ docker-compose -p proj1 -f docker-compose.vol-named.yml up -d
Creating network "proj1_default" with the default driver
Creating volume "proj1_data" with default driver
Creating proj1_test_1 ...
Creating proj1_test_1 ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
71f2eb516f71 busybox "tail -f /dev/null" 5 seconds ago Up 2 seconds proj1_test_1
$ docker exec -it 71f /bin/sh
/ # ls /data
/ # echo "Hello proj1" >/data/data.txt
/ # exit
Volume is now populated, lets stop and start a new container to show it persist:
$ docker-compose -p proj1 -f docker-compose.vol-named.yml down
Stopping proj1_test_1 ... done
Removing proj1_test_1 ... done
Removing network proj1_default
$ docker-compose -p proj1 -f docker-compose.vol-named.yml up -d
Creating network "proj1_default" with the default driver
Creating proj1_test_1 ...
Creating proj1_test_1 ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
311900fd3d27 busybox "tail -f /dev/null" 5 seconds ago Up 3 seconds proj1_test_1
$ docker exec -it 311 /bin/sh
/ # cat /data/data.txt
Hello proj1
/ # exit
There's the expected persistent volume, lets run a different project at the same time to show the volume would be independent:
$ docker-compose -p proj2 -f docker-compose.vol-named.yml up -d
Creating network "proj2_default" with the default driver
Creating volume "proj2_data" with default driver
Creating proj2_test_1 ...
Creating proj2_test_1 ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d39e6fc51436 busybox "tail -f /dev/null" 4 seconds ago Up 2 seconds proj2_test_1
311900fd3d27 busybox "tail -f /dev/null" 33 seconds ago Up 32 seconds proj1_test_1
$ docker exec -it d39 /bin/sh
/ # ls -al /data
total 8
drwxr-xr-x 2 root root 4096 Nov 6 19:56 .
drwxr-xr-x 1 root root 4096 Nov 6 19:56 ..
/ # exit
The volume is completely empty in the new project. Let's cleanup.
$ docker-compose -p proj2 -f docker-compose.vol-named.yml down -v
Stopping proj2_test_1 ...
Stopping proj2_test_1 ... done
Removing proj2_test_1 ... done
Removing network proj2_default
Removing volume proj2_data
$ docker volume ls
DRIVER VOLUME NAME
local proj1_data
Note the volume is there in proj1 from before.
$ docker-compose -p proj1 -f docker-compose.vol-named.yml down -v
Stopping proj1_test_1 ... done
Removing proj1_test_1 ... done
Removing network proj1_default
Removing volume proj1_data
$ docker volume ls
DRIVER VOLUME NAME
But doing a down -v deletes the volume.

How to re-mount a docker volume without overriding existing files?

When running Docker, you can mount files and directories using the --volume option. E.g.:
docker run --volume /remote ./local myimage
I'm running a docker image that defines VOLUMESs in the Dockerfile. I need to access a config file that happens to be inside one of the defined volumes. I'd like to have that file "synced" on the host so that I can edit it. I know I could run docker exec ..., but I hope to circumvent that overhead for only editing one file. I found out that the volumes created by the VOLUMES line are stored in /var/lib/docker/volumes/<HASH>/_data.
Using docker inspect I was able to find the directory that is mounted:
docker inspect gitlab-runner | grep -B 1 '"Destination": "/etc/gitlab-runner"' | head -n 1 | cut -d '"' -f 4
Output:
/var/lib/docker/volumes/9c233c085c36380c6c33035222c16e5d061368c5060cc81dda2a9a713a2b2b3b/_data
So the question is:
Is there a way to re-mount volumes defined in an image? OR to somehow get the directory easier than my oneliner above?
EDIT after comments by zeppelin I've tried rebinding the volume with no success:
$ mkdir etc
$ docker run -d --name test1 gitlab/gitlab-runner
$ docker run -d --name test2 -v ~/etc:/etc/gitlab-runner gitlab/gitlab-runner
$ docker exec test1 ls /etc/gitlab-runner/
certs
config.toml
$ docker exec test2 ls /etc/gitlab-runner/
# empty. no files
$ ls etc
# also empty
docker inspect shows correctly that the volume is bound to ~/etc, but the files inside the container at /etc/gitlab-runner/ seem lost.
$ docker run -d --name test1 gitlab/gitlab-runner
$ docker run -d --name test2 -v ~/etc:/etc/gitlab-runner gitlab/gitlab-runner
You've got two different volume types there. One I call an anonymous volume (a very long uuid visible when you run docker volume ls). The second is a host volume or bind mount that maps a directory on the host directly into the container. So each container you spun up is looking at different places.
Anonymous volumes and named volumes (docker run -d -v mydata:/etc/gitlab-runner gitlab/gitlab-runner) get initialized to the contents of the image at that directory location. This initialization only happens when the volume is empty and is mounted into a new container. Host volumes, as you've seen, only get the contents of the host filesystem, even if it's empty at that location.
With that background, the short answer to your question is no, you cannot mount a file inside the container back out to your host. But you can copy the file out with several methods, assuming you don't overlay the source of the file with a host volume mount. With a running container, there's the docker cp command. Personally, I like:
docker run --rm -v ~/etc:/target gitlab/gitlab-runner \
cp -av /etc/gitlab-runner/. /target/.
If you have a named volume with data you want to copy in or out, you can use any image with the tools you need to do the copy:
docker run --rm -v mydata:/source -v ~/etc:/target busybox \
cp -av /source/. /target/.
Try to avoid modifying data inside a container from the host directly, much nicer is when you wrap your task into another container that you then start with "--volumes-from" option when possible in your case.
Not sure I understood your problem, anyway, as for the documentation you mention,
The VOLUME instruction creates a mount point with the specified name
and marks it as holding externally mounted volumes from native host or
other containers. [...] The docker run command initializes the newly
created volume with any data that exists at the specified location
within the base image.
So, following the example Dockerfile , after having built the image
docker build -t mytest .
and having the container running
docker run -d -ti --name mytestcontainer mytest /bin/bash
you can access it from the container itself, e.g.
docker exec -ti mytestcontainer ls -l /myvol/greeting
docker exec -ti mytestcontainer cat /myvol/greeting
Hope it helps.

What is the right way to add data to an existing named volume in Docker?

I was using Docker in the old way, with a volume container:
docker run -d --name jenkins-data jenkins:tag echo "data-only container for Jenkins"
But now I changed to the new way by creating a named volume:
docker volume create --name my-jenkins-volume
I bound this new volume to a new Jenkins container.
The only thing I've left is a folder in which I have the /var/jenkins_home of my previous jenkins container. (by using docker cp)
Now I want to fill my new named volume with the content of that folder.
Can I just copy the content of that folder to /var/lib/jenkins/volume/my-jenkins-volume/_data?
You can certainly copy data directly into /var/lib/docker/volumes/my-jenkins-volume/_data, but by doing this you are:
Relying on physical access to the docker host. This technique won't work if you're interacting with a remote docker api.
Relying on a particular aspect of the volume implementation would could change in the future, breaking any processes you have that rely on it.
I think you are better off relying on things you can accomplish using the docker api, via the command line client. The easiest solution is probably just to use a helper container, something like:
docker run -v my-jenkins-volume:/data --name helper busybox true
docker cp . helper:/data
docker rm helper
You don't need to start some container to add data to already existing named volume, just create a container and copy data there:
docker container create --name temp -v my-jenkins-volume:/data busybox
docker cp . temp:/data
docker rm temp
You can reduce the accepted answer to one line using, e.g.
docker run --rm -v `pwd`:/src -v my-jenkins-volume:/data busybox cp -r /src /data
Here are steps for copying contents of ~/data to docker volume named my-vol
Step 1. Attach the volume to a "temporary" container. For that run in terminal this command :
docker run --rm -it --name alpine --mount type=volume,source=my-vol,target=/data alpine
Step 2. Copy contents of ~/data into my-vol . For that run this commands in new terminal window :
cd ~/data
docker cp . alpine:/data
This will copy contents of ~/data into my-vol volume. After copy exit the temporary container.
You can add this BASH function to your .bashrc to copy files to a existing Docker volume without running a container
# Usage: copy-to-docker-volume SRC_PATH DEST_VOLUME_NAME [DEST_PATH]
copy-to-docker-volume() {
SRC_PATH=$1
DEST_VOLUME_NAME=$2
DEST_PATH="${3:-}"
# create smallest Docker image possible
echo -e 'FROM scratch\nLABEL empty=""' | docker build -t empty -
# create temporary container to be able to mount volume
CONTAINER_ID=$(docker container create -v my-volume:/data empty cmd)
# copy files to volume
docker cp "${SRC_PATH}" "${CONTAINER_ID}":"/data/${DEST_PATH}"
# remove temporary container
docker rm "${CONTAINER_ID}"
}
Example
# create volume as destination
docker volume create my-volume
# create directory to copy
mkdir my-dir
echo "hello file1" > my-dir/my-file-1
# copy directory to volume
copy-to-docker-volume my-dir my-volume
# list directory on volume
docker run --rm -it -v my-volume:/data busybox ls -la /data/my-dir
# show file content on volume
docker run --rm -it -v my-volume:/data busybox cat /data/my-dir/my-file-1
# create another file to copy
echo "hello file2" > my-file-2
# copy file to directory on volume
copy-to-docker-volume my-file-2 my-volume my-dir
# list (updated) directory on volume
docker run --rm -it -v my-volume:/data busybox ls -la /data/my-dir
# check volume content
docker run --rm -it -v my-volume:/data busybox cat /data/my-dir/my-file-2
If you don't want to create a docker and you can access as privileged user to , simply do (on Linux systems):
docker volume create my_named_volume
sudo cp -p . /var/lib/docker/volumes/my_named_volume/_data/
Furthermore, it also allows you to access data in docker runtime or also with docker containers stopped.
If you don't want to create a temp helper container on windows docker desktop (backed by wsl2) then
copy the files to below location
\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\my-volume\_data
here my-volume is the name of your named volume. browse the above path from address bar in your file explorer. This is a internal network created by wsl in windows.
Note: it might be better to use docker API like mentioned by larsks, but I have not faced any issues on windows.
Similarly on linux files can be copied to
/var/lib/docker/volumes/my-volume/_data/

Resources