I want to make a backup of all my containers and volumes, so the easiest way would be to copy /var/lib/docker to another location.
However this directory also includes all the images, and I don't want to include them since they all can easily be re-downloaded from public sources.
So how can I copy this directory while excluding the images?
You have to differentiate between container backup and vol backup:
Backing up a container, that is, its configurations like labels, envs, etc.
You do that by committing the container as an image:
$ docker container commit <container-name/id> <name-of-new-image>
Better give it also some metainfo:
$ docker container ... -m "very important container config state" -a "John Doe"
Backing up a volume
Let's say the volume of interest <my-vol> is bound to a container <other-container> - which may have been created like: docker container run -v /my-vol other-container ...
So first you have to bind the volume also to a newly created temporary container with the --volumes-from flag. With the -v option you mount a local path (of the host) into the container:
$ docker container run -rm --volumes-from <other-container> \
-v <dir/on/host>:<mountpath/in/container> \
<ubuntu/centos/whatever-base-image> tar cvf <mountpath/in/container>/backup.tar /<my-vol>
After completing the command the container stops and with that it will also be deleted because of the -rm option.
Whith all that the steps are:
bind the volume to a temp container
mount a hostpath into the container
make a tarbal (or whatever kind of backup)
of the volume in the container
container stops and is deleted after the backup command has finished
the backup tarbal is left on the mounted dir of the container host.
see also: https://docs.docker.com/storage/volumes/
Shell Command
.. the other - not recommended - way would be to do it just with os level commands:
shopt -s extglob
cp -r var/lib/docker/!(image) your/path/backup
For that you have to stop all involved containers to prevent read/write issues.
Related
When using -v switch the files from container should be copied to localhost volume right? But it seems like the directory jenkins_home isn't created at all.
If I create the jenkins_home directory manually and then mount it, the directory is still empty.
I want to preserve the jenkins configs so I could re-run image later.
docker run -p 8080:8080 -p 50000:50000 -d -v jenkins_home:/var/jenkins_home jenkins/jenkins:latest
If you docker run -v jenkins_home:... where the first half of the -v option has no slashes in it at all, that syntax creates a named Docker volume; it isn't a bind mount.
If you docker run -v "$PWD/jenkins_home:..." then that host directory is mounted over the corresponding container directory. At startup time, nothing is ever copied into the host directory; if the host directory is empty, that empty directory gets mounted into the container, hiding everything that was in the image.
If you use the docker run -v named-volume:... syntax, and the named volume is empty, then in this case only, and only the very first time the container is run, the contents of the image are copied into the named volume. This doesn't work for bind mounts, and it doesn't work if there is already data in the volume (perhaps from a previous docker run). It also does not work in other container environments such as Kubernetes. I do not recommend relying on this behavior.
Probably the easiest way to make this work is to launch a one-off container to export the contents of the image, and then use bind-mount syntax:
cd jenkins_home
docker run \
--rm \ # clean up this container when done
-w /var/jenkins_home \ # set the current container directory
jenkins/jenkins \ # the image to run
tar cf - . \ # write a tar file to stdout
| tar xf - # and unpack it on the host
# Now launch the container as normal
docker run -d -p ... -v "$PWD:/var/jenkins_home" jenkins/jenkins
Figured it out.
Turned out that by default it creates the volume in /var/lib/docker/volumes/jenkins_home/ instead of in the current directory.
Also I had tried docker volume create jenkins_home before running the docker image to mount. So not sure if it was the -v jenkins_home:/var/jenkins_home or if it was docker create volume that created the directory in /var/lib/docker/volumes/.
I am trying to export a docker container that uses a mounted local volume as its root, which I run with docker run --privileged -v /path/to/local/files:/root --name cse303dev -it cse303 to mount that local directory.
What is the best way of exporting the container AND all of the contents of that local mounted directory into a simple tar file of some sort? And then how would I easily re-import this container and run it so that I can see and use all of those files copied from the local machine that exported it? Is this possible?
You almost never "export" a container per se. Containers are generally intended to be freely destroyed and recreated.
In your case you already have the data you care about stored outside the container (in a bind-mounted folder, which is the easy case), so you can just copy that directory tree to wherever else and then run a new copy of the container there.
(cd /path/to/local/files; tar cvzf ~/local-files.tar.gz .)
scp local-files.tar.gz there:
ssh there
mkdir files
(cd files; tar xvzf ../local-files.tar.gz)
docker run -v $PWD/files:/root cse303
This is trickier if you're storing the data in a named volume. The Docker documentation describes how to back up the contents of a named volume and you'd have to go through that procedure.
If you want to export the entire contents of the container (including the mounted volumes, which might be a bad idea depending upon what you have mounted), then you want to run tar inside the container and pipe out the data to a file:
docker run --privileged -v /path/to/local/files:/root ${IMAGE_NAME} \
tar -cf - -C / --exclude=proc --exclude=sys . | gzip > myfile.tgz
I would highly recommend excluding /proc and /sys (as in the above example) or you will likely encounter issues.
I have Jenkins running in a Docker container. The home directory is in a host volume, in order to ensure that the build history is preserved when updates to the container are actioned.
I have updated the container, to create an additional file in the home directory. When the new container is pulled, I cannot see the changed file.
ENV JENKINS_HOME=/var/jenkins_home
RUN mkdir -p ${JENKINS_HOME}/.m2
COPY settings.xml ${JENKINS_HOME}/.m2/settings.xml
RUN chown -R jenkins:jenkins ${JENKINS_HOME}/.m2
VOLUME ["/var/jenkins_home"]
I am running the container like this:
docker run -v /host/directory:/var/jenkins_home -p 80:8080 jenkins
I had previous run Jenkins and so the home directory already exists on the host. When I pull the new container and run it, I see that the file .m2/settings.xml is not created. Why is this please?
Basically when you run:
docker run -v /host-src-dir:/container-dest-dir my_image
You will overlay your /container-dest-dir with what is in /host-src-dir
From Docs
$ docker run -d -P --name web -v /src/webapp:/webapp training/webapp python app.py
This command mounts the host directory, /src/webapp, into the
container at /webapp. If the path /webapp already exists inside the
container’s image, the /src/webapp mount overlays but does not remove
the pre-existing content. Once the mount is removed, the content is
accessible again. This is consistent with the expected behavior of the
mount command.
This SO question is also relevant docker mounting volumes on host
It seems you want it the other way around (i.e. the container is source and the host is destination).
Here is a workaround:
Create the volume in your Dockerfile
Run it without -v i.e.: docker run --name=my_container my_image
Run docker inspect --format='{{json .Mounts}}' my_container
This will give you output similar to:
[{"Name":"5e2d41896b9b1b0d7bc0b4ad6dfe3f926c73","Source":"/var/lib/docker/volumes/5e2d41896b9b1b0d7bc0b4ad6dfe3f926c73/_data","Destination":"/var/jenkins_home","Driver":"local","Mode":"","RW":true,"Propagation":""}]
Which means your dir as it is on container was mounted into the host directory /var/lib/docker/volumes/5e2d41896b9b1b0d7bc0b4ad6dfe3f926c73/_data
Unfortunately, I do not know a way to make it mount on a specific host directory instead.
I tried to share data between the docker container and the host, for example by adding the parameter -v /Users/name/Desktop/Tutorials:/cntk/Tutorials to the docker run command, but I noticed that it also deletes all the files on the docker contained in /cntk/Tutorials.
My question is how to make the same link, but having instead all the files in /cntk/Tutorials copied to the host (at /Users/name/Desktop/Tutorials)
Thank you
Unfortunately that it is not possible, take a look here. That is because this is how mounting works in Linux.
It is not correct to say that the files were deleted. They are still present in the underlying image, but the act of mounting another directory at the same path has obscured them. They exist, but are not accessible in this condition.
One way you can accomplish this is by mounting a volume into your container at a different path, and then copying the container's files to that path. Something like this.
Mount a host volume using a different path than the one the container already has for the files you are interested in.
docker run -v /Users/name/Desktop/Tutorials:/cntk/Tutorials2 [...]
Now, execute a command that will copy the files already in the docker image, into the mounted volume from the outside host.
docker exec <container-id> cp -r /cntk/Tutorials /cntk/Tutorials2
The docker cp command allows you to copy files/folders on demand between host and the container:
docker cp [OPTIONS] CONTAINER:SRC_PATH DEST_PATH|-
docker cp [OPTIONS] SRC_PATH|- CONTAINER:DEST_PATH
docker cp ContainerName:/home/data.txt . <== copy from container to host
docker cp ./test.txt ContainerName:/test.txt <== copy from host to container
docker cp ContainerName:/test.txt ./test2.txt <== copy from container to host
For details run docker cp --help
I am wondering if I can map the volume in the docker to another folder in my linux host. The reason why I want to do this is, if I don't misunderstand, the default mapping folder is in /var/lib/docker/... and I don't have the access to this folder. So I am thinking about changing that to a host's folder I have access (for example /tmp/) when I create the image. I'm now able to modify Dockerfile if this can be done before creating the image. Or must this be done after creating the image or after creating the container?
I found this article which helps me to use a local directory as the volume in docker.
https://docs.docker.com/engine/userguide/containers/dockervolumes/
Command I use while creating a new container:
docker run -d -P -name randomname -v /tmp/localfolder:/volumepath imageName
Docker doesn't have any tools I know of to map named or container volumes back to the host, though they are just sub directories under /var/lib/docker so writing your own tool wouldn't be impossible, but you'd need root access to run it. Note that with access to docker on the host, there are likely a lot of ways to access root privileges on the host. Creating a hard link to the target folder should be all that's needed if both source and target are on the same file system.
The docker way to access the named volume would be to create a disposable container to access your files. You can even create an additional host volume to export the data. E.g.
docker run -it --rm \
-v test:/source -v `pwd`/data:/target \
busybox /bin/sh -c "tar -cC /source . | tar -xC /target"'
Where "test" is the named volume you want to export/copy. You may need to also run a chown -R $uid /target in a container to change everything to your uid on the host.