In docker compose I use volumes_from to load database container data from another container.
mssql-server-linux:
image: mssql-server-linux:2017-latest
depends_on:
- mssql-server-linux-data
volumes_from:
- mssql-server-linux-data
mssql-server-linux-data:
image: mssql-server-linux-data-keva:latest
entrypoint: /bin/sh
volumes:
- /var/opt/mssql
As a result, the database container uses the following volume to store it's data
$ docker volume ls
DRIVER VOLUME NAME
local e0368e9ae230de6578bb18a3a70823d93ecc4acab7905f96380aff4689024c25
I would like to backup and restore database changes with new named image and volume. How do I do this?
EDIT
I tried to back up the volume in tar with no luck either
$ docker run --rm --volumes-from emma_mssql-server-linux_1 -v /c/temp:/backup ubuntu tar cvf /backup/backup.tar /var/opt/mssql
tar: Cannot connect to C: resolve failed
I am not sure this is the root cause for your failure - but have you tried to perform the rename and backup in separate steps?
Remove the old volume
docker volume rm <old-volume-name>
Create a new volume with the name you want to have
docker volume create --name <new-volume-name>
Backup the newly created volume
docker run --rm --volumes-from <container-name> -v $(pwd):/backup busybox tar cvf /backup/<volume-name>.tar /<volume-path>
please share if this resolves it for you.
Please Note: the assumption is that the volume is local on your system - otherwise, it might be a permissions issue and in that case, you may want to create a local copy of the volume before you back it up:
docker cp <container-name>:/<volume-path>/<volume-name>.tar <local-directory>
Related
Docker keeps creating random volumes instead of using the I specify when running docker run....
I'll start out with no volumes.
$ docker volume ls
DRIVER VOLUME NAME
I'll create one
docker volume create myvol
It'll get created
$ docker volume ls
DRIVER VOLUME NAME
local myvol
I'll start a container using the volume
$ docker run -d \
--name myapp \
--publish 1337:1337 \
--volume myvol:/my-work-dir/.tmp \
foo/bar:tag
I'll go and check my volumes again and I have the one I created and a new one.
$ docker volume ls
DRIVER VOLUME NAME
local 9f7ffe30c24821c8c2cf71b4228a1ec7bc3ad6320c05451e42661a4e3c2c0fb7
local myvol
Why isn't myvol being use? Why is a new volume being created?
This happens when the image you are using defines a VOLUME in the Dockerfile to a container path that you do not define as a volume in your run command. Docker creates the guid for the volume name when you have a volume without a source, aka an anonymous volume. You can use docker image inspect on the image to see the volumes defined in that image. If you inspect the container (docker container inspect), you'll see that your volume is being used, it's just that there's a second anonymous volume to a different path also being used.
How to work with volumes:
To create a volume:
docker volume create my-vol
To use this volume:
docker run -d --name devtest -v my-vol:/app nginx:latest
Create jenkins container and bind volume - jenkins-data
docker run --name myJenkins1 -p 8080:8080 -p 50000:50000 -v jenkins-data:/var/jenkins_home jenkins/jenkins:lts
make changes - update plugins, run builds etc
login to jenkins in browser etc
now export the whole container as a tar
docker export 2c8b996d3088 > jenkinsContainerAndVolume.tar
Since this includes the jenkins image, it seems quite large. I am going to need the jenkins image anyway, but wondered if there is a better practice or standard to save just the volume data?
The docker-export command doesn't save the container's volumes.
To backup the named volume you could use tar like this:
docker run -v jenkins-data:/dbdata -v $(pwd):/backup ubuntu tar zcvf /backup/backup.tar.gz /dbdata
In case you need to migrate this container with all its volumes to another host I use this script:
https://github.com/ricardobranco777/docker-volumes.sh
I was using Docker in the old way, with a volume container:
docker run -d --name jenkins-data jenkins:tag echo "data-only container for Jenkins"
But now I changed to the new way by creating a named volume:
docker volume create --name my-jenkins-volume
I bound this new volume to a new Jenkins container.
The only thing I've left is a folder in which I have the /var/jenkins_home of my previous jenkins container. (by using docker cp)
Now I want to fill my new named volume with the content of that folder.
Can I just copy the content of that folder to /var/lib/jenkins/volume/my-jenkins-volume/_data?
You can certainly copy data directly into /var/lib/docker/volumes/my-jenkins-volume/_data, but by doing this you are:
Relying on physical access to the docker host. This technique won't work if you're interacting with a remote docker api.
Relying on a particular aspect of the volume implementation would could change in the future, breaking any processes you have that rely on it.
I think you are better off relying on things you can accomplish using the docker api, via the command line client. The easiest solution is probably just to use a helper container, something like:
docker run -v my-jenkins-volume:/data --name helper busybox true
docker cp . helper:/data
docker rm helper
You don't need to start some container to add data to already existing named volume, just create a container and copy data there:
docker container create --name temp -v my-jenkins-volume:/data busybox
docker cp . temp:/data
docker rm temp
You can reduce the accepted answer to one line using, e.g.
docker run --rm -v `pwd`:/src -v my-jenkins-volume:/data busybox cp -r /src /data
Here are steps for copying contents of ~/data to docker volume named my-vol
Step 1. Attach the volume to a "temporary" container. For that run in terminal this command :
docker run --rm -it --name alpine --mount type=volume,source=my-vol,target=/data alpine
Step 2. Copy contents of ~/data into my-vol . For that run this commands in new terminal window :
cd ~/data
docker cp . alpine:/data
This will copy contents of ~/data into my-vol volume. After copy exit the temporary container.
You can add this BASH function to your .bashrc to copy files to a existing Docker volume without running a container
# Usage: copy-to-docker-volume SRC_PATH DEST_VOLUME_NAME [DEST_PATH]
copy-to-docker-volume() {
SRC_PATH=$1
DEST_VOLUME_NAME=$2
DEST_PATH="${3:-}"
# create smallest Docker image possible
echo -e 'FROM scratch\nLABEL empty=""' | docker build -t empty -
# create temporary container to be able to mount volume
CONTAINER_ID=$(docker container create -v my-volume:/data empty cmd)
# copy files to volume
docker cp "${SRC_PATH}" "${CONTAINER_ID}":"/data/${DEST_PATH}"
# remove temporary container
docker rm "${CONTAINER_ID}"
}
Example
# create volume as destination
docker volume create my-volume
# create directory to copy
mkdir my-dir
echo "hello file1" > my-dir/my-file-1
# copy directory to volume
copy-to-docker-volume my-dir my-volume
# list directory on volume
docker run --rm -it -v my-volume:/data busybox ls -la /data/my-dir
# show file content on volume
docker run --rm -it -v my-volume:/data busybox cat /data/my-dir/my-file-1
# create another file to copy
echo "hello file2" > my-file-2
# copy file to directory on volume
copy-to-docker-volume my-file-2 my-volume my-dir
# list (updated) directory on volume
docker run --rm -it -v my-volume:/data busybox ls -la /data/my-dir
# check volume content
docker run --rm -it -v my-volume:/data busybox cat /data/my-dir/my-file-2
If you don't want to create a docker and you can access as privileged user to , simply do (on Linux systems):
docker volume create my_named_volume
sudo cp -p . /var/lib/docker/volumes/my_named_volume/_data/
Furthermore, it also allows you to access data in docker runtime or also with docker containers stopped.
If you don't want to create a temp helper container on windows docker desktop (backed by wsl2) then
copy the files to below location
\\wsl$\docker-desktop-data\version-pack-data\community\docker\volumes\my-volume\_data
here my-volume is the name of your named volume. browse the above path from address bar in your file explorer. This is a internal network created by wsl in windows.
Note: it might be better to use docker API like mentioned by larsks, but I have not faced any issues on windows.
Similarly on linux files can be copied to
/var/lib/docker/volumes/my-volume/_data/
I would like to persist some configuration data from a container and am following the tutorial on data volumes.
I'm successfully running the app with:
docker run -it --privileged -v /app/config -p 8083:8083 myapp-ubuntu:2.2.2
Where -v /app/config is the directory inside the container that contains the config that should survive a container restart.
Also the result of running the container creates a volume in /var/lib/docker/volumes.
# ls /var/lib/docker/volumes
5e60d70dc15bcc53aa13cfd84507b5758842c7743d43da2bfa2fc121b2f32479
However, if I kill the container and rerun it no data is persisted and a new volume is created in /var/lib/docker/volumes:
# ls /var/lib/docker/volumes
5e60d70dc15bcc53aa13cfd84507b5758842c7743d43da2bfa2fc121b2f32479 (FIRST RUN)
82de3aa910bc38157a6dc20a516b770bd0264860ae83093d471212f69960d02a (SECOND RUN)
I would expect that these would be the steps for persisting, am I missing something here?
I think you can solve it with named volumes:
docker run -it --privileged -v some_named_volume:/app/config -p 8083:8083 myapp-ubuntu:2.2.2
Or you can use Dockerfile
with directive COPY
I am now learning docker and according to the documentation a shared data volume shall solely be destroyed when the last container holding a link to the shared volume is removed with the -v flag. Nevertheless, in my initial tests this is not the behaviour that I saw.
From the documentation:
Managing Data in Containers
If you remove containers that mount volumes, including the initial dbdata container, or the subsequent containers db1 and db2, the volumes will not be deleted. To delete the volume from disk, you must explicitly call docker rm -v against the last container with a reference to the volume. This allows you to upgrade, or effectively migrate data volumes between containers.
I did the following:
docker run -d -v /dbdata --name dbdata ubuntu:14.04 echo Data-only container for postgres
docker run -d --volumes-from dbdata --name db1 ubuntu:14.04 /bin/bash
Created some files on the /dbdata directory
Exited the db1 container
docker run -d --volumes-from dbdata --name db2 ubuntu:14.04 /bin/bash
I could access the files created on item 3 and create some new files
Exited the db2 container
docker run -d --volumes-from dbdata --name db3 ubuntu:14.04 /bin/bash
I could access the files created on item 3 and 6 and create some new files
Exited the db3 container
Removed all containers without the -v flag
Created the db container again, but the data was not there.
As stated in the user manual:
This allows you to upgrade, or effectively migrate data volumes between containers.
I wonder what I am doing wrong.
You are doing nothing wrong. In step 12, you are creating a new container with the same name. It has a different volume, which initially is empty.
Maybe the following example can illustrate what is happening (ids and paths will/may vary on your system or in other docker versions):
$ docker run -d -v /dbdata --name dbdata ubuntu:14.04 echo Data-only container for postgres
7c23cc1e6637e29f36c6cdd4c1461f6e1742b201e05227279ac3db55328da674
Run a container that has a volume /dbdata and give it the name dbdata. The Id is returned (your Id will be different).
Now lets inspect the container and print the "Volumes" information:
$ docker inspect --format "{{ .Volumes }}" dbdata
map[/dbdata:/var/lib/docker/vfs/dir/248641a5f51a80b5004f72f622a7329835e93881e9915a01b3c7112189d0b55e]
We can see that your /dbdata volume is located at /var/lib/docker/vfs/dir/248641...
Let's create some new data inside the container's volume:
$ docker run --rm --volumes-from dbdata ubuntu:14.04 /bin/bash -c "echo fuu >> /dbdata/test"
And check if it is available
$ docker run --rm --volumes-from dbdata -it ubuntu:14.04 cat /dbdata/test
fuu
Afterwards you delete the containers, without the -v flag.
$ docker rm dbdata
The dbdata container (with id 7c23cc1e6637) is gone, however is still present on your filesystem, as you can see if you inspect the folder:
$ cat /var/lib/docker/vfs/dir/248641a5f51a80b5004f72f622a7329835e93881e9915a01b3c7112189d0b55e/test
fuu
(Please note: if you use the -v flag and delete the container with docker rm -v dbdata the files of the volume on your host filesystem will be deleted and the above cat command would result in a No such file or directory message or similar)
Finally, in step 12. you start a new container with a different volume and give it the same name: dbdata.
docker run -d -v /dbdata --name dbdata ubuntu:14.04 echo Data-only container for postgres
2500731848fd6f2093243da3be064db79e76d731904e6f5349c3f00f054e5f8c
Inspection yields a different volume, which is initially empty.
docker inspect --format "{{ .Volumes }}" dbdata
map[/dbdata:/var/lib/docker/vfs/dir/faffba00358060024026412203a1562125f73d2bdd69a2202483e858dda04740]
If you want to re-use the volume, you have to create a new container and import/restore the data from the filesystem into the data container. In your case, you should not delete the data container in the first place, as you want to reuse the volume from it.