I am trying to create a wrapper container to build and run a set of containers using a docker-compose I cannot modify. The docker-compose mounts several volumes, but when starting the docker-compose from inside of the wrapper docker, the volumes are still mounted from the host since the docker .sock is volume mounted to be the host's docker.sock.
I would like to not have to use full docker-in-docker due to all the problems associated with it outlined in jpetazzo's article.
I would also like to avoid volume-from since I cannot edit the docker-compose file mentioned previously.
Is there a way to get this snippet to correctly use the parent docker's file instead of going to the host filesystem and mounting it from there?
FROM docker:latest
RUN mkdir -p /tmp/parent/ && echo "This is from the parent docker" > /tmp/parent/parent.txt
CMD docker run -v /tmp/parent/parent.txt:/root/parent.txt --rm ubuntu:18.04 bash -c "cat /root/parent.txt"
when run with a command akin to this:
docker build -t parent . && docker run --rm -v /var/run/docker.sock:/var/run/docker.sock parent
Make your paths the same on the host and inside of the docker image, e.g.
docker run -v /var/run/docker.sock:/var/run/docker.sock \
-v /home/user:/home/user -w /home/user/project parent_image ...
By mounting the volume as /home/user in the same location inside the image, a command like docker-compose up with relative bind mounts will use the container path names when talking to the docker socket, which will match the paths on the host.
I am looking at this example
docker run --rm --volumes-from myredis -v $(pwd)/backup:/backup debian cp /data/dump.rdb /backup/
from Using Docker book.
Why do we need --rm flag?
Why do we have --volumes-from?
The idea here is that
you have a redis container named myredis which has some volumes for persistent storage (that you'd like to backup).
you run a temporary debian container that will save the backup to your_current_dir/backup and get removed.
docker run --rm ... debian runs the container and removes it after it exits
--volumes-from myredis this way the debian container will have access to the database
-v $(pwd)/backup:/backup this second volume is used to put the backup at your current dir $(pwd)/backup. If it wasn't used, the backup would have only been copied to /backup (inside the container) and later been removed together with the container. This way the backup persists.
cp /data/dump.rdb /backup/ copies the actual files
The --rm flag tells Docker Engine to remove the container once it exits. Without this flag, you need to manually remove the container after you stop it.
The --volumes-from flag mounts all the defined volumes from the referenced containers, it ensures the two containers mounts same volumes.
Please bear with me as I learn my way around docker. I'm using v1.11.1
I am making a Dockerfile and would like to specify that a folder of the container should be persisted, this should only be persisted per user (computer running the container). I originally thought that including:
VOLUME /path/to/dir/to/persist
would be enough, but when I start my container with docker run -t -i myimage:latest bash and manually add files in then exit I expect to be able to find my files again. But when I run the image again (as per above) the added files are no longer there.
I've read around but answers seem either outdated in regards to the use of VOLUMES, or suggest things I would rather not do, which is:
I don't want to use -v in the run command
I would rather not make a volume container (seems like overkill for my one tiny folder)
What is it that I'm doing wrong? Any help would be greatly appreciated.
Cheers guys.
Update: I can persist data using a named volume ie: docker run -v name:/path/to/persist -t -i myimage:latest bash But building with a Dockerfile that contains VOLUME name:/path/to/persist does not work.
What is not very obvious is that you are creating a brand new container every time you do a "docker run". Each new container would then have a fresh volume.
So your data is being persisted, but you're not reading the data from the container you wrote it to.
Example to illustrate the problem
Sample Dockerfile
FROM ubuntu
VOLUME /data
built as normal
$ docker build . -t myimage
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM ubuntu
---> bd3d4369aebc
Step 2 : VOLUME /data
---> Running in db84d80841de
---> 7c94335543b8
Now run it twice
$ docker run -ti myimage echo hello world
$ docker run -ti myimage echo hello world
And take a look at the volumes
$ docker volume ls
DRIVER VOLUME NAME
local 078820609d31f814cd5704cf419c3f579af30672411c476c4972a4aad3a3916c
local cad0604d02467a02f2148a77992b1429bb655dba8137351d392b77a25f30192b
The "docker rm" command has a special "-v" option that will cleanup any volumes associated with containers.
$ docker rm -v $(docker ps -qa)
How to use a data container
Using the same docker image, built in the previous example create a container whose sole purpose is to persist data via it's volume
$ docker create --name mydata myimage
Launch another container that saves some data into the "/data" volume
$ docker run -it --rm --volumes-from mydata myimage bash
root#a1227abdc212:/# echo hello world > /data/helloworld.txt
root#a1227abdc212:/# exit
Launch a second container that retrieves the data
$ docker run -it --rm --volumes-from mydata myimage cat /data/helloworld.txt
hello world
Cleanup, simply remove the container and specify the "-v" option to ensure its volume is cleaned up.
$ docker rm -v mydata
Notes:
The "volumes-from" parameter means all data is saved into the underlying volume associated with the "mydata" container
When running the containers the "rm" option will ensure they are automatically removed, useful for once-off containers.
I have been trying to setup a graph database using orientdb. So I tried using volumes by the following command
docker run -d -p 2424:2424 -p 2480:2480 -v config:/orientdb/config -v database:/orientdb/databases -v backup:/orientdb/backup -e ORIENTDB_ROOT_PASSWORD=mypasswdhere orientdb:latest
My prime motive behind using volumes was to store data in database after I kill the container.
But I used this command frequently to start the server.
Now it has hogged my disk space so I guess it creates a new copy each time this command is executed.
Can someone indicate a correct way to use existing volumes to use stored data in docker and to clean up the redundant data recreated by frequent execution of this command?
You can create named volumes with docker volume create
$ docker volume create --name hello
$ docker run -d -v hello:/world busybox ls /world
That way, only one volume in /var/lib/docker/volumes will be used each time you launch that container.
See also "Mount a shared-storage volume as a data volume".
In the meantime, to remove dangling volumes:
docker volume ls -qf "dangling=true" | xargs docker volume rm
As far as I understand, you aren't re-using the container, instead you start a new one each time.
After the first run, you can stop and the restart it with docker stop/start commands.
My question is related to this question on copying files from containers to hosts; I have a Dockerfile that fetches dependencies, compiles a build artifact from source, and runs an executable. I also want to copy the build artifact (in my case it's a .zip produced by sbt dist in '../target/`, but I think this question also applies to jars, binaries, etc.
docker cp works on containers, not images; do I need to start a container just to get a file out of it? In a script, I tried running /bin/bash in interactive mode in the background, copying the file out, and then killing the container, but this seems kludgey. Is there a better way?
On the other hand, I would like to avoid unpacking a .tar file after running docker save $IMAGENAME just to get one file out (but that seems like the simplest, if slowest, option right now).
I would use docker volumes, e.g.:
docker run -v hostdir:out $IMAGENAME /bin/cp/../blah.zip /out
but I'm running boot2docker in OSX and I don't know how to directly write to my mac host filesystem (read-write volumes are mounting inside my boot2docker VM, which means I can't easily share a script to extract blah.zip from an image with others. Thoughts?
To copy a file from an image, create a temporary container, copy the file from it and then delete it:
id=$(docker create image-name)
docker cp $id:path - > local-tar-file
docker rm -v $id
Unfortunately there doesn't seem to be a way to copy files directly from Docker images. You need to create a container first and then copy the file from the container.
However, if your image contains a cat command (and it will do in many cases), you can do it with a single command:
docker run --rm --entrypoint cat yourimage /path/to/file > path/to/destination
If your image doesn't contain cat, simply create a container and use the docker cp command as suggested in Igor's answer.
docker cp $(docker create --name tc registry.example.com/ansible-base:latest):/home/ansible/.ssh/id_rsa ./hacked_ssh_key && docker rm tc
wanted to supply a one line solution based on pure docker functionality (no bash needed)
edit: container does not even has to be run in this solution
edit2: thanks to #Jonathan Dumaine for --rm so the container will be removed after, i just never tried, because it sounded illogical to copy something from somewhere which has been already removed by the previous command, but i tried it and it works
edit3: due the comments we found out --rm is not working as expected, it does not remove the container because it never runs, so I added functionality to delete the created container afterwards(--name tc=temporary-container)
edit 4: this error appeared, seems like a bug in docker, because t is in a-z and this did not happen a few months before.
Error response from daemon: Invalid container name (t), only [a-zA-Z0-9][a-zA-Z0-9_.-] are allowed
A much faster option is to copy the file from running container to a mounted volume:
docker run -v $PWD:/opt/mount --rm --entrypoint cp image:version /data/libraries.tgz /opt/mount/libraries.tgz
real 0m0.446s
** VS **
docker run --rm --entrypoint cat image:version /data/libraries.tgz > libraries.tgz
real 0m9.014s
Parent comment already showed how to use cat. You could also use tar in a similar fashion:
docker run yourimage tar -c -C /my/directory subfolder | tar x
Another (short) answer to this problem:
docker run -v $PWD:/opt/mount --rm -ti image:version bash -c "cp /source/file /opt/mount/"
Update - as noted by #Elytscha Smith this only works if your image has bash built in
Not a direct answer to the question details, but in general, once you pulled an image, the image is stored on your system and so are all its files. Depending on the storage driver of the local Docker installation, these files can usually be found in /var/lib/docker/overlay2 (requires root access). overlay2 should be the most common storage driver nowadays, but the path may differ.
The layers associated with an image can be found using $ docker inspect image IMAGE_NAME:TAG, look for a GraphDriver attribute.
At least in my local environment, the following also works to quickly see all layers associated with an image:
docker inspect image IMAGE_NAME:TAG | jq ".[0].GraphDriver.Data"
In one of these diff directories, the wanted file can be found.
So in theory, there's no need to create a temporary container. Ofc this solution is pretty inconvenient.
First pull docker image using docker pull
docker pull <IMG>:<TAG>
Then, create a container using docker create command and store the container id is a variable
img_id=$(docker create <IMG>:<TAG>)
Now, run the docker cp command to copy folders and files from docker container to host
docker cp $img_id:/path/in/container /path/in/host
Once the files/folders are moved, delete the container using docker rm
docker rm -v $img_id
You essentially had the best solution already. Have the container copy out the files for you, and then remove itself when it's complete.
This will copy the files from /inside/container/ to your machine at /path/to/hostdir/.
docker run --rm -v /path/to/hostdir:/mnt/out "$IMAGENAME" /bin/cp -r /inside/container/ /mnt/out/
Update - here's a better version without the tar file:
$id = & docker create image-name
docker cp ${id}:path .
docker rm -v $id
Old answer
PowerShell variant of Igor Bukanov's answer:
$id = & docker create image-name
docker cp ${id}:path - > local-file.tar
docker rm -v $id
I am using boot2docker on MacOS. I can assure you that scripts based on "docker cp" are portable. Because any command is relayed inside boot2docker but then the binary stream is relayed back to the docker command line client running on your mac. So write operations from the docker client are executed inside the server and written back to the executing client instance!
I am sharing a backup script for docker volumes with any docker container I provide and my backup scripts are tested both on linux and MacOS with boot2docker. The backups can be easily exchanged between platforms. Basically I am executing the following command inside my script:
docker run --name=bckp_for_volume --rm --volumes-from jenkins_jenkins_1 -v /Users/github/jenkins/backups:/backup busybox tar cf /backup/JenkinsBackup-2015-07-09-14-26-15.tar /jenkins
Runs a new busybox container and mounts the volume of my jenkins container with the name jenkins_jenkins_1. The whole volume is written to the file backups/JenkinsBackup-2015-07-09-14-26-15.tar
I have already moved archives between the linux container and my mac container without any adjustments to the backup or restore script. If this is what you want you find the whole script an tutorial here: blacklabelops/jenkins
You could bind a local path on the host to a path on the container, and then cp the desired file(s) to that path at the end of your script.
$ docker run -d \
-it \
--name devtest \
--mount type=bind,source="$(pwd)"/target,target=/app \
nginx:latest
Then there is no need to copy afterwards.