I'm trying to create a new Docker image that no longer uses volumes from a running container that does use images. The volumes were created using docker-compose file, not Dockerfile. The problem is, when I launch a new container via new docker-compose.yml file it still has the volumes mapped. I still need to keep these volumes and the original containers/images that use them. Also, if possible I would like to continue to use the same docker image, just add a new version, or :latest. Here's the steps I used:
New version of an existing image:
docker commit <image id> existingImage:new-version
Create a new image from current running container:
docker commit <Image ID> newimage
Create new docker-compose.yml with no volumes defined and run docker-compose with a different project name
docker-compose -p <new project name>
Running without docker-compose, just use docker run:
docker run -d -p 8093:80 <img>:<version>
Any time I run any combination of these the volumes are still mapped from the original image. So my question is, how to I create a container from an image that once had mapped volumes but I no longer want to use the volumes?
Edit:
Additional things I've tried:
Stop container, remove container, restart docker, run docker compose again. No luck.
Edit 2:
Decided to start over on the image. Using a base image, launched a container with an updated docker compose file that uses the now unrelated image. Run docker-compose -f up -d -> STILL has these same volumes mapped even though the image does not (and never has) any volumes mapped, and the current docker-compose.yml file does not map files. It looks like docker-compose caches what volumes are mapped for projects.
After searching for caching options in docker-compose, I came across this article: How to get docker-compose to always re-create containers from fresh images?
which seems to solve the problem of caching images but not containers caching volumes
According to another SO post what I am trying to do is not possible. For future reference, one cannot attach volumes to an image, and then later decide to remove them. A new image must be created without the volumes instead. Reference:
How to remove configure volumes in docker images
To remove volumes along with the containers used by docker-compose, use docker-compose down -v.
To start containers with docker-compose, leave your existing volumes intact, but not use those volumes, you should change your project name. You can use docker-compose -p new_project_name up -d for that.
Edit: here's an example showing how docker-compose does not reuse named volumes between different projects, but it does reuse and persist the volume unless you do a down -v:
$ docker-compose -p proj1 -f docker-compose.vol-named.yml up -d
Creating network "proj1_default" with the default driver
Creating volume "proj1_data" with default driver
Creating proj1_test_1 ...
Creating proj1_test_1 ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
71f2eb516f71 busybox "tail -f /dev/null" 5 seconds ago Up 2 seconds proj1_test_1
$ docker exec -it 71f /bin/sh
/ # ls /data
/ # echo "Hello proj1" >/data/data.txt
/ # exit
Volume is now populated, lets stop and start a new container to show it persist:
$ docker-compose -p proj1 -f docker-compose.vol-named.yml down
Stopping proj1_test_1 ... done
Removing proj1_test_1 ... done
Removing network proj1_default
$ docker-compose -p proj1 -f docker-compose.vol-named.yml up -d
Creating network "proj1_default" with the default driver
Creating proj1_test_1 ...
Creating proj1_test_1 ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
311900fd3d27 busybox "tail -f /dev/null" 5 seconds ago Up 3 seconds proj1_test_1
$ docker exec -it 311 /bin/sh
/ # cat /data/data.txt
Hello proj1
/ # exit
There's the expected persistent volume, lets run a different project at the same time to show the volume would be independent:
$ docker-compose -p proj2 -f docker-compose.vol-named.yml up -d
Creating network "proj2_default" with the default driver
Creating volume "proj2_data" with default driver
Creating proj2_test_1 ...
Creating proj2_test_1 ... done
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d39e6fc51436 busybox "tail -f /dev/null" 4 seconds ago Up 2 seconds proj2_test_1
311900fd3d27 busybox "tail -f /dev/null" 33 seconds ago Up 32 seconds proj1_test_1
$ docker exec -it d39 /bin/sh
/ # ls -al /data
total 8
drwxr-xr-x 2 root root 4096 Nov 6 19:56 .
drwxr-xr-x 1 root root 4096 Nov 6 19:56 ..
/ # exit
The volume is completely empty in the new project. Let's cleanup.
$ docker-compose -p proj2 -f docker-compose.vol-named.yml down -v
Stopping proj2_test_1 ...
Stopping proj2_test_1 ... done
Removing proj2_test_1 ... done
Removing network proj2_default
Removing volume proj2_data
$ docker volume ls
DRIVER VOLUME NAME
local proj1_data
Note the volume is there in proj1 from before.
$ docker-compose -p proj1 -f docker-compose.vol-named.yml down -v
Stopping proj1_test_1 ... done
Removing proj1_test_1 ... done
Removing network proj1_default
Removing volume proj1_data
$ docker volume ls
DRIVER VOLUME NAME
But doing a down -v deletes the volume.
Related
I have installed Docker and have running some Ubuntu image with command:
sudo docker run ubuntu
I would like to create some text file on it and find it next time the same image will run. How to achieve that?
UPD.
Got problems with attaching to docker.
I have running docker
docker ps -a
aef01293fdc9 ubuntu "/bin/bash" 6 hours ago Up 6 hours priceless_ramanujan
Since it is Up mode, I suppose I don't need to execute command:
docker start priceless_ramanujan
So, I run command attach
docker attach priceless_ramanujan
And got nothing in output while command not returns.
Why I can't get to container's bash?
Simple example:
$ docker run -it ubuntu
root#4d5643e8c1a8:/# echo "test" > test.txt
root#4d5643e8c1a8:/# cat test.txt
test
root#4d5643e8c1a8:/# exit
exit
$ docker run -it ubuntu
root#cdb44750bffc:/# cat test.txt
cat: test.txt: No such file or directory
root#cdb44750bffc:/#
docker run image_name
This command creates and starts a new container based on the provided image_name. If a name is not set for the container, a random one is generated and assigned by docker. In the above example 2 containers were created based on ubuntu.
with docker ps -a we can see that modest_jennings and optimistic_leakey are the random names created:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cdb44750bffc ubuntu "/bin/bash" About a minute ago Exited (1) 4 seconds ago optimistic_leakey
4d5643e8c1a8 ubuntu "/bin/bash" 2 minutes ago Exited (0) 2 minutes ago modest_jennings
cat test.txt failed the 2nd time because the file didn't exist. The container started from a "clean" ubuntu image.
Actually, we created test.txt inside modest_jennings only.
docker start container_name
This command starts a stopped container. So, in our case, the file is still there:
$ docker start modest_jennings
modest_jennings
$ docker attach modest_jennings
root#4d5643e8c1a8:/# cat test.txt
test
root#4d5643e8c1a8:/#
docker commit container_name image_name
This command is to create a new image, so that you can use it later and run containers based on that image. Continuing our example...
$ docker commit modest_jennings my_ubuntu
sha256:a4357f37153ac0b94e37315595f1a3b540538283adc3721df4d4e3b39bf8334f
$ docker run -it my_ubuntu
root#2e38616d532a:/# cat test.txt
test
root#2e38616d532a:/#
If you want a custom image, you can create a Dockerfile
`FROM ubuntu:16.04
ADD ./test.txt /tmp/`
after you can build it docker build -t ubuntu:custom .
and finally run your custom image docker run --name myubuntu ubuntu:custom sleep 3000
You can check your file with docker exec -it myubuntu /bin/bash and more /tmp/test.txt
When running Docker, you can mount files and directories using the --volume option. E.g.:
docker run --volume /remote ./local myimage
I'm running a docker image that defines VOLUMESs in the Dockerfile. I need to access a config file that happens to be inside one of the defined volumes. I'd like to have that file "synced" on the host so that I can edit it. I know I could run docker exec ..., but I hope to circumvent that overhead for only editing one file. I found out that the volumes created by the VOLUMES line are stored in /var/lib/docker/volumes/<HASH>/_data.
Using docker inspect I was able to find the directory that is mounted:
docker inspect gitlab-runner | grep -B 1 '"Destination": "/etc/gitlab-runner"' | head -n 1 | cut -d '"' -f 4
Output:
/var/lib/docker/volumes/9c233c085c36380c6c33035222c16e5d061368c5060cc81dda2a9a713a2b2b3b/_data
So the question is:
Is there a way to re-mount volumes defined in an image? OR to somehow get the directory easier than my oneliner above?
EDIT after comments by zeppelin I've tried rebinding the volume with no success:
$ mkdir etc
$ docker run -d --name test1 gitlab/gitlab-runner
$ docker run -d --name test2 -v ~/etc:/etc/gitlab-runner gitlab/gitlab-runner
$ docker exec test1 ls /etc/gitlab-runner/
certs
config.toml
$ docker exec test2 ls /etc/gitlab-runner/
# empty. no files
$ ls etc
# also empty
docker inspect shows correctly that the volume is bound to ~/etc, but the files inside the container at /etc/gitlab-runner/ seem lost.
$ docker run -d --name test1 gitlab/gitlab-runner
$ docker run -d --name test2 -v ~/etc:/etc/gitlab-runner gitlab/gitlab-runner
You've got two different volume types there. One I call an anonymous volume (a very long uuid visible when you run docker volume ls). The second is a host volume or bind mount that maps a directory on the host directly into the container. So each container you spun up is looking at different places.
Anonymous volumes and named volumes (docker run -d -v mydata:/etc/gitlab-runner gitlab/gitlab-runner) get initialized to the contents of the image at that directory location. This initialization only happens when the volume is empty and is mounted into a new container. Host volumes, as you've seen, only get the contents of the host filesystem, even if it's empty at that location.
With that background, the short answer to your question is no, you cannot mount a file inside the container back out to your host. But you can copy the file out with several methods, assuming you don't overlay the source of the file with a host volume mount. With a running container, there's the docker cp command. Personally, I like:
docker run --rm -v ~/etc:/target gitlab/gitlab-runner \
cp -av /etc/gitlab-runner/. /target/.
If you have a named volume with data you want to copy in or out, you can use any image with the tools you need to do the copy:
docker run --rm -v mydata:/source -v ~/etc:/target busybox \
cp -av /source/. /target/.
Try to avoid modifying data inside a container from the host directly, much nicer is when you wrap your task into another container that you then start with "--volumes-from" option when possible in your case.
Not sure I understood your problem, anyway, as for the documentation you mention,
The VOLUME instruction creates a mount point with the specified name
and marks it as holding externally mounted volumes from native host or
other containers. [...] The docker run command initializes the newly
created volume with any data that exists at the specified location
within the base image.
So, following the example Dockerfile , after having built the image
docker build -t mytest .
and having the container running
docker run -d -ti --name mytestcontainer mytest /bin/bash
you can access it from the container itself, e.g.
docker exec -ti mytestcontainer ls -l /myvol/greeting
docker exec -ti mytestcontainer cat /myvol/greeting
Hope it helps.
I have been trying to setup a graph database using orientdb. So I tried using volumes by the following command
docker run -d -p 2424:2424 -p 2480:2480 -v config:/orientdb/config -v database:/orientdb/databases -v backup:/orientdb/backup -e ORIENTDB_ROOT_PASSWORD=mypasswdhere orientdb:latest
My prime motive behind using volumes was to store data in database after I kill the container.
But I used this command frequently to start the server.
Now it has hogged my disk space so I guess it creates a new copy each time this command is executed.
Can someone indicate a correct way to use existing volumes to use stored data in docker and to clean up the redundant data recreated by frequent execution of this command?
You can create named volumes with docker volume create
$ docker volume create --name hello
$ docker run -d -v hello:/world busybox ls /world
That way, only one volume in /var/lib/docker/volumes will be used each time you launch that container.
See also "Mount a shared-storage volume as a data volume".
In the meantime, to remove dangling volumes:
docker volume ls -qf "dangling=true" | xargs docker volume rm
As far as I understand, you aren't re-using the container, instead you start a new one each time.
After the first run, you can stop and the restart it with docker stop/start commands.
I have multiple docker containers, and all of them must have a common content in a subdirectory. This common content is pretty much standard, so I would enjoy being able to package it somehow.
One option would be to create a volume, put the files in the volume, and then bind the containers to the volume, however from what I understand of docker volumes, the result would be that the volume is shared among containers. Any changes a container does to the volume's content, will appear in the other container. I don't want this.
Note that keeping it readonly is not an option. I want the data to be read-write, I simply don't want it to be shared among containers, and at the same time I don't want to keep them in the image.
Is this possible in Docker?
As long as you don't re-use the same volume for other containers, you can use a docker image as template, and use it to "propagate" the volume data;
1. Create a directory for all your template files;
mkdir data
# just creating some dummy files here to illustrate the concept
touch data/foo data/bar data/baz data/bla data/bla2
2. Create a Dockerfile for building the template image
This image contains default data to be used for the containers to used
We're using a tiny image ("hello-world") as it requires a command to
allow a container to be created from it
FROM hello-world
COPY . /data/
3. Build the template image
docker build -t template-data .
4. Create a new volume, and propagate the data
Then, you can create volume, create a container from the image, and
attach the volume to it. The first time the volume is used and is still
empty, the files are copied from the container to the volume.
After the volume is created, and propagated, we don't really need the
container anymore (the data is copied to the volume), so we're passing the
--rm flag as well, so that the container (not the volume, because it's a
"named" volume) is removed directly after it exits
# create an empty volume
docker volume create --name data-volume1
# start the container (which copies the data), and remove the container
docker run -it --rm -v data-volume1:/data template-data
5. Use the volume for your application
Then start your application container, and attach the volume (which now
contains the template data) to it.
For this example, I just start an alpine container and show the contents
of the volume, but normally this would be your application;
docker run --rm -v data-volume1:/somewhere alpine ls -l /somewhere
And you can see the data is there;
docker run --rm -v data-volume1:/somewhere alpine ls -l /somewhere
total 0
-rw-r--r-- 1 root root 0 Jun 2 20:14 bar
-rw-r--r-- 1 root root 0 Jun 2 20:14 baz
-rw-r--r-- 1 root root 0 Jun 2 20:14 bla
-rw-r--r-- 1 root root 0 Jun 2 20:14 bla2
-rw-r--r-- 1 root root 0 Jun 2 20:14 foo
You can do this multiple times, but you need to create a new volume
for each project/application, otherwise they share the same volume,
so are working on the same data;
docker volume create --name data-volume2
docker volume create --name data-volume3
docker volume create --name data-volume4
docker run -it --rm -v data-volume2:/data template-data
docker run -it --rm -v data-volume3:/data template-data
docker run -it --rm -v data-volume4:/data template-data
docker run --rm -v data-volume2:/somewhere alpine ls -l /somewhere
docker run --rm -v data-volume3:/somewhere alpine ls -l /somewhere
docker run --rm -v data-volume4:/somewhere alpine ls -l /somewhere
Hope this helps!
I am trying to create a busybox docker image to save the logs of my rails application, including nginx and unicorn logs. In order to create that container, I use the following command:
docker run --name app-logs -v /logs busybox /bin/sh
However the created container exits immediately with the code 0:
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
75e2f2efdc77 busybox "/bin/sh" 6 seconds ago Exited (0) 4 seconds ago app-logs
The command docker logs is not giving any output and I can't find out what the problem is.
Thanks in advance.
You need to run docker in the foreground if you want to use a shell.
-t=false : Allocate a pseudo-tty
-i=false : Keep STDIN open even if not attached
So
docker run -ti --name app-logs -v /logs busybox /bin/sh
Data + Volumes
If you want to keep a what's called a data volume container, you need to have at least one container that has a reference to the volume. There's no need to keep it running. An exited container is still saved on your system (docker ps -a).
docker run --name app-logs -v /logs busybox /bin/true
Then you can mount your data container volume from your app containers
docker run -d --volumes-from app-logs --name app busybox ruby yourapp.rb
The other way to achieve the same is to use the host to store the data by mounting a host volume everywhere
docker run --name app -v /logs:/logs busybox ruby yourapp.rb
I've found storing data on the host outside of docker to be beneficial when dockers storage has issues. I can blow all the docker data away and start again and easily keep and re mount all my stateful/app data that's stored on the host.