-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
Does docker-compose up -V not apply to named volumes?
I have a service that, at image build time, pulls some files from SVN. It then creates a named volume.
I can docker-compose build --no-cache to recreate the image and pull latest files from SVN. But the volume doesn't get updated on docker-compose up -V, unless I remove it beforehand.
I just want a clean and simple way to update files in a named volume
Sure, I could manually remove the volume, and then run everything, but I really wanted it to be compose-driven. This leads me to a second problem.
There is a docker-compose down -v that will also remove volumes, but I cannot run that against a single service (only all or nothing).
So I need to somehow figure out the named volumes of just 1 service from compose-file, and then use some extra command (docker volume rm?) to remove just that one volume.
If you don't care about the content of a named volume, either don't create it in the first place (remove the named volume line in the compose file) or delete it when you stop the project with:
docker-compose down -v
Related
I want to recreate a service, including its volumes. The closest I got was the following commands:
docker-compose stop foo
docker-compose rm -f foo
docker-compose up --renew-anon-volumes -d foo
docker-compose start foo
The issue here is --renew-anon-volumes recreates all services that have anonymous volumes, not just foo's volumes. If I don't use --renew-anon-volumes, then I think I need a named volume to do docker volume rm myvolume. However, with named volumes, Docker Compose always prepends a project name. Since my script doesn't know the project name, I can't programmatically delete the volume. I can't enforce that the user uses a particular project name. I know I can set the project name using an environment variable, but there's no guarantee that the user won't run Docker Compose with a different project name.
I think there are 2 potential solutions:
Make --renew-anon-volumes only recreate the volumes for the service I specified
Use a named volume and somehow figure out the correct prefix
Are either of these doable, or is there another solution?
Many roads leading to Rome, depending on your prerequisites:
Do docker volume ls and regex the result for your named volume (just working if volume name is unique)
Use external volumes and volume create them with known names by bootstrap script before running docker-compose up (not working if volumes must be instantiated)
Set project name to a known value. Normally it takes the folder name, but can explicitly given in docker-compose command (-p NAME) or by environment variable (COMPOSE_PROJECT_NAME=NAME).
Setup a dummy compose-file just containing this single service with its volumes for your script. Doing a docker-compose -f 'your-down-file.yml' down -v which removes all named and anonymous volumes belonging to this service and docker-compose -f .. up on this file.
Edit (#DavidMaze):
You're right, docker compose recognizes that fact. But it does NOT remove it, just warning. If you want to remove all "orphans" you need the flag --remove-orphans.
But for some reasons the down does not remove volumes then, even if flag -v is given. This could be reported because it is not behaving like described.
And errata: the flag -f must go before up/down and not after!
docker-compose rm has a -v option to delete anonymous volumes attached to a container, and also a -s option to stop the container. For your particular use case it should be enough to:
docker-compose rm -s -f -v foo
docker-compose up -d foo
This will only help for anonymous volumes, that is, where the Compose file has volumes: with only a container path and there is no corresponding top-level volumes: entry. I don't immediately see a Compose option to list, remove, or otherwise manage named volumes that Compose created.
I was looking for an explanation on the VOLUME entry when writing a Dockerfile and came across this statement
A volume is a persistent data stored in /var/lib/docker/volumes/...
You can either declare it in a Dockerfile, which means each time a container is started from the image, the volume is created (empty), even if you don't have any -v option.
You can declare it on runtime docker run -v [host-dir:]container-dir.
combining the two (VOLUME + docker run -v) means that you can mount the content of a host folder into your volume persisted by the container in /var/lib/docker/volumes/...
docker volume create creates a volume without having to define a Dockerfile and build an image and run a container. It is used to quickly allow other containers to mount said volume.
But I'm having a hard time understanding this line:
...combining the two (VOLUME + docker run -v) means that you can mount the content of a host folder into your volume persisted by the container in /var/lib/docker/volumes/...
For example, let's say I have a config file on my host machine and I run the container based off the image I made with the Dockerfile I wrote. Will it copy the config file into where the volume that I stated in my the volume entry?
Would it be something like (pseudocode)
#dockerfile
From Ubuntu
Run apt-get update
Run apt-get install mysql
Volume . /etc/mysql/conf.d
Cmd systemcl start MySQL
And when I run it
docker run -it -v /path/to/config/file: ubuntu_based_image
Is this what they mean?
You probably don't want VOLUME in your Dockerfile. It's not necessary to mount files or directories at runtime, and it has confusing side effects like making subsequent RUN commands silently lose state.
If an image does have a VOLUME, and you don't mount anything else there when you start the container, Docker will create an anonymous volume and mount it for you. This can result in space leaks if you don't clean these volumes up.
You can use a docker run -v option on any container directory regardless of whether or not it's declared as a VOLUME.
If you docker run -v /host/path:/container/path, the two directories are actually the same; nothing is copied, and writes to one are (supposed to be) immediately visible on the other.
docker run -v /host/path:/container/path bind mounts aren't visible in /var/lib/docker at all.
You shouldn't usually be looking at content in /var/lib/docker (and can't if you're not on a native-Linux host). If you need to access the volume file content directly, use a bind mount rather than a named or anonymous volume.
Bind mounts like you've shown are appropriate for injecting config files into containers, and for reading log files back out. Named volumes are appropriate for stateful applications' storage, like the data for a MySQL database. Neither type of volume is appropriate for code or libraries; build these directly into Docker images instead.
I am trying to capture the state of a docker container as an image, in a way that includes files I have added to a volume within the container. So, if I run the original container in this way:
$ docker run -ti -v /cookbook ubuntu:14.04 /bin/bash
root#b78f3599d936:/# cd cookbook
root#b78f3599d936:/cookbook# touch foo.txt
Now, if I either export, or commit the container as a new docker image, and then run a container from the new image, then the file, foo.txt is never included in the /cookbook directory.
My question is whether there is a way to create an image from a container in a way that allows the image to include file content within its volumes.
whether there is a way to create an image from a container in a way that allows the image to include file content within its volumes?
No, because volume is designed to manage data inside and between your Docker containers, it's used to persist and share data. What's in image is usually your program(artifacts, executables, libs. e.g) with its whole environment, building/updating data to image does not make much sense.
And in docs of volumes, they told us:
Changes to a data volume will not be included when you update an image.
Also in docs of docker commit:
The commit operation will not include any data contained in volumes mounted inside the container.
Well, by putting the changes in a volume, you're excluding them from the actual container. The documentation for docker export includes this:
The docker export command does not export the contents of volumes associated with the container. If a volume is mounted on top of an existing directory in the container, docker export will export the contents of the underlying directory, not the contents of the volume.
Refer to Backup, restore, or migrate data volumes in the user guide for examples on exporting data in a volume.
This points to this documentation. Please follow the steps there to export the information stored in the volume.
You're probably looking for something like this:
docker run --rm --volumes-from <containerId> -v $(pwd):/backup ubuntu tar cvf /backup/backup.tar /cookbook
This would create a file backup.tar with the contents of the container's /cookbook directory and store it in the current directory of the host. You could then use this tar file to import it in another container.
Essentially, there are three ways to do persistence in Docker:
You can keep files in a volume, which is a filesystem managed by Docker. This is what happens in your example: because the /cookbook directory is part of a volume, your file does not get commited/exported with the image. It does however get stored in the volume, so if you remount the same volume in a different container, you will find your file there. You can list your volumes using docker volume ls. As you can see, you should probably give your volumes names if you plan to reuse them. You can mount an existing volume, or create a new one, if the name does not exist, with
docker run -v name:/directory ubuntu
You can keep files as part of the image. If you commit the container, all changes to its file hierarchy are stored in the new image except those made to mounted volumes. So if you just get rid of the -v flag, your file shows up in the commit.
You can bind mount a directory from the host machine to the container, by using the -v /hostdir:/targetdir syntax. The container then simply has access to a directory of the host machine.
Docker commit allows you to create an image from a container and its data (mounted volumes will be ignored)
I'm trying to build a Dockerfile for a webapp that uses a file-based database. I would like to be able to mount the file from the host*
The file is in the root of the complete software install, so it's not really ideal to mount that complete dir.
Another problem is that before the first use, the database-file isn't created yet. A first time user won't have a database, but another user might. I can't 'mount' anything during a build** I believe.
It could probably work like this:
First/new database start:
Start the container (without mount).
The webapp creates a database.
Stop the container
subsequent starts:
Start the container using a -v to mount the file
It would be better if that extra start/stop isn't needed for a user. Even if it is, I'm still looking for a way to do this userfriendly, possibly having 2 'methods' of starting it (maybe I can define a first-boot thing in docker-compose as well as a 'normal' method?).
How can I do this in a simpel way, so that it's clear for any first time users?
* The reason is that you can copy your Dockerfile and the database file as a backup, and be up and running with just those 2 elements.
** How to mount host volumes into docker containers in Dockerfile during build
One approach that may work is:
Start the database in the build file in such a way that it has time to create the default file before exiting.
Declare a VOLUME in the Dockerfile for the file after the above instruction. This will cause the file to be copied into the volume when a container is started, assuming you don't explicitly provide a host path
Use data-containers rather than volumes. So the normal usage would be:
docker run --name data_con my_db echo "my_db data container"
docker run -d --volumes-from data_con my_db
...
The first container should exit immediately but set up the volume that is used in the second container.
I was trying to achieve something similar and managed to do it by mounting a folder, instead of the file, and creating a symlink in the Dockerfile, initially pointing to a non-existing file:
docker-compose.yml
version: '3.0'
services:
bash:
build: .
volumes:
- ./data:/data
command: ['bash']
Dockerfile
FROM bash:latest
RUN ln -s /data/.bash_history /root/.bash_history
Then you can run the container with:
docker-compose run --rm bash
With this setup, you can push an empty "data" folder into the repository for example (and exclude its content with .gitignore). In the first run, inside the container /root/.bash_history will be a "broken" symlink, pointing to a file that does not exist. When you exit the shell, bash will write the history to /root/.bash_history, which will end up in /data/.bash_history.
This is probably not the correct approach.
If you have multiple containers that are trying to share some information through the file-system, you should probably let them share some directory.
That way, the flow is simple and very hard to get wrong.
You simply mount the same directory, say /data (from the host's perspective) into all the containers that are trying to use it.
When an application starts and it can't find anything inside that directory, it can gracefully stop and exit with a code that says: "Cannot start, DB not initialized yet".
You can then configure some mechanism with a growing timeout to try and restart that container until you're successful.
On the other hand, the app that creates the DB can start and create it inside the directory or find an existing file to use.
I want to commit the data in a container's shared volume to an image. I cannot seem to do it? I kind of get the impression this perhaps is not possible in Docker but that seems totally at odds with the whole philosophy of not leaving data on the host so part of me thinks there must be a way to do this.
1. Terminal 1
Start up a container in Terminal 1 with a volume.
$ docker run -it -v /data ubuntu:14.10 /bin/bash
root#19fead4f6a68:/# echo "Hello Docker Volumes." > /data/foo.txt
2. Terminal 2
Start up a second container in Terminal 2 the file from container 1 is there so docker volumes are all working.
$ docker run -it --volumes-from 19fead4f6a68 ubuntu:14.10 /bin/bash
root#5c7cdbfc67d8:/# cat /data/foo.txt
Hello Docker Volumes.
3. Terminal 3
My understanding is that I can only commit diffs to images so I check what the diffs are on both the containers. For some bizarre reason my changes do not show up!??
$ docker diff 19fead4f6a68
A /data
$ docker diff 5c7cdbfc67d8
A /data
4. Back in Terminal 1
I create a file outside of the volume folder
root#19fead4f6a68:/# echo "Docker you are a very strange beast...." > /var/beast.txt
5. Back in Terminal 3
We now have some changes we can commit although I am rather frustrated as this is not the data from the volume I needed to share with my other container.
$ docker diff 19fead4f6a68
A /data
C /var
A /var/beast.txt
Clearly this is by design. Anyone have any ideas as to why docker don't allow me to save volume data to a commit? Is there anyway at all to share files between containers and then save them to an image? I feel like there must be something I am missing? Especially to the ends of sharing data whilst avoiding host dependencies.
Volumes are outside of container images. That's exactly what they are for - bringing data inside a container that isn't in the image.
From the Docker docs:
A data volume is a specially-designated directory within one or more containers that bypasses the Union File System to provide several useful features for persistent or shared data:
Data volumes can be shared and reused between containers
Changes to a data volume are made directly
Changes to a data volume will not be included when you update an image
If you want to save some changes as part of an image, make the changes inside the image and not in a volume. If you want to share changes across multiple containers, put that data in a volume but you have to make your own arrangements for snapshots, rollback, etc., because Docker doesn't have that feature.
Maybe you would be interested in Flocker.
It looks as though there is an open issue around adding volume layers to docker:
https://github.com/docker/docker/issues/9382