https://docs.docker.com/storage/#more-details-about-mount-types
Good use cases for volumes
When you want to store your container’s data on a remote host or a cloud provider, rather than locally.
How is this accomplished with docker volume? Aren't docker volume under hosts's /var/lib/docker?
Could you give me an example of "docker volume create" of this and how it could be utilized?
Yes, volumes are created under /var/lib/docker/volumes/ so you need to link this volume with the folder you want to persist or where you have your data to persist.
Example:
You have your image named ImageExample and your project under /var/www/MyProject/.
First, you need to create new volume and assign a name.
$ docker volume create --name VolumeExample
# if you run: docker volume ls, they list all your volumes available
$ docker volume ls
DRIVER VOLUME NAME
local JbpmVolume1
local VolumeExample
Second, you have to link your new volume to a folder in your container.
$ docker run -v VolumeExample:/var/www/MyProject/ -p 8080:8080 MyImage
Where run is the command to create the container, -p is to map the local and host ports, MyImage is the image used in this example, VolumeExample is the volume created before and /var/www/MyProject/ is the example folder which you need to persist.
You can use this volume to store application configuration, database data or configuration too and so on. Maybe, depends on what you need to store, you can use bind mount or volumes or if your host is in linux, you can use tmpfs mounts.
As simple as that, you can read more about in docker webpage but basically this is how to work with a volume. Every time you stop/start or create/delete the container, the data in your volume will persist.
I do it in this way, because this is not the "happy path" you want. You have to mount before you store the data in the folder, because when you mount the volume, the folder will be empty because the volume is empty. If you have data in the folder before you mount the volume, the data will be not visible for you. So it depends on your project the way you will create the volume, but basically, with this two commands you mount the volume into the host container.
Related
docker run -ti --rm -v DataVolume3:/var ubuntu
Lets say I have a volume DataVolume 3 which pulls the contents of /var in the ubuntu container
even after killing this ubuntu container the volume remains and I can use this volume DataVolume3 to mount it to other containers.
This means with the deletion of container the volume mounts are not deleted.
How does this work ?
Does that volume mount mean that it copies the contents of /var into some local directory because this does not look like a symbolic link ?
If I have the container running and I create a file in the container then the same file gets copied to the host path ?
How does this whole process of volume mount from container to host and host to container work ?
Volumes are used for persistent storage and the volumes persists independent of the lifecycle of the container.
We can go through a demo to understand it clearly.
First, let's create a container using the named volumes approach as:
docker run -ti --rm -v DataVolume3:/var ubuntu
This will create a docker volume named DataVolume3 and it can be viewed in the output of docker volume ls:
docker volume ls
DRIVER VOLUME NAME
local DataVolume3
Docker stores the information about these named volumes in the directory /var/lib/docker/volumes/ (*):
ls /var/lib/docker/volumes/
1617af4bce3a647a0b93ed980d64d97746878564b141f30b6110d0818bf32b76 DataVolume3
Next, let's write some data from the ubuntu container at the mounted path var:
echo "hello" > var/file1
root#2b67a89a0050:/# cat /var/file1
hello
We can see this data with cat even after deleting the container:
cat /var/lib/docker/volumes/DataVolume3/_data/file1
hello
Note: Although, we are able to access the volumes like shown above but it not a recommended practice to access volumes data like this.
Now, next time when another container uses the same volume then the data from the volume gets mounted at the container directory specified as part of -v flag.
(*) The location may vary based on OS as pointed by David and probably can be seen by the docker volume inspect command.
Docker has a concept of a named volume. By default the storage for this lives somewhere on your host system and you can't directly access it from outside Docker (*). A named volume has its own lifecycle, it can be independently docker volume rm'd, and if you start another container mounting the same volume, it will have the same persistent content.
The docker run -v option takes some unit of storage, either a named volume or a specific host directory, and mounts it (as in the mount(8) command) in a specific place in the container filesystem. This will hide what was originally in the image and replace it with the volume content.
As you note, if the thing you mount is an empty named volume, it will get populated from the image content at container initialization time. There are some really important caveats on this functionality:
Named volume initialization happens only if the volume is totally empty.
The contents of the named volume never automatically update.
If the volume isn't empty, the volume contents completely replace what's in the image, even if it's changed.
The initialization happens only on native Docker, and not for example in Kubernetes.
The initialization happens only on named volumes, and not for bind-mounted host directories.
With all of these caveats, I'd avoid relying on this functionality.
If you need to mount a volume into a container, assume it will be empty when your entrypoint or the main container command starts. If you need a particular directory layout or file structure there, an entrypoint script can create it; if you're expecting it to hold particular data, keep a copy of it somewhere else in your image and copy it in if it's not already there (or, perhaps, always).
(*) On native Linux you can find a filesystem location for it, but accessing this isn't a best practice. On other OSes this will be hidden inside a virtual machine or other opaque storage. If you need to directly access the data (or inject config files, or read log files) a docker run -v /host/path:/container/path bind mount is a better choice.
Volumes are part of neither the container nor the host. Well, technically everything resides in the host machine. But the docker directories are only accessible by users in "docker" group. The files in these directories are separately managed by docker.
"Volumes are stored in a part of the host filesystem which is managed by Docker (/var/lib/docker/volumes/ on Linux)."
Hence volumes are like the union of files under the docker container and the host itself. Any addition on either end will be added to the volume(/var/lib/docker/volumes), not hard copy, rather something like symbol link
As volumes can be shared across different containers, deleting a container does not cascade to the volumes associated with it.
To remove unused volumes:
docker volume prune .
I'm confused with what is different between creating docker volume create my-vol and VOLUME ["/var/www"].
My understanding is:
1) docker volume create my-vol creates a persistent volume on our machine and each container could be linked to my-vol.
2) VOLUME ["/var/www"] creates a volume inside its container.
And when I create another container, I could link my-vol as follows:
when running a container
$ docker run -d --name devtest --mount source=myvol2,target=/app nginx:latest
At that time, if I added VOLUME ["/var/www"] in my Dockerfile, all data of this docker file will be stored in both myvol2 and /var/www?
The Dockerfile VOLUME command says two things:
If the operator doesn't explicitly mount a volume on the specific container directory, create an anonymous one there anyways.
No Dockerfile step will ever be able to make further changes to that directory tree.
As an operator, you can mount a volume (either a named volume or a host directory) into a container with the docker run -v option. You can mount it over any directory in the container, regardless of whether or not there was a VOLUME declared for it in the Dockerfile.
(Since you can use docker run -v regardless of whether or not you declare a VOLUME, and it has confusing side effects, I would generally avoid declaring VOLUME in Dockerfiles.)
Just like in ordinary Linux, only one thing can be (usefully) mounted on any given directory. With the setup you describe, data will be stored in the myvol2 you create and mount, and it will be visible in /var/www in the container, but the data will only actually be stored in one place. If you deleted and recreated the container without the volume mount the data would not be there any more.
There are two types of persistent storage used in Docker,the first one is Docker Volumes and the second one is bind mounts. The differebce between them is that volumes are internal to Docker and stored in the Docker store (which is usually all under /var/lib/docker) and bind mounts use a physical location on your machine to store persistent data.
If you want to use a Docker Volume for nginx:
docker volume create nginx-vol
docker run -d --name devtest -v nginx-vol:/usr/share/nginx/html nginx
If you want to use a bind mount:
docker run -d --name devtest -v [path]:/usr/share/nginx/html nginx
[path] is the location in which you want to store the container's data.
I want to share a file storage between two containers. From the documentation, I've seen that you can create and use volumes like this:
docker volume create --name DataVolume1
docker run -ti --rm -v DataVolume1:/datavolume1 ubuntu
However, I want containers to be able to access an initial set of shared data. Does docker support publishing of volumes? If not, does this mean I should write the initial data manually, after creating the volume, or is there another solution for publishing the data along with the images?
With a named volume (not with a host volume, aka bind mount) docker will initialize an empty named volume to the contents of the image at the location you mount it. So if you have files in your image at /datavolume1, and DataVolume1 is empty, docker will copy those files into the named volume.
I am using Docker to deploy some services and I want to share the Docker volumes between different containers.
Suppose I have a Docker container A which mounts a volume at /data. Here is its Dockerfile:
VOLUME /data
From my understanding, this will attach a volume to the container but it will not mount a host directory to the container. So the data inside this volume is still inside the container A.
I have another container B which provides an FTP service. It accesses the data under volume /public. Its Dockerfile is:
VOLUME /public
Now I want to link them together so that I can use B to manage A's data. From the Docker doc https://docs.docker.com/engine/userguide/containers/dockervolumes/ I shall use the --volumes-from flag to mount A's data volume to B. But this command will mount A's data to /data in B instead of /public, and in this case, the container B is not able to access the data. I didn't see any way to rename the mount point.
Any suggestions or best practices to handle this case?
The data-only container gives a good solution for this case. But if you want to use volumes-from and mount the data to different mount point, this question may be helpful!
How to map volume paths using Docker's --volumes-from?
You may find a lot of pointers mentioning data-only containers and --volumes-from. However, since docker 1.9, volumes have become first class citizens, they can have names, and have more flexibility:
It's now easy to achieve the behavior you want, here's an example :
Create a named data volume with name service-data:
docker volume create --name service-data
You can then create a container that mounts it in your /public folder by using the -v flag:
docker run -t -i -v service-data:/public debian:jessie /bin/bash
For testing purpose we create a small text file in our mapped folder:
cd public
echo 'hello' > 'hello.txt'
You may then attach your named volume to a second container, but this time under the data folder:
docker run -t -i -v service-data:/data debian:jessie /bin/bash
ls /data #-->shows "hello.txt"
Just remember, if both containers are using different images, be careful with ownership and permissions!
Take a typical data only Docker container:
FROM stackbrew/busybox:latest
RUN mkdir /data
VOLUME /data
Now I have seen a great deal of them that are run like this:
docker run -name my-data data true
The true command exits as soon as it runs, and so does the container. But surprisingly it continues to serve the volume when you connect it with another container via --volumes-from my-data.
My question is, how does that work? How does a stopped container still allow access in it's volumes?
Volumes in docker are not a top-level thing. They are "simply" part of container's meta-data.
When you have VOLUME in your dockerfile or start a container with -v, Docker will create a directory in /var/lib/docker/volumes* with a random ID (this is the exact same process as creating an image with commit except it is empty) and add that random ID to the container's metadata.
When the container starts, Docker will mount-bind the directory /var/lib/docker/volumes/* at the given location for that volume.
When you use volumes-from, Docker will just lookup the volume id and the location from an other container, running or not and mount-bind the directory at the set location.
Volumes are not linked with the runtime, it is just directories that are mounted.
* With newer versions, Docker now uses the vfs driver for storage and /var/lib/docker/volumes/ is used only for metadatas like size, create time, etc. The actual data are stored in /var/lib/docker/vfs/dir/<volume id>