I declare a new volume in runtime with -v flag in docker run command. What is the created volume? Partition or only a subdirectory? Could be it maybe formatted to another filesystem(If Docker supports other fs, than AUFS)?
Using "-v" is created a link between the host folder to container. The files created in container linked folder, that documents will be stored in folder of host.
About other FS. Docker supports AUFS, btrfs, vfs, and DeviceMapper:
https://docs.docker.com/introduction/understanding-docker/
Related
docker run -ti --rm -v DataVolume3:/var ubuntu
Lets say I have a volume DataVolume 3 which pulls the contents of /var in the ubuntu container
even after killing this ubuntu container the volume remains and I can use this volume DataVolume3 to mount it to other containers.
This means with the deletion of container the volume mounts are not deleted.
How does this work ?
Does that volume mount mean that it copies the contents of /var into some local directory because this does not look like a symbolic link ?
If I have the container running and I create a file in the container then the same file gets copied to the host path ?
How does this whole process of volume mount from container to host and host to container work ?
Volumes are used for persistent storage and the volumes persists independent of the lifecycle of the container.
We can go through a demo to understand it clearly.
First, let's create a container using the named volumes approach as:
docker run -ti --rm -v DataVolume3:/var ubuntu
This will create a docker volume named DataVolume3 and it can be viewed in the output of docker volume ls:
docker volume ls
DRIVER VOLUME NAME
local DataVolume3
Docker stores the information about these named volumes in the directory /var/lib/docker/volumes/ (*):
ls /var/lib/docker/volumes/
1617af4bce3a647a0b93ed980d64d97746878564b141f30b6110d0818bf32b76 DataVolume3
Next, let's write some data from the ubuntu container at the mounted path var:
echo "hello" > var/file1
root#2b67a89a0050:/# cat /var/file1
hello
We can see this data with cat even after deleting the container:
cat /var/lib/docker/volumes/DataVolume3/_data/file1
hello
Note: Although, we are able to access the volumes like shown above but it not a recommended practice to access volumes data like this.
Now, next time when another container uses the same volume then the data from the volume gets mounted at the container directory specified as part of -v flag.
(*) The location may vary based on OS as pointed by David and probably can be seen by the docker volume inspect command.
Docker has a concept of a named volume. By default the storage for this lives somewhere on your host system and you can't directly access it from outside Docker (*). A named volume has its own lifecycle, it can be independently docker volume rm'd, and if you start another container mounting the same volume, it will have the same persistent content.
The docker run -v option takes some unit of storage, either a named volume or a specific host directory, and mounts it (as in the mount(8) command) in a specific place in the container filesystem. This will hide what was originally in the image and replace it with the volume content.
As you note, if the thing you mount is an empty named volume, it will get populated from the image content at container initialization time. There are some really important caveats on this functionality:
Named volume initialization happens only if the volume is totally empty.
The contents of the named volume never automatically update.
If the volume isn't empty, the volume contents completely replace what's in the image, even if it's changed.
The initialization happens only on native Docker, and not for example in Kubernetes.
The initialization happens only on named volumes, and not for bind-mounted host directories.
With all of these caveats, I'd avoid relying on this functionality.
If you need to mount a volume into a container, assume it will be empty when your entrypoint or the main container command starts. If you need a particular directory layout or file structure there, an entrypoint script can create it; if you're expecting it to hold particular data, keep a copy of it somewhere else in your image and copy it in if it's not already there (or, perhaps, always).
(*) On native Linux you can find a filesystem location for it, but accessing this isn't a best practice. On other OSes this will be hidden inside a virtual machine or other opaque storage. If you need to directly access the data (or inject config files, or read log files) a docker run -v /host/path:/container/path bind mount is a better choice.
Volumes are part of neither the container nor the host. Well, technically everything resides in the host machine. But the docker directories are only accessible by users in "docker" group. The files in these directories are separately managed by docker.
"Volumes are stored in a part of the host filesystem which is managed by Docker (/var/lib/docker/volumes/ on Linux)."
Hence volumes are like the union of files under the docker container and the host itself. Any addition on either end will be added to the volume(/var/lib/docker/volumes), not hard copy, rather something like symbol link
As volumes can be shared across different containers, deleting a container does not cascade to the volumes associated with it.
To remove unused volumes:
docker volume prune .
In docker, where is the container data, apart from the mounted volume, is stored till the container is running.
So lets say /usr/local is volume mounted, so it would be shared between the host and the container. Where is everything else stored?
You should inspect your docker container
docker inspect [ID_CONTAINER]
and check for the fields MergedDir, LowerDir and UpperDir. Docker uses OverlayFS file system to store the data.
OverlayFS layers two directories on a single Linux host and presents them as a single directory. These directories are called layers and the unification process is referred to a a union mount. OverlayFS refers to the lower directory as lowerdir and the upper directory a upperdir. The unified view is exposed through its own directory called merged.
Check the doc here.
You can run the following command to get container written data
ll `podman inspect ContainerName --format "{{ .GraphDriver.Data.MergedDir }}"`
https://docs.docker.com/storage/#more-details-about-mount-types
Good use cases for volumes
When you want to store your container’s data on a remote host or a cloud provider, rather than locally.
How is this accomplished with docker volume? Aren't docker volume under hosts's /var/lib/docker?
Could you give me an example of "docker volume create" of this and how it could be utilized?
Yes, volumes are created under /var/lib/docker/volumes/ so you need to link this volume with the folder you want to persist or where you have your data to persist.
Example:
You have your image named ImageExample and your project under /var/www/MyProject/.
First, you need to create new volume and assign a name.
$ docker volume create --name VolumeExample
# if you run: docker volume ls, they list all your volumes available
$ docker volume ls
DRIVER VOLUME NAME
local JbpmVolume1
local VolumeExample
Second, you have to link your new volume to a folder in your container.
$ docker run -v VolumeExample:/var/www/MyProject/ -p 8080:8080 MyImage
Where run is the command to create the container, -p is to map the local and host ports, MyImage is the image used in this example, VolumeExample is the volume created before and /var/www/MyProject/ is the example folder which you need to persist.
You can use this volume to store application configuration, database data or configuration too and so on. Maybe, depends on what you need to store, you can use bind mount or volumes or if your host is in linux, you can use tmpfs mounts.
As simple as that, you can read more about in docker webpage but basically this is how to work with a volume. Every time you stop/start or create/delete the container, the data in your volume will persist.
I do it in this way, because this is not the "happy path" you want. You have to mount before you store the data in the folder, because when you mount the volume, the folder will be empty because the volume is empty. If you have data in the folder before you mount the volume, the data will be not visible for you. So it depends on your project the way you will create the volume, but basically, with this two commands you mount the volume into the host container.
I have a directory in my Docker container, and I'm trying to make it available locally using -v screenshots:/srv/screenshots in my docker run command but it's not available.
Do I need to add something else to my command?
Host volumes are mapped from the host into the container, not the other way around. This is one way to have persistent storage (so the data don't disappear when the container is re-created).
You can copy the screenshot folder to your host with docker cp and map them in.
You will have your screenshots in the local screenshots folder. Mapping them in with -v screenshots:/srv/screenshots makes them appear in /srv/screenshots in the container, but these files are really on the host.
See: Mount a host directory as data volume
Let's take an example.
The following is the VOLUME instruction for the nginx image:
VOLUME ["/etc/nginx/sites-enabled", "/etc/nginx/certs", "/etc/nginx/conf.d", "/var/log/nginx", "/var/www/html"]
Here are my questions:
When you start the container, will these directories show up on my host? And when I stop my container, the directories will stay?
If some (or all) of these directories already exist in my host, what will happen? For example, let's say the image comes with a default config file within the /etc/nginx directory of the container, and I also have a config file within /etc/nginx on my host. When the container starts, which of these files will get priority?
What's the key difference between -v /host/dir:container/dir and VOLUME?
References:
https://github.com/dockerfile/nginx/blob/master/Dockerfile
http://www.tech-d.net/2014/11/03/docker-indepth-volumes/
How to mount host volumes into docker containers in Dockerfile during build
http://jpetazzo.github.io/2015/01/19/dockerfile-and-data-in-volumes/
A container's volumes are just directories on the host regardless of what method they are created by. If you don't specify a directory on the host, Docker will create a new directory for the volume, normally under /var/lib/docker/vfs.
However the volume was created, it's easy to find where it is on the host by using the docker inspect command e.g:
$ ID=$(docker run -d -v /data debian echo "Data container")
$ docker inspect -f {{.Mounts}} $ID
[{0d7adb21591798357ac1e140735150192903daf3de775105c18149552a26f951 /var/lib/docker/volumes/0d7adb21591798357ac1e140735150192903daf3de775105c18149552a26f951/_data /data local true }]
We can see that Docker has created a directory for the volume at /var/lib/docker/volumes/0d7adb21591798357ac1e140735150192903daf3de775105c18149552a26f951/_data.
You are free to modify/add/delete files in this directory from the host, but note that you may need to use sudo for permissions.
Docker will only delete volume directories in two circumstances:
If the --rm option is given to docker run, any volumes will be deleted when the container exits
If a container is deleted with docker rm -v CONTAINER, any volumes will be removed.
In both cases, volumes will only be deleted if no other containers refer to them. Volumes mapped to specific host directories (the -v HOST_DIR:CON_DIR syntax) are never deleted by Docker. However, if you remove the container for a volume, the naming scheme means you will have a hard time figuring out which directory contains the volume.
So, specific questions:
Yes and yes, with above caveats.
Each Docker managed volume gets a new directory on the host
The VOLUME instruction is identical to -v without specifying the host dir. When the host dir is specified, Docker does not create any directories for the volume, will not copy in files from the image and will never delete the volume (docker rm -v CONTAINER will not delete volumes mapped to user-specified host directories).
More information here:
https://blog.container-solutions.com/understanding-volumes-docker