what does docker do, when you bind-mount a volume in your docker "run" command,
which is already a managed volume defined in the docker build file/ image?
Example:
dockerfile defines /myvolume as managed volume
then: docker run -v /< my_host_dir >:/myvolume ... /< image >
What I see is that the managed volume is no longer created.
Instead the bind-mount comes through and mounts the host-dir into the container.
What goes on behind the scenes?
Is this documented somewhere and therefor something one can count on?
br volker
The VOLUME statement in a Dockerfile just marks the directory as to be mounted from somewhere else to help users of the image. For example when you create a Database-Image, the user of that image usually wants to persist the date outside of the container.
If you (as the creator of the Image/writer of the Dockerfile) marked a directory as a VOLUME, the user of the image (the one who executes docker run or similar) has an idea, where in the container he should mount a directory from outside.
Related
We have a system in which the user can start sessions inside a number of docker containers. When they do this their home directory is automatically mounted to the docker container. We can't modify the system which starts the docker containers and mounts the directory.
Our goal is to have one image not automatically mount this container. Is there something that I can do to the image to basically make one directory unmountable?
No. If you can docker run a container, you can always use docker run -v to mount any host directory over any directory, and the original contents of the image will be hidden.
Docker's general model is that the image has somewhat limited powers, but you can specify most things when you start the container. Trying to prevent a volume mount (more frequently asked, trying to force a volume mount) is the opposite of this model; the image has no way to prevent how it will eventually be used.
docker run -ti --rm -v DataVolume3:/var ubuntu
Lets say I have a volume DataVolume 3 which pulls the contents of /var in the ubuntu container
even after killing this ubuntu container the volume remains and I can use this volume DataVolume3 to mount it to other containers.
This means with the deletion of container the volume mounts are not deleted.
How does this work ?
Does that volume mount mean that it copies the contents of /var into some local directory because this does not look like a symbolic link ?
If I have the container running and I create a file in the container then the same file gets copied to the host path ?
How does this whole process of volume mount from container to host and host to container work ?
Volumes are used for persistent storage and the volumes persists independent of the lifecycle of the container.
We can go through a demo to understand it clearly.
First, let's create a container using the named volumes approach as:
docker run -ti --rm -v DataVolume3:/var ubuntu
This will create a docker volume named DataVolume3 and it can be viewed in the output of docker volume ls:
docker volume ls
DRIVER VOLUME NAME
local DataVolume3
Docker stores the information about these named volumes in the directory /var/lib/docker/volumes/ (*):
ls /var/lib/docker/volumes/
1617af4bce3a647a0b93ed980d64d97746878564b141f30b6110d0818bf32b76 DataVolume3
Next, let's write some data from the ubuntu container at the mounted path var:
echo "hello" > var/file1
root#2b67a89a0050:/# cat /var/file1
hello
We can see this data with cat even after deleting the container:
cat /var/lib/docker/volumes/DataVolume3/_data/file1
hello
Note: Although, we are able to access the volumes like shown above but it not a recommended practice to access volumes data like this.
Now, next time when another container uses the same volume then the data from the volume gets mounted at the container directory specified as part of -v flag.
(*) The location may vary based on OS as pointed by David and probably can be seen by the docker volume inspect command.
Docker has a concept of a named volume. By default the storage for this lives somewhere on your host system and you can't directly access it from outside Docker (*). A named volume has its own lifecycle, it can be independently docker volume rm'd, and if you start another container mounting the same volume, it will have the same persistent content.
The docker run -v option takes some unit of storage, either a named volume or a specific host directory, and mounts it (as in the mount(8) command) in a specific place in the container filesystem. This will hide what was originally in the image and replace it with the volume content.
As you note, if the thing you mount is an empty named volume, it will get populated from the image content at container initialization time. There are some really important caveats on this functionality:
Named volume initialization happens only if the volume is totally empty.
The contents of the named volume never automatically update.
If the volume isn't empty, the volume contents completely replace what's in the image, even if it's changed.
The initialization happens only on native Docker, and not for example in Kubernetes.
The initialization happens only on named volumes, and not for bind-mounted host directories.
With all of these caveats, I'd avoid relying on this functionality.
If you need to mount a volume into a container, assume it will be empty when your entrypoint or the main container command starts. If you need a particular directory layout or file structure there, an entrypoint script can create it; if you're expecting it to hold particular data, keep a copy of it somewhere else in your image and copy it in if it's not already there (or, perhaps, always).
(*) On native Linux you can find a filesystem location for it, but accessing this isn't a best practice. On other OSes this will be hidden inside a virtual machine or other opaque storage. If you need to directly access the data (or inject config files, or read log files) a docker run -v /host/path:/container/path bind mount is a better choice.
Volumes are part of neither the container nor the host. Well, technically everything resides in the host machine. But the docker directories are only accessible by users in "docker" group. The files in these directories are separately managed by docker.
"Volumes are stored in a part of the host filesystem which is managed by Docker (/var/lib/docker/volumes/ on Linux)."
Hence volumes are like the union of files under the docker container and the host itself. Any addition on either end will be added to the volume(/var/lib/docker/volumes), not hard copy, rather something like symbol link
As volumes can be shared across different containers, deleting a container does not cascade to the volumes associated with it.
To remove unused volumes:
docker volume prune .
According to the Docker documentation regarding the VOLUME instruction:
The docker run command initializes the newly created volume with any data that exists at the specified location within the base image. For example, consider the following Dockerfile snippet:
FROM ubuntu
RUN mkdir /myvol
RUN echo "hello world" > /myvol/greeting
VOLUME /myvol
This Dockerfile results in an image that causes docker run to create a new mount point at /myvol and copy the greeting file into the newly created volume.
I am unable to replicate this behaviour using this exact Dockerfile and running:
$ docker run --volume ~/testdir:/myvol 159b3387c7eb
The testdir directory is created by does not contain the expected greeting file.
What am I doing wrong ?
There are several types of volumes:
Host volume, which bind mounts a directory from the host into a container. Docker never initializes these. Whatever the state of the host becomes the directory contents in that directory. This is the type of volume you created. They have the downside of having uid/gid permission issues when the container uid/gid does not match that of your host user.
Anonymous volume, which is a named volume with a long guid for a name. There is no information of what the target of an anonymous volume was, especially when the container has been deleted, so you'll want to give your volumes a name at least. These are the default if you define a volume with VOLUME and then forget to define a volume when running the container.
Named volume, which is created by giving it a valid name rather than a path as a source. These are recommended for most scenarios, unless you need external access to your data with a host volume. The advantage of named volumes is that docker initializes them upon creation to the contents of your image at that location. This initialization only happens when the volume is first created, so changes to the image will not be reflected unless you have something like an entrypoint to copy data from another directory back out to your volume.
If you want to see a named volume with the data initialized, that would look like:
$ docker run --volume testvol:/myvol 159b3387c7eb
to create a volume called testvol.
When you declare a VOLUME that doesn't pre-populate the volume with anything, it just describes an intended mount-point.
When you call Docker with the --volume argument it replaces anything in that target directory with your local directory, effectively mounting that local directory on top of the target location. Anything in that original location is obscured by the mount and inaccessible.
If you need to pre-populate the volume with content either do it before you mount it, or have an ENTRYPOINT script that initializes that directory when the container boots up.
I've searched the docs but nothing came up so time to test it. But for a quick future reference...
Is the host folder populated with the container folder contents?
Is it the opposite?
Are both folder contents merged? (In that case: What happens when a file with the same name is in both folders?)
Or does it produce an error? Is the error thrown on launch or is it thrown when you try to build an image with a VOLUME pointing to an existing populated folder on the container?
Also, another thing that isn't in the docs: Do I have to define the container path as a VOLUME in the Dockerfile in order to use -v against it when launching the container or can I create volumes on the fly?
When you run a container and mount a volume from the host, all you see in the container is what is on the host - the volume mount points at the host directory, so if there was anything in the directory in the image it gets bypassed.
With an image from this Dockerfile:
FROM ubuntu
WORKDIR /vol
RUN touch /vol/from-container
VOLUME /vol
When you run it without a host mount, the image contents get copied into the volume:
> docker run vol-test ls /vol
from-container
But mount the volume from the host and you only see the host's content:
> ls $(pwd)/host
from-host
> docker run -v $(pwd)/host:/vol vol-test ls /vol
from-host
And no, you don't need the VOLUME instruction. The behaviour is the same without it.
Whenever a Docker container is created with a volume mounted on the host, e.g.:
docker run -v /path/on/host:/data container-image
Any contents that are already in /data due to the image build process are always completely discarded, and whatever is currently at /path/on/host is used in its place. (If /path/on/host does not exist, it is created as an empty directory, though I think some aspect of that behavior may currently be deprecated.)
Pre-defining a volume in the Dockerfile with VOLUME is not necessary; all VOLUME does is cause any containers run from the image to have an implicit -v /volume/path (Note lack of host mount path) argument added to their docker run command which is ignored if an explicit -v /host/path:/volume/path is used.
Take a typical data only Docker container:
FROM stackbrew/busybox:latest
RUN mkdir /data
VOLUME /data
Now I have seen a great deal of them that are run like this:
docker run -name my-data data true
The true command exits as soon as it runs, and so does the container. But surprisingly it continues to serve the volume when you connect it with another container via --volumes-from my-data.
My question is, how does that work? How does a stopped container still allow access in it's volumes?
Volumes in docker are not a top-level thing. They are "simply" part of container's meta-data.
When you have VOLUME in your dockerfile or start a container with -v, Docker will create a directory in /var/lib/docker/volumes* with a random ID (this is the exact same process as creating an image with commit except it is empty) and add that random ID to the container's metadata.
When the container starts, Docker will mount-bind the directory /var/lib/docker/volumes/* at the given location for that volume.
When you use volumes-from, Docker will just lookup the volume id and the location from an other container, running or not and mount-bind the directory at the set location.
Volumes are not linked with the runtime, it is just directories that are mounted.
* With newer versions, Docker now uses the vfs driver for storage and /var/lib/docker/volumes/ is used only for metadatas like size, create time, etc. The actual data are stored in /var/lib/docker/vfs/dir/<volume id>