Is there any difference between:
Mounting a host directory into a container path (the path is not exposed as a volume), e.g.:
docker run -v /host/directory:/container/directory my_image command_to_run
Dockerfile of my_image does not include VOLUME ["/container/directory"]
Mounting a host directory into a a container path exposed as a volume:
docker run -v /host/directory:/container/directory my_image command_to_run
Dockerfile of my_image includes VOLUME ["/container/directory"]
I know that volume data persists independent of the container life-cycle. However, since I want to work on my local data from within a container, does that make any difference if the mount-point inside the container is a volume?
There is no difference if you mount the path from the host into the container. The filesystem from the host will be mounted over top of that directory inside the container.
The difference between listing the volume and not listing it inside the image is the behavior of docker when you create an image without specifying a volume. When the volume is defined on the image, docker will create an "anonymous" volume you can see with docker volume ls as a long uuid string. These volumes are rarely useful, so I recommend against defining a volume in the image and instead only defining them on your docker run command or docker-compose.yml definition.
Downsides of defining a volume in the image include:
Later lines in the Dockerfile or in descendant Dockerfile's may not be able to change the contents at this location. Docker's behavior with this varies by scenario and version, so for predictability, once a volume is defined in an image, I consider that directory off limits.
Creation of anonymous volumes are difficult to use and are likely to clutter up the filesystem.
I posted a blog on this topic a while back if you're interested in more details.
Related
Is it currently possible with docker to do something like this conceptually?
docker run --mount type=xxx,image=imageX,subdir=/somedir,dst=/mnt-here imageY ...
I understand this can be done during at docker build time with COPY --from=...., however, in my use-case it would only really be beneficial if it can be done at container creation time.
The only things it's possible to mount into a container arbitrary host directories, tmpfs directories, and Docker named volumes. You can make a named volume use anything you could mount with the Linux mount(8) command. Potentially you can install additional volume drivers to mount other things. But these are all of the possible options.
None of these options allow you to mount image or container content into a different container. The COPY --from=other-image syntax you suggest is probably the best approach here.
If you really absolutely needed it in a volume, one option is to create a volume yourself, copy the content from the source image, and then mount that into the destination image.
docker volume create some-volume
# Since the volume is empty, mounting it into the container will
# copy the contents from the image into the volume. This only happens
# with native Docker volumes and only if the volume is totally empty.
# Docker will never modify the contents of this volume after this.
# Create an empty temporary container to set up the volume
docker run -v some-volume:/somedir --rm some-image /bin/true
# Now you can mount the volume into the actual container
docker run -v some-volume:/mnt-here ...
docker run -ti --rm -v DataVolume3:/var ubuntu
Lets say I have a volume DataVolume 3 which pulls the contents of /var in the ubuntu container
even after killing this ubuntu container the volume remains and I can use this volume DataVolume3 to mount it to other containers.
This means with the deletion of container the volume mounts are not deleted.
How does this work ?
Does that volume mount mean that it copies the contents of /var into some local directory because this does not look like a symbolic link ?
If I have the container running and I create a file in the container then the same file gets copied to the host path ?
How does this whole process of volume mount from container to host and host to container work ?
Volumes are used for persistent storage and the volumes persists independent of the lifecycle of the container.
We can go through a demo to understand it clearly.
First, let's create a container using the named volumes approach as:
docker run -ti --rm -v DataVolume3:/var ubuntu
This will create a docker volume named DataVolume3 and it can be viewed in the output of docker volume ls:
docker volume ls
DRIVER VOLUME NAME
local DataVolume3
Docker stores the information about these named volumes in the directory /var/lib/docker/volumes/ (*):
ls /var/lib/docker/volumes/
1617af4bce3a647a0b93ed980d64d97746878564b141f30b6110d0818bf32b76 DataVolume3
Next, let's write some data from the ubuntu container at the mounted path var:
echo "hello" > var/file1
root#2b67a89a0050:/# cat /var/file1
hello
We can see this data with cat even after deleting the container:
cat /var/lib/docker/volumes/DataVolume3/_data/file1
hello
Note: Although, we are able to access the volumes like shown above but it not a recommended practice to access volumes data like this.
Now, next time when another container uses the same volume then the data from the volume gets mounted at the container directory specified as part of -v flag.
(*) The location may vary based on OS as pointed by David and probably can be seen by the docker volume inspect command.
Docker has a concept of a named volume. By default the storage for this lives somewhere on your host system and you can't directly access it from outside Docker (*). A named volume has its own lifecycle, it can be independently docker volume rm'd, and if you start another container mounting the same volume, it will have the same persistent content.
The docker run -v option takes some unit of storage, either a named volume or a specific host directory, and mounts it (as in the mount(8) command) in a specific place in the container filesystem. This will hide what was originally in the image and replace it with the volume content.
As you note, if the thing you mount is an empty named volume, it will get populated from the image content at container initialization time. There are some really important caveats on this functionality:
Named volume initialization happens only if the volume is totally empty.
The contents of the named volume never automatically update.
If the volume isn't empty, the volume contents completely replace what's in the image, even if it's changed.
The initialization happens only on native Docker, and not for example in Kubernetes.
The initialization happens only on named volumes, and not for bind-mounted host directories.
With all of these caveats, I'd avoid relying on this functionality.
If you need to mount a volume into a container, assume it will be empty when your entrypoint or the main container command starts. If you need a particular directory layout or file structure there, an entrypoint script can create it; if you're expecting it to hold particular data, keep a copy of it somewhere else in your image and copy it in if it's not already there (or, perhaps, always).
(*) On native Linux you can find a filesystem location for it, but accessing this isn't a best practice. On other OSes this will be hidden inside a virtual machine or other opaque storage. If you need to directly access the data (or inject config files, or read log files) a docker run -v /host/path:/container/path bind mount is a better choice.
Volumes are part of neither the container nor the host. Well, technically everything resides in the host machine. But the docker directories are only accessible by users in "docker" group. The files in these directories are separately managed by docker.
"Volumes are stored in a part of the host filesystem which is managed by Docker (/var/lib/docker/volumes/ on Linux)."
Hence volumes are like the union of files under the docker container and the host itself. Any addition on either end will be added to the volume(/var/lib/docker/volumes), not hard copy, rather something like symbol link
As volumes can be shared across different containers, deleting a container does not cascade to the volumes associated with it.
To remove unused volumes:
docker volume prune .
From https://hub.docker.com/r/gitlab/gitlab-runner/dockerfile:
VOLUME ["/etc/gitlab-runner", "/home/gitlab-runner"]
I have read Understanding "VOLUME" instruction in DockerFile and docker volume and VOLUME inside Dockerfile, however, I still cannot figure why does this image specify these two paths as VOLUME.
The VOLUME command in the Dockerfile exposes these directories as VOLUMES. Meaning it will auto-create volumes on docker run to persist the data inside the container(s).
Since containers are suppose to be stateless you want to persistent configuration and e.g. build-cache on the host.
I want to mount a folder of the host system to the container and it need to be defined in the Dockerfile so that user doesn't need to do it manually by passing the argument in the command line to run the container. How to achieve this ?
This simply cannot be done. Docker images are designed to be portable. Host mounts are host specific. Thus if you are able to specify a host mount at build time, it will make the image non-portable across machine that don't have this mount folder. Thus this is why this option is not available.
You can use docker compose to help the user not choose the mount folder. Take a look at How do I mount a host directory as a volume in docker compose
Dockerfile is for create images not containers.
You can not define a volume on a image. The volume must be defined on execution time when the container is created.
docker run -v /host-folder:/root/containerfolder -i -t imagename
I understand that using the VOLUME command within a Dockerfile, defines a mount point within container.
FROM centos:6
VOLUME /html
However I noticed that without that VOLUME definition, it's still possible to mount on that VOLUME point regardless of defining it
docker run -ti -v /path/to/my/html:/html centos:6
What is the purpose of defining VOLUME mount points in the dockerfile? I suspect it's for readability so people can read the Dockerfile and instantly know what is meant to be mounted?
VOLUME instruction used within a Dockerfile does not allow us to do host mount, that is where we mount a directory from the host OS into a container.
However other containers can still mount into the volumes of a container using the --from-container=<container name>, created with the VOLUMES instruction in the Dockerfile
I understand that using the VOLUME command within a Dockerfile,
defines a mount point within container.
That's not right. In that case the volume is defined for an image, not for a container.
When a volume is defined in the Dockerfile, it's set for an image, so every container run from that image gets that volume defined.
If you define the volume in the command line (docker run -v ...) the volume is defined just for that specific container.