I'm trying to avoid slow osxfs when using docker. So I'm running docker-sync volume container.
I want to mount only subdir of this volume to another container.
Current example errors out:
docker: Error response from daemon: create docker-sync/www/marcopolo.front: "docker-sync/www/marcopolo.front" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed.
See 'docker run --help'.
As you've seen, docker does not currently provide a way to mount only a sub directory from inside of a named volume. You'll have to mount to entire volume and then either pick your specific files, or you can mount to volume in a separate location and include a symbolic link to your specific subdirectory as part of the image.
Related
docker run -ti --rm -v DataVolume3:/var ubuntu
Lets say I have a volume DataVolume 3 which pulls the contents of /var in the ubuntu container
even after killing this ubuntu container the volume remains and I can use this volume DataVolume3 to mount it to other containers.
This means with the deletion of container the volume mounts are not deleted.
How does this work ?
Does that volume mount mean that it copies the contents of /var into some local directory because this does not look like a symbolic link ?
If I have the container running and I create a file in the container then the same file gets copied to the host path ?
How does this whole process of volume mount from container to host and host to container work ?
Volumes are used for persistent storage and the volumes persists independent of the lifecycle of the container.
We can go through a demo to understand it clearly.
First, let's create a container using the named volumes approach as:
docker run -ti --rm -v DataVolume3:/var ubuntu
This will create a docker volume named DataVolume3 and it can be viewed in the output of docker volume ls:
docker volume ls
DRIVER VOLUME NAME
local DataVolume3
Docker stores the information about these named volumes in the directory /var/lib/docker/volumes/ (*):
ls /var/lib/docker/volumes/
1617af4bce3a647a0b93ed980d64d97746878564b141f30b6110d0818bf32b76 DataVolume3
Next, let's write some data from the ubuntu container at the mounted path var:
echo "hello" > var/file1
root#2b67a89a0050:/# cat /var/file1
hello
We can see this data with cat even after deleting the container:
cat /var/lib/docker/volumes/DataVolume3/_data/file1
hello
Note: Although, we are able to access the volumes like shown above but it not a recommended practice to access volumes data like this.
Now, next time when another container uses the same volume then the data from the volume gets mounted at the container directory specified as part of -v flag.
(*) The location may vary based on OS as pointed by David and probably can be seen by the docker volume inspect command.
Docker has a concept of a named volume. By default the storage for this lives somewhere on your host system and you can't directly access it from outside Docker (*). A named volume has its own lifecycle, it can be independently docker volume rm'd, and if you start another container mounting the same volume, it will have the same persistent content.
The docker run -v option takes some unit of storage, either a named volume or a specific host directory, and mounts it (as in the mount(8) command) in a specific place in the container filesystem. This will hide what was originally in the image and replace it with the volume content.
As you note, if the thing you mount is an empty named volume, it will get populated from the image content at container initialization time. There are some really important caveats on this functionality:
Named volume initialization happens only if the volume is totally empty.
The contents of the named volume never automatically update.
If the volume isn't empty, the volume contents completely replace what's in the image, even if it's changed.
The initialization happens only on native Docker, and not for example in Kubernetes.
The initialization happens only on named volumes, and not for bind-mounted host directories.
With all of these caveats, I'd avoid relying on this functionality.
If you need to mount a volume into a container, assume it will be empty when your entrypoint or the main container command starts. If you need a particular directory layout or file structure there, an entrypoint script can create it; if you're expecting it to hold particular data, keep a copy of it somewhere else in your image and copy it in if it's not already there (or, perhaps, always).
(*) On native Linux you can find a filesystem location for it, but accessing this isn't a best practice. On other OSes this will be hidden inside a virtual machine or other opaque storage. If you need to directly access the data (or inject config files, or read log files) a docker run -v /host/path:/container/path bind mount is a better choice.
Volumes are part of neither the container nor the host. Well, technically everything resides in the host machine. But the docker directories are only accessible by users in "docker" group. The files in these directories are separately managed by docker.
"Volumes are stored in a part of the host filesystem which is managed by Docker (/var/lib/docker/volumes/ on Linux)."
Hence volumes are like the union of files under the docker container and the host itself. Any addition on either end will be added to the volume(/var/lib/docker/volumes), not hard copy, rather something like symbol link
As volumes can be shared across different containers, deleting a container does not cascade to the volumes associated with it.
To remove unused volumes:
docker volume prune .
According to the Docker documentation regarding the VOLUME instruction:
The docker run command initializes the newly created volume with any data that exists at the specified location within the base image. For example, consider the following Dockerfile snippet:
FROM ubuntu
RUN mkdir /myvol
RUN echo "hello world" > /myvol/greeting
VOLUME /myvol
This Dockerfile results in an image that causes docker run to create a new mount point at /myvol and copy the greeting file into the newly created volume.
I am unable to replicate this behaviour using this exact Dockerfile and running:
$ docker run --volume ~/testdir:/myvol 159b3387c7eb
The testdir directory is created by does not contain the expected greeting file.
What am I doing wrong ?
There are several types of volumes:
Host volume, which bind mounts a directory from the host into a container. Docker never initializes these. Whatever the state of the host becomes the directory contents in that directory. This is the type of volume you created. They have the downside of having uid/gid permission issues when the container uid/gid does not match that of your host user.
Anonymous volume, which is a named volume with a long guid for a name. There is no information of what the target of an anonymous volume was, especially when the container has been deleted, so you'll want to give your volumes a name at least. These are the default if you define a volume with VOLUME and then forget to define a volume when running the container.
Named volume, which is created by giving it a valid name rather than a path as a source. These are recommended for most scenarios, unless you need external access to your data with a host volume. The advantage of named volumes is that docker initializes them upon creation to the contents of your image at that location. This initialization only happens when the volume is first created, so changes to the image will not be reflected unless you have something like an entrypoint to copy data from another directory back out to your volume.
If you want to see a named volume with the data initialized, that would look like:
$ docker run --volume testvol:/myvol 159b3387c7eb
to create a volume called testvol.
When you declare a VOLUME that doesn't pre-populate the volume with anything, it just describes an intended mount-point.
When you call Docker with the --volume argument it replaces anything in that target directory with your local directory, effectively mounting that local directory on top of the target location. Anything in that original location is obscured by the mount and inaccessible.
If you need to pre-populate the volume with content either do it before you mount it, or have an ENTRYPOINT script that initializes that directory when the container boots up.
I want to mount a folder of the host system to the container and it need to be defined in the Dockerfile so that user doesn't need to do it manually by passing the argument in the command line to run the container. How to achieve this ?
This simply cannot be done. Docker images are designed to be portable. Host mounts are host specific. Thus if you are able to specify a host mount at build time, it will make the image non-portable across machine that don't have this mount folder. Thus this is why this option is not available.
You can use docker compose to help the user not choose the mount folder. Take a look at How do I mount a host directory as a volume in docker compose
Dockerfile is for create images not containers.
You can not define a volume on a image. The volume must be defined on execution time when the container is created.
docker run -v /host-folder:/root/containerfolder -i -t imagename
Is there any difference between:
Mounting a host directory into a container path (the path is not exposed as a volume), e.g.:
docker run -v /host/directory:/container/directory my_image command_to_run
Dockerfile of my_image does not include VOLUME ["/container/directory"]
Mounting a host directory into a a container path exposed as a volume:
docker run -v /host/directory:/container/directory my_image command_to_run
Dockerfile of my_image includes VOLUME ["/container/directory"]
I know that volume data persists independent of the container life-cycle. However, since I want to work on my local data from within a container, does that make any difference if the mount-point inside the container is a volume?
There is no difference if you mount the path from the host into the container. The filesystem from the host will be mounted over top of that directory inside the container.
The difference between listing the volume and not listing it inside the image is the behavior of docker when you create an image without specifying a volume. When the volume is defined on the image, docker will create an "anonymous" volume you can see with docker volume ls as a long uuid string. These volumes are rarely useful, so I recommend against defining a volume in the image and instead only defining them on your docker run command or docker-compose.yml definition.
Downsides of defining a volume in the image include:
Later lines in the Dockerfile or in descendant Dockerfile's may not be able to change the contents at this location. Docker's behavior with this varies by scenario and version, so for predictability, once a volume is defined in an image, I consider that directory off limits.
Creation of anonymous volumes are difficult to use and are likely to clutter up the filesystem.
I posted a blog on this topic a while back if you're interested in more details.
I've searched the docs but nothing came up so time to test it. But for a quick future reference...
Is the host folder populated with the container folder contents?
Is it the opposite?
Are both folder contents merged? (In that case: What happens when a file with the same name is in both folders?)
Or does it produce an error? Is the error thrown on launch or is it thrown when you try to build an image with a VOLUME pointing to an existing populated folder on the container?
Also, another thing that isn't in the docs: Do I have to define the container path as a VOLUME in the Dockerfile in order to use -v against it when launching the container or can I create volumes on the fly?
When you run a container and mount a volume from the host, all you see in the container is what is on the host - the volume mount points at the host directory, so if there was anything in the directory in the image it gets bypassed.
With an image from this Dockerfile:
FROM ubuntu
WORKDIR /vol
RUN touch /vol/from-container
VOLUME /vol
When you run it without a host mount, the image contents get copied into the volume:
> docker run vol-test ls /vol
from-container
But mount the volume from the host and you only see the host's content:
> ls $(pwd)/host
from-host
> docker run -v $(pwd)/host:/vol vol-test ls /vol
from-host
And no, you don't need the VOLUME instruction. The behaviour is the same without it.
Whenever a Docker container is created with a volume mounted on the host, e.g.:
docker run -v /path/on/host:/data container-image
Any contents that are already in /data due to the image build process are always completely discarded, and whatever is currently at /path/on/host is used in its place. (If /path/on/host does not exist, it is created as an empty directory, though I think some aspect of that behavior may currently be deprecated.)
Pre-defining a volume in the Dockerfile with VOLUME is not necessary; all VOLUME does is cause any containers run from the image to have an implicit -v /volume/path (Note lack of host mount path) argument added to their docker run command which is ignored if an explicit -v /host/path:/volume/path is used.