I need to store the project on the mounted drive using the docker-compose file, placed at the root of that drive. I want to use docker volumes, not as bounded folders, but as named volumes. But I do not want to store it physically on the default location (/var/lib/docker/volume), because the system drive is small. How can I set up named volumes in docker-compose and ask the docker to create volumes in the same place where the docker-compose file is located?
Recommended: If system drive is small then you should move the docker data-root to some other location. This will keep compose file same across all users/systems.
Not Recommended: For having a named volume on desired location, you can specify driver_opts for the volume driver.
e.g. for local driver, declare top level volumes in compose file as
volumes:
my_volume:
driver: local
driver_opts:
type: none
device: "/full/host/path/"
o: bind
This change will apply to all systems your docker compose is used, with small system drives or large.
Related
I have a docker compose stack of several containers that all need to access the same network drive shared via cifs. The applications need access to several subfolders within this drive and the mount points cannot be changed, because they expect the folders to reside in specific locations.
I know it's not possible to access subfolders of named volumes such as named_vol:/subdir:/container_path, but I cannot seem to find a way around that. The only thing I can think of is to manually mount each folder as a network drive, but I would like to avoid that, as I would end up with 15-20 mounted network drives. Below is a partial example of what I need to achieve, were it possible to access subdirs of a named volume. How can I fix the below get the desired volume mapping?
node1:
...
volumes:
# - shared_vol:/data/media:/media
# - shared_vol:/tmp/:/tmp
...
node10:
...
volumes:
# - shared_vol:/data/media:/media
# - shared_vol:/data/config:/config
volumes:
shared_vol:
driver_opts:
type: "cifs"
o: "username=xxx,password=xxx,uid=xxx,gid=xxx,vers=3.0"
device: "//192.168.1.69/shared_vol"
Example (many options omitted for brevity):
version: "3"
volumes:
traefik:
driver: local
driver_opts:
type: nfs
o: "addr=192.168.1.100,soft,rw,nfsvers=4,async"
device: ":/volume/docker/traefik"
services:
traefik:
volumes:
- traefik/traefik.toml:/traefik.toml
This errors out as there is no volume with the name traefik/traefik.toml meaning that the volume name must be the full path to the file (i.e. you can't append a path to the volume name)?
Trying to set device: ":/volume/docker/traefik/traefik.toml" just returns a not a directory error.
Is there a way to take a single file and mount it into a container?
You cannot mount a file or sub-directory within a named volume, the source is either the named volume or a host path. NFS itself, along with most filesystems you'd mount in Linux, require you to mount an entire filesystem, not a single file, and when you get down to the inode level, this is often a really good thing.
The options remaining that I can think of are to mount the entire directory somewhere else inside your container, and symlink to the file you want. Or to NFS mount the directory to the host and do a host mount (bind mount) to a specific file.
However considering the example you presented, using a docker config would be my ideal solution, removing the NFS mount entirely, and getting a read only copy of the file that's automatically distributed to whichever node is running the container.
More details on configs: https://docs.docker.com/engine/swarm/configs/
I believe I found the issue!
Wrong:
volumes:
- traefik/traefik.toml:/traefik.toml
Correct:
volumes:
- /traefik/traefik.toml:/traefik.toml
Start the volume with "/"
What's the difference between declaring in the docker-compose.yml file a volume section and just using the volumes keyword under a service?
For example, I map a volume this way for a container:
services:
mysqldb:
volumes:
- ./data:/var/lib/mysql
This will map to the folder called data from my working directory.
But I could also map a volume by declaring a volume section and use its alias for the container:
services:
mysqldb:
volumes:
- data_volume:/var/lib/mysql
volumes:
data_volume:
driver: local
In this method, the actual location of where the mapped files are stored appears to be somewhat managed by docker compose.
What are the differences between these 2 methods or are they the same? Which one should I really use?
Are there any benefits of using one method over the other?
The difference between the methods you've described is that first method is a bind mount, and the other is a volume. These are more of Docker functions (rather than Docker Compose), and there are several benefits volumes provide over mounting a path from your host's filesystem. As described in the documentation, they:
are easier to back up or migrate
can be managed with docker volumes or the API (as opposed to the raw filesystem)
work on both Linux and Windows containers
can be safely shared among multiple containers
can have content pre-populated by a container (with bind mounts sometimes you have to copy data out, then restart the container)
Another massive benefit to using volumes are the volume drivers, which you'd specify in place of local. They allow you to store volumes remotely (i.e. cloud, etc) or add other features like encryption. This is core to the concept of containers, because if the running container is stateless and uses remote volumes, then you can move the container across hosts and it can be run without being reconfigured.
Therefore, the recommendation is to use Docker volumes. Another good example is the following:
services:
webserver_a:
volumes:
- ./serving/prod:/var/www
webserver_b:
volumes:
- ./serving/prod:/var/www
cache_server:
volumes:
- ./serving/prod:/cache_root
If you move the ./serving directory somewhere else, the bind mount breaks because it's a relative path. As you noted, volumes have aliases and have their path managed by Docker, so:
you wouldn't need to find and replace the path 3 times
the volume using local stores data somewhere else on your system and would continue mounting just fine
TL;DR: try and use volumes. They're portable, and encourage practices that reduce dependencies on your host machine.
In the Docker Compose documentation, here, you have the following example related to the volumes section of docker-compose.yml files:
volumes:
# (1) Just specify a path and let the Engine create a volume
- /var/lib/mysql
# (2) Specify an absolute path mapping
- /opt/data:/var/lib/mysql
# (3) Path on the host, relative to the Compose file
- ./cache:/tmp/cache
# (4) User-relative path
- ~/configs:/etc/configs/:ro
# (5) Named volume
- datavolume:/var/lib/mysql
Which syntaxes produce a bind mount and which produce a docker volume?
At some place of the documentation, the two concepts are strictly differentiated but at this place they are mixed together... so it is not clear to me.
Whenever you see "volume" in the comment, that will create a volume: so (1) and (5).
If there is not a volume in the comment, this is about a bind mount.
The documentation regarding volumes in docker-compose is here:
Mount host paths or named volumes, specified as sub-options to a service.
You can mount a host path as part of a definition for a single service, and there is no need to define it in the top level volumes key.
But, if you want to reuse a volume across multiple services, then define a named volume in the top-level volumes key.
The top-level volumes key defines a named volume and references it from each service’s volumes list. This replaces volumes_from in earlier versions of the Compose file format. See Use volumes and Volume Plugins for general information on volumes.
Those are two completely different concepts. A volume means that given directory will be persisted between container runs. Imagine MySQL database. You don’t want to lose your data. On the other hand there’s a bind mount where you attach your local directory to the directory in the container. If the container writes something there it will appear in your file system and vice versa (synchronization).
As a side note a volume is nothing more than a symlink to the directory on your machine :) (to a /var/lib/docker/volumes/... directory by default)
I have a docker-compose.yml file with the following:
volumes:
- .:/usr/app/
- /usr/app/node_modules
First option maps current host directory to /usr/app, but what does the second option do?
[Refreshing this answer since it seems others have similar questions]
There are three kinds of volumes in docker:
Host volumes: these map a path from the host into the container with a bind mount. They have the short syntax /path/on/host:/path/in/container. Whatever exists on the host is what will be visible in the container, there's no merging of files or initialization from the image, and uid/gid's do not get any special mapping so you need to take care to allow the container uid/gid read and write access to this location (an exception is Docker for Mac with OSXFS). If the path on the host does not exist, docker will create an empty directory as root, and if it is a file, you can mount a single file into the container this way.
Named volumes: these have a name, instead of a host path as the source. They have the short syntax name:/path/in/container and in a compose file, you also need to define the named volume used in containers at the top level. By default, these are also a bind mount, but to a docker specific directory under /var/lib/docker/volumes that should be considered internal. However these defaults can be changed to allow things like NFS mounts, mounting disks, or even your own bind mounts to other locations. Named volumes also have a feature in docker, when they are new or empty and first used, docker copies the contents from the image into named volume before mounting it. This includes files, directories, uid/gid owners, and permissions. After that, they behave identical to a host volume, whatever is inside the volume overlays the image location.
Anonymous volumes: these only have a path inside the container. They are in the form /path/in/container and docker will create a default named volume with a guid as the name. They share the behaviors of named volumes, storing files under /var/lib/docker/volumes, initializing with the contents of the image, except they have a randomly generated guid that gives you no indication of how or even if they are being used. You can mount the volume in another container and inspect the contents, or you can find the container using the volume by inspecting each container to find the guid. If you create a container with the --rm flag, anonymous volumes will also be deleted automatically.
tmpfs: Wait, I said 3, and this is 4? That's because tmpfs isn't considered a volume, the syntax to mount it is different. The result is a pointer to an empty in memory filesystem. This is useful if you have temporary files you don't wish to save, they are relatively small, and you either need speed or want to be sure they aren't saved to disk.
In the OP's case:
/usr/app is mounted from the host, commonly used for development
/usr/app/node_modules is an anonymous volume initialized from the image
Why do this? Likely because you do not want to modify the node_modules directory on the host, particularly if there's platform specific data and you're running on Docker desktop where it's Mac/Win on the host and Linux in the container. It's also possible there's data in the image you want to get access to within the directory structure of the other volume mount.
Are there downsides to anonymous volumes? Two that I can think of:
If there's anything in /usr/app/node_modules that you want to reuse in a future container, you're unlikely to find the old volume. I tend to consider any data written to these as likely lost.
You'll often find the volumes on the host full of guids over time, and it's unclear which are in use and which can be deleted. Unused anonymous volumes are one of several causes of excessive disk use in docker.
For more details on docker volumes, see: https://docs.docker.com/storage/
Original answer:
The second one creates an anonymous volume. It will be listed in docker volume ls with a long unique id rather than a name. Docker-compose will be able to reuse this if you update your image, but it's easy to lose track of which volume belongs to what with those names, so I recommend always giving your volume a name.
Just to complement the accepted answer, according to Docker's Knowledge Base there are three types of volumes: host, anonymous, and named:
A host volume lives on the Docker host's filesystem and can be
accessed from within the container. Example volume path:
/path/on/host:/path/in/container
An anonymous volume is useful for when you would rather have
Docker handle where the files are stored. It can be difficult,
however, to refer to the same volume over time when it is an
anonymous volumes. Example volume path:
/path/in/container
A named volume is similar to an anonymous volume. Docker manages
where on disk the volume is created, but you give it a volume name. Example volume path:
name:/path/in/container
The path used in your example is an anonymous volume.
I had the same question while I was going through this tutorial, and the answer to what those lines could actually be doing is this:
Without the anonymous volume ('/usr/src/app/node_modules'), the node_modules directory would essentially disappear by the mounting of the host directory at runtime:
Build - The node_modules directory is created.
Run - The current directory is copied into the container, overwriting the node_modules that were just installed when the container was built.
The docker-compose.yml file for this:
version: '3.5'
services:
something-clever:
container_name: something-clever
build:
context: .
dockerfile: Dockerfile
volumes:
- '.:/usr/src/app'
- '/usr/src/app/node_modules'
ports:
- '4200:4200'