Mount volume from host in Dockerfile long format - docker

When configuring a docker-compose.yml, I can easily mount a volume that maps a folder from the host machine to the container:
...
volumes:
- "/host/folder/:/container/folder/"
...
However, if I try to use the long syntax, that doesn't work anymore:
...
volumes:
- type: volume
source: /host/folder
target: /container/folder/
...
According to the docs
source: the source of the mount, a path on the host for a bind mount,
or the name of a volume defined in the top-level volumes key. Not
applicable for a tmpfs mount.
So, it seems that in the long syntax I have to use the bind type to mount a host path. Does this make sense, different features according to syntax?
Furthermore, again, according to the docs,
Volumes are the preferred mechanism for persisting data generated by
and used by Docker containers. While bind mounts are dependent on the
directory structure and OS of the host machine, volumes are completely
managed by Docker.
So, I much prefer having volumes instead of binds. Do I have to use the short syntax for that?

The type: field says whether it's a named volume, a bind mount, or a couple of other things. Since you're mounting a host directory, you need to specify type: bind in the extended syntax.
volumes:
- type: bind # <-- not "volume"
source: /host/folder
target: /container/folder/
according to the docs, "volumes are [...] preferred...."
IMHO the Docker documentation is very enthusiastic about named volumes and glosses over their downsides. Since you can't access the contents of a named volume from outside of Docker, they're harder to back up and manage, and a poor match for tasks like injecting config files and reviewing logs. I would not automatically reach for a named volume because the Docker documentation suggests it's preferred.
version: '3.8'
services:
some-application:
volumes:
# Use a bind mount to inject configuration files; you need to
# directly edit them on the host
- type: bind
source: ./config
target: /app/config
# Use a bind mount to read back log files; you need to read them
# on the host
- type: bind
source: ./log
target: /app/data
# Use a named volume for opaque application data; you do not
# need to manipulate files on the host, and on MacOS/Windows
# a named volume will be faster
- type: volume
source: app-data # when type: volume, this is a volume name
target: /app/data
# Do not mount anything over /app or the code in the image.
volumes:
app-data:

Try this:
...
services:
some-service:
image: some-image
volumes:
- my-volume-name:/some-path-in-container
volumes:
my-volume-name:
driver: local
driver_opts:
type: 'none'
o: 'bind'
device: '/my-host-folder'
...

Related

invalid mount config for type "bind": bind source path does not exist: /container/tdarr-server/server

I am trying to use an nfs share to use docker swarm with a single endpoint server for a drive, the NFS share its self does work as i can create files on it, however when trying to use it on a stack i get a bind source path error. my nfs share is set to /container on both machines so each machine can find it at the same location. here is what i have as a volume in my docker compose file:
volumes:
- /container/tdarr-server/server:/app/server
- /container/tdarr-server/configs:/app/configs
- /container/tdarr-server/logs:/app/logs
- /container/plex/media:/media
- /container/tdarr-server/transcode:/transcode
There are two ways to do this:
On each server mount a nfs share. Assuming you have a nfs server sharing a volume "docker_volumes" you could mount that as "/mnt/volumes"
Then your stack file could look like this:
version: "3.9"
volumes:
prometheus:
driver: local
driver_opts:
o: bind
type: none
device: /mnt/volumes/prometheus-data
services:
prometheus:
image: prom/prometheus:latest
volumes:
- prometheus:/data
NB. Docker will NOT create missing volumes for you. Each missing directory (e.g. ./prometheus-data) needs to be manually created on the nfs share before docker will start the service.
As an alternative you can - rather than pre-mounting the nfs volume in a defined location - provide docker with the nfs connection details so it can mount the nfs share on the fly:
volumes:
data:
driver_opts:
type: "nfs"
o: "addr=10.40.0.199,nolock,soft,rw"
device: ":/docker/example"
Again, if you provide a path into the nfs share as part of the device description, docker will not create it if it does not exist. The admin must pre-create any specific sub folders referenced before services or containers will be able to use the volume definition.

Share a FUSE mount between containers in docker compose without persisting anything on host

It's quite straightforward how to share a volume between containers when they share the FS, however it gets more complicated when one endpoint targets a directory that not only has a different FS but also this FS changes after the container runs, and the other one wants to replicate it (ro mode).
An important part is that the shared volume should not be stored on the host machine. It should only "live" between the containers and not be persistent.
The snippet below won't work, it's just to give an idea on what I am trying to achieve (see comments).
version: '3.7'
services:
# This service spins up with /mnt being an empty, unmounted dir.
# It then runs an executable that will mount a FUSE filesystem
# under /mnt
app:
# application
volumes:
- type: bind
source: shared_mnt
target: /mnt
bind:
propagation: shared
privileged: true
devices:
- '/dev/fuse'
# This service wants to read (read-only) the contents of this
# FUSE fs from `app` service, thus it should act as a slave
nginx:
# nginx
volumes:
- type: bind
source: shared_mnt
target: /whateve # a read-only replica of `app`'s /mnt dir (does not have to be a fuse FS)
bind:
propagation: shared
volumes:
shared_mnt:

Mounting a single file from an NFS docker volume into a container

Example (many options omitted for brevity):
version: "3"
volumes:
traefik:
driver: local
driver_opts:
type: nfs
o: "addr=192.168.1.100,soft,rw,nfsvers=4,async"
device: ":/volume/docker/traefik"
services:
traefik:
volumes:
- traefik/traefik.toml:/traefik.toml
This errors out as there is no volume with the name traefik/traefik.toml meaning that the volume name must be the full path to the file (i.e. you can't append a path to the volume name)?
Trying to set device: ":/volume/docker/traefik/traefik.toml" just returns a not a directory error.
Is there a way to take a single file and mount it into a container?
You cannot mount a file or sub-directory within a named volume, the source is either the named volume or a host path. NFS itself, along with most filesystems you'd mount in Linux, require you to mount an entire filesystem, not a single file, and when you get down to the inode level, this is often a really good thing.
The options remaining that I can think of are to mount the entire directory somewhere else inside your container, and symlink to the file you want. Or to NFS mount the directory to the host and do a host mount (bind mount) to a specific file.
However considering the example you presented, using a docker config would be my ideal solution, removing the NFS mount entirely, and getting a read only copy of the file that's automatically distributed to whichever node is running the container.
More details on configs: https://docs.docker.com/engine/swarm/configs/
I believe I found the issue!
Wrong:
volumes:
- traefik/traefik.toml:/traefik.toml
Correct:
volumes:
- /traefik/traefik.toml:/traefik.toml
Start the volume with "/"

docker volume type - bind vs volume

TLDR
In docker-compose, what's the difference between
volumes:
- type: volume
source: mydata
target: /data
and
volumes:
- type: bind
source: mydata
target: /data
?
The question in long:
When you specify the volumes option in your docker-compose file, you can use the long-syntax style
According to the docs, the type option accepts 3 different values: volume, bind and tmpfs:
I understand the tmpfs option - it means that the volume will not be saved after the container is down..
But I fail to find any reference in the docs about the difference between the other 2 options: bind and volume, could someone enlighten me about that?
When bind mounts are files coming from your host machine, volumes are something more like the nas of docker.
Bind mounts are files mounted from your host machine (the one that runs your docker daemon) onto your container.
Volumes are like storage spaces totally managed by docker.
You will find, in the literature, two types of volumes:
named volumes (you provide the name of it)
anonymous volumes (usual UUID names from docker, like you can find them on container or untagged images)
Those volumes come with their own set of docker commands; you can also consult this list via
docker volume --help
You can see your existing volumes via
docker volume ls
You can create a named volume via
docker volume create my_named_volume
But you can also create a volume via a docker-compose file
version: "3.3"
services:
mysql:
image: mysql
volumes:
- type: volume
source: db-data
target: /var/lib/mysql/data
volumes:
db-data:
Where this is the part saying please docker, mount me the volume named db-data on top of the container directory /var/lib/mysql/data
- type: volume
source: db-data
target: /var/lib/mysql/data
And this is the part saying to docker please create me a volume named db-data
volumes:
db-data:
Docker documentation about the three mount types:
https://docs.docker.com/storage/bind-mounts/
https://docs.docker.com/storage/volumes/
https://docs.docker.com/storage/tmpfs/
If I understood you correctly, you're asking in other words: What is the difference between Volumes and bind mounts?
Differences in management and isolation on the host
Bind mounts exist on the host file system and being managed by the host maintainer. Applications / processes outside of Docker can also modify it.
Volumes can also be implemented on the host, but Docker will manage them for us and they can not be accessed outside of Docker.
Volumes are a much wider solution
Although both solutions help us to separate the data lifecycle from containers,
by using Volumes you gain much more power and flexibility over your system.
With Volumes we can design our data effectively and decouple it from the host and other parts of the system by storing it dedicated remote locations (Cloud for example) and integrate it with external services like backups, monitoring, encryption and hardware management.
More Volumes advantages over bind mounts:
No host concerns.
Can be managed using Docker CLI.
Volumes can save you some uid/gid issues related permissions which occur in cases like when a container user's uid does not match the host gid.
A new volume’s contents can be pre-populated by a container.
Examples
Lets take 2 scenarios.
Case 1: Web server.
We want to provide our web server a configuration file that might change frequently. For example: exposing ports according to the current environment.
We can rebuild the image each time with the relevant setup or create 2 different images for each environment. Both of this solutions aren’t very efficient.
With Bind mounts Docker mounts the given source directory into a location inside the container.
(The original directory / file in the read-only layer inside the union file system will simply be overridden).
For example - binding a dynamic port to nginx:
version: "3.7"
services:
web:
image: nginx:alpine
volumes:
- type: bind #<-----Notice the type
source: ./mysite.template
target: /etc/nginx/conf.d/mysite.template
ports:
- "9090:8080"
environment:
- PORT=8080
command: /bin/sh -c "envsubst < /etc/nginx/conf.d/mysite.template >
/etc/nginx/conf.d/default.conf && exec nginx -g 'daemon off;'"
(*) Notice that this example could also be solved using Volumes.
Case 2 : Databases.
Docker containers do not store persistent data: any data that will be written to the writable layer in container’s union file system will be lost once the container stop running.
But what if we have a database running on a container, and the container stops - that means that all the data will be lost?
Volumes to the rescue.
Those are named file system trees which are managed for us by Docker.
For example - persisting Postgres SQL data:
services:
db:
image: postgres:latest
volumes:
- "dbdata:/var/lib/postgresql/data"
volumes:
- type: volume #<-----Notice the type
source: dbdata
target: /var/lib/postgresql/data
volumes:
dbdata:
Notice that in this case, for named volumes, the source is the name of the volume
(For anonymous volumes, this field is omitted).
Feature
Bind
Volume                                 
Internal soul
Bind mounts attach a user-specified location on host filesystem to a specific point in a container file tree.
Volume attach with disk storage on the host filesystem or cloud storage.
command
--mount type=bind,src="",dst=""
Docker CLI docker volume command
Dependency
dependent on location on to the host filesystem.
Container-independent data management
Separation of concerns
No
Yes
Conflict with other containers
Yes Example: multiple instances of Cassandra that all use the same host location as a bind mount for data storage. In that case, each of the instances would compete for the same set of files. Without other tools such as file locks, that would likely result in corruption of the database.
No. By default, Docker creates volumes by using the local volume plugin.
When to choose
1- Bind mounts are useful when the host provides a file or directory that is needed by a program running in a container, or when that containerized program produces a file or log that is processed by users or programs running outside containers. 2- appropriate tools for workstations, machines with specialized concerns 3- systems with more traditional configuration management tooling.
Working with Persistent storage 1. Databases 2. Cloud storage
When not to choose
Better to avoid these kinds of specific bindings in generalized platforms or hardware pools.
To be written

Relative path not working with named volumes in the docker-compose.yml

I need to make a named volume use a relative path to the folder where the docker-compose command is executed.
Here is the volume definition in the docker-compose.yml
volumes:
esdata1:
driver: local
driver_opts:
type: none
device: ./esdata1
o: bind
It seems that docker-compose do not create the folder if it does not exist, but even when the folder is created before lauching docker I'm always getting this error:
ERROR: for esdata Cannot create container for service esdata: error while mounting volume with options: type='none' device='./esdata1' o='bind': no such file or directory
NOTE: This is maybe silly, but esdata is the service that use the named volume
esdata:
...
volumes:
- esdata1:/usr/share/elasticsearch/data
...
What I'm missing here?
Maybe the relative path ./ does not point to the folder where the docker-compose is executed (I've tried with ~/ to use a folder relative the user's home but I got the same error).
Thanks in advance,
PS: If I use an absolute path it works like a charm
I encountered exactly the same issue. It seems that you have done nothing wrong. This is just not yet implemented in Docker : https://github.com/docker/compose/issues/6343
Not very nice for portability...
If you use a named bind mount like this, you must include the full path to the file, e.g.:
volumes:
esdata1:
driver: local
driver_opts:
type: none
device: /home/username/project/esdata1
o: bind
That folder must also exist in advance. This is how the linux bind mount syscall works, and when you pass flags like this, you are talking directly to Linux without any path expansion by docker or compose.
If you just want to mount the directory from the host, using a host volume will be expanded for the relative path by compose:
esdata:
...
volumes:
- ./esdata1:/usr/share/elasticsearch/data
...
While the host volume is easier for portability since the path is automatically expanded, you will lose the feature from named volumes where docker initializes an empty named volume with the image contents (including file permissions/ownership).

Resources