The docker-compose.yml contains a volume definition and a service that uses it.
#(This is a reduced but representative example)
services:
mongo:
image: mongo:3.6.11
...
volumes:
- "mongodb:/data/db/"
volumes:
mongodb:
driver: local
driver_opts:
type: "none"
o: "bind"
device: "$PWD/.storage/mongodb"
I've removed the entire .storage/mongodb directory, wanting to force the next docker-compose up not reuse any volume.
Instead, I received an error: docker was missing its volume data.
Creating mongo ... error
ERROR: for mongo Cannot create container for service mongo: failed to mount local volume: mount <...>/.storage/mongodb:/var/lib/docker/volumes/mongodb/_data, flags: 0x1000: no such file or directory
How I tried to resolve that:
docker-build up --build --force-recreate. Container's build and restart don't work.
docker volume ls -q | while read id; do docker volume rm "$id"; done. Removing volumes didn't seem to help either.
I assume that docker is able to recover from the removed volumes data, since the very first docker-compose up runs without initialised .storage directory.
Basically, I want docker-compose up to work.
It also would be fantastic to understand the issue.
Look forward to your help!
Have you tried docker-compose up with -V option?
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
Related
I am writing a docker-compose.yaml file for my project. I have checked the volumes documentation here .
I also understand the concept of volume in docker that I can mount a volume e.g. -v my-data/:/var/lib/db where my-data/ is a directory on my host machine while /var/lib/db is the path inside database container.
My confuse is with the link I put above. There it has the following sample:
version: "3.9"
services:
db:
image: db
volumes:
- data-volume:/var/lib/db
backup:
image: backup-service
volumes:
- data-volume:/var/lib/backup/data
volumes:
data-volume:
I wonder does it mean that I have to create a directory named data-volume on my host machine? What if I have a directory on my machine with path temp/my-data/ and I want to mount that path to the database container /var/lib/db ? Should I do something like below?
version: "3.9"
services:
db:
image: db
volumes:
- temp/my-data/:/var/lib/db
volumes:
temp/my-data/:
My main confusion is the volumes: section at the bottom, I am not sure whether the volume name should be the path of my directory or should be just literally a name I give & if it is the latter case then how could the given name be mapped with temp/my-data/ on my machine? The sample doesn't indicate that & is ambiguous to clarify that.
Could someone please clarify it for me?
P.S. I tried with above docker-compose I guessed, ended up with the error:
ERROR: The Compose file './docker-compose.yaml' is invalid because:
volumes value 'temp/my-data/' does not match any of the regexes: '^[a-zA-Z0-9._-]+$'
Mapped volumes can either be files/directories on the host machine (sometimes called bind mounts in the documentation) or they can be docker volumes that can be managed using docker volume commands.
The volumes: section in a docker-compose file specify docker volumes, i.e. not files/directories. The first docker-compose in your post uses such a volume.
If you want to map a file or directory (like in your last docker-compose file), you don't need to specify anything in the volumes: section.
Docker volumes (the ones specified in the volumes: section or created using docker volume create) are of course also stored somewhere on your host computer, but docker manages that and you shouldn't normally need to know where or what the format is.
This part of the documentation is pretty good about explaining it, I think https://docs.docker.com/storage/volumes/
As #HansKilian mentions, you don't need both volumes and services.volumes. To use services.volumes, map the host directory to the container directory like this:
services:
db:
image: db
volumes:
- /host/path/lib/db:/container/path/lib/db
With that, the directory /host/path/lib/db on the host machine will be used by the container and available at /container/path/lib/db.
Now, if you're like me, I get really confused with fake examples, so let's say the real directory on your host machine is /var/lib/db and you just want to see it at /db when you run a shell in Docker (i.e., docker exec -it /bin/bash container-id).
docker-compose.yaml would look like this:
services:
db:
image: db
volumes:
- /var/lib/db:/db
Now when you run the shell, cd /logs and ls, you'll see the same results as if you'd cd /var/lib/db on the host.
If you want to use the volumes section to indicate a global volume to use, you first have to create that volume using docker volume create. The documentation Hans linked includes steps to do this. The syntax of /host/path:/container/path is replaced by volume-name:/container/path. Then, once defined, you'd alter your docker-compose.yaml to be more like this:
services:
db:
image: db
volumes:
- your-global-volume-name:/db
volumes:
your-global-volume-name:
external: true
Note that I have not tested or used the this configuration. I'm assuming it's correct based on the other method working and the few changes I can identify in the docs.
In my Docker environment I have always to run the command to create volumes manually like
docker volume create --name= ...
I would like a way to speed up this process with a script shell which could help me to run at once.
If I could see a possible solution would be great as I have many volumes to create manually
A possible solution would be to use docker-compose and have a docker_compose.yml file composed only of volumes but no services:
version: "3.8"
volumes:
logvolume01: {}
logvolume02: {}
logvolume03: {}
When run, this creates the volumes accordingly:
$ docker-compose up
Creating volume "docker_logvolume01" with default driver
Creating volume "docker_logvolume02" with default driver
Creating volume "docker_logvolume03" with default driver
Attaching to
$ docker volume ls
DRIVER VOLUME NAME
local docker_logvolume01
local docker_logvolume02
local docker_logvolume03
If you need a more complex set of options while creating your volumes, you can find them in the documentation.
Just a little quirk to note here: per default, when you are using docker-compose, the volumes will be prefixed with the name of the folder you are in, this is done by Docker so there is no collision between different Docker projects.
This is the reason why, in the example above, the volumes are starting with docker_, because the folder I am in, is called docker.
To fix this, just give a name to your volumes:
version: "3.8"
volumes:
logvolume01:
name: logvolume01
logvolume02:
name: logvolume02
logvolume03:
name: logvolume03
Running this modified version gives:
$ docker-compose up
Creating volume "logvolume01" with default driver
Creating volume "logvolume02" with default driver
Creating volume "logvolume03" with default driver
Attaching to
$ docker volume ls
DRIVER VOLUME NAME
local logvolume01
local logvolume02
local logvolume03
I'm setting to set up a dokcer-compose file, and want to access a docker volume which is on another server. How do I specify that external volume in my docker-compose file?
I've tried using the driver_opts in my docker-compose file but without luck. I always get this error:
ERROR: for api Cannot start service api: error while mounting volume '/var/lib/docker/volumes/data-api-media': error while mounting volume with options: type='' device='remote-path' o='addr=...,rw': no such device
And with external I get:
Volume data-api-media declared as external, but could not be found. Please create the volume manually using docker volume create --name=data-api-media and try again.
version: '3.4'
services:
api:
build: .
entrypoint:
- ./docker-entrypoint.sh
volumes:
- data-api-media:/usr/src/app/media/
ports:
- "1095:1095"
volumes:
data-api-media:
driver_opts:
o: "addr=...,rw"
device: remote-path
I expect to mount the external docker volume from a different server to my docker-compose service and access the files in it.
You can create docker volume through vieux/sshfs plugin at your host and map to another host.
Use a volume driver
When you create a volume using docker volume create, or when you start a container which uses a not-yet-created volume, you can specify a volume driver. The following examples use the vieux/sshfs volume driver, first when creating a standalone volume, and then when starting a container which creates a new volume.
Initial set-up
This example assumes that you have two nodes, the first of which is a Docker host and can connect to the second using SSH.
On the Docker host, install the vieux/sshfs plugin:
$ docker plugin install --grant-all-permissions vieux/sshfs
Create a volume using a volume driver
This example specifies a SSH password, but if the two hosts have shared keys configured, you can omit the password. Each volume driver may have zero or more configurable options, each of which is specified using an -o flag.
$ docker volume create --driver vieux/sshfs \
-o sshcmd=test#node2:/home/test \
-o password=testpassword \
sshvolume
docker-compose setup
volumes:
- type: volume
driver: vieux/sshfs
source: sshvolume
target: /target
How does mixing named volumes and bind mounts work? Using the following setup will the paths that are being bind mounted still be available inside the bind mount as they exist in the bind mount?
/var/www/html/wp-content/uploads
Using a separate container which I attach to the named volumes, seems to show that it is not the case as those paths are completely empty from the view of the separate container. Is there a way for this to work in a sense?
volumes:
- "wordpress:/var/www/html"
- "./wordpress/uploads:/var/www/html/wp-content/uploads"
- "./wordpress/plugins:/var/www/html/wp-content/plugins"
- "./wordpress/themes:/var/www/html/wp-content/themes"
Host volumes: For a host volume, defined with a path in your docker compose file like:
volumes:
- "./wordpress/uploads:/var/www/html/wp-content/uploads"
you will not receive any initialization of the host directory from the image contents. This is by design.
Named volumes: You can define a named volume that maps back to a local directory:
version: "2"
services:
your-service:
volumes:
- uploads:/var/www/html/wp-content/uploads
volumes:
uploads:
driver: local
driver_opts:
type: none
o: bind
device: /path/on/host/to/wordpress/uploads
This will provide the initialization properties of a named volume. When your host directory is empty empty, on container creation docker will copy the contents of the image at /var/www/html/wp-content/uploads to /path/on/host/to/wordpress/uploads.
Nested mounts with Docker: If you have multiple nested volume mounts, docker will still copy from the image directory contents, not from a parent volume.
Here's an example of that initialization. Starting with the filesystem:
testvol/
data-image/
sub-dir/
from-image
data-submount/
Dockerfile
docker-compose.yml
The Dockerfile contains:
FROM busybox
COPY data-image/ /data
The docker-compose.yml contains:
version: "2"
services:
test:
build: .
image: test-vol
command: find /data
volumes:
- data:/data
- subdir:/data/sub-dir
volumes:
data:
subdir:
driver: local
driver_opts:
type: none
o: bind
device: /path/on/host/test-vol/data-submount
And the named volume has been initialized:
$ docker run -it --rm -v testvol_data:/data busybox find /data
/data
/data/sub-dir
/data/sub-dir/from-named-vol
Running the test shows the copy comes from-image rather than from-named-vol:
$ docker-compose -f docker-compose.bind.yml up
...
Attaching to testvol_test_1
test_1 | /data
test_1 | /data/sub-dir
test_1 | /data/sub-dir/from-image
testvol_test_1 exited with code 0
And docker has copied this to the host filesystem:
$ ls -l data-submount/
total 0
-rw-r--r-- 1 root root 0 Jan 15 08:08 from-image
Nested mounts in Linux: From your question, there appears to be some confusion on how a mount itself works in Linux. Each volume mount runs in the container's mount namespace. This namespace gives the container its own view of a filesystem tree. When you mount a volume into that tree, you do not modify the contents from the parent filesystem, it simply covers up the contents of the parent at that location. All changes happen directly in that newly mounted directory, and if you were to unmount it, the parent directories will then be visible in their original state.
Therefore, if you mount two nested directories in one container, e.g. /data and /data/a, and then mount /data in a second container, you will not see /data/a from your first container in the second container, only the contents of /data will be there, including any folders that were mounted over top of.
I believe the answer is to configure bind propagation.
will report back.
Edit: Seems you can only configure bind propagation on bind mounted volumes and only on linux host system.
I've tried to get this to work for hours, but I've come to the conclusion that it just won't. My case was adding a specific plugin to a CMS as a volume for local development. I want to post this here because I haven't come across this workaround anywhere.
So the following would suffer from the overlapping volumes issue, causing the folders to be empty.
services:
your-service:
volumes:
- web-data:/var/www/html
- ./wordpress/plugins:/var/www/html/wp-content/plugins
- ./wordpress/themes:/var/www/html/wp-content/themes
This is how you avoid that, by binding your themes and plugins to a different directory, not inside /var/www/html.
services:
your-service:
volumes:
- web-data:/var/www/html
- ./wordpress/plugins:/tmp/plugins
- ./wordpress/themes:/tmp/themes
But now you have to get these files in the correct place, and have them still be in sync with the files on your host.
Simple version
Note: These examples assume you have a shell script as your entrypoint.
In your Docker entrypoint:
#!/bin/bash
ln -s /tmp/plugins/my-plugin /var/www/html/wp-content/plugins/my-plugin
ln -s /tmp/themes/my-theme /var/www/html/wp-content/themes/my-theme
This should work as long as your system/software resolves symlinks.
More modular solution
I only wrote this for plugins, but you could process themes the same way. This finds all plugins in the /tmp/plugins folder and symlinks them to /var/www/html/wp-content/plugins/<plugin>, without you having to write hard-coded folder/plugin names in your script.
#!/bin/bash
TMP_PLUGINS_DIR="/tmp/plugins"
CMS_PLUGINS_DIR="/var/www/html/wp-content/plugins"
# Loop through all paths in the /tmp/plugins folder.
for path in $TMP_PLUGINS_DIR/*/; do
# Ignore anything that's not a directory.
[ -d "${path}" ] || continue
# Get the plugin name from the path.
plugin="$(basename "${path}")"
# Symlink the plugin to the real plugins folder.
ln -sf $TMP_PLUGINS_DIR/$plugin CMS_PLUGINS_DIR/$plugin
# Anything else you might need to do for each plugin, like installing/enabling it in your CMS.
done
Is this possible from docker-compose - or do I have identify the volumes location and delete them?
I can only find functionality to delete ALL the volumes associated with the docker-compose.yaml config file.
Compose can only remove all volumes defined in a yml file, or none
$ docker-compose down -v
If you don't want to investigate which volumes are used for the current compose file and remove them with docker volume rm X, then create a cutdown compose file that only lists the volumes you want to remove
docker-compose.yml
version: "2.1"
volumes:
one:
two:
three:
docker-compose-cleanup.yml
version: "2.1"
volumes:
two:
Then you can act on different sets depending on the compose file
$ docker-compose up
Creating volume "composevolumes_three" with default driver
Creating volume "composevolumes_two" with default driver
Creating volume "composevolumes_one" with default driver
Attaching to
$ docker-compose -f docker-compose-cleanup.yml down -v
Removing volume composevolumes_two