Mixing named volumes and bind mounting in Docker? - docker

How does mixing named volumes and bind mounts work? Using the following setup will the paths that are being bind mounted still be available inside the bind mount as they exist in the bind mount?
/var/www/html/wp-content/uploads
Using a separate container which I attach to the named volumes, seems to show that it is not the case as those paths are completely empty from the view of the separate container. Is there a way for this to work in a sense?
volumes:
- "wordpress:/var/www/html"
- "./wordpress/uploads:/var/www/html/wp-content/uploads"
- "./wordpress/plugins:/var/www/html/wp-content/plugins"
- "./wordpress/themes:/var/www/html/wp-content/themes"

Host volumes: For a host volume, defined with a path in your docker compose file like:
volumes:
- "./wordpress/uploads:/var/www/html/wp-content/uploads"
you will not receive any initialization of the host directory from the image contents. This is by design.
Named volumes: You can define a named volume that maps back to a local directory:
version: "2"
services:
your-service:
volumes:
- uploads:/var/www/html/wp-content/uploads
volumes:
uploads:
driver: local
driver_opts:
type: none
o: bind
device: /path/on/host/to/wordpress/uploads
This will provide the initialization properties of a named volume. When your host directory is empty empty, on container creation docker will copy the contents of the image at /var/www/html/wp-content/uploads to /path/on/host/to/wordpress/uploads.
Nested mounts with Docker: If you have multiple nested volume mounts, docker will still copy from the image directory contents, not from a parent volume.
Here's an example of that initialization. Starting with the filesystem:
testvol/
data-image/
sub-dir/
from-image
data-submount/
Dockerfile
docker-compose.yml
The Dockerfile contains:
FROM busybox
COPY data-image/ /data
The docker-compose.yml contains:
version: "2"
services:
test:
build: .
image: test-vol
command: find /data
volumes:
- data:/data
- subdir:/data/sub-dir
volumes:
data:
subdir:
driver: local
driver_opts:
type: none
o: bind
device: /path/on/host/test-vol/data-submount
And the named volume has been initialized:
$ docker run -it --rm -v testvol_data:/data busybox find /data
/data
/data/sub-dir
/data/sub-dir/from-named-vol
Running the test shows the copy comes from-image rather than from-named-vol:
$ docker-compose -f docker-compose.bind.yml up
...
Attaching to testvol_test_1
test_1 | /data
test_1 | /data/sub-dir
test_1 | /data/sub-dir/from-image
testvol_test_1 exited with code 0
And docker has copied this to the host filesystem:
$ ls -l data-submount/
total 0
-rw-r--r-- 1 root root 0 Jan 15 08:08 from-image
Nested mounts in Linux: From your question, there appears to be some confusion on how a mount itself works in Linux. Each volume mount runs in the container's mount namespace. This namespace gives the container its own view of a filesystem tree. When you mount a volume into that tree, you do not modify the contents from the parent filesystem, it simply covers up the contents of the parent at that location. All changes happen directly in that newly mounted directory, and if you were to unmount it, the parent directories will then be visible in their original state.
Therefore, if you mount two nested directories in one container, e.g. /data and /data/a, and then mount /data in a second container, you will not see /data/a from your first container in the second container, only the contents of /data will be there, including any folders that were mounted over top of.

I believe the answer is to configure bind propagation.
will report back.
Edit: Seems you can only configure bind propagation on bind mounted volumes and only on linux host system.

I've tried to get this to work for hours, but I've come to the conclusion that it just won't. My case was adding a specific plugin to a CMS as a volume for local development. I want to post this here because I haven't come across this workaround anywhere.
So the following would suffer from the overlapping volumes issue, causing the folders to be empty.
services:
your-service:
volumes:
- web-data:/var/www/html
- ./wordpress/plugins:/var/www/html/wp-content/plugins
- ./wordpress/themes:/var/www/html/wp-content/themes
This is how you avoid that, by binding your themes and plugins to a different directory, not inside /var/www/html.
services:
your-service:
volumes:
- web-data:/var/www/html
- ./wordpress/plugins:/tmp/plugins
- ./wordpress/themes:/tmp/themes
But now you have to get these files in the correct place, and have them still be in sync with the files on your host.
Simple version
Note: These examples assume you have a shell script as your entrypoint.
In your Docker entrypoint:
#!/bin/bash
ln -s /tmp/plugins/my-plugin /var/www/html/wp-content/plugins/my-plugin
ln -s /tmp/themes/my-theme /var/www/html/wp-content/themes/my-theme
This should work as long as your system/software resolves symlinks.
More modular solution
I only wrote this for plugins, but you could process themes the same way. This finds all plugins in the /tmp/plugins folder and symlinks them to /var/www/html/wp-content/plugins/<plugin>, without you having to write hard-coded folder/plugin names in your script.
#!/bin/bash
TMP_PLUGINS_DIR="/tmp/plugins"
CMS_PLUGINS_DIR="/var/www/html/wp-content/plugins"
# Loop through all paths in the /tmp/plugins folder.
for path in $TMP_PLUGINS_DIR/*/; do
# Ignore anything that's not a directory.
[ -d "${path}" ] || continue
# Get the plugin name from the path.
plugin="$(basename "${path}")"
# Symlink the plugin to the real plugins folder.
ln -sf $TMP_PLUGINS_DIR/$plugin CMS_PLUGINS_DIR/$plugin
# Anything else you might need to do for each plugin, like installing/enabling it in your CMS.
done

Related

Exclude sub-folder when mounting host to volume docker [duplicate]

Supposed I have a Docker container and a folder on my host /hostFolder. Now if I want to add this folder to the Docker container as a volume, then I can do this either by using ADD in the Dockerfile or mounting it as a volume.
So far, so good.
Now /hostFolder contains a sub-folder, /hostFolder/subFolder.
I want to mount /hostFolder into the Docker container (whether as read-write or read-only does not matter, works both for me), but I do NOT want to have it included /hostFolder/subFolder. I want to exclude this, and I also want the Docker container be able to make changes to this sub-folder, without the consequence of having it changed on the host as well.
Is this possible? If so, how?
Using docker-compose I'm able to use node_modules locally, but ignore it in the docker container using the following syntax in the docker-compose.yml
volumes:
- './angularApp:/opt/app'
- /opt/app/node_modules/
So everything in ./angularApp is mapped to /opt/app and then I create another mount volume /opt/app/node_modules/ which is now empty directory - even if in my local machine ./angularApp/node_modules is not empty.
If you want to have subdirectories ignored by docker-compose but persistent, you can do the following in docker-compose.yml:
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules
This will mount your current directory as a shared volume, but mount a persistent docker volume in place of your local node_modules directory. This is similar to the answer by #kernix, but this will allow node_modules to persist between docker-compose up runs, which is likely the desired behavior.
For those trying to get a nice workflow going where node_modules isn't overridden by local this might help.
Change your docker-compose to mount an anonymous persistent volume to node_modules to prevent your local overriding it. This has been outlined in this thread a few times.
services:
server:
build: .
volumes:
- .:/app
- /app/node_modules
This is the important bit we were missing. When spinning up your stack use docker-compose -V. Without this if you added a new package and rebuilt your image it would be using the node_modules from your initial docker-compose launch.
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
To exclude a file, use the following
volumes:
- /hostFolder:/folder
- /dev/null:/folder/fileToBeExcluded
With the docker command line:
docker run \
--mount type=bind,src=/hostFolder,dst=/containerFolder \
--mount type=volume,dst=/containerFolder/subFolder \
...other-args...
The -v option may also be used (credit to Bogdan Mart), but --mount is clearer and recommended.
First, using the ADD instruction in a Dockerfile is very different from using a volume (either via the -v argument to docker run or the VOLUME instruction in a Dockerfile). The ADD and COPY commands just take a copy of the files at the time docker build is run. These files are not updated until a fresh image is created with the docker build command. By contrast, using a volume is essentially saying "this directory should not be stored in the container image; instead use a directory on the host"; whenever a file inside a volume is changed, both the host and container will see it immediately.
I don't believe you can achieve what you want using volumes, you'll have to rethink your directory structure if you want to do this.
However, it's quite simple to achieve using COPY (which should be preferred to ADD). You can either use a .dockerignore file to exclude the subdirectory, or you could COPY all the files then do a RUN rm bla to remove the subdirectory.
Remember that any files you add to image with COPY or ADD must be inside the build context i.e. in or below the directory you run docker build from.
for the people who also had the issue that the node_modules folder would still overwrite from your local system and the other way around
volumes:
node_modules:
services:
server:
volumes:
- .:/app
- node_modules:/app/node_modules/
This is the solution, With the trailing / after the node_modules being the fix.
Looks like the old solution doesn't work anymore(at least for me).
Creating an empty folder and mapping target folder to it helped though.
volumes:
- ./angularApp:/opt/app
- .empty:/opt/app/node_modules/
I found this link which saved me: Working with docker bind mounts and node_modules.
This working solution will create a "exclude" named volume in docker volumes manager. The volume name "exclude" is arbitrary, so you can use a custom name for the volume intead exclude.
services:
node:
command: nodemon index.js
volumes:
- ./:/usr/local/app/
# the volume above prevents our host system's node_modules to be mounted
- exclude:/usr/local/app/node_modules/
volumes:
exclude:
You can see more infos about volumes in Official docs - Use a volume with docker compose
To exclude a mounted file contained in the volume of your machine, you will have to overwrite it by allocating a volume to this same file.
In your config file:
services:
server:
build : ./Dockerfile
volumes:
- .:/app
An example in you dockerfile:
# Image Location
FROM node:13.12.0-buster
VOLUME /app/you_overwrite_file

Shared volume between docker-compose files

I want to shared a volume between 2 docker-compose files. There are 2 interconnected apps and I need to create a symlink between them.
I tried using named volumes and the external feature.
On the first container, I can see the contents of the /var/www/s1 folder, but the /var/www/s2 folder is empty, while on the second container I can see the contents of the /var/www/s2 folder, but the /var/www/s1 folder seems empty. Since I can't see the contents of the folder created by the other app in /var/www, I can't do a symlink.
I made some dummy docker-compose files to try to expose the problem in an easier way.
In /var/www/s1 there should be a "magazine.txt" file, while in /var/www/s2 there should be a "paper.txt" file.
The first docker-compose file looks like this:
services:
nginx:
image: nginx
container_name: nginx
volumes:
- ../:/var/www/s1
- shared-s:/var/www
volumes:
shared-s:
name: shared-s
The second docker-compose file looks like this:
version: '3.8'
services:
php:
image: php
container_name: php
command: tail -F /var/www/s2/paper.txt
volumes:
- ../:/var/www/s2
- shared-s:/var/www
volumes:
shared-s:
external:
name: shared-s
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
80b83a60a0e5 php "docker-php-entrypoi…" 2 seconds ago Up 1 second php
05addf1fc24e nginx "/docker-entrypoint.…" 9 seconds ago Up 8 seconds 80/tcp nginx
8c596d21cf7b portainer/portainer "/portainer" 2 hours ago Up About a minute 9000/tcp, 0.0.0.0:9001->9001/tcp portainer
$ docker exec -it 05addf1fc24e sh
# cd /var/www
# ls
s1 s2
# cd s1
# ls
docker magazine.txt
# cd ..
# cd s2
# ls
# exit
$ docker exec -it 80b83a60a0e5 sh
# cd /var/www
# ls
s1 s2
# cd s1
# ls
# cd ..
# cd s2
# ls
docker paper.txt
# exit
At a mechanical level, volumes and bind mounts don't "nest" in the way you're suggesting. The named volume shared-s will wind up containing only empty directories s1 and s2, but none of the content from either host directory.
What happens is something like this:
Docker starts (say) the nginx container first. It sorts the volumes: mounts on that container from shortest to longest.
Since the shared-s volume is empty, the content from the nginx base image in /var/www is copied to the volume; then the volume is mounted on /var/www in the container.
Docker creates the mount point /var/www/s1 (in the volume), then bind-mounts the host directory there (without modifying the volume at all).
Docker starts the php container and sorts its volumes:.
Since the shared-s volume is not empty, Docker just mounts it into the container, hiding any content that might have been in /var/www in the image.
Docker creates the mount point /var/www/s2 (in the volume), then bind-mounts the host directory there (without modifying the volume at all).
You'll notice a couple of problems with this sequence. Other mounted volumes' content never gets copied into the "shared" volume, which breaks the file sharing you're attempting here. Whichever container starts up first copies content from its image into the shared volume, and anything in that image in the other container gets lost. For that matter, if there is an update in the base image, Docker will ignore that in favor of the (old) content that's in the shared volume.
I'd suggest avoiding volumes here entirely. Build a separate image for each container, COPYing your application code into it. If you can use a static file server in the backend application, that will be much easier than trying to copy files from one container to the other. If that's not avoidable, you can use the COPY --from=image syntax that's normally used with multi-stage builds to also copy content from one built image to another.

Mounted directory empty with docker-compose and custom Dockerfile

I am very (read very) new to Docker so experimenting. I have created a very basic Dockerfile to pull in Laravel:
FROM composer:latest
RUN composer_version="$(composer --version)" && echo $composer_version
RUN composer global require laravel/installer
WORKDIR /var/www
RUN composer create-project --prefer-dist laravel/laravel site
My docker-compose.yml file looks like:
version: '3.7'
services:
laravel:
build:
context: .
dockerfile: laravel.dockerfile
container_name: my_laravel
network_mode: host
restart: on-failure
volumes:
- ./site:/var/www/site
When I run docker-compose up, the ./site directory is created but the contents are empty. I've put this in docker-compose as I plan on on including other things like nginx, mysql, php etc
The command:
docker run -v "/where/i/want/data/site:/var/www/site" my_laravel
Results in the same behaviour.
I know the install is successful as I modified my dockerfile with the follwing two lines appended to it:
WORKDIR /var/www/site
RUN ls -la
Which gives me the correct listing.
Clearly misunderstanding something here. Any help appreciated.
EDIT: So, I was able to get this to work... although, it slightly more difficult than just specifying a path..
You can accomplish this by specifying a volume in docker-compose.yml.. The path to the directory (on the host) is labeled as device in the compose file.. It appears that the root of the path has to be an actual volume (possibly a share would work) but the 'destination' of the path can be a directory on the specified volume..
I created a new volume called docker on my machine but I suppose you could do this with your existing disk/volume..
I am on a Mac and this docker-compose.yml file worked for me:
version: '3.7'
services:
nodemon-test:
container_name: my-nodemon-test
image: oze4/nodemon-docker-test
ports:
- "1337:1337"
volumes:
- docker_test_app:/app # see comment below on which name to use here
volumes:
docker_test_app: # use this name under `volumes:` for the service
name: docker_test_app
driver: local
driver_opts:
o: bind
type: none
device: /Volumes/docker/docker_test_app
The container specified exists in my DockerHub.. this is the source code for it, just in case you are worried about anything malicious. I created it like two weeks ago to help someone else on StackOverflow.
Shows files from the container on my machine (the host)..
You can read more about Docker Volume configs here if you would like.
ORIGINAL ANSWER:
It looks like you are trying to share the build directory with your host machine.. After some testing, it appears Docker will overwrite the specified path on the container with the contents of the path on the host.
If you run docker logs my_laravel you should see an error about missing files at /var/www/site.. So, even though the build is successful - once Docker mounts the directory from your machine (./site) onto the container (/var/www/site) it overwrites the path within the container (/var/www/site) with the contents of the path on your host (./site) - which is empty.
To test and make sure the contents of /var/www/site are in fact being overwritten, you can run docker exec -it /bin/bash (you may need to replace /bin/bash with /bash).. This will give you command line access inside of the container. From there you can do ls -a /var/www/site..
Furthermore, you can also pre-stage ./site to have a random test file in it (test.txt or whatever), then docker-compose up -d, then run the same commands from the step above docker exec -it ... and see if the staged test.txt file is now inside the container - this gives you definitive evidence that when you run volumes, the data on your host overwrites data in the container.
With that being said, doing something like this and sharing a log directory will work... the volume path specified on the container is still overwritten, the difference is the container is writing to that path.. it doesn't rely on it for config files/app files.
Hope this helps.

use volume defined in Dockerfile from docker-compose

I have for example this service and volume defined in my docker-compose file
postgres:
image: postgres:9.4
volumes:
- db_data:/var/lib/postgresql/data
volumes:
blue_prod_db:
driver: rancher-nfs
Then. if you define a volume inside a Dockerfile like this:
RUN mkdir /stuff
COPY ./stuff/* /stuff/
VOLUME /stuff
How can you later access it through the docker-compose configuration and add it to a container?
When configured in the Dockerfile, a volume will result in any container started from that image, including temporary containers later in the build process from the RUN command, to have a volume defined at the specified location, e.g. /stuff. If you do not define a source for that volume at run time, you will get an anonymous volume created by docker for you at that location. However, you can always define a volume with a source at run time (even without the volume being defined) by specifying the location in your compose file:
version: "3"
services:
app:
image: your_image
volumes:
- data:/stuff
volumes:
data:
Note that there are two volumes sections, one for a specific service that specifies where the volume is mounted inside the container, and another at the top level where you can specify the source of the volume. Without specifying a source, you'll get a local volume driver with a directory under /var/lib/docker bind mounted into the container.
I do not recommend specifying volumes inside the Dockerfile in general, it breaks the ability to extend the image in later steps for child images, and clutters the filesystem with anonymous volumes that are not easy to track back to their origin. It's best to define them at runtime with something like a compose file.

How to mount Docker directory into host directory with docker-compose

Imagine I have a Docker container containing some static data.
Now for development purpose I want the content of the container directory /resources mounted to my local working directory ..
docker-compose.yml:
version: '3.2'
services:
resources:
image: <private_registry>/resources:latest
volumes:
- ./resources:/resources
When running docker-compose up the folder resources is created in my working directory, but it has no content, whereas the container has content in /resources/
When using a named volume and inspecting it, it works like expected.
Docker provides initialization of the the volume source to the content of your image in a specific scenario:
It must be a named volume, not a host volume (mapping a path into the container)
The volume source must be empty, once there is data inside the directory it will not be changed by docker
Only on creation of the container (while the container is running it won't reinitialize the folder)
The option to disable the copy has not been set (this is the "nocopy" option in the compose file).
You are currently stuck at the first requirement, but it is possible to map any folder from the host into the container using a named volume that performs a bind mount. Here are some examples of three different ways to do that:
# create the volume in advance
$ docker volume create --driver local \
--opt type=none \
--opt device=/home/user/test \
--opt o=bind \
test_vol
# create on the fly with --mount
$ docker run -it --rm \
--mount type=volume,dst=/container/path,volume-driver=local,volume-opt=type=none,volume-opt=o=bind,volume-opt=device=/home/user/test \
foo
# inside a docker-compose file
...
volumes:
bind-test:
driver: local
driver_opts:
type: none
o: bind
device: /home/user/test
...
Your example would look more like:
version: '3.2'
services:
resources:
image: <private_registry>/resources:latest
volumes:
- resources:/resources
volumes:
resources:
driver: local
driver_opts:
type: none
o: bind
device: /full/path/to/resources
Note that this directory must exist on the host in advance. The bind mount will fail without it, and unlike a host mount, docker will not create it for you.
There are a couple of things here. First, when you mount a host directory it 'shades' any existing content on the given path, effectively replacing it with the contents of the mount. So, your resources directory on your host is hiding any content in your container.
There is no easy solution to your problem. When I want to edit files in a container and on the host, I keep the files on the host and mount them in the container. If I want a copy of a container, I mount a host dir to a different dir in the container and arrange for the files to be copied.

Resources