Allowing multiple services in docker-compose to share a merged volume - docker

Question

Given a docker-compose.yml file like below, I'm looking for a way that both service a and b can have access to a shared volume which consists of the merged contents of both containers.
version: '3'
volumes:
shared-merged-volume:
services:
a:
volumes:
- shared-merged-volume:/shared
b:
volumes:
- shared-merged-volume:/shared
Let's say service a has a directory at /shared/dir-from-a and service b has a similar /shared-dir-from-b directory. The desired result is to end up with:
$ ls /shared # from either container
dir-from-a
dir-from-b
What I find is that one of the containers "wins" and only one of those two directories is ever present. I can work around the issue like this but is more verbose and requires modification if directory contents ever changes:
version: '3'
volumes:
service-a-shared-volume:
service-b-shared-volume:
services:
a:
volumes:
- service-a-shared-volume:/shared/dir-from-a
- service-b-shared-volume:/shared/dir-from-b
b:
volumes:
- service-a-shared-volume:/shared/dir-from-a
- service-b-shared-volume:/shared/dir-from-b
Thanks in advance for any help!

Answer 1

Is using a named volume a requirement?
If not, then to accomplish such merging I usually just map directories to one location on the host drive, instead of using volumes, and it merges with no problems. Tested on big loads and multiple containers writing simultaneously.
proposed compose file:
version: '3'
volumes:
shared-merged-volume:
services:
a:
volumes:
- /location/on/host/system:/shared
b:
volumes:
- /location/on/host/system:/shared
Edit from comments
This method mounts everything that's in the local host directory to the /shared, meaning if it's empty - it'll mount the empty dir, and whatever was there - will be overwritten by the empty dir. Everything that will be written inside that mount after your service starts, will be persisted and merged across services as expected.

Answer 2

If both containers are creating different folders, I don't see how they can be contending to create their own respective folders, unless they both delete the contents of /shared first, then they create the folders? But that would mean that the use of volumes in this case is null because the contents will be deleted every time the container starts?
In any case, I find that it is often useful to persuade the containers to share the same folder by use of path redirection. I will share two ways of accomplishing this:
If you have access to the code that creates the folders in /shared, then you can use environment variables to change the expected location of /shared for each service
version: '3'
volumes:
shared-merged-volume:
services:
a:
environment:
SHARED_VOLUME_PATH: /shared/a/
volumes:
- shared-merged-volume:/shared
b:
environment:
SHARED_VOLUME_PATH: /shared/b/
volumes:
- shared-merged-volume:/shared
You may need to have the services create SHARED_VOLUME_PATH, but now they can both live peaceably with each other.
If you are unable to change the location of /shared, which means each service will always want to use that path, another way to create path redirection is to use symbolic links. For this to work, you will have to override the entrypoint of your services or do this step during the build process of the image.
version: '3'
volumes:
shared-merged-volume:
services:
a:
entrypoint: [ "ln", "-sf", "/symshared/a/", "/shared/" ]
volumes:
- shared-merged-volume:/symshared
b:
entrypoint: [ "ln", "-sf", "/symshared/b/", "/shared/" ]
volumes:
- shared-merged-volume:/symshared
Alternatively, build the images ahead of time, and add a simple RUN command in the Dockerfile which creates this symbolic link:
...
ARG SHARED_VOLUME_PATH
RUN ln -sf ${SHARED_VOLUME_PATH} /shared/
What this allows you to do is that each container will keep using /shared as they used to, but you will still be able to store it's content in the volume, without interfering with what other containers want to do with their own version of /shared.
Needless to say, the ln command only works on linux and other unixes, and in some cases, you may need to install it prior. If your container image is based on something else like windows for example, then find something else that can be used to create symlinks.