I would like to define a general mount volume - along with all the options I would like to have it associated - that can be reused across multiple services. In fact, I'm developing a project which uses the same source for several microservices.
That way, the volume will be simpler to manage and modify.
To start off, I used the old way which took advantage of volumes_from:
shared:
image: phusion/baseimage
volumes:
- ./code:/var/www/html
nginx:
build: docker/nginx
ports:
- "8080:80"
links:
- php
volumes_from:
- shared
This works, but I had to define a shared service to make it work. As of the 3.0 version, volumes can be used, so I would like to define a general volume and use it into my nginx service, but I'm not finding the right syntax:
version: '3.3'
volumes:
vol_test:
type: bind
source: ./code
target: /var/www/html
volume:
nocopy: true
services:
nginx:
build: docker/nginx
ports:
- "8080:80"
volumes:
- vol_test
Update
I've found that defining a volume the way I want could not be possible, since the following definition:
volumes:
data-volume:
type: bind
source: ./code
target: /var/www/html
volume:
nocopy: true
will produce this output when calling docker-compose up:
ERROR: The Compose file './docker-compose.yml' is invalid because:
volumes.data-volume value Additional properties are not allowed ('volume', 'source', 'type', 'target' were unexpected)
I guess I still have to use the volumes_from way then. Can anybody confirm that?
I can confirm your observation: If you want to mount a host directory, you'll have to use the bind mount syntax.
You do not have to use volume_from anymore. Form
https://docs.docker.com/compose/compose-file/#volumes:
But, if you want to reuse a volume across multiple services, then
define a named volume in the top-level volumes key. Use named volumes
with services, swarms, and stack files.
They even specifically address your issue:
Note: The top-level volumes key defines a named volume and references it from each service’s volumes list. This replaces
volumes_from in earlier versions of the Compose file format. See Use
volumes and Volume Plugins for general information on volumes.
So, for a MWE I defined an empty volume and referenced it in two services. This is the compose file:
version: '3.3'
volumes:
vol_test:
services:
reader:
image: ubuntu
tty: true
command: bash -c "sleep 1; cat /volume/file;exit 0"
volumes:
- vol_test:/volume
writer:
image: ubuntu
tty: true
command: bash -c "date | tee /volume/file; exit 0"
volumes:
- vol_test:/volume
This gives us the following behavior:
$ date; docker-compose up
So 27. Aug 11:54:13 CEST 2017
Creating network "dockercomposetest_default" with the default driver
Creating dockercomposetest_writer_1 ...
Creating dockercomposetest_reader_1 ...
Creating dockercomposetest_writer_1
Creating dockercomposetest_reader_1 ... done
Attaching to dockercomposetest_writer_1, dockercomposetest_reader_1
writer_1 | 27 09:54:15 UTC 2017
reader_1 | 27 09:54:15 UTC 2017
Gracefully stopping... (press Ctrl+C again to force)
Related
I am trying to externalise my runtime data from my applications to be saved in OpenMediaVault shared folder.
I was able to create shared folder and configure NFS or at least I think so. The config I see in OMV/Services/NFS/Shares is:
Shared folder: NasFolder[on /dev/sda1, nas/]
Client: 192.168.50.0/24
Privelage: Read/Write
Extra options: subtree_check,insecure
Now in that shared folder I have this structure(I checked it using windows SMB/CIFS config)
\\nfs-ip\NasFolder
|- mysql
| \- some my sql folders...
|- TEST.txt
I want to use this mysql folder to store MariaDB runtime data(I know names are messed up I am in a middle of a migration to Maria...). And meaby create some other folders for other services. This is my config from docker-compose.yml:
version: '3.2'
services:
mysqldb:
image: arm64v8/mariadb:latest
ports:
- 3306:3306
restart: on-failure:3
volumes:
- type: volume
source: nfs-volume
target: /mysql
volume:
nocopy: true
environment:
- MYSQL_ROOT_PASSWORD=my-secret-pw
command: --character-set-server=utf8 --collation-server=utf8_general_ci
volumes:
nfs-volume:
driver: local
driver_opts:
type: "nfs"
o: addr=192.168.50.70,nolock,soft,rw
device: ":/NasFolder"
Now when I run docker stack deploy -c docker-compose.yml --with-registry-auth maprealm on my manager node I get error on maprealm_mysqldb.1 that looks like this:
"Err": "starting container failed: error while mounting volume '/var/lib/docker/volumes/maprealm_nfs-volume/_data': failed to mount local volume: mount :/NasFolder:/var/lib/docker/volumes/maprealm_nfs-volume/_data, data: addr=192.168.50.70,nolock,soft: permission denied",
I am pretty new to integration stuff. This is my home server and I just can't find good tutorials that 'get through my thick skull' how to configure those NFS paths and permissions or at least how can I debug it beside just getting this error. I know that volumes.nfs-volume.driver_opts.device is supposed to be a path but I am not sure what path should that be.
I was trying to adapt config from here: https://gist.github.com/ruanbekker/4a9c0d250bce9f84482f2a788ce92131
Edit1) Few additional details:
Docker swarm has 3 nodes and only one node is manager with availability pause.
OMV is running on a separet machine that is not a part of a cluster
Ok so if someone would be looking for solution:
OMV by default has /export/ for NFS so volume needed to be updated. I needed to update volume for mysql and update volumes.mysql-volume.driver_opts.device to include that /export/ prefix and I also added path to mysql folder to have volume for mysqldb service use only:
volumes:
mysql-volume:
driver: local
driver_opts:
type: "nfs"
o: addr=192.168.50.70,nolock,soft,rw
device: ":/export/NasFolder/mysql"
After those changes there was need to update volume config on mysql/mariadb:
mysqldb:
image: arm64v8/mariadb:latest
ports:
- 3306:3306
restart: on-failure:3
volumes:
- type: volume
source: mysql-volume
target: /var/lib/mysql
volume:
nocopy: true
environment:
- MYSQL_ROOT_PASSWORD=my-secret-pw
command: --character-set-server=utf8 --collation-server=utf8_general_ci
mysqldb.volumes.source points to name of your volume defined in step 1 - mysql-volume
mysqldb.volumes.target is where inside container runtime data is stored. In mysql/mariadb databases runtime data is stored in /var/lib/mysql so you want to point to that and you can only use full path.
Since I used default OMV config there were problems with permissions. So I updated OMV/Services/NFS/Shares to this:
Shared folder: NasFolder[on /dev/sda1, nas/]
#here you can see note 'The location of the files to share. The share will be accessible at /export/.'
Client: 192.168.50.0/24
Privelage: Read/Write
Extra options: rw,sync,no_root_squash,anonuid=1000,anongid=1000,no_acl
I have this docker-compose.yml
version: 2.4
services:
foo:
image: ubuntu:focal
container_name: oof
volumes:
- ./foo:/foo
bar:
image: ubuntu:focal
container_name: rab
volumes_from:
- container:oof
With compose 1.29.2 and docker 20.10.8. But it does not start.
I get an error
ERROR: Service "bar" mounts volumes from "oof", which is not the name of a service or container.
How do I properly reference volumes by container name in compose?
The best way is to explicitly spell out the volumes you're reusing. If the foo image has VOLUME directives, you'll also mount the automatically-created anonymous volumes; you also have no control over where the volumes will be mounted, and if the two containers aren't similar enough, there could be conflicts. (This is essentially the logic behind Compose file version 3 removing volumes_from:.)
version: 2.4
services:
foo:
volumes:
- ./foo:/foo
bar:
volumes:
- ./foo:/foo # or a different container path if that suits your needs
In general in Compose, you shouldn't need to manually specify container_name:. Compose will generate unique container names, within the docker-compose.yml file if you need to refer to things you can use the service name, and there are docker-compose wrapper commands that know how to find the right container. If you do need to use the older volumes_from: syntax use the service name instead:
version: '2.4' # does not work in version 3
services:
foo:
image: ...
volumes:
- ./foo:/foo
# no container_name:
bar:
image: ...
volumes_from: foo
How to be with orphan images when you have 2 independent projects and you want them to work at the same time or at least to build running docker-compose up -d without --remove-orphans flag when images are already built for another project.
docker compose file1:
version: '2'
services:
applications:
image: tianon/true
volumes:
- ../../:/var/www/vhosts/project1
nginx:
build: ./images/nginx
image: project1/nginx:latest
ports:
- "80:80"
volumes_from:
- applications
networks:
appnet:
aliases:
- project1.app
- admin.project1.app
php:
image: project1/php:latest
ports:
- "7778:7778"
build:
context: ./images/php
dockerfile: Dockerfile
volumes_from:
- applications
networks:
- appnet
mysql:
image: project1/mysql:latest
build: ./images/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
volumes:
- mysqldata:/var/lib/mysql
networks:
- appnet
ports:
- "33066:3306"
workspace:
image: project1/workspace:latest
build:
context: ./images/workspace
volumes_from:
- applications
working_dir: /var/www/vhosts/project1
networks:
- appnet
networks:
appnet:
driver: "bridge"
volumes:
mysqldata:
driver: "local"
the second docker compose file:
version: '2'
services:
project2_applications:
image: tianon/true
volumes:
- ../../:/var/www/vhosts/project2
project2_nginx:
build: ./images/nginx
image: project2/nginx:latest
ports:
- "8080:80"
volumes_from:
- project2_applications
networks:
project2_appnet:
aliases:
- project2.app
- admin.project2.app
project2_php:
image: project2/php:latest
ports:
- "7777:7777"
build:
context: ./images/php
dockerfile: Dockerfile
volumes_from:
- project2_applications
networks:
- project2_appnet
project2_mysql:
image: project2/mysql:latest
build: ./images/mysql
environment:
MYSQL_ROOT_PASSWORD: secret
volumes:
- project2_mysqldata:/var/lib/mysql
networks:
- project2_appnet
ports:
- "33067:3306"
project2_workspace:
image: project2/workspace:latest
build:
context: ./images/workspace
volumes_from:
- project2_applications
working_dir: /var/www/vhosts/videosite
networks:
- project2_appnet
networks:
project2_appnet:
driver: "bridge"
volumes:
project2_mysqldata:
driver: "local"
And now when I have already built project1 and trying to run docker-compose up -d for the second project I see warning:
WARNING: Found orphan containers (docker_workspace_1, docker_nginx_1, docker_php_1, docker_mysql_1, docker_memcached_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
I have a supposition that it's because container names for project1 should be more specific and I need to add some prefixes like I'm doing for project2, but project1 is in use by many other developers and I do not want to change it.
Is there any way to turn off orphan check?
And the second thing: is just a warning message but for some reason, after it appearing compose is failing with error:
ERROR: Encountered errors while bringing up the project.
And to make it work I need to run docker-compose up -d --remove-orphans
Compose uses the project name (which defaults to the basename of the project directory) internally to isolate projects from each other. The project name is used to create unique identifiers for all of the project's containers and other resources. For example, if your project name is myapp and it includes two services db and web, then Compose starts containers named myapp_db_1 and myapp_web_1 respectively.
You get the "Found orphan containers" warning because docker-compose detects some containers which belong to another project with the same name.
To prevent different projects from interfering with each other (and suppress the warning) you can set a custom project name by using any of the following options:
The -p command line option.
COMPOSE_PROJECT_NAME environment variable. This environment variable can also be set via an environment file (.env in the current working directory by default).
Top-level name element in the Compose file. Note: if you pass multiple files to docker-compose via the -f option, then the value from the last file will be used.
docker-compose takes the name of the directory it is in as the default project name.
You can set a different project name by using -p or --project-name.
https://docs.docker.com/compose/reference/#use--p-to-specify-a-project-name
I had a similar problem because my projects all had the docker/docker-compose.yml structure.
To build on other answers, I create a .env file with my docker compose projects. I have a number of projects that all use the docker directory but are different projects.
To use docker-compose -p is a bit error prone, so creating .env file in the same directory as the docker-compose.yml:
-rw-rw-r-- 1 auser auser 1692 Aug 22 20:34 docker-compose.yml
-rw-rw-r-- 1 auser auser 31 Aug 22 20:44 .env
alleviates the necessary overhead of remembering -p.
In the .env file, I can now set the COMPOSE_PROJECT_NAME variable:
COMPOSE_PROJECT_NAME=myproject
On running:
docker-compose up -d
the COMPOSE_PROJECT_NAME is substituted without the use of -p.
Reference:
https://docs.docker.com/compose/env-file/
docker-compose up --remove-orphans
you can run this command to clean orphan containers. As specified in the warning
If the orphaned containers are expected and not intended to remove, you can set COMPOSE_IGNORE_ORPHANS variable to true.
Consise but just right away working source is here.
One option is to put it as a line into .env file next to docker-compose.yml like this:
COMPOSE_IGNORE_ORPHANS=True
Another option is pass or set it as an environment variable.
sh:
COMPOSE_IGNORE_ORPHANS=True docker-compose up -d
or
export COMPOSE_IGNORE_ORPHANS=True
docker-compose up -d
cmd:
SET COMPOSE_IGNORE_ORPHANS=True&& docker-compose up -d
powershell:
$env:COMPOSE_IGNORE_ORPHANS = 'True'; & docker-compose up -d
TL;DR
You can also add a unique name: myproject to each of your compose files.
My journey
In case this helps anybody else scrounging around to find help for the above issue (This is in support of the already good comments here):
I have several config files in the same directory
redis.yml
mariadb.yml
...
and I kept getting the same error about orphan containers when I ran
docker-compose -f <one of my configs>.yml up
as of now you can simply put each yml file into a separate project. This is simply done using the command like parameter "-p my_project_name" as has already been mentioned before. BUT the name must be in all lowercase!
This got me a little closer but I also kept forgetting that to bring the docker container down using docker-compose I needed to include that parameter as well.
For example to start the container:
docker-compose -p myproject-d redis.yml up -d
and to destroy the container
docker-compose -p myproject-d redis.yml down
Today I found that I can simply add the name: bit into the yml config. Here is an example for redis:
version: '3.9'
name: redis
services:
redis_0:
...
Now I can simply start the container with the following and don't have to worry about project names again:
docker-compose -f redis.yml <up/down>
This happens when your docker-compose file has got updated. I received similar error on Docker startup and found out that another team member updated the docker-compose.yml as part of cleanup.
To fix this, I deleted the docker group using the Delete button in Docker Desktop and started it again. This fixed the error for me.
As a complement for the existing answers, if you're using docker-compose with the -f option, to my surprise docker-compose will use the name of the parent folder of the first file passed via -f as the project name.
For example, assuming the following folder structure:
/
└── Users/
└── papb/
├── a.yml
└── foo/
└── b.yml
If you're in /Users and run docker-compose -f papb/a.yml -f papb/foo/b.yml:
The project name will be inferred as papb
Any relative paths you have in both files will be resolved against /Users/papb
If you're in /Users and run docker-compose -f papb/foo/b.yml -f papb/a.yml:
The project name will be inferred as foo
Any relative paths you have in both files will be resolved against /Users/papb/foo
If you're in /Users/papb and run docker-compose -f foo/b.yml -f a.yml:
The project name will be inferred as foo
Any relative paths you have in both files will be resolved against /Users/papb/foo
With v2 of docker-compose synthax, we were able to do something like this:
version: '2'
services:
app:
image: tianon/true
volumes:
- ../app:/var/www/app
nginx:
image: nginx
volumes_from:
- app
php:
image: php
volumes_from:
- app
In v3.2 volumes_from is now invalid option. The documentation is all for using new top-level volumes synthax, which is all the ways better.
I've read some comments on github, and the only solution that people propose is
version: '3.2'
services:
nginx:
image: nginx
volumes:
- app:/var/www/app
php:
image: php
volumes:
- app:/var/www/app
volumes:
app:
driver_opts:
type: none
device: ../app
o: bind
Which looks worse obviously, and it even doesn't work for me. It gives me an error: no such file or directory. So what else should I try? It seems like I can still use links instead of top-level volumes, but it's considered as legacy option in documentation. So how to do it right with new syntax?
EDIT:
Question has been identified as a possible duplicate, but I don't agree. See my comment bellow for explanation.
As the topic starter already mentions, volumes_from has been removed from the new docker-compose syntax, according to the documentation in favour of named volumes defined in the top level key volumes. The documentation also states the difference between volumes and bind mounts, one of which is who manages the contents:
By contrast, when you use a volume, a new directory is created within Docker’s storage directory on the host machine, and Docker manages that directory’s contents.
If this is the case, then it does not make sense to bind mount a host folder into a volume and let it be controlled by the host's file system and by Docker simultaneously.
If you still want to bind mount the same folder into two or more containers you could try something like:
version: '3.2'
services:
nginx:
image: nginx
volumes:
- type: bind
source: ../app
target: /var/www/app
php:
image: php
volumes:
- type: bind
source: ../app
target: /var/www/app
I'm trying to create a docker-compose.yml file that contains a --volumes-from instruction. Does anyone know the syntax?
I have been looking online for some time now, and it appears that the --volumes-from command is only available as a docker command. I hope I'm wrong.
Aug. 2022:
brandt points out in the comments to the updated docker-compose documentation.
Note August 2017: with docker-compose version 3, regarding volumes:
The top-level volumes key defines a named volume and references it from each service’s volumes list.
This replaces volumes_from in earlier versions of the Compose file format. See Use volumes and Volume Plugins for general information on volumes.
Example:
version: "3.2"
services:
web:
image: nginx:alpine
volumes:
- type: volume
source: mydata
target: /data
volume:
nocopy: true
- type: bind
source: ./static
target: /opt/app/static
db:
image: postgres:latest
volumes:
- "/var/run/postgres/postgres.sock:/var/run/postgres/postgres.sock"
- "dbdata:/var/lib/postgresql/data"
volumes:
mydata:
dbdata:
This example shows a named volume (mydata) being used by the web service, and a bind mount defined for a single service (first path under db service volumes).
The db service also uses a named volume called dbdata (second path under db service volumes), but defines it using the old string format for mounting a named volume.
Named volumes must be listed under the top-level volumes key, as shown.
February 2016:
The docs/compose-file.md mentions:
Mount all of the volumes from another service or container, optionally specifying read-only access(ro) or read-write(rw).
(If no access level is specified, then read-write will be used.)
volumes_from:
- service_name
- service_name:ro
- container:container_name
- container:container_name:rw
For instance (from this issue or this one)
version: "2"
services:
...
db:
image: mongo:3.0.8
volumes_from:
- dbdata
networks:
- back
links:
- dbdata
dbdata:
image: busybox
volumes:
- /data/db