docker-compose v2 named volumes and volumes on host - docker

I am looking for guidance on how what is the cleanest way to make a docker-compose.yml version 2 that:
Has container state clearly separated from the container.
Has container state mounted to the host for simplicity (single data point, simply backup /data on the host and you're done). I'm open to be wrong about this, see questions below).
The app is a classic web app with a mysql & redis database for the backend, and with a webserver that is behind a proxy that serves static assets directly. Some details like depends_on, environment variables and the networks are intentionally left out.
Here is what I use at the moment:
version: "2"
services:
proxy:
build:
context: ./apps/nginx
ports:
- "80:80"
- "443:443"
volumes:
- /etc/localtime:/etc/localtime:ro
- ./data/web/assets:/var/www/assets:ro
- ./data/web/puma:/var/run/puma
web:
build:
context: ./apps/rails
volumes:
- /etc/localtime:/etc/localtime:ro
- ./data/web/assets:/srv/app/public/assets
- ./data/web/puma:/var/run/puma
db:
image: mysql:5.7
volumes:
- /etc/localtime:/etc/localtime:ro
- ./data/mysql:/var/lib/mysql
redis:
image: redis
volumes:
- /etc/localtime:/etc/localtime:ro
- ./data/redis:/data
Here is what I plan to use for the next release:
version: "2"
services:
proxy:
build:
context: ./apps/nginx
ports:
- "80:80"
- "443:443"
volumes_from:
- localtime
- web-assets-data:ro
- web-puma-data
web:
build:
context: ./apps/rails
volumes_from:
- localtime
- web-assets-data
- web-puma-data
db:
image: mysql:5.7
volumes_from:
- localtime
- db-data
redis:
image: redis
volumes_from:
- localtime
- redis-data
web-assets-data:
image: ubuntu:14.04
volumes:
- ./data/web/assets:/srv/app/public/assets
web-puma-data:
image: ubuntu:14.04
volumes:
- ./data/web/puma:/var/run/puma
db-data:
image: ubuntu:14.04
volumes:
- ./data/mysql:/var/lib/mysql
redis-data:
image: ubuntu:14.04
volumes:
- ./data/redis:/data
localtime:
image: ubuntu:14.04
volumes:
- /etc/localtime:/etc/localtime:ro
I think the benefits of the new version are:
It's more clear where the data is.
It's easier to share data among multiple containers (no need to remember the exact paths like in the current version).
So, my questions are:
Is it problematic to use different images between the container and it's container-data? for example, should db-data use mysql:5.7 instead of ubuntu:14.04?
Is it correct to say that there's no way of having "data stored at a specific path on the host" with a top level volumes: key?
What are the advantages and inconvenients of using a named volume (with a top-level "volumes" key)? Should I prefer using a named volume over a host mount? Workflow comparisons would be nice.

Is it problematic to use different images between the container and it's container-data
Not at all, this is normal.
Is it correct to say that there's no way of having "data stored at a specific path on the host" with a top level volumes: key?
Correct. The top level volumes key is for named volumes, but you can't name host volumes.
What are the advantages and inconveniences of using a named volume (with a top-level "volumes" key)? Should I prefer using a named volume over a host mount? Workflow comparisons would be nice.
Named volumes let you use volume drivers, so you could have the data stored somewhere other than the local filesystem. However named volumes need to be initialized with data, so you might have to add a script or something to do so.

Related

Docker volumes vs configs?

I start under docker and I do not understand the difference between volumes and configs.
https://docs.docker.com/compose/compose-file/#volumes-top-level-element
https://docs.docker.com/compose/compose-file/#configs-top-level-element (added in version 3.3 of docker compose)
Should the configs properties be defined if, for example, a configuration file is used on different services ?
In what case does it apply ?
For example to share the Apache Root Document (/usr/local/apache2/htdocs) with volumes :
version: '3.8'
services:
apache:
image: httpd:2.4
restart: always
ports:
- 8000:80
volumes:
- ./:/usr/local/apache2/htdocs
- php-socket:/run/php
depends_on:
- php-fpm
networks:
- code
php-fpm:
image: php:7.4-fpm
restart: always
ports:
- 9000:9000
volumes:
- ./:/usr/local/apache2/htdocs
- ./.docker/php-fpm/zz-docker.conf:/usr/local/etc/php-fpm.d/zz-docker.conf
- php-socket:/run/php
networks:
- code
volumes:
php-socket:
networks:
code:
What is the difference with the configs properties ? :
version: '3.8'
services:
apache:
image: httpd:2.4
restart: always
ports:
- 8000:80
volumes:
- php-socket:/run/php
configs:
- source: apache-www
target: /usr/local/apache2/htdocs
depends_on:
- php-fpm
networks:
- code
php-fpm:
image: php:7.4-fpm
restart: always
ports:
- 9000:9000
volumes:
- ./.docker/php-fpm/zz-docker.conf:/usr/local/etc/php-fpm.d/zz-docker.conf
- php-socket:/run/php
configs:
- source: apache-www
target: /usr/local/apache2/htdocs
networks:
- code
volumes:
php-socket:
configs:
apache-www:
file: ./
networks:
code:
The two examples above work, But I don't understand the difference between volumes and configs ?
Can anyone explain to me ? Thanks !
Configs were added for Swarm Mode. They are immutable objects added into swarm, that get pushed to the worker node when needed, and mounted as a file in the container. They solved the issue that the file you wanted to mount may not be on the worker node in the cluster.
If you aren't using Swarm Mode, you likely don't need any of the v3 syntax and can stick to either v2 or compose-spec and mount the file directly as a volume. You'll find newer version of compose have added compatibility features to treat compose files written for Swarm with comparable features in compose, effectively treating these as volume mounts.
It is the same thing.
The config attribute is just syntactic sugar and does the same thing as volumes.

How to config volumes in docker-compose.yml?

Is a very simple question I guess, but I could not find an answer.
This is my docker-compose.yml
version: '3'
services:
db:
volumes:
- db:/var/lib/mysql
volumes:
db:
nextcloud:
The question is, I want to specify the value of "db" or "nextcloud" in "volumes", and reference them in "Services".like this
services:
db:
volumes:
- db:/var/lib/mysql
nextcloud:
volumes:
- nextcloud:/var/www/html
volumes:
db: /home/roj/DataDisk/nextcloud-insecure/db
nextcloud: /home/roj/DataDisk/nextcloud-insecure/disk
but I got problemERROR: In file './docker-compose.yml', volume 'db' must be a mapping not a string.
how can i fix it ?
The top-level volumes section is not meant to specify mounts but volume driver configuration (see official documention on that matter). ie. this is incorrect
volumes:
db: /home/roj/DataDisk/nextcloud-insecure/db # incorrect
nextcloud: /home/roj/DataDisk/nextcloud-insecure/disk # incorrect
If you want to mount host directories to you container, you must specify it in the volumes section of your services, eg.
services:
db:
volumes:
- /home/roj/DataDisk/nextcloud-insecure/db:/var/lib/mysql
nextcloud:
volumes:
- /home/roj/DataDisk/nextcloud-insecure/disk:/var/www/html
See official documentation on services volumes for more information on that.
Your syntax in the outer volumes instruction is incorrect.
If you want to mount to a docker-managed volume, do this:
services:
test:
image: alpine
volumes:
- db:/app
volumes:
db:
If you want to mount to a local path, do this (you can replace the dot in .:/app with any other local path, like: /home/you:/server/path):
services:
test:
image: alpine
volumes:
- .:/app
If it starts with a dot or a slash, it will be treated as a path, otherwise, as a docker-managed named volume.
These are the common usage patterns, but you can read more about volumes in compose for some additional information.

How to create 2 different running app with the same docker-compose.yml file?

I already have a docker-compose.yml file like this:
version: "3.1"
services:
memcached:
image: memcached:alpine
container_name: dl-memcached
redis:
image: redis:alpine
container_name: dl-redis
mysql:
image: mysql:5.7.21
container_name: dl-mysql
restart: unless-stopped
working_dir: /application
environment:
- MYSQL_DATABASE=dldl
- MYSQL_USER=docker
- MYSQL_PASSWORD=docker
- MYSQL_ROOT_PASSWORD=docker
volumes:
- ./../:/application
ports:
- "8007:3306"
phpmyadmin:
image: phpmyadmin/phpmyadmin
container_name: dl-phpmyadmin
environment:
- PMA_ARBITRARY=1
- PMA_HOST=dl-mysql
- PMA_PORT=3306
- MYSQL_USER=docker
- MYSQL_PASSWORD=docker
- MYSQL_ROOT_PASSWORD=docker
restart: always
ports:
- 8002:80
volumes:
- /application
links:
- mysql
elasticsearch:
build: phpdocker/elasticsearch
container_name: dl-es
volumes:
- ./phpdocker/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
ports:
- "8003:9200"
webserver:
image: nginx:alpine
container_name: dl-webserver
working_dir: /application
volumes:
- ./../:/application:delegated
- ./phpdocker/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
- ./logs:/var/log/nginx:delegated
ports:
- "9003:80"
php-fpm:
build: phpdocker/php-fpm
container_name: dl-php-fpm
working_dir: /application
volumes:
- ./../:/application:delegated
- ./phpdocker/php-fpm/php-ini-overrides.ini:/etc/php/7.2/fpm/conf.d/99-overrides.ini
- ./../docker/php-fpm/certs/store_stock/:/usr/local/share/ca-certificates/
- ./logs:/var/log:delegated # nginx logs
- /application/var/cache
environment:
XDEBUG_CONFIG: remote_host=host.docker.internal
PHP_IDE_CONFIG: "serverName=dl"
node:
build:
dockerfile: dl/phpdocker/node/Dockerfile
context: ./../
container_name: dl-node
working_dir: /application
ports:
- "8008:3000"
volumes:
- ./../:/application:cached
tty: true
My goal is to have 2 isolate environments working at the same time in the same server with the same docker-compose file? I wonder if it's possible?
I want to be able to stop and update one env. while the other one is still running and getting the traffic.
Maybe I need another approach in my case?
There are a couple of problems with what you're trying to do. If your goal is to put things behind a load balancer, I think that rather than trying to start multiple instances of your project, a better solution would be to use the scaling features available to docker-compose. In particular, if your goal is to put some services behind a load balancer, you probably don't want multiple instances of things like your database.
If you combine this with a dynamic front-end proxy like Traefik, you can make the configuration largely automatic.
Consider a very simple example consisting of a backend container running a simple webserver and a traefik frontend:
---
version: "3"
services:
webserver:
build:
context: web
labels:
traefik.enable: true
traefik.port: 80
traefik.frontend.rule: "PathPrefix:/"
frontend:
image: traefik
command:
- --api
- --docker
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
ports:
- "80:80"
- "127.0.0.1:8080:8080"
If I start it like this, I get a single backend and a single frontend:
docker-compose up
But I can also ask docker-compose to scale out the backend:
docker-compose up --scale webserver=3
In this case, I get a single frontend and three backend servers. Traefik will automatically discover the backends and will round-robin connections between them. You can download this example and try it out.
Caveats
There are a few aspects of your configuration that would need to change in order to make this work (and in fact, you would need to change them even if you were to create multiple instances of your project as you have proposed in your question).
Conflicting paths
Take for example the configuration of your webserver container:
volumes:
- ./logs:/var/log/nginx:delegated
If you start two instances of this service, both containers will mount ./logs on /var/log/nginx. If they both attempt to write to /var/log/nginx/access.log, you're going to have problems.
The easiest solution here is to avoid bind mounts for things like log directories (and any other directories to which you will be writing), and instead use named docker volumes.
Hardcoding container names
In some places, you are hardcoding the container name, like this:
mysql:
image: mysql:5.7.21
container_name: dl-mysql
This will cause problems if you attempt to start multiple instances of this project or multiple instances of the mysql container. Don't statically set the container name.
Deprecated links syntax
Your configuration is using the deprecated links syntax:
links:
- mysql
Don't do that. In modern docker, containers on the same network can simply refer to each other by name. In other words, if your compose configuration has:
mysql:
image: mysql:5.7.21
restart: unless-stopped
working_dir: /application
environment:
- MYSQL_DATABASE=dldl
- MYSQL_USER=docker
- MYSQL_PASSWORD=docker
- MYSQL_ROOT_PASSWORD=docker
volumes:
- ./../:/application
ports:
- "8007:3306"
Other containers in your compose stack can simply use the hostname mysql to refer to this service.
You won't be able to run same compose file on a host without changing the port mappings because that will cause port conflict. I'd recommend creating a base compose file and using extends to override port mappings for different environments.

Confused about volumes_from and volumes usage in docker-compose.yml

I am trying to understand - maybe I already did maybe not - the differences between volumes_from and volumes usage in a docker-compose.yml file. I have read docs already but from there is not so clear to me so I am doing a real exercise.
I have the following setup:
a root directory
a directory named php-apache with a Dockerfile under root
a directory named mongo with a Dockerfile under root
a docker-compose.yml file under root
Note: If it's not clear to you, take a look here and everything exposed down below is right there as well (mongodb-test branch)
At php-apache/Dockerfile I have the following entry:
VOLUME /data /data
At mongo/Dockerfile I have the following entry:
VOLUME /data/db /data/configdb
At docker-compose.yml I have the following:
version: '2'
services:
php-apache:
container_name: "php55-dev"
image: reynierpm/php55-dev
ports:
- "80:80"
environment:
PHP_ERROR_REPORTING: 'E_ALL & ~E_DEPRECATED & ~E_NOTICE'
volumes:
- ~/mmi:/var/www
volumes_from:
- volumes_data
mongo:
container_name: "mongodb"
image: reynierpm/mongodb
ports:
- "27017:27017"
volumes_from:
- volumes_data
volumes_data:
image: tianon/true
volumes:
- ~/data/mongo:/data/db
- ~/data:/data
This is what I am understanding from that setup:
The image reynierpm/php55-dev will expose a /data directory and this will be mapped to ~data:/data in tianon/true image
The image reynierpm/mongodb will expose a /data/db to the outside and mapped to /data/configdb internally then the /data/db is mapped to ~/data/mongo:/data/db in tianon/true image.
Is a mess in my head right now because what do I want to achieve is the following:
Keep mapped the code on the host to the container (this line <path_on_host>:/var/www on docker-compose.yml)
Keep data stored on a local directory in the host
So, it's fine what I am doing? Feel free to made any modification at this setup since I am still learning.
The image reynierpm/php55-dev will expose a /data directory and this will be mapped to ~data:/data in tianon/true image
It's better to say it will mapped to your ~/data on docker host. Please note that there will be a /data/db from the second volume too.
The image reynierpm/mongodb will expose a /data/db to the outside and mapped to /data/configdb internally then the /data/db is mapped to ~/data/mongo:/data/db in tianon/true image.
This container will be the same as php-apache in terms of volumes from
volume_data container.
In case of your objectives:
If your code is in ~/mni/ you are fine. You are mounting mongoDB database directory to php-apache container, I don't think you need that.
You need to create a user defined network for your container connectivity or link containers (legacy). To create user defined network:
docker network create --driver bridge <yournetwork name>
You don't need a DOC. Thats why I removed the third container. I also fixed the unnecessary volume mappings.
Updated Docker file:
version: '2'
services:
php-apache:
container_name: "php55-dev"
image: reynierpm/php55-dev
ports:
- "80:80"
environment:
PHP_ERROR_REPORTING: 'E_ALL & ~E_DEPRECATED & ~E_NOTICE'
volumes:
- ~/mmi:/var/www
volumes_from:
- volumes_data
mongo:
container_name: "mongodb"
image: reynierpm/mongodb
ports:
- "27017:27017"
volumes_from:
- volumes_data
volumes_data:
image: tianon/true
volumes:
- ~/data/mongo:/data/db
- ~/data:/data
networks:
default:
external:
name: <your network name>
Please note you have to call your mongo container from your web application by its name, in your case mongodb.

Docker Compose - How to store database data?

I am new to docker and developing a project using docker compose. From the documentation I have learned that I should be using data only containers to keep data persistant but I am unable to do so using docker-compose.
Whenever I do docker-compose down it removes the the data from db but by doing docker-compose stop the data is not removed. May be this is because that I am not creating named data volume and docker-compose down hardly removes all the containers. So I tried naming the container but it threw me errors.
Please have a look at my yml file:
version: '2'
services:
data_container:
build: ./data
#volumes:
# - dataVolume:/data
db:
build: ./db
ports:
- "5445:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_DB=postgres
# - PGDATA=/var/lib/postgresql/data/pgdata
volumes_from:
# - container:db_bus
- data_container
geoserver:
build: ./geoserver
depends_on:
- db
ports:
- "8004:8080"
volumes:
- ./geoserver/data:/opt/geoserverdata_dir
web:
build: ./web
volumes:
- ./web:/code
ports:
- "8000:8000"
depends_on:
- db
command: python manage.py runserver 0.0.0.0:8000
nginx:
build: ./nginx
ports:
- "83:80"
depends_on:
- web
The Docker file for the data_container is:
FROM stackbrew/busybox:latest
MAINTAINER Tom Offermann <tom#offermann.us>
# Create data directory
RUN mkdir /data
# Create /data volume
VOLUME /data
I tried this but by doing docker-compose down, the data is lost. I tried naming the data_container as you can see the commented line, it threw me this error:
ERROR: Named volume "dataVolume:/data:rw" is used in service "data_container" but no declaration was found in the volumes section.
So right now what I am doing is I created a stand alone data only named container and put that in the volumes_from value of the db. It worked fine and didn't remove any data even after doing docker-compose down.
My queries:
What is the best approach to make containers that can store database's data using the docker-compose and to use them properly ?
My conscious is not agreeing with me on approach that I have opted, the one by creating a stand alone data container. Any thoughts?
docker-compose down
does the following
Stops containers and removes containers, networks, volumes, and images
created by up
So the behaviour you are experiencing is expected.
Use docker-compose stop to shutdown containers created with the docker-compose file but not remove their volumes.
Secondly you don't need the data-container pattern in version 2 of docker compose. So remove that and just use
db:
...
volumes:
- /var/lib/postgresql/data
docker-compose down stops containers but also removes them (with everything: networks, ...).
Use docker-compose stop instead.
I think the best approach to make containers that can store database's data with docker-compose is to use named volumes:
version: '2'
services:
db: #https://hub.docker.com/_/mysql/
image: mysql
volumes:
- "wp-db:/var/lib/mysql:rw"
env_file:
- "./conf/db/mysql.env"
volumes:
wp-db: {}
Here, it will create a named volume called "wp-db" (if it doesn't exist) and mount it in /var/lib/mysql (in read-write mode, the default). This is where the database stores its data (for the mysql image).
If the named volume already exists, it will be used without creating it.
When starting, the mysql image look if there are databases in /var/lib/mysql (your volume) in order to use them.
You can have more information with the docker-compose file reference here:
https://docs.docker.com/compose/compose-file/#/volumes-volume-driver
To store database data make sure your docker-compose.yml will look like
if you want to use Dockerfile
version: '3.1'
services:
php:
build:
context: .
dockerfile: Dockerfile
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
mysql-data:
your docker-compose.yml will looks like
if you want to use your image instead of Dockerfile
version: '3.1'
services:
php:
image: php:7.4-apache
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
if you want to store or preserve data of mysql then
must remember to add two lines in your docker-compose.yml
volumes:
- mysql-data:/var/lib/mysql
and
volumes:
mysql-data:
after that use this command
docker-compose up -d
now your data will persistent and will not be deleted even after using this command
docker-compose down
extra:- but if you want to delete all data then you will use
docker-compose down -v
to verify or check database data list by using this command
docker volume ls
DRIVER VOLUME NAME
local 35c819179d883cf8a4355ae2ce391844fcaa534cb71dc9a3fd5c6a4ed862b0d4
local 133db2cc48919575fc35457d104cb126b1e7eb3792b8e69249c1cfd20826aac4
local 483d7b8fe09d9e96b483295c6e7e4a9d58443b2321e0862818159ba8cf0e1d39
local 725aa19ad0e864688788576c5f46e1f62dfc8cdf154f243d68fa186da04bc5ec
local de265ce8fc271fc0ae49850650f9d3bf0492b6f58162698c26fce35694e6231c
local phphelloworld_mysql-data

Resources