I have my docker-compose as below:
version: '3.8'
services:
mongodb:
image: 'mongo'
restart: 'always'
volumes:
- data:/data/db
backend:
build: ./api
ports:
- '5000:5000'
volumes:
- upload:/app/upload
depends_on:
- mongodb
frontend:
build: ./client
ports:
- '3000:3000'
volumes:
- ./client:/app
- /app/node_modules
stdin_open: true
tty: true
depends_on:
- backend
volumes:
data:
upload:
After running docker-compose up -d it creates 4 volumes:
DRIVER VOLUME NAME
local 86ade5422b5ec2ab35fc3b2ea76099596d8b4a36f96c4428fbab4559ae5cff74
local de8c5f9680c5c4f73b6a3d4a1b3d4d2f962cc498d34a49dbfbbbda4cc7652288
local easyhome_data
local easyhome_upload
The 2 anonymous volumes seem to target to the /data/configdb and /app/node_modules
If I shut it down then run it again docker-compose down && docker-compose up -d it will create 2 more anonymous volumes identical to the previous two
DRIVER VOLUME NAME
local 5a1ead27ac78cee770aad636c156818c246377132b50a8b773b35be46146195f
local 86ade5422b5ec2ab35fc3b2ea76099596d8b4a36f96c4428fbab4559ae5cff74
local bc3063ff80f63f9e7ed5c1e319ee43a6d92ccdb2bfa9aec7e6be35378a566638
local de8c5f9680c5c4f73b6a3d4a1b3d4d2f962cc498d34a49dbfbbbda4cc7652288
local easyhome_data
local easyhome_upload
I dont want to run docker-compose down -v because it will remove my 2 named volumes as well. Please provide guidance!
You need to explicitly ask docker-compose to remove volumes:
docker-compose down -v
Reason is: volumes are meant to persist data useful to the user.
Read more about available options:
docker-compose down --help
Output:
Usage: docker compose down
Stop and remove containers, networks
Options:
--remove-orphans Remove containers for services not defined in the Compose file.
--rmi string Remove images used by services. "local" remove only images that don't have a custom tag ("local"|"all")
-t, --timeout int Specify a shutdown timeout in seconds (default 10)
-v, --volumes volumes Remove named volumes declared in the volumes section of the Compose file and anonymous volumes
attached to containers.
Related
I am using docker-compose and here is my docker-compose.yaml file:
version: "3.7"
services:
node:
container_name: my-app
image: my-app
build:
context: ./my-app-directoty
dockerfile: Dockerfile
command: npm run dev
environment:
MONGO_URL: my-database
port: 3000
volumes:
- ./my-app-directory/src:/app/src
- ./my-app-directory/node_modules:/app/node_modules
ports:
- "3000:3000"
networks:
- my-app-network
depends_on:
- my-database
my-database:
container_name: my-database
image: mongo
ports:
- "27017:27017"
networks:
- my-app-network
networks:
my-app-network:
driver: bridge
I expect to find a clear and newly created database each time I run the following command:
docker-compose build
docker-compose up
But this is not the case. When I bring the containers up with docker-compose up, my database has the exact state of the last time I shut it down with docker-compose down command. And since I have not specified a volume prop in my-database object, is this normal behaviour? Does this mean that no other action to persisting database state is required? And can I use this in production if I ever choose to use docker-compose?
The mongo image define the following volumes:
/data/configdb
/data/db
So docker-volume will create and use a unamed volume for data/db.
If you want to have a new one, use:
docker-compose down -v
docker-compose up -d --build
Or use a mount point mounted on the volume location like:
volumes:
- ./db:/data/db:rw
And drop your local db directories when you want to start over.
I am trying to learn kong, using docker-compose, i am able to run kong+konga and create services. But whenever i do docker-compose down and then up again i lose all my data:
kong:
container_name: kong
image: kong:2.1.4-alpine
restart: unless-stopped
networks:
kong-net:
ipv4_address: 172.1.1.40
volumes:
- kong_data:/usr/local/kong/declarative
environment:
KONG_DATABASE: postgres
KONG_PG_HOST: kong-database
KONG_PG_USER: kong
KONG_PG_PASSWORD: password
KONG_ADMIN_LISTEN: "0.0.0.0:8001, 0.0.0.0:8444 ssl"
KONG_DB_UPDATE_FREQUENCY: 1m
KONG_PROXY_ACCESS_LOG: /dev/stdout
KONG_ADMIN_ACCESS_LOG: /dev/stdout
KONG_PROXY_ERROR_LOG: /dev/stderr
KONG_ADMIN_ERROR_LOG: /dev/stderr
depends_on:
- kong-migration
ports:
- "8001:8001"
- "8444:8444"
- "8000:8000"
- "8443:8443"
Looks like volume mapping not working. pleasE help
If you want to keep data when your kong docker-compose is down it is better to use kong in database mode.
So then you will create a persistent volume for your database and it will keep your changes.
By the kong manual you will find there are two type of database supported: postgresql and cassandra
Postgresql is my choice for small project as I'm not planning for huge horizontal scale with cassandra database.
As you will find in the manual starting your project with docker and database is very simple.
But remember to add a volume to your database service as in the sample mentioned in manual there is no volume.
For postgresql you can add: -v /custom/mount:/var/lib/postgresql/data in docker run command
or
volumes:
postgress-data:
driver: local
services:
postgress:
restart: unless-stopped
image: postgres:latest
environment:
- POSTGRES_USER=your_db_user
- POSTGRES_DB=kong
- POSTGRES_PASSWORD=your_db_password
volumes:
- postgres-data:/var/lib/postgresql/data
Answer : You should use docker volume for having persistent data
As reference says :
Volumes are the preferred mechanism for persisting data generated by and used by Docker containers
First step is to create a volume that you want your host and docker container communicate using :
docker volume create new-volume
Second step is to use that volume in a docker-compose (in your case)
A single docker compose service with a volume looks like this:
version: "3.9"
services:
frontend:
image: node:lts
volumes:
- myapp:/home/node/app
volumes:
myapp:
On the first invocation of docker-compose up the volume will be created. The same volume will be reused on following invocations.
A volume may be created directly outside of compose with docker volume create and then referenced inside docker-compose.yml as follows:
version: "3.9"
services:
frontend:
image: node:lts
volumes:
- myapp:/home/node/app
volumes:
myapp:
external: true
I'm running this on debian 9
I'm using sudo docker volume create db to create a volume I'm using in my docker-compose.yml. But I still get the error db_1_d89b59353579 | mkdir: cannot create directory '/var/lib/mysql': Permission denied.
How can I set permissions for the user using that volume. And how to get the user?
Docker-Compose:
version: '2'
volumes:
nextcloud:
db:
services:
db:
image: mariadb
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
restart: always
volumes:
- db:/var/lib/mysql:z
environment:
- MYSQL_ROOT_PASSWORD=***
- MYSQL_PASSWORD=***
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
app:
image: nextcloud
ports:
- 8080:80
links:
- db
volumes:
- nextcloud:/var/www/html
restart: always
I got an install.sh file where I run:
...
sudo docker volume create db
sudo docker-compose build
docker-compose up -d
Try to first change the mounts to local folders and see if that fixes your issue:
version: '2'
volumes:
nextcloud:
db:
services:
db:
...
volumes:
- ./db:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=***
- MYSQL_PASSWORD=***
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
app:
...
volumes:
- ./nextcloud:/var/www/html
restart: always
If that does then check that the volumes are correctly removed by docker-compose down. Run docker volume ls. If they still persist then remove them by hand and rerun your containers with the volumes.
Regarding the difference between mounting to a volume (db:/var/lib/mysql) and mounting to a host path (./db:/var/lib/mysql):
In the first case it is a volume managed by Docker. It is meant for persistence but getting to the files is a bit more tricky. In the second case it is a path on the host and it makes it a lot easier to retrieve persisted files. I recommend to run "docker-compose config" for both situations and see the difference in how docker-compose internally transforms the statement.
Question
How do you access a volume across remote hosts so the data persists even if the instance goes down?
My Setup
This was setup on a Digital Ocean droplet with -
Create the docker machine from the cli
eval $(docker-machine env <name_of_machine>)
docker-compose -f production.yml build
docker-compose -f production.yml up
Docker-Compose file
My Postgres service is writing data to two different volumes - one for the database, and the other for backups.
version: '3'
volumes:
production_postgres_data: {}
production_postgres_data_backups: {}
production_caddy: {}
services:
django: &django
build:
context: .
dockerfile: ./compose/production/django/Dockerfile
image: mom_production_django
depends_on:
- postgres
- redis
env_file:
- ./.envs/.production/.django
- ./.envs/.production/.postgres
command: /start
postgres:
build:
context: .
dockerfile: ./compose/production/postgres/Dockerfile
image: mom_production_postgres
volumes:
- production_postgres_data:/var/lib/postgresql/data
- production_postgres_data_backups:/backups
env_file:
- ./.envs/.production/.postgres
Concern
These volumes are within that droplet, and my concern is that if anything happens to the droplet, the volumes will go with it.
So how do you create a volume that can persist and be shared across remote hosts or in this case droplets?
You need to add persistence block storage. For reference please refer the link below.
add persistence volume support
I'm trying to use Docker to containerize a MySQL (MariaDB actually) database. I figured out how to store MySQL data (/var/lib/mysql) in a volume mounted from a host directory.
However, because the underlying filesystem is different from host to host there are some inconsistencies, for example table names are case insensitive on NTFS (Windows). Also, it looks like if the database is created on a Linux host it doesn't work on a Windows host (haven't figured out why exactly).
Therefore, I want to store the data on a disk image and mount it inside the container, i.e. db-data.img formatted as ext4. But I'm facing a strange problem, when mounting this image inside the container:
$ docker run -v $PWD:/outside --rm -it ubuntu /bin/bash
# dd if=/dev/zero of=/test.img bs=1M count=100
# mkfs.ext4 test.img
# mount -o loop -t ext4 test.img /mnt
mount: /mnt: mount failed: Operation not permitted.
Using another directory instead of /mnt didn't work either.
Why does it refuse to mount the img file?
I would suggest to use docker-compose and just use a volume declared in the docker-compose.yml configuration.
Something like this:
version: '3'
services:
mysql:
image: mysql
environment:
MYSQL_ROOT_PASSWORD: $MYSQL_ROOT_PASSWORD
MYSQL_USER: $MYSQL_USER
MYSQL_PASS: $MYSQL_PASSWORD
volumes:
- mysql-data:/var/lib/mysql
volumes:
mysql-data:
The mysql-data volume should be stored as a separate volume, independent from the host operating system. The difference to just mounting a directory on the host, it's basically mounting a volume container (which you could also do without docker-compose, but it's more work).
It will not work inside of docker image, Docker blocks access to mouning filesystems (and loop devices). Should be easier create these image earlier, mount and connect to docker as folder by -v.
P.S. Another option is dump your database to sql and restore from windows.
I managed to solve this by using the privileged option in docker-compose.yml:
privileged: true
(or --privileged in the docker command)
Here is my final docker-compose.yml:
version: '3'
services:
db:
build: ./db
image: my_db
container_name: db
privileged: true
ports:
- "3306:3306"
environment:
- MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
- MYSQL_USER=${MYSQL_USER}
- MYSQL_PASSWORD=${MYSQL_PASSWORD}
- MYSQL_DATABASE=${MYSQL_DATABASE}
volumes:
- ${MYSQL_DATA_IMG}:/data.img
restart: always
Dockerfile:
FROM mariadb
COPY my-custom.cnf /etc/mysql/conf.d/custom.cnf
COPY run.sh /usr/local/bin/run-mariadb.sh
ENTRYPOINT ["run-mariadb.sh"]
and a custom entry point script that executes mount (run.sh):
#!/bin/sh
# For this mount comamnd to work the DB container must be started
# with --privileged.
mount -o loop /data.img /var/lib/mysql
# Call the entry point script of MariaDB image.
exec /usr/local/bin/docker-entrypoint.sh mysqld
for storing database data make docker-compose.yml will look like
if you want to use Dockerfile
version: '3.1'
services:
php:
build:
context: .
dockerfile: Dockerfile
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
mysql-data:
your docker-compose.yml will looks like
if you want to use your image instead of Dockerfile
version: '3.1'
services:
php:
image: php:7.4-apache
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
if you want to store or preserve data of mysql then
must remember to add two lines in your docker-compose.yml
volumes:
- mysql-data:/var/lib/mysql
and
volumes:
mysql-data:
after that use this command
docker-compose up -d
now your data will persistent and will not be deleted even after using this command
docker-compose down
extra:- but if you want to delete all data then you will use
docker-compose down -v
plus you can check your data list by this command
docker volume ls
DRIVER VOLUME NAME
local 35c819179d883cf8a4355ae2ce391844fcaa534cb71dc9a3fd5c6a4ed862b0d4
local 133db2cc48919575fc35457d104cb126b1e7eb3792b8e69249c1cfd20826aac4
local 483d7b8fe09d9e96b483295c6e7e4a9d58443b2321e0862818159ba8cf0e1d39
local 725aa19ad0e864688788576c5f46e1f62dfc8cdf154f243d68fa186da04bc5ec
local de265ce8fc271fc0ae49850650f9d3bf0492b6f58162698c26fce35694e6231c
local phphelloworld_mysql-data