In my docker-compose file i have created a database service. There i have my local (outside) folder named ´database´ under volumes. I have copied my sql file (sql.sql) into this folder. But when I go into the database container, I don't see it. what did I do wrong?
version: '3.9'
services:
database:
image: mariadb:latest
container_name: mariadb
ports:
- 3306:3306
environment:
- MYSQL_USER=me
- MYSQL_PASSWORD=password
- MYSQL_ROOT_PASSWORD=password
volumes:
- ./database:/home
Folder structure
/database
sql.sql
/docker-compose.yaml
Related
I have an application where I need to reset the database (wipe it completely).
I ran all the commands I could find
docker system prune
docker system prune -a -f
docker volume prune
Using docker volume ls, I copied the volume ID and then ran
docker volume rm "the volume id"
When I do docker system df nothing is shown anymore. However, once I run my app again
docker-compose up --build
the database still contains old values.
What am I doing wrong?
EDIT here is my compose file
version: "3"
services:
nftapi:
env_file:
- .env
build:
context: .
ports:
- '5000:5000'
depends_on:
- postgres
networks:
- postgres
extra_hosts:
- "host.docker.internal:host-gateway"
restart: always
postgres:
container_name: postgres
image: postgres:latest
ports:
- "5432:5432"
volumes:
- /data/postgres:/var/lib/postgresql/data
env_file:
- docker.env
networks:
- postgres
pgadmin:
links:
- postgres:postgres
container_name: pgadmin
image: dpage/pgadmin4
ports:
- "8080:80"
env_file:
- docker.env
networks:
- postgres
networks:
postgres:
driver: bridge
It seems the database in your config is mapped to a directory on your host system:
volumes:
- /data/postgres:/var/lib/postgresql/data
so the data in the containers /var/lib/postgresql/data is read from and written to your local /data/postgres directory.
If you want to delete it you should empty out that directory. (Or move it until you are sure that you can delete it)
Below is my docker-compose file in which I specified, that the postgres data is saved into my folder in the host called volumes/db_data/. But this folder stays empty. Instead, the data seems to be saved into the default folder var/lib/docker/volumes. Also, the folder user_models is not saved locally in volumes/user_models. The api_server.log is also not saved. Therefore I guess there is something wrong with my understanding of docker volumes in general.
version: "3.7"
services:
db:
image: postgres
environment:
POSTGRES_PASSWORD: xxxxxxxxxxx
POSTGRES_USER: postgres
POSTGRES_DB: user-db
volumes:
- ./volumes/db_data:/var/lib/postgresql/data
ports:
- 5432:5432
api-server:
image: api-server
volumes:
- ./volumes/user_models:/classification/user_models
depends_on:
- db
ports:
- 1337:1337
model-trainer:
image: webapp-trainer
volumes:
- ./volumes/user_models:/user_models
- ./volumes/api_server.log:/api_server.log
depends_on:
- db
react-client:
image: webapp-client
depends_on:
- api-server
ports:
- 3000:3000
Everything else is working fine, just the data in user_models is not saved and the postgres data is saved in the wrong place. What am I doing wrong?
Edit:
docker-compose up prints this at the start:
WARNING: Service "db" is using volume "/var/lib/postgresql/data" from the previous container. Host mapping "/home/theo/Documents/Programming/cbi-webapp/compose/volumes/db_data"
yet db_data is empty
I'm trying to set up a docker-compose file for running Apache Guacamole.
The compose file has 3 services, 2 for guacamole itself and 1 database image. The problem is that the database has to be initialized before the guacamole container can use it, but the files to initialize the database are in the guacamole image. The solution I came up with is this:
version: "3"
services:
init:
image: guacamole/guacamole:latest
command: ["/bin/sh", "-c", "cp /opt/guacamole/postgresql/schema/*.sql /init/" ]
volumes:
- dbinit:/init
database:
image: postgres:latest
restart: unless-stopped
volumes:
- dbinit:/docker-entrypoint-initdb.d
- dbdata:/var/lib/postgresql/data
environment:
POSTGRES_USER: guac
POSTGRES_PASSWORD: guac
depends_on:
- init
guacd:
image: guacamole/guacd:latest
restart: unless-stopped
guacamole:
image: guacamole/guacamole:latest
restart: unless-stopped
ports:
- "8080:8080"
environment:
GUACD_HOSTNAME: guacd
POSTGRES_HOSTNAME: database
POSTGRES_DATABASE: guac
POSTGRES_USER: guac
POSTGRES_PASSWORD: guac
depends_on:
- database
- guacd
volumes:
dbinit:
dbdata:
So I have one container whose job is to copy the database initialization files into a volume and then I mount that volume in the database. The problem is that this creates a race condition and is ugly. Is there some elegant solution for this? Is it possible to mount the files from the guacamole image into the database container? I would rather avoid having an extra sql file with the docker-compose file.
Thanks in advance!
I have the following docker-compose.yml file:
# some other services here
......
......
#############################
# Setup PostgreSQL container
#############################
service-postgres:
image: postgres:10-alpine
container_name: service-postgres-server
volumes:
- ./data/postgres:/var/lib/postgresql
- ./docker/initdb:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
environment:
- POSTGRES_PASSWORD=root
- POSTGRES_DB=db_name
Each time I try to rebuild the containers running force-recreate I lose my database schema and data:
docker-compose up -d --force-recreate --build
But when I look under data/postgres I found that files and settings are there:
-$ ls data/postgres
PG_VERSION pg_commit_ts pg_ident.conf pg_notify pg_snapshots pg_subtrans pg_wal postgresql.conf
base pg_dynshmem pg_logical pg_replslot pg_stat pg_tblspc pg_xact postmaster.opts
global pg_hba.conf pg_multixact pg_serial pg_stat_tmp pg_twophase postgresql.auto.conf postmaster.pid
How can I solve the issue and persist my database schema and data each time I need to rebuild the system.
Why the database data get lost?
I tried different things and compared the MySQL image with the PostgreSQL image. Now I found the problem. There is no difference using a named volume or a mounted volume but you have to bind the correct folder on the container to persist the database data:
services:
service-postgres:
image: postgres:10-alpine
container_name: service-postgres-server
volumes:
- ./data/postgres/data:/var/lib/postgresql/data
- ./docker/initdb:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: root
POSTGRES_DB: db_name
You have to bind the /var/lib/postgresql/data folder to the host instead of the /var/lib/postgresql folder. You can also mount both folders to the host (the data folder and postgres folder) but you need the data folder to persist. On the MySQL image you only need the /var/lib/mysql folder to persist the data on the host.
Just for comparison: The MySQL image:
services:
service-mysql:
image: mysql:5.7
container_name: service-mysql-server
volumes:
- ./data/mysql-db:/var/lib/mysql
- ./docker/initdb:/docker-entrypoint-initdb.d
ports:
- "3306:3306"
environment:
MYSQL_PASSWORD: root
MYSQL_DATABASE: db_name
With the above configuration the MySQL data is persisted on the host.
Using a named volume:
services:
service-postgres:
image: postgres:10-alpine
container_name: service-postgres-server
volumes:
- postgresql-data:/var/lib/postgresql/data
- ./docker/initdb:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: root
POSTGRES_DB: db_name
volumes:
postgresql-data:
With named volumes you can use docker-compose down -v to remove it. You can't remove a mounted volume with the docker-compose down command.
You can find the volume on the list of volumes with docker volume ls. You can also inspect the volume using docker volume inspect <volume-name> to get the mount point on the host machine.
I am new to docker and developing a project using docker compose. From the documentation I have learned that I should be using data only containers to keep data persistant but I am unable to do so using docker-compose.
Whenever I do docker-compose down it removes the the data from db but by doing docker-compose stop the data is not removed. May be this is because that I am not creating named data volume and docker-compose down hardly removes all the containers. So I tried naming the container but it threw me errors.
Please have a look at my yml file:
version: '2'
services:
data_container:
build: ./data
#volumes:
# - dataVolume:/data
db:
build: ./db
ports:
- "5445:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_DB=postgres
# - PGDATA=/var/lib/postgresql/data/pgdata
volumes_from:
# - container:db_bus
- data_container
geoserver:
build: ./geoserver
depends_on:
- db
ports:
- "8004:8080"
volumes:
- ./geoserver/data:/opt/geoserverdata_dir
web:
build: ./web
volumes:
- ./web:/code
ports:
- "8000:8000"
depends_on:
- db
command: python manage.py runserver 0.0.0.0:8000
nginx:
build: ./nginx
ports:
- "83:80"
depends_on:
- web
The Docker file for the data_container is:
FROM stackbrew/busybox:latest
MAINTAINER Tom Offermann <tom#offermann.us>
# Create data directory
RUN mkdir /data
# Create /data volume
VOLUME /data
I tried this but by doing docker-compose down, the data is lost. I tried naming the data_container as you can see the commented line, it threw me this error:
ERROR: Named volume "dataVolume:/data:rw" is used in service "data_container" but no declaration was found in the volumes section.
So right now what I am doing is I created a stand alone data only named container and put that in the volumes_from value of the db. It worked fine and didn't remove any data even after doing docker-compose down.
My queries:
What is the best approach to make containers that can store database's data using the docker-compose and to use them properly ?
My conscious is not agreeing with me on approach that I have opted, the one by creating a stand alone data container. Any thoughts?
docker-compose down
does the following
Stops containers and removes containers, networks, volumes, and images
created by up
So the behaviour you are experiencing is expected.
Use docker-compose stop to shutdown containers created with the docker-compose file but not remove their volumes.
Secondly you don't need the data-container pattern in version 2 of docker compose. So remove that and just use
db:
...
volumes:
- /var/lib/postgresql/data
docker-compose down stops containers but also removes them (with everything: networks, ...).
Use docker-compose stop instead.
I think the best approach to make containers that can store database's data with docker-compose is to use named volumes:
version: '2'
services:
db: #https://hub.docker.com/_/mysql/
image: mysql
volumes:
- "wp-db:/var/lib/mysql:rw"
env_file:
- "./conf/db/mysql.env"
volumes:
wp-db: {}
Here, it will create a named volume called "wp-db" (if it doesn't exist) and mount it in /var/lib/mysql (in read-write mode, the default). This is where the database stores its data (for the mysql image).
If the named volume already exists, it will be used without creating it.
When starting, the mysql image look if there are databases in /var/lib/mysql (your volume) in order to use them.
You can have more information with the docker-compose file reference here:
https://docs.docker.com/compose/compose-file/#/volumes-volume-driver
To store database data make sure your docker-compose.yml will look like
if you want to use Dockerfile
version: '3.1'
services:
php:
build:
context: .
dockerfile: Dockerfile
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
mysql-data:
your docker-compose.yml will looks like
if you want to use your image instead of Dockerfile
version: '3.1'
services:
php:
image: php:7.4-apache
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
if you want to store or preserve data of mysql then
must remember to add two lines in your docker-compose.yml
volumes:
- mysql-data:/var/lib/mysql
and
volumes:
mysql-data:
after that use this command
docker-compose up -d
now your data will persistent and will not be deleted even after using this command
docker-compose down
extra:- but if you want to delete all data then you will use
docker-compose down -v
to verify or check database data list by using this command
docker volume ls
DRIVER VOLUME NAME
local 35c819179d883cf8a4355ae2ce391844fcaa534cb71dc9a3fd5c6a4ed862b0d4
local 133db2cc48919575fc35457d104cb126b1e7eb3792b8e69249c1cfd20826aac4
local 483d7b8fe09d9e96b483295c6e7e4a9d58443b2321e0862818159ba8cf0e1d39
local 725aa19ad0e864688788576c5f46e1f62dfc8cdf154f243d68fa186da04bc5ec
local de265ce8fc271fc0ae49850650f9d3bf0492b6f58162698c26fce35694e6231c
local phphelloworld_mysql-data