I have MariaDB on docker
Trying to setup ssl I managed to completely break my user; I ended up with user duplicates and a new user called testssl
I tried resetting everything by deleting my database; deleting the image but nothing works. Every time I connect to the database and list users I get the same old list; testssl is still there
Where are those settings stored and how do I reset MariaDB to a completely clean state on my docker?
app:
container_name: app
image: "${APP_IMAGE}"
restart: always
build: build/app
env_file: .env
networks:
- app_network
volumes:
- "${APP_HOST_DIR}:${APP_CONTAINER_DIR}"
depends_on:
- database
database:
container_name: mariadb
image: "mariadb:${MARIADB_VERSION}"
restart: always
env_file: .env
volumes:
- "${SQL_INIT}:/docker-entrypoint-initdb.d"
- type: bind
source: ${MARIADB_DATA_DIR}
target: /var/lib/mysql
- type: bind
source: ${MARIADB_LOG_DIR}
target: /var/logs/mysql
- type: bind
source: ${MARIADB_CERTS_DIR}
target: /etc/certs/
environment:
MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}"
MYSQL_DATABASE: "${MYSQL_DATABASE}"
MYSQL_USER: "${MYSQL_USER}"
MYSQL_PASSWORD: "${MYSQL_PASSWORD}"
.env
MARIADB_DATA_DIR=./build/database/files/database
MARIADB_LOG_DIR=./build/database/files/logs
MARIADB_CERTS_DIR=./build/database/certs
MariaDB stores all of its runtime configuration (users, databases, etc.) in its data directory.
You've told your MariaDB container to use the host build/database/files/database directory for its data, so resetting the container won't do much since the data is, well, there.
Assuming the container is well-behaved and will initialize the database system if there's nothing in the data directory, you should be able to move that build/database/files/database folder to e.g. build/database/files/database.old, and create a new empty build/database/files/database, then retry.
If you are using docker volumes you can:
Rename the docker volume
From
mariadb:
image: mariadb:10.6.11
...
volumes:
- mysqldataprevious:/var/lib/mysql
...
To
mariadb:
image: mariadb:10.6.11
...
volumes:
- mysqldatanew:/var/lib/mysql
...
Restart docker services
docker-compose up -d
Remove previous volume (optional)
docker volume prune
This will install a fresh mariadb in the new volume and with docker volume prune if your previous volume is not used by another container it will be deleted
Related
I'm running this on debian 9
I'm using sudo docker volume create db to create a volume I'm using in my docker-compose.yml. But I still get the error db_1_d89b59353579 | mkdir: cannot create directory '/var/lib/mysql': Permission denied.
How can I set permissions for the user using that volume. And how to get the user?
Docker-Compose:
version: '2'
volumes:
nextcloud:
db:
services:
db:
image: mariadb
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
restart: always
volumes:
- db:/var/lib/mysql:z
environment:
- MYSQL_ROOT_PASSWORD=***
- MYSQL_PASSWORD=***
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
app:
image: nextcloud
ports:
- 8080:80
links:
- db
volumes:
- nextcloud:/var/www/html
restart: always
I got an install.sh file where I run:
...
sudo docker volume create db
sudo docker-compose build
docker-compose up -d
Try to first change the mounts to local folders and see if that fixes your issue:
version: '2'
volumes:
nextcloud:
db:
services:
db:
...
volumes:
- ./db:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=***
- MYSQL_PASSWORD=***
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
app:
...
volumes:
- ./nextcloud:/var/www/html
restart: always
If that does then check that the volumes are correctly removed by docker-compose down. Run docker volume ls. If they still persist then remove them by hand and rerun your containers with the volumes.
Regarding the difference between mounting to a volume (db:/var/lib/mysql) and mounting to a host path (./db:/var/lib/mysql):
In the first case it is a volume managed by Docker. It is meant for persistence but getting to the files is a bit more tricky. In the second case it is a path on the host and it makes it a lot easier to retrieve persisted files. I recommend to run "docker-compose config" for both situations and see the difference in how docker-compose internally transforms the statement.
I have the following docker-compose.yml file:
# some other services here
......
......
#############################
# Setup PostgreSQL container
#############################
service-postgres:
image: postgres:10-alpine
container_name: service-postgres-server
volumes:
- ./data/postgres:/var/lib/postgresql
- ./docker/initdb:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
environment:
- POSTGRES_PASSWORD=root
- POSTGRES_DB=db_name
Each time I try to rebuild the containers running force-recreate I lose my database schema and data:
docker-compose up -d --force-recreate --build
But when I look under data/postgres I found that files and settings are there:
-$ ls data/postgres
PG_VERSION pg_commit_ts pg_ident.conf pg_notify pg_snapshots pg_subtrans pg_wal postgresql.conf
base pg_dynshmem pg_logical pg_replslot pg_stat pg_tblspc pg_xact postmaster.opts
global pg_hba.conf pg_multixact pg_serial pg_stat_tmp pg_twophase postgresql.auto.conf postmaster.pid
How can I solve the issue and persist my database schema and data each time I need to rebuild the system.
Why the database data get lost?
I tried different things and compared the MySQL image with the PostgreSQL image. Now I found the problem. There is no difference using a named volume or a mounted volume but you have to bind the correct folder on the container to persist the database data:
services:
service-postgres:
image: postgres:10-alpine
container_name: service-postgres-server
volumes:
- ./data/postgres/data:/var/lib/postgresql/data
- ./docker/initdb:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: root
POSTGRES_DB: db_name
You have to bind the /var/lib/postgresql/data folder to the host instead of the /var/lib/postgresql folder. You can also mount both folders to the host (the data folder and postgres folder) but you need the data folder to persist. On the MySQL image you only need the /var/lib/mysql folder to persist the data on the host.
Just for comparison: The MySQL image:
services:
service-mysql:
image: mysql:5.7
container_name: service-mysql-server
volumes:
- ./data/mysql-db:/var/lib/mysql
- ./docker/initdb:/docker-entrypoint-initdb.d
ports:
- "3306:3306"
environment:
MYSQL_PASSWORD: root
MYSQL_DATABASE: db_name
With the above configuration the MySQL data is persisted on the host.
Using a named volume:
services:
service-postgres:
image: postgres:10-alpine
container_name: service-postgres-server
volumes:
- postgresql-data:/var/lib/postgresql/data
- ./docker/initdb:/docker-entrypoint-initdb.d
ports:
- "5432:5432"
environment:
POSTGRES_PASSWORD: root
POSTGRES_DB: db_name
volumes:
postgresql-data:
With named volumes you can use docker-compose down -v to remove it. You can't remove a mounted volume with the docker-compose down command.
You can find the volume on the list of volumes with docker volume ls. You can also inspect the volume using docker volume inspect <volume-name> to get the mount point on the host machine.
I'm trying to setup a Drupal site template however I have an issue, this is my current docker-compose:
version: '2'
services:
database:
image: mysql
container_name: database
command: mysqld --user=root --verbose
ports:
- "3306:3306"
environment:
MYSQL_DATABASE: "db"
MYSQL_USER: "user"
MYSQL_PASSWORD: "pass"
MYSQL_ROOT_PASSWORD: "root"
MYSQL_ALLOW_EMPTY_PASSWORD: "yes"
restart: always
site:
image: drupal
container_name: site
ports:
- "555:80"
volumes:
- ./drupal:/var/www/html
links:
- database:database
working_dir: /app
restart: always
volumes:
db:
Now if I do that the site doesn't work, no files are in the /var/www/html directory and the site 404's on everything. However if I remove the volume in the site container, it works perfectly and I can start setting it up as if it was a regular site.
What am I missing?
When you don't map a volume in the site service, that means Drupal is using whatever's already in /var/www/html from the drupal image. When you map the volume, you're overwriting /var/www/html with whatever's in ./drupal on the host machine. The results you're seeing imply there may be something wrong with the contents of ./drupal. To start with, I would run the service without mapping a volume and then copy the exact contents of /var/www/html into your local folder:
docker cp compose_site_1:/var/www/html/ ./drupal
Then try running the service again, this time with the volume mapped and see if that works. If it works, that tells you the problem was with the contents of ./drupal.
I can't seem to get MySQL data to persist if I run $ docker-compose down with the following .yml
version: '2'
services:
# other services
data:
container_name: flask_data
image: mysql:latest
volumes:
- /var/lib/mysql
command: "true"
mysql:
container_name: flask_mysql
restart: always
image: mysql:latest
environment:
MYSQL_ROOT_PASSWORD: 'test_pass' # TODO: Change this
MYSQL_USER: 'test'
MYSQL_PASS: 'pass'
volumes_from:
- data
ports:
- "3306:3306"
My understanding is that in my data container using volumes: - /var/lib/mysql maps it to my local machines directory where mysql stores data to the container and because of this mapping the data should persist even if the containers are destroyed. And the mysql container is just a client interface into the db and can see the local directory because of volumes_from: - data
Attempted this answer and it did not work. Docker-Compose Persistent Data Trouble
EDIT
Changed my .yml as shown below and created a the dir ./data but now when I run docker-compose up --build the mysql container wont start throws error saying
data:
container_name: flask_data
image: mysql:latest
volumes:
- ./data:/var/lib/mysql
command: "true"
mysql:
container_name: flask_mysql
restart: always
image: mysql:latest
environment:
MYSQL_ROOT_PASSWORD: 'test_pass' # TODO: Change this
MYSQL_USER: 'test'
MYSQL_PASS: 'pass'
volumes_from:
- data
ports:
- "3306:3306"
flask_mysql | mysqld: Can't create/write to file '/var/lib/mysql/is_writable' (Errcode: 13 - Permission denied)
flask_mysql | 2016-08-26T22:29:21.182144Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
flask_mysql | 2016-08-26T22:29:21.185392Z 0 [ERROR] --initialize specified but the data directory exists and is not writable. Aborting.
The data container is a superfluous workaround. Data-volumes would do the trick for you. Alter your docker-compose.yml to:
version: '2'
services:
mysql:
container_name: flask_mysql
restart: always
image: mysql:latest
environment:
MYSQL_ROOT_PASSWORD: 'test_pass' # TODO: Change this
MYSQL_USER: 'test'
MYSQL_PASS: 'pass'
volumes:
- my-datavolume:/var/lib/mysql
volumes:
my-datavolume:
Docker will create the volume for you in the /var/lib/docker/volumes folder. This volume persist as long as you are not typing docker-compose down -v
There are 3 ways:
First way
You need specify the directory to store mysql data on your host machine. You can then remove the data container. Your mysql data will be saved on you local filesystem.
Mysql container definition must look like this:
mysql:
container_name: flask_mysql
restart: always
image: mysql:latest
environment:
MYSQL_ROOT_PASSWORD: 'test_pass' # TODO: Change this
MYSQL_USER: 'test'
MYSQL_PASS: 'pass'
volumes:
- /opt/mysql_data:/var/lib/mysql
ports:
- "3306:3306"
Second way
Would be to commit the data container before typing docker-compose down:
docker commit my_data_container
docker-compose down
Third way
Also you can use docker-compose stop instead of docker-compose down (then you don't need to commit the container)
first, you need to delete all old mysql data using
docker-compose down -v
after that add two lines in your docker-compose.yml
volumes:
- mysql-data:/var/lib/mysql
and
volumes:
mysql-data:
your final docker-compose.yml will looks like
version: '3.1'
services:
php:
build:
context: .
dockerfile: Dockerfile
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
mysql-data:
after that use this command
docker-compose up -d
now your data will persistent and will not be deleted even after using this command
docker-compose down
extra:- but if you want to delete all data then you will use
docker-compose down -v
You have to create a separate volume for mysql data.
So it will look like this:
volumes_from:
- data
volumes:
- ./mysql-data:/var/lib/mysql
And no, /var/lib/mysql is a path inside your mysql container and has nothing to do with a path on your host machine. Your host machine may even have no mysql at all. So the goal is to persist an internal folder from a mysql container.
Adding on to the answer from #Ohmen, you could also add an external flag to create the data volume outside of docker compose. This way docker compose would not attempt to create it. Also you wouldn't have to worry about losing the data inside the data-volume in the event of $ docker-compose down -v.
The below example is from the official page.
version: "3.8"
services:
db:
image: postgres
volumes:
- data:/var/lib/postgresql/data
volumes:
data:
external: true
Actually this is the path and you should mention a valid path for this to work. If your data directory is in current directory then instead of my-data you should mention ./my-data, otherwise it will give you that error in mysql and mariadb also.
volumes:
./my-data:/var/lib/mysql
Feasible bind mount solution:
mariadb:
image: mariadb:latest
restart: unless-stopped
environment:
- MARIADB_ROOT_PASSWORD=${MARIADB_ROOT_PASSWORD}
volumes:
- type: bind
source: /host/dir
target: /var/lib/mysql
I am new to docker and developing a project using docker compose. From the documentation I have learned that I should be using data only containers to keep data persistant but I am unable to do so using docker-compose.
Whenever I do docker-compose down it removes the the data from db but by doing docker-compose stop the data is not removed. May be this is because that I am not creating named data volume and docker-compose down hardly removes all the containers. So I tried naming the container but it threw me errors.
Please have a look at my yml file:
version: '2'
services:
data_container:
build: ./data
#volumes:
# - dataVolume:/data
db:
build: ./db
ports:
- "5445:5432"
environment:
- POSTGRES_USER=postgres
- POSTGRES_DB=postgres
# - PGDATA=/var/lib/postgresql/data/pgdata
volumes_from:
# - container:db_bus
- data_container
geoserver:
build: ./geoserver
depends_on:
- db
ports:
- "8004:8080"
volumes:
- ./geoserver/data:/opt/geoserverdata_dir
web:
build: ./web
volumes:
- ./web:/code
ports:
- "8000:8000"
depends_on:
- db
command: python manage.py runserver 0.0.0.0:8000
nginx:
build: ./nginx
ports:
- "83:80"
depends_on:
- web
The Docker file for the data_container is:
FROM stackbrew/busybox:latest
MAINTAINER Tom Offermann <tom#offermann.us>
# Create data directory
RUN mkdir /data
# Create /data volume
VOLUME /data
I tried this but by doing docker-compose down, the data is lost. I tried naming the data_container as you can see the commented line, it threw me this error:
ERROR: Named volume "dataVolume:/data:rw" is used in service "data_container" but no declaration was found in the volumes section.
So right now what I am doing is I created a stand alone data only named container and put that in the volumes_from value of the db. It worked fine and didn't remove any data even after doing docker-compose down.
My queries:
What is the best approach to make containers that can store database's data using the docker-compose and to use them properly ?
My conscious is not agreeing with me on approach that I have opted, the one by creating a stand alone data container. Any thoughts?
docker-compose down
does the following
Stops containers and removes containers, networks, volumes, and images
created by up
So the behaviour you are experiencing is expected.
Use docker-compose stop to shutdown containers created with the docker-compose file but not remove their volumes.
Secondly you don't need the data-container pattern in version 2 of docker compose. So remove that and just use
db:
...
volumes:
- /var/lib/postgresql/data
docker-compose down stops containers but also removes them (with everything: networks, ...).
Use docker-compose stop instead.
I think the best approach to make containers that can store database's data with docker-compose is to use named volumes:
version: '2'
services:
db: #https://hub.docker.com/_/mysql/
image: mysql
volumes:
- "wp-db:/var/lib/mysql:rw"
env_file:
- "./conf/db/mysql.env"
volumes:
wp-db: {}
Here, it will create a named volume called "wp-db" (if it doesn't exist) and mount it in /var/lib/mysql (in read-write mode, the default). This is where the database stores its data (for the mysql image).
If the named volume already exists, it will be used without creating it.
When starting, the mysql image look if there are databases in /var/lib/mysql (your volume) in order to use them.
You can have more information with the docker-compose file reference here:
https://docs.docker.com/compose/compose-file/#/volumes-volume-driver
To store database data make sure your docker-compose.yml will look like
if you want to use Dockerfile
version: '3.1'
services:
php:
build:
context: .
dockerfile: Dockerfile
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
mysql-data:
your docker-compose.yml will looks like
if you want to use your image instead of Dockerfile
version: '3.1'
services:
php:
image: php:7.4-apache
ports:
- 80:80
volumes:
- ./src:/var/www/html/
db:
image: mysql
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_ROOT_PASSWORD: example
volumes:
- mysql-data:/var/lib/mysql
adminer:
image: adminer
restart: always
ports:
- 8080:8080
volumes:
if you want to store or preserve data of mysql then
must remember to add two lines in your docker-compose.yml
volumes:
- mysql-data:/var/lib/mysql
and
volumes:
mysql-data:
after that use this command
docker-compose up -d
now your data will persistent and will not be deleted even after using this command
docker-compose down
extra:- but if you want to delete all data then you will use
docker-compose down -v
to verify or check database data list by using this command
docker volume ls
DRIVER VOLUME NAME
local 35c819179d883cf8a4355ae2ce391844fcaa534cb71dc9a3fd5c6a4ed862b0d4
local 133db2cc48919575fc35457d104cb126b1e7eb3792b8e69249c1cfd20826aac4
local 483d7b8fe09d9e96b483295c6e7e4a9d58443b2321e0862818159ba8cf0e1d39
local 725aa19ad0e864688788576c5f46e1f62dfc8cdf154f243d68fa186da04bc5ec
local de265ce8fc271fc0ae49850650f9d3bf0492b6f58162698c26fce35694e6231c
local phphelloworld_mysql-data