MariaDB on Docker cannot write to named volume - docker

Question

I have been trying to get mariadb to work with named volumes, but I keep running into the problem that it cannot write to it.
My docker host
Docker Version: 20.10.3 on Synology DSM 7
My docker-compose.yml
---
version: "3.8"
services:
bitwarden:
depends_on:
- database
env_file:
- /volume1/docker/bitwardenextvol/settings.env
image: bitwarden/self-host:beta
restart: unless-stopped
ports:
- "4088:4080"
- "4449:4443"
volumes:
- bitwarden:/etc/bitwarden
database:
environment:
MARIADB_USER: "bitwarden"
MARIADB_PASSWORD: "****************"
MARIADB_DATABASE: "bitwarden_vault"
MARIADB_RANDOM_ROOT_PASSWORD: "true"
image: mariadb:latest
user: 1026:100
restart: unless-stopped
volumes:
- data:/var/lib/mysql
volumes:
bitwarden:
external: true
data:
external: true
My named volumes
[
{
"CreatedAt": "2023-01-14T18:32:39+01:00",
"Driver": "local",
"Labels": null,
"Mountpoint": "/volume1/#docker/volumes/bitwarden/_data",
"Name": "bitwarden",
"Options": {
"device": "//diskstation.diesveld.lan/docker/bitwardenextvol/bitwarden",
"o": "addr=diskstation.diesveld.lan,username=harald,password=********,vers=3.0",
"type": "cifs"
},
"Scope": "local"
}
]
[
{
"CreatedAt": "2023-01-14T18:33:24+01:00",
"Driver": "local",
"Labels": null,
"Mountpoint": "/volume1/#docker/volumes/data/_data",
"Name": "data",
"Options": {
"device": "//diskstation.diesveld.lan/docker/bitwardenextvol/data",
"o": "addr=diskstation.diesveld.lan,username=harald,password=********,vers=3.0",
"type": "cifs"
},
"Scope": "local"
}
]
The host directory
harald#diskstation:/volume1/docker$ ls -al | grep bitwardenextvol
drwxrwxrwx+ 1 harald users 116 Jan 14 18:36 bitwardenextvol
Inside the host directory
harald#diskstation:/volume1/docker/bitwardenextvol$ ls -al
total 16
drwxrwxrwx+ 1 harald users 116 Jan 14 18:36 .
drwxrwxrwx+ 1 root root 294 Jan 14 18:41 ..
drwxrwxrwx+ 1 harald users 94 Jan 14 18:39 bitwarden
drwxrwxrwx+ 1 harald users 30 Jan 14 18:37 data
-rwxrwxrwx+ 1 harald users 698 Jan 14 22:07 docker-compose.yml
-rwxrwxrwx+ 1 harald users 6148 Jan 14 18:11 .DS_Store
drwxrwxrwx+ 1 root users 94 Jan 14 18:12 #eaDir
-rwxrwxrwx+ 1 harald users 1940 Jan 14 18:18 settings.env
My user account
harald#diskstation:/volume1/docker/bitwardenextvol$ id harald
uid=1026(harald) gid=100(users) groups=100(users),101(administrators)
The error message that I get when running sudo docker-compose up
harald#diskstation:/volume1/docker/bitwardenextvol$ sudo docker-compose up
Creating bitwardenextvol_database_1 ... done
Creating bitwardenextvol_bitwarden_1 ... done
Attaching to bitwardenextvol_database_1, bitwardenextvol_bitwarden_1
database_1 | 2023-01-14 23:24:33+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.10.2+maria~ubu2204 started.
bitwarden_1 | addgroup: gid '100' in use
database_1 | 2023-01-14 23:24:33+00:00 [Note] [Entrypoint]: Initializing database files
database_1 | 2023-01-14 23:24:33 0 [Warning] Can't create test file /var/lib/mysql/a68bd83e89af.lower-test
database_1 | 2023-01-14 23:24:33 0 [ERROR] mariadbd: Can't create/write to file './ddl_recovery.log' (Errcode: 13 "Permission denied")
database_1 | 2023-01-14 23:24:33 0 [ERROR] DDL_LOG: Failed to create ddl log file: ./ddl_recovery.log
database_1 | 2023-01-14 23:24:33 0 [ERROR] Aborting
database_1 |
database_1 | Installation of system tables failed! Examine the logs in
database_1 | /var/lib/mysql/ for more information.
database_1 |
database_1 | The problem could be conflicting information in an external
database_1 | my.cnf files. You can ignore these by doing:
database_1 |
database_1 | shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf
database_1 |
database_1 | You can also try to start the mysqld daemon with:
database_1 |
database_1 | shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &
database_1 |
database_1 | and use the command line tool /usr/bin/mariadb
database_1 | to connect to the mysql database and look at the grant tables:
database_1 |
database_1 | shell> /usr/bin/mysql -u root mysql
database_1 | mysql> show tables;
database_1 |
database_1 | Try 'mysqld --help' if you have problems with paths. Using
database_1 | --general-log gives you a log in /var/lib/mysql/ that may be helpful.
database_1 |
database_1 | The latest information about mysql_install_db is available at
database_1 | ******/kb/en/installing-system-tables-mysql_install_db
database_1 | You can find the latest source at ******** and
database_1 | the maria-discuss email list at ********/~maria-discuss
database_1 |
database_1 | Please check all of the above before submitting a bug report
database_1 | at *******/jira
database_1 |
bitwardenextvol_database_1 exited with code 1
database_1 | 2023-01-14 23:24:35+00:00 [Note] [Entrypoint]: Initializing database files
bitwarden_1 | 2023-01-14 23:24:35,560 INFO Included extra file "/etc/supervisor.d/admin.ini" during parsing
bitwarden_1 | 2023-01-14 23:24:35,560 INFO Included extra file "/etc/supervisor.d/api.ini" during parsing
bitwarden_1 | 2023-01-14 23:24:35,560 INFO Included extra file "/etc/supervisor.d/events.ini" during parsing
bitwarden_1 | 2023-01-14 23:24:35,560 INFO Included extra file "/etc/supervisor.d/icons.ini" during parsing
bitwarden_1 | 2023-01-14 23:24:35,560 INFO Included extra file "/etc/supervisor.d/identity.ini" during parsing
bitwarden_1 | 2023-01-14 23:24:35,560 INFO Included extra file "/etc/supervisor.d/nginx.ini" during parsing
bitwarden_1 | 2023-01-14 23:24:35,560 INFO Included extra file "/etc/supervisor.d/notifications.ini" during parsing
bitwarden_1 | 2023-01-14 23:24:35,560 INFO Included extra file "/etc/supervisor.d/scim.ini" during parsing
bitwarden_1 | 2023-01-14 23:24:35,561 INFO Included extra file "/etc/supervisor.d/sso.ini" during parsing
database_1 | 2023-01-14 23:24:35 0 [Warning] Can't create test file /var/lib/mysql/a68bd83e89af.lower-test
bitwarden_1 | 2023-01-14 23:24:35,571 INFO RPC interface 'supervisor' initialized
bitwarden_1 | 2023-01-14 23:24:35,571 CRIT Server 'unix_http_server' running without any HTTP authentication checking
bitwarden_1 | 2023-01-14 23:24:35,572 INFO supervisord started with pid 48
database_1 | 2023-01-14 23:24:35 0 [ERROR] mariadbd: Can't create/write to file './ddl_recovery.log' (Errcode: 13 "Permission denied")
database_1 | 2023-01-14 23:24:35 0 [ERROR] DDL_LOG: Failed to create ddl log file: ./ddl_recovery.log
database_1 | 2023-01-14 23:24:35 0 [ERROR] Aborting
database_1 |
database_1 | Installation of system tables failed! Examine the logs in
database_1 | /var/lib/mysql/ for more information.
database_1 |
database_1 | The problem could be conflicting information in an external
database_1 | my.cnf files. You can ignore these by doing:
database_1 |
database_1 | shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf
database_1 |
database_1 | You can also try to start the mysqld daemon with:
database_1 |
database_1 | shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &
database_1 |
database_1 | and use the command line tool /usr/bin/mariadb
database_1 | to connect to the mysql database and look at the grant tables:
database_1 |
database_1 | shell> /usr/bin/mysql -u root mysql
database_1 | mysql> show tables;
database_1 |
database_1 | Try 'mysqld --help' if you have problems with paths. Using
database_1 | --general-log gives you a log in /var/lib/mysql/ that may be helpful.
database_1 |
database_1 | The latest information about mysql_install_db is available at
database_1 | ********/kb/en/installing-system-tables-mysql_install_db
database_1 | You can find the latest source at https://downloads.mariadb.org and
database_1 | the maria-discuss email list at https://launchpad.net/~maria-discuss
database_1 |
database_1 | Please check all of the above before submitting a bug report
database_1 | at ******
database_1 |
bitwarden_1 | 2023-01-14 23:24:36,574 INFO spawned: 'identity' with pid 49
bitwarden_1 | 2023-01-14 23:24:36,576 INFO spawned: 'admin' with pid 50
bitwarden_1 | 2023-01-14 23:24:36,578 INFO spawned: 'api' with pid 51
bitwarden_1 | 2023-01-14 23:24:36,579 INFO spawned: 'icons' with pid 52
bitwarden_1 | 2023-01-14 23:24:36,583 INFO spawned: 'nginx' with pid 53
bitwarden_1 | 2023-01-14 23:24:36,586 INFO spawned: 'notifications' with pid 54
database_1 | 2023-01-14 23:24:36+00:00 [Note] [Entrypoint]: Initializing database files
database_1 | 2023-01-14 23:24:36 0 [Warning] Can't create test file /var/lib/mysql/a68bd83e89af.lower-test
database_1 | 2023-01-14 23:24:36 0 [ERROR] mariadbd: Can't create/write to file './ddl_recovery.log' (Errcode: 13 "Permission denied")
database_1 | 2023-01-14 23:24:36 0 [ERROR] DDL_LOG: Failed to create ddl log file: ./ddl_recovery.log
database_1 | 2023-01-14 23:24:36 0 [ERROR] Aborting
database_1 |
database_1 | Installation of system tables failed! Examine the logs in
database_1 | /var/lib/mysql/ for more information.
database_1 |
database_1 | The problem could be conflicting information in an external
database_1 | my.cnf files. You can ignore these by doing:
database_1 |
database_1 | shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf
database_1 |
database_1 | You can also try to start the mysqld daemon with:
database_1 |
database_1 | shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &
database_1 |
database_1 | and use the command line tool /usr/bin/mariadb
database_1 | to connect to the mysql database and look at the grant tables:
database_1 |
database_1 | shell> /usr/bin/mysql -u root mysql
database_1 | mysql> show tables;
database_1 |
database_1 | Try 'mysqld --help' if you have problems with paths. Using
database_1 | --general-log gives you a log in /var/lib/mysql/ that may be helpful.
database_1 |
database_1 | The latest information about mysql_install_db is available at
database_1 | *******/kb/en/installing-system-tables-mysql_install_db
database_1 | You can find the latest source at ****** and
database_1 | the maria-discuss email list at *******/~maria-discuss
database_1 |
database_1 | Please check all of the above before submitting a bug report
database_1 | at ******
database_1 |
bitwardenextvol_database_1 exited with code 1
The weird thing is it works perfectly well with bind mounts to the very same location on disk. Also, the bitwarden container is working perfectly fine with the described named volume, however, the mariadb container is throwing the errors like in above log
[ERROR] mariadbd: Can't create/write to file './ddl_recovery.log' (Errcode: 13 "Permission denied")
The directory that the named volume is pointing to is not getting written in at all.
I cannot view the MariaDB logs, as the container keeps restarting so I cannot SSH into it.
What I get from the official documentation is that MariaDB should work perfectly fine with named volumes, but for some reason it is not in my case. My hunch was that it has to do with permissions. I tried adding the --user 1026:100 in the MariaDB service declaration but that doesn't change things.
I know I could simply work with bind mounts, but I really want to figure out how to do it with named volumes.
Who has some tips or knows how to further debug this? Your help or insights are much appreciated.
Updates based on answers
I noticed something peculiar though, the user mysql is not 999:999, but rather 66:66
harald#diskstation:~$ id mysql uid=66(mysql) gid=66(mysql) groups=66(mysql)
As I am a bit out of my comfort zone here, I cannot exactly estimate the impact of this.
As I can see in the below output, this user is not running any processes
harald#diskstation:~$ ps -u mysql
PID TTY TIME CMD
But when I try to delete the user (Synology does not support deluser) I cannot get it done:
harald#diskstation:~$ sudo synouser --del mysql
Lastest SynoErr=[user_db_delete.c:38] synouser.c:798 SYNOLocalAccountUserDelete failed. synoerr=[0xB800].
Otherwise I could get MariaDB recreate the mysql user with the right creds perhaps.
Results from docker run command from answer of Dan shows that the named volume can be read:
harald#diskstation:~$ sudo docker run -v data:/var/lib/mysql --rm mariadb ls -laZ /var/lib/mysql
total 8
drwxr-xr-x 2 root root ? 0 Jan 15 14:57 .
drwxr-xr-x 1 root root ? 76 Dec 9 02:27 ..
-rwxr-xr-x 1 root root ? 6148 Jan 14 16:40 .DS_Store
-rwxr-xr-x 1 root root ? 0 Jan 15 14:57 just-to-show-this-file-in-the-volume.yml
mysql user is indeed 999:999 inside container, but it is 66:66 outside for some weird reason.
harald#diskstation:~$ sudo docker run --rm mariadb id mysql
uid=999(mysql) gid=999(mysql) groups=999(mysql)
Changing the mount options of CIFS to include cache=none appears to be
one solution. The other is adding --innodb_flush_method=fsync to the
mariadb container args.
Mounting volume with cache=none option and adding
innodb_flush_method: "fsync"
to the docker-compose.yml does not change the behaviour unfortunately.
This is the latest state, with above remarks taken into account and without the --user. Still the same results.
harald#diskstation:~$ sudo docker volume inspect data
[
{
"CreatedAt": "2023-01-15T16:42:40+01:00",
"Driver": "local",
"Labels": {},
"Mountpoint": "/volume1/#docker/volumes/data/_data",
"Name": "data",
"Options": {
"device": "//diskstation.diesveld.lan/docker/bitwardenextvol/data",
"o": "addr=diskstation.diesveld.lan,username=harald,password=***,vers=3.0,cache=none",
"type": "cifs"
},
"Scope": "local"
}
]
And this is my docker-compose.yml
---
version: "3.8"
services:
bitwarden:
depends_on:
- database
env_file:
- /volume1/docker/bitwardenextvol/settings.env
image: bitwarden/self-host:beta
restart: unless-stopped
ports:
- "4088:4080"
- "4449:4443"
volumes:
- bitwarden:/etc/bitwarden
database:
environment:
MARIADB_USER: "bitwarden"
MARIADB_PASSWORD: "***"
MARIADB_DATABASE: "bitwarden_vault"
MARIADB_RANDOM_ROOT_PASSWORD: "true"
innodb_flush_method: "fsync"
image: mariadb:latest
restart: unless-stopped
volumes:
- data:/var/lib/mysql
volumes:
bitwarden:
external: true
data:
external: true
The logs from this:
harald#diskstation:/volume1/docker/bitwardenextvol$ sudo docker-compose up
Creating bitwardenextvol_database_1 ... done
Creating bitwardenextvol_bitwarden_1 ... done
Attaching to bitwardenextvol_database_1, bitwardenextvol_bitwarden_1
database_1 | 2023-01-16 06:26:00+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.10.2+maria~ubu2204 started.
bitwarden_1 | addgroup: gid '100' in use
database_1 | 2023-01-16 06:26:00+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
database_1 | 2023-01-16 06:26:00+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.10.2+maria~ubu2204 started.
database_1 | 2023-01-16 06:26:00+00:00 [Note] [Entrypoint]: Initializing database files
database_1 | 2023-01-16 6:26:00 0 [Warning] Can't create test file /var/lib/mysql/4d6c9c2f7405.lower-test
database_1 | 2023-01-16 6:26:00 0 [ERROR] mariadbd: Can't create/write to file './ddl_recovery.log' (Errcode: 13 "Permission denied")
database_1 | 2023-01-16 6:26:00 0 [ERROR] DDL_LOG: Failed to create ddl log file: ./ddl_recovery.log
database_1 | 2023-01-16 6:26:00 0 [ERROR] Aborting
database_1 |
database_1 | Installation of system tables failed! Examine the logs in
database_1 | /var/lib/mysql/ for more information.
database_1 |
database_1 | The problem could be conflicting information in an external
database_1 | my.cnf files. You can ignore these by doing:
database_1 |
database_1 | shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf
database_1 |
database_1 | You can also try to start the mysqld daemon with:
database_1 |
database_1 | shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &
database_1 |
database_1 | and use the command line tool /usr/bin/mariadb
database_1 | to connect to the mysql database and look at the grant tables:
database_1 |
database_1 | shell> /usr/bin/mysql -u root mysql
database_1 | mysql> show tables;
database_1 |
database_1 | Try 'mysqld --help' if you have problems with paths. Using
database_1 | --general-log gives you a log in /var/lib/mysql/ that may be helpful.
database_1 |
database_1 | The latest information about mysql_install_db is available at
database_1 | https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
database_1 | You can find the latest source at https://downloads.mariadb.org and
database_1 | the maria-discuss email list at https://launchpad.net/~maria-discuss
database_1 |
database_1 | Please check all of the above before submitting a bug report
database_1 | at https://mariadb.org/jira
database_1 |
bitwardenextvol_database_1 exited with code 1
bitwarden_1 | 2023-01-16 06:26:02,303 INFO Included extra file "/etc/supervisor.d/admin.ini" during parsing
bitwarden_1 | 2023-01-16 06:26:02,303 INFO Included extra file "/etc/supervisor.d/api.ini" during parsing
bitwarden_1 | 2023-01-16 06:26:02,303 INFO Included extra file "/etc/supervisor.d/events.ini" during parsing
bitwarden_1 | 2023-01-16 06:26:02,303 INFO Included extra file "/etc/supervisor.d/icons.ini" during parsing
bitwarden_1 | 2023-01-16 06:26:02,303 INFO Included extra file "/etc/supervisor.d/identity.ini" during parsing
bitwarden_1 | 2023-01-16 06:26:02,303 INFO Included extra file "/etc/supervisor.d/nginx.ini" during parsing
bitwarden_1 | 2023-01-16 06:26:02,304 INFO Included extra file "/etc/supervisor.d/notifications.ini" during parsing
bitwarden_1 | 2023-01-16 06:26:02,304 INFO Included extra file "/etc/supervisor.d/scim.ini" during parsing
bitwarden_1 | 2023-01-16 06:26:02,304 INFO Included extra file "/etc/supervisor.d/sso.ini" during parsing
bitwarden_1 | 2023-01-16 06:26:02,314 INFO RPC interface 'supervisor' initialized
bitwarden_1 | 2023-01-16 06:26:02,314 CRIT Server 'unix_http_server' running without any HTTP authentication checking
bitwarden_1 | 2023-01-16 06:26:02,315 INFO supervisord started with pid 47
database_1 | 2023-01-16 06:26:02+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
database_1 | 2023-01-16 06:26:02+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.10.2+maria~ubu2204 started.
database_1 | 2023-01-16 06:26:02+00:00 [Note] [Entrypoint]: Initializing database files
database_1 | 2023-01-16 6:26:02 0 [Warning] Can't create test file /var/lib/mysql/4d6c9c2f7405.lower-test
database_1 | 2023-01-16 6:26:03 0 [ERROR] mariadbd: Can't create/write to file './ddl_recovery.log' (Errcode: 13 "Permission denied")
database_1 | 2023-01-16 6:26:03 0 [ERROR] DDL_LOG: Failed to create ddl log file: ./ddl_recovery.log
database_1 | 2023-01-16 6:26:03 0 [ERROR] Aborting
database_1 |
database_1 | Installation of system tables failed! Examine the logs in
database_1 | /var/lib/mysql/ for more information.
database_1 |
database_1 | The problem could be conflicting information in an external
database_1 | my.cnf files. You can ignore these by doing:
database_1 |
database_1 | shell> /usr/bin/mariadb-install-db --defaults-file=~/.my.cnf
database_1 |
database_1 | You can also try to start the mysqld daemon with:
database_1 |
database_1 | shell> /usr/sbin/mariadbd --skip-grant-tables --general-log &
database_1 |
database_1 | and use the command line tool /usr/bin/mariadb
database_1 | to connect to the mysql database and look at the grant tables:
database_1 |
database_1 | shell> /usr/bin/mysql -u root mysql
database_1 | mysql> show tables;
database_1 |
database_1 | Try 'mysqld --help' if you have problems with paths. Using
database_1 | --general-log gives you a log in /var/lib/mysql/ that may be helpful.
database_1 |
database_1 | The latest information about mysql_install_db is available at
database_1 | https://mariadb.com/kb/en/installing-system-tables-mysql_install_db
database_1 | You can find the latest source at https://downloads.mariadb.org and
database_1 | the maria-discuss email list at https://launchpad.net/~maria-discuss
database_1 |
database_1 | Please check all of the above before submitting a bug report
database_1 | at https://mariadb.org/jira
database_1 |
bitwarden_1 | 2023-01-16 06:26:03,318 INFO spawned: 'identity' with pid 48
bitwarden_1 | 2023-01-16 06:26:03,320 INFO spawned: 'admin' with pid 49
bitwarden_1 | 2023-01-16 06:26:03,322 INFO spawned: 'api' with pid 50
bitwarden_1 | 2023-01-16 06:26:03,324 INFO spawned: 'icons' with pid 51
bitwarden_1 | 2023-01-16 06:26:03,326 INFO spawned: 'nginx' with pid 52
bitwarden_1 | 2023-01-16 06:26:03,328 INFO spawned: 'notifications' with pid 53

Answer 1

Its either the UID mapping or a selinux label, (note the + in the ls -la output). With the selinux label its probably up to an selinux boolean to allow docker to access the volume. Specific selinux labels can also be applied at the mount time.
When checking uid mapping its helpful to look inside the container as the mappings may be different:
docker run -v data:/var/lib/mysql --rm mariadb ls -laZ /var/lib/mysql
Rather than putting user in compose you can get the CIFS to map the uid to the 999:999 user the mariadb container uses by default.
docker run --rm mariadb id mysql
uid=999(mysql) gid=999(mysql) groups=999(mysql)
Based of MDEV-26970, the conditions under which MariaDB under its default --innodb_flush_method=O_DIRECT work with CIFS is limited.
Changing the mount options of CIFS to include cache=none appears to be one solution.
The other is adding --innodb_flush_method=fsync to the mariadb container command.
command: --innodb_flush_method=fsync