MariaDB incremental backup generates large files for unchanged database - docker

Question

Cross-site dupe (no answer).
Basically I have to set up incremental backup system for mariadb. Everything is living in docker containers, and I'm trying to use wal-g (similar to how easy I can set it up with postgres and WAL archiving) together with mariabackup. Backup task is executed within separate docker container based on mariadb:10.8.3-jammy image with wal-g binary added (I'm running wal-g backup-push). During debugging I use exporting to data folder (mounted as docker volume to /bak).
I'm presenting more details below, but let me begin with the actual problem: incremental backups are large. If I disable all DB clients and repeat the same backup a few times in a row (so nothing changes between them), initial backup is approx. 30
Mb and all consequent incremental backups are 30 Mb each.
# Initial
$ sudo docker compose run --rm db_backup wal-g backup-push
[+] Running 1/0
⠿ Container reporting-db-1 Running 0.0s
INFO: 2022/11/20 19:24:01.865762 FILE PATH: stream_20221120T192400Z/stream.lz4
INFO: 2022/11/20 19:24:01.865984 Backup sentinel: {"StartLocalTime":"2022-11-20T19:24:00.366507Z","StopLocalTime":"2022-11-20T19:24:01.86589Z","UncompressedSize":350375370,"CompressedSize":227476734,"Hostname":"b08f12bbbb68"}
# Incremental
$ sudo docker compose run --rm db_backup wal-g backup-push
[+] Running 1/0
⠿ Container reporting-db-1 Running 0.0s
INFO: 2022/11/20 19:24:07.109202 FILE PATH: stream_20221120T192405Z/stream.lz4
INFO: 2022/11/20 19:24:07.109509 Backup sentinel: {"StartLocalTime":"2022-11-20T19:24:05.639259Z","StopLocalTime":"2022-11-20T19:24:07.109321Z","UncompressedSize":28104928,"CompressedSize":14609954,"Hostname":"3d9854550254"}
If I provide an empty database as data source, then the result is much more comical: both initial and incremental backups are of the same size (+- a few Kb).
I supposed that mariabackup dumps system tables, which use Aria storage engine and do not support incremental backup (source). However, manual verification seems to disagree: I tried converting all mysql.* tables to InooDB (ALTER TABLE ... engine=innodb) and re-creating the backup, getting 1.5x larger incremental dumps (though compressed results are slightly better).
# Initial
$ sudo docker compose run --rm db_backup wal-g backup-push
[+] Running 1/0
⠿ Container reporting-db-1 Running 0.0s
INFO: 2022/11/20 18:56:06.254782 FILE PATH: stream_20221120T185602Z/stream.lz4
INFO: 2022/11/20 18:56:06.254984 Backup sentinel: {"StartLocalTime":"2022-11-20T18:56:02.328882Z","StopLocalTime":"2022-11-20T18:56:06.254911Z","UncompressedSize":366630153,"CompressedSize":229793575,"Hostname":"843e27ceff9e"}
# Incremental 1
$ sudo docker compose run --rm db_backup wal-g backup-push
[+] Running 1/0
⠿ Container reporting-db-1 Running 0.0s
INFO: 2022/11/20 18:56:15.123822 FILE PATH: stream_20221120T185610Z/stream.lz4
INFO: 2022/11/20 18:56:15.124035 Backup sentinel: {"StartLocalTime":"2022-11-20T18:56:10.974406Z","StopLocalTime":"2022-11-20T18:56:15.12395Z","UncompressedSize":44359711,"CompressedSize":13798326,"Hostname":"26a6e38afb59"}
# Incremental 2
$ sudo docker compose run --rm db_backup wal-g backup-push
[+] Running 1/0
⠿ Container reporting-db-1 Running 0.0s
INFO: 2022/11/20 19:06:57.626672 FILE PATH: stream_20221120T190653Z/stream.lz4
INFO: 2022/11/20 19:06:57.626904 Backup sentinel: {"StartLocalTime":"2022-11-20T19:06:53.667772Z","StopLocalTime":"2022-11-20T19:06:57.626823Z","UncompressedSize":44360402,"CompressedSize":13799019,"Hostname":"184342c39610"}
Unfortunately, I cannot ask my client to switch to convenient DBMS and have to do something here. I do not want to store these 30 Mb for every backup, if it can be avoided.
Is my reasoning ok? What else can cause this weird behaviour?
Can I just convert all system tables to InnoDB? I found evidence that it can be harmful on mysql 5.7, but cannot find more recent references. It'd resolve the problem, because then everything will support incremental backup properly. (Dupe? Not really).
Are there any alternative backup solutions that can handle described situation better?
Can I give up and prevent mariabackup from backing system tables up? I doubt it can be a viable solution (because the more complete the backup is, the easier to live with it), but may be wrong.
Side questions:
How can I examine the binary stream outputted by mariabackup somehow to confirm that system tables are the actual problem (and perhaps find out which tables exactly)?
What can cause the dump size fluctuations? Whenever I run multiple incremental backups in a row, the compressed and uncompressed size is slightly different every time (it can either increase or decrease compared to the previous run). Backup should be a deterministic process, and all actions above are performed on the same database without any modifications in between (I started from local dump which was loaded to new mariadb container with clean volumes, and no clients have access to that instance, so nothing should differ) - then why do I observe this? I checked with mariabackup without wal-g tool, and the size is stable then. It is introduced on some higher level, and this is less interesting.
Everything described above reproduces with plain mariabackup as well, generating approx. 27 Mb files per incremental backup.
mariabackup wrapper script:
last_lsns=$(ls /bak/lsns/ | sort -rn | head -n1)
if [ -n "$last_lsns" ]; then
ex="/bak/lsns/lsn_$(date +%s)"
mkdir -p "$ex"
mariabackup -H"$WEB_DB_HOST" -uroot -p"$MYSQL_ROOT_PASSWORD" --backup \
--stream=xbstream --datadir=/var/lib/mysql \
--incremental-basedir=/bak/lsns/$last_lsns --extra-lsndir=$ex
else
mkdir -p /bak/lsns/initial
mariabackup -H$WEB_DB_HOST -uroot -p"$MYSQL_ROOT_PASSWORD" --backup \
--stream=xbstream --datadir=/var/lib/mysql \
--extra-lsndir=/bak/lsns/initial
fi
This script is used as WALG_STREAM_CREATE_COMMAND. I have also
WALG_MYSQL_DATASOURCE_NAME='root:$MYSQL_ROOT_PASSWORD#tcp($REPORTING_DB_HOST:$REPORTING_DB_PORT)/$REPORTING_DATABASE'
WALG_FILE_PREFIX='/bak/foo'
and these settings (in fact written in compose file, but it's probably not important) seem to be correct (backups are created as expected ad written to proper directories).
Here are storage types used:
> select table_schema, table_name, engine from information_schema.tables where table_schema <> 'performance_schema' and engine <> 'MEMORY';
+--------------------+---------------------------+--------+
| table_schema | table_name | engine |
+--------------------+---------------------------+--------+
| information_schema | ALL_PLUGINS | Aria |
| information_schema | CHECK_CONSTRAINTS | Aria |
| information_schema | COLUMNS | Aria |
| information_schema | EVENTS | Aria |
| information_schema | OPTIMIZER_TRACE | Aria |
| information_schema | PARAMETERS | Aria |
| information_schema | PARTITIONS | Aria |
| information_schema | PLUGINS | Aria |
| information_schema | PROCESSLIST | Aria |
| information_schema | ROUTINES | Aria |
| information_schema | SYSTEM_VARIABLES | Aria |
| information_schema | TRIGGERS | Aria |
| information_schema | VIEWS | Aria |
| mysql | slow_log | CSV |
| mysql | db | Aria |
| mysql | help_relation | Aria |
| mysql | general_log | CSV |
| mysql | innodb_index_stats | InnoDB |
| mysql | servers | Aria |
| mysql | time_zone_transition_type | Aria |
| mysql | gtid_slave_pos | InnoDB |
| mysql | time_zone | Aria |
| mysql | roles_mapping | Aria |
| mysql | transaction_registry | InnoDB |
| mysql | procs_priv | Aria |
| mysql | proxies_priv | Aria |
| mysql | global_priv | Aria |
| mysql | func | Aria |
| mysql | innodb_table_stats | InnoDB |
| mysql | help_topic | Aria |
| mysql | time_zone_leap_second | Aria |
| mysql | help_keyword | Aria |
| mysql | time_zone_transition | Aria |
| mysql | event | Aria |
| mysql | columns_priv | Aria |
| mysql | tables_priv | Aria |
| mysql | time_zone_name | Aria |
| mysql | plugin | Aria |
| mysql | table_stats | Aria |
| mysql | index_stats | Aria |
| mysql | proc | Aria |
| mysql | help_category | Aria |
| mysql | column_stats | Aria |
| test_reporting | merchant_configs | InnoDB |
| test_reporting | masterdata_prediction | InnoDB |
| test_reporting | aggregator_config | InnoDB |
| test_reporting | masterdata | InnoDB |
| test_reporting | fixed_costs | InnoDB |
| test_reporting | timeline | InnoDB |
| test_reporting | migrations | InnoDB |
| test_reporting | user_analytics | InnoDB |
| test_reporting | affiliate | InnoDB |
| test_reporting | vat_config | InnoDB |
| sys | sys_config | Aria |
| reporting | merchant_configs | InnoDB |
| reporting | masterdata_prediction | InnoDB |
| reporting | aggregator_config | InnoDB |
| reporting | masterdata | InnoDB |
| reporting | fixed_costs | InnoDB |
| reporting | timeline | InnoDB |
| reporting | migrations | InnoDB |
| reporting | user_analytics | InnoDB |
| reporting | affiliate | InnoDB |
| reporting | vat_config | InnoDB |
| reporting_web | record_change | InnoDB |
| reporting_web | jwt_expiry | InnoDB |
| reporting_web | forecasting | InnoDB |
| reporting_web | users | InnoDB |
| reporting_web | alembic_version | InnoDB |
| reporting_web | merchant_analytics | InnoDB |
| reporting_web | email_config | InnoDB |
| reporting_web | user_company | InnoDB |
| reporting_web | user_detail | InnoDB |
+--------------------+---------------------------+--------+

Answer 1

As referenced MDEV-23614, Aria system tables cannot be incrementally backed up.
As you saw, system tables can be changed to InnoDB in 10.4+. Azure do this by default.
Two small issues prevent this being default under --enforce-storage-engine=InnoDB, both related to help tables:
CREATE TABLE IF NOT EXISTS help_relation has a FK reference to help_keyword, but help_keyword isn't created (easy fix, swap order in mysql_system_tables.sql)
In fill_help_tables.sql, lock tables help_topic write, help_category write.. once its InnoDB, returns ER_WRONG_LOCK_OF_SYSTEM_TABLE, can be commented out, but really is a bug that needs reporting/fixing.
To save space, help table and the proc tables are the biggest ones.
help tables, are optional for HELP command syntax. And could be truncated/removed.
proc, by default, contains a bunch of gis functions you may not need.
both could be converted to InnoDB.
Alternatives:
Use binary logs as a PITR alternate mechanism and perform less incremental mariabackups further apart.
Contribute a patch to mariabackup.