RabbitMQ fails to boot from docker-compose - docker

Question

Im trying to set up rabbitmq instance from docker-compose command.
My docker compose yaml
version: '3.8'
services:
rabbitmq:
image: rabbitmq:3-management
hostname: rabbit
container_name: 'rabbitmq'
volumes:
- ./etc/rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
- ./data:/var/lib/rabbitmq/mnesia/rabbit#rabbit
- ./logs:/var/log/rabbitmq/log
- ./etc/ssl/CERT_LAB_CA.pem:/etc/rabbitmq/ssl/cacert.pem
- ./etc/ssl/CERT_LAB_RABBITMQ.pem:/etc/rabbitmq/ssl/cert.pem
- ./etc/ssl/KEY_LAB_RABBITMQ.pem:/etc/rabbitmq/ssl/key.pem
ports:
- 5672:5672
- 15672:15672
- 15671:15671
- 5671:5671
environment:
- RABBITMQ_DEFAULT_USER=secret
- RABBITMQ_DEFAULT_PASS=secret
When I run docker compose up for the first time, everything works fine. But when I add queues and exchanged(loaded from definitions.json), shut down and remove container and try to docker compose up again, I got this error
2022-09-29 13:32:09.522956+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0>
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0> BOOT FAILED
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0> ===========
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0> Error during startup: {error,
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0> {schema_integrity_check_failed,
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0> [{table_missing,rabbit_listener}]}}
2022-09-29 13:32:09.523096+00:00 [error] <0.229.0>
BOOT FAILED
===========
Error during startup: {error,
{schema_integrity_check_failed,
[{table_missing,rabbit_listener}]}}
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> crasher:
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> initial call: application_master:init/4
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> pid: <0.228.0>
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> registered_name: []
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> exception exit: {{schema_integrity_check_failed,
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> [{table_missing,rabbit_listener}]},
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> {rabbit,start,[normal,[]]}}
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> in function application_master:init/4 (application_master.erl, line 142)
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> ancestors: [<0.227.0>]
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> message_queue_len: 1
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> messages: [{'EXIT',<0.229.0>,normal}]
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> links: [<0.227.0>,<0.44.0>]
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> dictionary: []
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> trap_exit: true
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> status: running
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> heap_size: 2586
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> stack_size: 28
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> reductions: 180
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0> neighbours:
2022-09-29 13:32:10.524073+00:00 [error] <0.228.0>
And here is my rabbitmq.conf file
listeners.tcp.default = 5672
listeners.ssl.default = 5671
ssl_options.cacertfile = /etc/rabbitmq/ssl/cacert.pem
ssl_options.certfile = /etc/rabbitmq/ssl/cert.pem
ssl_options.keyfile = /etc/rabbitmq/ssl/key.pem
#Generate client cert and uncomment this if client has to provide cert.
#ssl_options.verify = verify_peer
#ssl_options.fail_if_no_peer_cert = true
collect_statistics_interval = 10000
#load_definitions = /path/to/exported/definitions.json
#definitions.skip_if_unchanged = true
management.tcp.port = 15672
management.ssl.port = 15671
management.ssl.cacertfile = /etc/rabbitmq/ssl/cacert.pem
management.ssl.certfile = /etc/rabbitmq/ssl/cert.pem
management.ssl.keyfile = /etc/rabbitmq/ssl/key.pem
management.http_log_dir = /var/log/rabbitmq/http
What am I missing?

Answer 1

Try to substitute ./data:/var/lib/rabbitmq/mnesia/rabbit#rabbit in your config with ./data:/var/lib/rabbitmq.
I had the same error and spent quite time trying to figure out the problem. My configuration was slightly different from yours and looked like this:
rabbitmq:
image: rabbitmq:3.11.2-management-alpine
hostname: rabbitmq
environment:
RABBITMQ_DEFAULT_USER: tester
RABBITMQ_DEFAULT_PASS: qwerty
RABBITMQ_MNESIA_DIR: /my-custom-data-folder-path-inside-container
RABBITMQ_NODENAME: rabbitmq
volumes:
- type: bind
source: /my-custom-data-folder-path-on-host
target: /my-custom-data-folder-path-inside-container
I'm not an expert in RabbitMQ, and my idea was just to make RabbitMQ to persist it's database in the /my-custom-data-folder-path-on-host folder on host. And just like in your case on the first run it was able to start successfully, but after container restart I was getting the following error:
BOOT FAILED
Error during startup: {error, {schema_integrity_check_failed, [{table_missing,rabbit_listener}]}}
I learned from the documentation is that rabbit_listener is a table inside the Mnesia database that is used by RabbitMQ and that "listeners" are the TCP-listeners that are configured in RabbitMQ to accept client connections.
For RabbitMQ to accept client connections, it needs to bind to one or more interfaces and listen on (protocol-specific) ports. One such interface/port pair is called a listener in RabbitMQ parlance. Listeners are configured using the listeners.tcp.* configuration option(s).
I wanted to dig into the Mnesia database to troubleshoot but not managed to do that without Erlang knowledge. It seems that for some reason on the first run RabbitMQ does not create "rabbit_listener" table, but on subsequent runs requires it.
Finally, I managed to workaround the problem by changing my initial configuration as follows:
service-bus:
image: rabbitmq:3.11.2-management-alpine
hostname: rabbitmq
environment:
RABBITMQ_DEFAULT_USER: tester
RABBITMQ_DEFAULT_PASS: qwerty
RABBITMQ_NODENAME: rabbitmq
volumes:
- type: bind
source: /my-custom-data-folder-path-on-host
target: /var/lib/rabbitmq
Instead of overriding just the RABBITMQ_MNESIA_DIR folder I've overridden the entire /var/lib/rabbitmq. This did the trick and now my RabbitMQ successfully endures restarts.

Answer 2

I hit this problem and I changed my docker-compose.yml file to use rabbitmq:3.9-management rather than rabbitmq:3-management.
The problem happened for me when I restarted the stack and the rabbitmq image went to 3.11.