rabbitmq docker wss not working even though https works - docker

Question

Update
I initially started the docker container using this command:
sudo docker run -d -it --hostname some-rabbitmq --name rabbitmq -p 5672:5672 -p 15672:15672 -p 15674:15674 --restart=unless-stopped -v /opt/rabbitmq-test:/etc/rabbitmq/ rabbitmq:3.9-management
However I needed to add tls without losing the info that was already saved on it, so I added more ports by stopping the docker container and service, updating the config.v2.json and hostconfig.json files to add the ports I wanted to expose, including 15673.
I'm using the official rabbitmq docker image as my rabbitmq server. I've configured tls but for some reason the wss:// connection fails completely. This is my rabbitmq.conf file
loopback_users.guest = false
management.cors.allow_origins.1 = *
management.tcp.port = 80
# ===========================================================================
# ====================== TLS for webstomp plugin ============================
# ===========================================================================
# source of tls settings for webstomp plugin: https://www.rabbitmq.com/web-stomp.html#tls-versions
web_stomp.ssl.port = 15673
web_stomp.ssl.backlog = 1024
web_stomp.ssl.cacertfile = /etc/letsencrypt/live/{example.com}/chain.pem
web_stomp.ssl.certfile = /etc/letsencrypt/live/{example.com}/cert.pem
web_stomp.ssl.keyfile = /etc/letsencrypt/live/{example.com}/privkey.pem
# web_stomp.ssl.password = changeme
web_stomp.ssl.honor_cipher_order = true
web_stomp.ssl.honor_ecc_order = true
web_stomp.ssl.client_renegotiation = false
web_stomp.ssl.secure_renegotiate = true
web_stomp.ssl.versions.1 = tlsv1.2
web_stomp.ssl.versions.2 = tlsv1.1
web_stomp.ssl.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384
web_stomp.ssl.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
web_stomp.ssl.ciphers.3 = ECDHE-ECDSA-AES256-SHA384
web_stomp.ssl.ciphers.4 = ECDHE-RSA-AES256-SHA384
web_stomp.ssl.ciphers.5 = ECDH-ECDSA-AES256-GCM-SHA384
web_stomp.ssl.ciphers.6 = ECDH-RSA-AES256-GCM-SHA384
web_stomp.ssl.ciphers.7 = ECDH-ECDSA-AES256-SHA384
web_stomp.ssl.ciphers.8 = ECDH-RSA-AES256-SHA384
web_stomp.ssl.ciphers.9 = DHE-RSA-AES256-GCM-SHA384
# ==========================================================================
# ====================== TLS for manangement ui ============================
# ==========================================================================
# source of tls settings for management ui: https://www.rabbitmq.com/management.html#single-listener-https
management.ssl.port = 443
# management.ssl.cacertfile = /opt/rabbitmq-test/tls-ca/letsencrypt.pem
management.ssl.cacertfile = /etc/letsencrypt/live/{example.com}/chain.pem
management.ssl.certfile = /etc/letsencrypt/live/{example.com}/cert.pem
management.ssl.keyfile = /etc/letsencrypt/live/{example.com}/privkey.pem
## This key must only be used if private key is password protected
# management.ssl.password = bunnies
# For RabbitMQ 3.7.10 and later versions
management.ssl.honor_cipher_order = true
management.ssl.honor_ecc_order = true
management.ssl.client_renegotiation = false
management.ssl.secure_renegotiate = true
management.ssl.versions.1 = tlsv1.2
management.ssl.versions.2 = tlsv1.1
management.ssl.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384
management.ssl.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
management.ssl.ciphers.3 = ECDHE-ECDSA-AES256-SHA384
management.ssl.ciphers.4 = ECDHE-RSA-AES256-SHA384
management.ssl.ciphers.5 = ECDH-ECDSA-AES256-GCM-SHA384
management.ssl.ciphers.6 = ECDH-RSA-AES256-GCM-SHA384
management.ssl.ciphers.7 = ECDH-ECDSA-AES256-SHA384
management.ssl.ciphers.8 = ECDH-RSA-AES256-SHA384
management.ssl.ciphers.9 = DHE-RSA-AES256-GCM-SHA384
## Usually RabbitMQ nodes do not perform peer verification of HTTP API clients
## but it can be enabled if needed. Clients then will have to be configured with
## a certificate and private key pair.
##
## See https://www.rabbitmq.com/ssl.html#peer-verification for details.
# management.ssl.verify = verify_peer
# management.ssl.fail_if_no_peer_cert = true
# ==========================================================================
# ========================= TLS for core server ============================
# ==========================================================================
# source of tls settings: https://www.rabbitmq.com/ssl.html#testssl-sh
listeners.ssl.default = 5671
# ssl_options.cacertfile = /opt/rabbitmq-test/tls-ca/letsencrypt.pem
ssl_options.cacertfile = /etc/letsencrypt/live/{example.com}/chain.pem
ssl_options.certfile = /etc/letsencrypt/live/{example.com}/cert.pem
ssl_options.keyfile = /etc/letsencrypt/live/{example.com}/privkey.pem
ssl_options.versions.1 = tlsv1.2
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
ssl_options.honor_cipher_order = true
ssl_options.honor_ecc_order = true
# These are highly recommended for TLSv1.2 but cannot be used
# with TLSv1.3. If TLSv1.3 is enabled, these lines MUST be removed.
ssl_options.client_renegotiation = false
ssl_options.secure_renegotiate = true
ssl_options.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.3 = ECDH-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.4 = ECDH-RSA-AES256-GCM-SHA384
ssl_options.ciphers.5 = DHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.6 = DHE-DSS-AES256-GCM-SHA384
ssl_options.ciphers.7 = ECDHE-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.8 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.9 = ECDH-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.10 = ECDH-RSA-AES128-GCM-SHA256
ssl_options.ciphers.11 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.12 = DHE-DSS-AES128-GCM-SHA256
If you didn't read all that, the web_stomp plugin is configured to accept wss:// connections on port 15673, and regular ws:// on 15674. Note that I'm not using amqp and therefore amqps is not configured.
With this configuration I'm able to connect to ws://example.com:15674/ws using a webstomp javascript library using this code:
const stompClient = new StompJs.Client({
brokerURL: url,
connectHeaders: {
login: userId,
passcode: password,
},
reconnectDelay: 20000,
heartbeatIncoming: 10000,
heartbeatOutgoing: 10000,
});
// ... set my onConnect and onError listeners
stompClient.activate();
I'm also able to access both http://example.com and https://example.com from the browser. But strangely I
cannot access wss://example.com:15673/ws using the same javascript code above.
When I run sudo docker logs rabbitmq (rabbitmq is the name of the running container), I see entries like this:
2022-02-08 16:06:45.314017+00:00 [info] <0.11704.33> accepting AMQP connection <0.11704.33> (162.142.125.221:54938 -> 172.17.0.2:5672)
2022-02-08 16:06:45.314245+00:00 [erro] <0.11704.33> closing AMQP connection <0.11704.33> (162.142.125.221:54938 -> 172.17.0.2:5672):
2022-02-08 16:06:45.314245+00:00 [erro] <0.11704.33> amqp1_0_plugin_not_enabled
2022-02-08 16:40:01.274168+00:00 [noti] <0.12580.33> TLS server: In state hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
2022-02-08 16:40:01.274168+00:00 [noti] <0.12580.33> - no_suitable_ciphers
2022-02-08 16:40:02.790021+00:00 [noti] <0.12597.33> TLS server: In state hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
2022-02-08 16:40:02.790021+00:00 [noti] <0.12597.33> - no_suitable_ciphers
2022-02-08 16:40:03.423216+00:00 [noti] <0.12604.33> TLS server: In state hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
2022-02-08 16:40:03.423216+00:00 [noti] <0.12604.33> - no_suitable_ciphers
2022-02-08 16:40:06.467195+00:00 [noti] <0.12610.33> TLS server: In state hello at tls_handshake.erl:346 generated SERVER ALERT: Fatal - Insufficient Security
2022-02-08 16:40:06.467195+00:00 [noti] <0.12610.33> - no_suitable_ciphers
2022-02-08 18:12:36.615274+00:00 [noti] <0.16116.33> TLS server: In state hello at tls_record.erl:564 generated SERVER ALERT: Fatal - Unexpected Message
2022-02-08 18:12:36.615274+00:00 [noti] <0.16116.33> - {unsupported_record_type,65}
2022-02-08 18:28:00.774093+00:00 [noti] <0.16535.33> TLS server: In state hello at tls_record.erl:564 generated SERVER ALERT: Fatal - Unexpected Message
2022-02-08 18:28:00.774093+00:00 [noti] <0.16535.33> - {unsupported_record_type,71}
None of these entries correspond to my actual attempts to connect. I don't know which connections are triggering these logs because my actual connection attempts do not show up in the logs.
RabbitMQ's website has a page for troubleshooting ssl, but when I go into the docker container (using docker exec -it rabbitmq bash) and try to use the rabbit-diagnostics command, I get:
Error: unable to perform an operation on node 'rabbit#name-rabbitmq-test.localdomain'. Please see diagnostics information and suggestions below.
I've also tried using the openssl s_client command mentioned in the troubleshooting page, but what I get is Error: verify error:num=2:unable to get issuer certificate. I've tried using both chain.pem and fullchain.pem (the CA files provided by letsencrypt) without any luck, it's the same error.
I'm not a rabbitmq expert so I'm not really sure how to proceed. I tried to include additional info about what I've tried, but please remember that the main problem is that I cannot connect using wss:// even though I can connect with ws:// and https://. Any ideas as to why this happening or what can help me resolve the problem? Thanks.

Answer 1

Many thanks to #jhmckimm for the idea. The problem was that port 15673 from the docker container was not exposed by the host. To fix this I had to add that port to the ExposedPorts in the /var/lib/docker/containers/[hash_of_the_container]/config.v2.json file of the docker system. See here for more information.