pika rabbitmq docker with tls-gen connection reset, no logs - docker

Question

I've configured a docker w rabbitmq, copied the certs from tls-gen there, stop/started and I can only get the connection established and a server reset right after sending a TLS Client Hello. No useful message from ssl. Logs all empty. Troubleshoot TLS guide didn't help. Thanks for any help.
docker run -d -it --hostname=[REDACTED] -e RABBITMQ_LOGS=/var/log/rabbitmq/ -e RABBITMQ_SASL_LOGS=/var/log/rabbitmq/ -e RABBITMQ_DEFAULT_USER=[REDACTED] -e RABBITMQ_DEFAULT_PASS=[REDACTED] --name rabbitmq1 -p [REDACTED]:5672 -p [REDACTED]:15672 -v /scratch/databases/rabbitmq/var/lib/rabbitmq:/var/lib/rabbitmq rabbitmq:3-management
docker cp tls-gen/basic/result/ca_certificate.pem rabbitmq1:/etc/rabbitmq/
docker cp tls-gen/basic/result/server_key.pem rabbitmq1:/etc/rabbitmq/
docker cp tls-gen/basic/result/server_certificate.pem rabbitmq1:/etc/rabbitmq/
docker exec -it rabbitmq1 /bin/bash -c "cd /etc/rabbitmq;chown rabbitmq:rabbitmq *.pem"
docker cp rabbitmq.conf rabbitmq1:/etc/rabbitmq/
docker exec -it rabbitmq1 /bin/bash -c "cd /etc/rabbitmq;chown rabbitmq:rabbitmq rabbitmq.conf"
I was sure to restart the container so the changes were applied. Also double checked cert permissions.
docker:/etc/rabbitmq/
-rw-rw-r--. 1 rabbitmq rabbitmq 1.2K Nov 22 02:29 ca_certificate.pem
drwxrwxrwx. 1 rabbitmq rabbitmq 61 Nov 6 01:51 conf.d
-rw-r--r--. 1 root root 71 Nov 22 07:34 enabled_plugins
-rw-r--r--. 1 rabbitmq rabbitmq 578 Nov 22 07:34 rabbitmq.conf
-rw-rw-r--. 1 rabbitmq rabbitmq 1.3K Nov 22 02:29 server_certificate.pem
-rw-------. 1 rabbitmq rabbitmq 1.9K Nov 22 02:29 server_key.pem
Redirected logs here knowing that the container start env variables should overide. Either way, logs seemed empty.
/etc/rabbitmq/rabbitmq.conf
loopback_users.guest = false
# listeners.tcp.default = 5672
management.tcp.port = 15672
listeners.tcp = none
listeners.ssl.default = 5671
ssl_options.cacertfile = /etc/rabbitmq/ca_certificate.pem
ssl_options.certfile = /etc/rabbitmq/server_certificate.pem
ssl_options.keyfile = /etc/rabbitmq/server_key.pem
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
ssl_options.password = [REDACTED]
log.dir = /tmp/
log.file = r.log
log.file.level = debug
Tried multiple configurations for context.options to no avail.
receive.py (python)
#!/usr/bin/env python
import ssl, pika, sys, os
import traceback
from pika.credentials import ExternalCredentials
def main():
connection = None
credentials = pika.PlainCredentials('[REDACTED]', '[REDACTED]')
context = ssl.create_default_context(cafile="tls-gen/basic/result/ca_certificate.pem")
context.load_cert_chain("tls-gen/basic/result/client_certificate.pem", "tls-gen/basic/result/client_key.pem")
#context.options = dict(ssl_version=ssl.PROTOCOL_TLSv1, cert_reqs=ssl.CERT_REQUIRED)
ssl_options = pika.SSLOptions(context, "[REDACTED]")
#ssl_options = pika.SSLOptions(ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT), "local")
#ssl_options = pika.SSLOptions(ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER), "localhost")
#ssl_options = ssl_options | pika.SSLOptions(ssl.SSLContext(ssl.PROTOCOL_TLSv1_2), "[REDACTED]")
try:
connection = pika.BlockingConnection(pika.ConnectionParameters(host='[REDACTED]',
port=[REDACTED],
virtual_host='/',
ssl_options = ssl_options)),
credentials=credentials))
#credentials=ExternalCredentials()))
except Exception as e:
exc_type, _, exc_tb = sys.exc_info()
fname = os.path.split(exc_tb.tb_frame.f_code.co_filename)[1]
print("receive: ({0}) {1} {2} {3}".format(str(e), exc_type, fname, exc_tb.tb_lineno))
traceback.print_exc()
sys.exit(1)
channel = connection.channel()
channel.queue_declare(queue='hello')
def callback(ch, method, properties, body):
print(" [x] Received %r" % body)
channel.basic_consume(queue='hello', on_message_callback=callback, auto_ack=True)
print(' [*] Waiting for messages. To exit press CTRL+C')
channel.start_consuming()
if __name__ == '__main__':
try:
main()
except KeyboardInterrupt:
print('Interrupted')
try:
sys.exit(0)
except SystemExit:
os._exit(0)
A wrong PEM password here generates a very different message, I'm sure I'm using the right one.
python3 receive.py
Enter PEM pass phrase:
receive: ([Errno 104] Connection reset by peer) <class 'ConnectionResetError'> receive.py 18
Traceback (most recent call last):
File "receive.py", line 18, in main
connection = pika.BlockingConnection(pika.ConnectionParameters(host='[REDACTED]',
File "[REDACTED]/rabbitmq/pika/venv/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 359, in __init__
self._impl = self._create_connection(parameters, _impl_class)
File "[REDACTED]/rabbitmq/pika/venv/lib/python3.8/site-packages/pika/adapters/blocking_connection.py", line 450, in _create_connection
raise self._reap_last_connection_workflow_error(error)
File "[REDACTED]/rabbitmq/pika/venv/lib/python3.8/site-packages/pika/adapters/utils/io_services_utils.py", line 636, in _do_ssl_handshake
self._sock.do_handshake()
File "/usr/local/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer
It looks like openssl likes the connection.
openssl s_client -connect localhost:5671
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 CN = TLSGenSelfSignedtRootCA, L = $$$$
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=0 CN = [REDACTED], O = server
verify return:1
---
Certificate chain
0 s:CN = [REDACTED], O = server
i:CN = TLSGenSelfSignedtRootCA, L = $$$$
1 s:CN = TLSGenSelfSignedtRootCA, L = $$$$
i:CN = TLSGenSelfSignedtRootCA, L = $$$$
---
Server certificate
-----BEGIN CERTIFICATE-----
[REDACTED]
-----END CERTIFICATE-----
subject=CN = [REDACTED], O = server
issuer=CN = TLSGenSelfSignedtRootCA, L = $$$$
---
Acceptable client certificate CA names
CN = TLSGenSelfSignedtRootCA, L = $$$$
Client Certificate Types: ECDSA sign, RSA sign, DSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2338 bytes and written 431 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: [REDACTED]
Session-ID-ctx:
Master-Key: [REDACTED]
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1606031943
Timeout : 7200 (sec)
Verify return code: 19 (self signed certificate in certificate chain)
Extended master secret: no
---
closed

Answer 1

here the steps I took to fix this:
Get rid of the docker and install a local rabbitmq on mac for debugging this configuration. For some reason, the rabbitmq docker wasn't logging anything;
I've added a real valid letsencrypt certificate to the server;
first error I saw in the client after that was the following:
ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1076)
This is because verify_peer is enabled, in that case we needed to bundle the ca_certificate.pem from tls-gen/basic/result/ca_certificate.pem to the chain.pem
cat ../tls-gen/basic/result/ca_certificate.pem chain.pem > chain_bundle.pem
rabbitmq.conf
...
ssl_options.cacertfile = /usr/local/etc/rabbitmq/chain_bundle.pem # <- bundled w self signed cert
ssl_options.certfile = /usr/local/etc/rabbitmq/cert.pem
ssl_options.keyfile = /usr/local/etc/rabbitmq/privkey.pem
...
after that, a different error on pika showed:
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)
That's because pika didn't like tls-gen/basic/result/ca_certificate.pem, instead, I used venv//lib/python3.7/site-packages/certifi/cacert.pem after installing pip install certifi.
Later on, I bundled tls-gen/basic/result/ca_certificate.pem + venv//lib/python3.7/site-packages/certifi/cacert.pem, and added a password to context.load_cert_chain
Wireshark shows messages fully encrypted, now I just need to deploy.
EDIT:
I was able to run it via docker after re-trying the configuration that worked:
#/bin/bash
docker run -d -it --hostname=\
-e RABBITMQ_LOGS=/tmp/log\
-e RABBITMQ_DEFAULT_USER=\
-e RABBITMQ_DEFAULT_PASS=\
-e RABBITMQ_SSL_CACERTFILE=/cert/chain_bundle2.pem \
-e RABBITMQ_SSL_CERTFILE=/cert/cert.pem\
-e RABBITMQ_SSL_KEYFILE=/cert/privkey.pem\
-e RABBITMQ_SSL_FAIL_IF_NO_PEER_CERT=true\
-e RABBITMQ_SSL_VERIFY=verify_peer\
--name rabbitmq1-verifypeer -p ... :5671 -p ... :15672\
-v ... /rabbitmq/var/log:/tmp/log\
-v ... /rabbitmq/var/lib/rabbitmq:/var/lib/rabbitmq\
-v ... /cert/:/cert/ rabbitmq:3.8.9-management