Traefik IngressRouteTCP can't handle HTTP2 - azure-aks

I have an IngressRouteTCP configured for Traefik running in an AKS cluster (behind an Azure load balancer). I'm trying to do routing based on SNI, rather than on the hostname header. The certificate is generated by Cloudflare for test.example.com.
As you can see below, it doesn't work. What does work, is setting a TLSOption with alpnProtocols to http/1.1. But that would default to http/1 as I understand it. My application supports http2, so I'd prefer to use that.
I'm not sure why it fails? Is it Traefik, curl or the application?
Testing it with curl -svk --connect-to test.example.com:443:my-azure-load-balancer.cloudapp.azure.com:443 https://test.example.com gives this:
* Connecting to hostname: my-azure-load-balancer.cloudapp.azure.com
* Connecting to port: 443
* Trying x.x.x.x:443...
* TCP_NODELAY set
* Connected to my-azure-load-balancer.cloudapp.azure.com (x.x.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: O=CloudFlare, Inc.; OU=CloudFlare Origin CA; CN=CloudFlare Origin Certificate
* start date: Sep 20 12:40:00 2022 GMT
* expire date: Sep 16 12:40:00 2037 GMT
* issuer: C=US; O=CloudFlare, Inc.; OU=CloudFlare Origin SSL Certificate Authority; L=San Francisco; ST=California
* SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x558d30129860)
> GET / HTTP/2
> Host: test.example.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* http2 error: Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.
* Connection #0 to host my-azure-load-balancer.cloudapp.azure.com left intact
I'm using Traefik version 2.9.1
This is my Kubernetes configuration
apiVersion: apps/v1
kind: Deployment
metadata:
name: whoami
spec:
replicas: 2
selector:
matchLabels:
app: whoami
template:
metadata:
labels:
app: whoami
spec:
containers:
- name: whoami
image: traefik/whoami:v1.6.0
imagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:
name: whoami
labels:
app: whoami
spec:
type: ClusterIP
ports:
- port: 80
name: whoami
selector:
app: whoami
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
name: whoami
spec:
entryPoints:
- websecure
routes:
- match: HostSNI(`test.example.com`)
services:
- name: whoami
port: 80
tls:
secretName: cloudflare-cert
These are the only logs related to this that I can find in the logs (the second log line doesn't always come though):
traefik-c757597b9-2xv65 time="2022-10-31T08:52:59Z" level=debug msg="Handling connection from 10.9.3.227:61988 to 10.9.3.73:80"
traefik-c757597b9-2xv65 time="2022-10-31T08:52:59Z" level=debug msg="Error during connection: read tcp 10.9.3.58:34400->10.9.3.73:80: read: connection reset by peer"
Those IPs are:
10.9.3.227 - The Kubernetes node
10.9.3.58 - Traefik
10.9.3.73 - The service for the whoami pod
I initially tested this behind Cloudflare, and received this:

Related

Traefik problem setting up an MQTT broker + TLS + LETSENCRYPT certificate

I have been trying to get my mqtt + TLS broker behind Traefik to work for hours but without convincing results.
The broker's administration web server works perfectly in https with letsencrypt but I can't produce the equivalent with the MQTT connection on port 8883
Could someone please help me?
Here is my setup
version: '3.4'
services:
reverse-proxy:
image: traefik:${TRAEFIK_VERSION}
restart: unless-stopped
ports:
- 80:80
- 443:443
- 8883:8883
command:
- --log.level=DEBUG
- --providers.docker=true
- --providers.docker.exposedbydefault=false
# Entrypoints
- --entrypoints.web.address=:80
- --entrypoints.webSecure.address=:443
- --entrypoints.mqtt.address=:8883
# Redirect http to https
- --entrypoints.web.http.redirections.entrypoint.to=webSecure
- --entrypoints.web.http.redirections.entrypoint.scheme=https
# Let's encrypt configuration
- --certificatesresolvers.le.acme.email=contact#hexa-ai.fr
- --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
- --certificatesresolvers.le.acme.tlschallenge=true
volumes:
- v_traefik:/letsencrypt
- /var/run/docker.sock:/var/run/docker.sock
emqx-service:
image: emqx/emqx:${EMQX_VERSION}
restart: unless-stopped
environment:
- EMQX_LOADED_PLUGINS="emqx_recon,emqx_retainer,emqx_management,emqx_dashboard,emqx_auth_http"
- EMQX_AUTH__HTTP__AUTH_REQ__URL=${EMQX_AUTH__HTTP__AUTH_REQ__URL}
- EMQX_AUTH__HTTP__AUTH_REQ__PARAMS=clientId=%c,username=%u,password=%P
- EMQX_AUTH__HTTP__ACL_REQ__URL=${EMQX_AUTH__HTTP__ACL_REQ__URL}
- EMQX_AUTH__HTTP__ACL_REQ__PARAMS=access=%A,username=%u,clientId=%c,ipaddr=%a,topic=%t,mountpoint=%m
- EMQX_ZONE__EXTERNAL__PUBLISH_LIMIT=${EMQX_ZONE__EXTERNAL__PUBLISH_LIMIT}
- EMQX_MQTT__MAX_PACKET_SIZE=10MB
volumes:
- v_emqx-data:/opt/emqx/data
- v_emqx-etc:/opt/emqx/etc
- v_emqx-log:/opt/emqx/log
#ports:
# - 8883:8883
# - 3000:18083
labels:
- "traefik.enable=true"
# - "traefik.http.routers.emqx-service.rule=Host(`mqtt.hexa-data.fr`)"
# - "traefik.http.routers.emqx-service.tls.certresolver=le"
# - "traefik.http.services.emqx-service.loadbalancer.server.port=18083"
- "traefik.tcp.routers.emqx-service.rule=HostSNI(`*`)"
- "traefik.tcp.routers.emqx-service.tls.certresolver=le"
- "traefik.tcp.services.emqx-service.loadbalancer.server.port=8883"
# Entrypoints
- "traefik.tcp.routers.emqx-service.entrypoints=mqtt"
#- "traefik.http.routers.emqx-service.entrypoints=webSecure"
extra_hosts:
- "host.docker.internal:host-gateway"
Server logs
time="2022-09-13T10:04:02Z" level=debug msg="Handling connection from 91.164.235.89:38383"
time="2022-09-13T10:04:06Z" level=debug msg="Handling connection from 91.164.235.89:33834"
time="2022-09-13T10:04:06Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:44948->172.19.0.15:8883: remote error: tls: expired certificate"
time="2022-09-13T10:04:06Z" level=debug msg="Error while terminating connection: tls: CloseWrite called before handshake complete"
time="2022-09-13T10:04:11Z" level=debug msg="Handling connection from 82.64.242.74:52648"
time="2022-09-13T10:04:11Z" level=debug msg="Handling connection from 91.164.235.89:45091"
time="2022-09-13T10:04:17Z" level=debug msg="Handling connection from 91.164.235.89:40677"
time="2022-09-13T10:04:21Z" level=debug msg="Handling connection from 91.164.235.89:33836"
time="2022-09-13T10:04:21Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:44972->172.19.0.15:8883: remote error: tls: expired certificate"
time="2022-09-13T10:04:21Z" level=debug msg="Error while terminating connection: tls: CloseWrite called before handshake complete"
time="2022-09-13T10:04:26Z" level=debug msg="Handling connection from 82.64.242.74:52710"
time="2022-09-13T10:04:26Z" level=debug msg="Error while setting deadline: set tcp 172.19.0.16:44976: use of closed network connection"
time="2022-09-13T10:04:26Z" level=debug msg="Handling connection from 91.164.235.89:37076"
time="2022-09-13T10:04:33Z" level=debug msg="Handling connection from 91.164.235.89:47424"
time="2022-09-13T10:04:36Z" level=debug msg="Handling connection from 91.164.235.89:33838"
time="2022-09-13T10:04:36Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:44990->172.19.0.15:8883: remote error: tls: expired certificate"
time="2022-09-13T10:04:36Z" level=debug msg="Error while terminating connection: tls: CloseWrite called before handshake complete"
time="2022-09-13T10:04:41Z" level=debug msg="Handling connection from 82.64.242.74:50496"
time="2022-09-13T10:04:42Z" level=debug msg="Handling connection from 91.164.235.89:45524"
time="2022-09-13T10:04:48Z" level=debug msg="Handling connection from 91.164.235.89:43400"
time="2022-09-13T10:04:51Z" level=debug msg="Handling connection from 91.164.235.89:33840"
time="2022-09-13T10:04:51Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:45010->172.19.0.15:8883: remote error: tls: expired certificate"
time="2022-09-13T10:04:51Z" level=debug msg="Error while terminating connection: tls: CloseWrite called before handshake complete"
time="2022-09-13T10:04:54Z" level=debug msg="Handling connection from 91.164.235.89:41826"
time="2022-09-13T10:04:55Z" level=debug msg="Handling connection from 91.164.235.89:38537"
time="2022-09-13T10:04:56Z" level=debug msg="Error while setting deadline: set tcp 172.19.0.16:45016: use of closed network connection"
time="2022-09-13T10:04:56Z" level=debug msg="Handling connection from 82.64.242.74:46692"
time="2022-09-13T10:04:56Z" level=debug msg="Error while setting deadline: set tcp 172.19.0.16:45018: use of closed network connection"
time="2022-09-13T10:04:57Z" level=debug msg="Handling connection from 91.164.235.89:37987"
Client conf
Client logs
[2022-09-13 12:04:56] [INFO] MQTTX client with ID 1fa93978-f254-438e-bc39-e2830403351f assigned
[2022-09-13 12:04:56] [INFO] Connect client PFC2004G, MQTT/SSL connection: mqtts://mqtt.hexa-data.fr:8883
[2022-09-13 12:04:56] [INFO] PFC2004G connect close, MQTT.js onClose trigger
The important part of the log would appear to be this part from server.log
time="2022-09-13T10:04:06Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:44948->172.19.0.15:8883: remote error: tls: expired certificate"
This implies the client has closed the connection to the broker because the certificate presented has expired.
I would first check the time/timezone for the server and the client.
Also I'm not clear how you've set the hostname (or domain, since it's set up as a wildcard hostname) the for LetsEncrypt certificate for the MQTT connection.
I would use something like openssl s_client to check what certificate is being presented
openssl s_client -connect mqtt.hexa-data.fr:8883

Traefik failing TLS handshake with Let's Encrypt Certificate

I am attempting to have Traefik serve as a reverse proxy for services running in Docker containers. I've been following the documentation that Traefik provides and have a small docker environment configured via docker compose that successfully serves data via HTTP. Traefik sits behind HAProxy running in TCP mode forwarding packets received from the Internet to Traefik.
However when I tried to add a new router for serving the same content via HTTPS, I receive the following esoteric (to me) error when I run a curl directed to https://my.domain.tld/: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Full curl output:
curl -v https://my.domain.tld/
* Trying <IP Address of Domain>...
* TCP_NODELAY set
* Connected to my.domain.tld (<IP Address of Domain>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
When I attempt to browse to the site via Firefox (web browser) I receive an error code of SSL_ERROR_RX_RECORD_TOO_LONG. When googling this error I was unable to find a post that seemed to have my specific issue.
Below is the docker-compose for the setup I am using to configure the applications
version: "3.9"
secrets:
cloudflare_dns_token:
file: ./secrets/cf_dns_api_token.txt
networks:
socket_proxy:
name: socket_proxy
driver: bridge
ipam:
config:
- subnet: 192.168.0.0/24
container_bridge:
name: container_bridge
driver: bridge
ipam:
config:
- subnet: 192.168.1.0/24
services:
socket-proxy:
image: tecnativa/docker-socket-proxy
container_name: socket-proxy
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
socket_proxy:
ipv4_address: 192.168.0.2 # Static IP address
environment:
EVENTS: 1
PING: 1
VERSION: 1
CONTAINERS: 1
NETWORKS: 1
traefik:
# The official v2 Traefik docker image
image: traefik:v2.8.1
container_name: traefik-proxy
command:
# Log Level for Traefik
- "--log.level=DEBUG"
# Enables the web UI
- "--api.insecure=true"
# Traefik enables docker as the provider to look for services
- "--providers.docker=true"
# Traefik will use the Docker Socket proxy to communicate with the docker socket
- "--providers.docker.endpoint=tcp://192.168.0.2:2375"
# Traefik will not expose services if they aren't labled for export
- "--providers.docker.exposedByDefault=false"
# Port where Traefik will listen for web (http) traffic for routing
- "--entrypoints.web.address=:80"
# Port where Traefik will listen for web secure (https) traffic for routing
- "--entrypoints.websecure.address=:443"
# Trust Proxy Protocol Packets from only the listed IP address
- "--entryPoints.web.proxyProtocol.trustedIPs=10.0.8.1/32"
# Trust Proxy Protocol Packets from only the listed IP address
- "--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.8.1/32"
# Enable a ACME DNS challenge named "letsencrypt"
- "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
# Tell Traefik which provider to use for DNS Challenge
- "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare"
# Staging environment for let's encrypt for testing
- "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
# Email to provide to let's encrypt
- "--certificatesresolvers.letsencrypt.acme.email=${EMAIL}"
# Tell Traefik to store the certificate on a path under our volume
- "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
networks:
# Tells Traefik to connect to both the socket proxy network and the container bridge network where the other containers will be connected
socket_proxy:
container_bridge:
ports:
# The HTTP port
- "80:80"
# The HTTPS port
- "443:443"
# The Web UI (enabled by --api.insecure=true)
- "8080:8080"
secrets:
- "cloudflare_dns_token"
environment:
# To Be Removed Need Secret Working Properly
- "CF_DNS_API_TOKEN=${CF_DNS_TOKEN}"
#- "CF_DNS_API_TOKEN=/run/secrets/cloudflare_dns_token"
volumes:
# Create a letsencrypt dir within the folder where the docker-compose file is
- "./letsencrypt:/letsencrypt"
whoami:
image: traefik/whoami
container_name: whoami-server
networks:
container_bridge:
labels:
# Tells Traefik to proxy to the service (container)
- "traefik.enable=true"
#####################################################################
#
# Labels for HTTPS Proxying
#
#####################################################################
# Explicitly stating 'whoami-secure' route is HTTPS
- "traefik.http.routers.whoami-secure.tls=true"
# Rule for determing when to route requests to this service for the secure http router
- "traefik.http.routers.whoami-secure.rule=Host(`whoami.${FQDN}`)"
# Entry point for requests to this service for the secure http router
- "traefik.http.routers.whoami-secure.entrypoints=websecure"
# Uses the Host rule to define which certificate to issue
- "traefik.http.routers.whoami-secure.tls.certresolver=letsencrypt"
#####################################################################
#
# Labels for HTTP Proxying
#
#####################################################################
# Rule for determing when to route requests to this service for the unsecure http router
- "traefik.http.routers.whoami.rule=Host(`whoami.${FQDN}`)"
# Entry point for requests to this service for the unsecure http router
- "traefik.http.routers.whoami.entrypoints=web"
My expectation is that Traefik would gracefully handle the request via HTTPS and manage the TLS handshake without issue. I can confirm that Traefik is able to successfully generate a certificate via Let's Encrypt DNS Challenge for Cloudflare. I am using the Let's Encrypt staging environment at the moment so I did expect an error about the certificate being served as invalid, but it seems that TLS handshake errors out before a determination of validity.
EDIT #1: Running OpenSSL and Wireshark
OpenSSL returns the following when run ```openssl s_client -connect my.domain.tld:443``
CONNECTED(00000003)
140330304906560:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:331:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 315 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
Wireshark logs show the following:
3 TCP packets preceding the first TLSv1 Client Hello
The Client Hello is acknowledged by the server
Then the server returns a HTTP 400 error - Bad Request
Wireshark dump of error
Transmission Control Protocol, Src Port: 443, Dst Port: 50085, Seq: 1, Ack: 518, Len: 207
Source Port: 443
Destination Port: 50085
[Stream index: 0]
[Conversation completeness: Complete, WITH_DATA (63)]
[TCP Segment Len: 207]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 1488204866
[Next Sequence Number: 208 (relative sequence number)]
Acknowledgment Number: 518 (relative ack number)
Acknowledgment number (raw): 500352812
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 1... = Push: Set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: ·······AP···]
Window: 501
[Calculated window size: 64128]
[Window size scaling factor: 128]
Checksum: 0x1155 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (207 bytes)
Hypertext Transfer Protocol
[Expert Info (Warning/Security): Unencrypted HTTP protocol detected over encrypted port, could indicate a dangerous misconfiguration.]
[Unencrypted HTTP protocol detected over encrypted port, could indicate a dangerous misconfiguration.]
[Severity level: Warning]
[Group: Security]
HTTP/1.1 400 Bad request\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 400 Bad request\r\n]
[HTTP/1.1 400 Bad request\r\n]
[Severity level: Chat]
[Group: Sequence]
Response Version: HTTP/1.1
Status Code: 400
[Status Code Description: Bad Request]
Response Phrase: Bad request
Content-length: 90\r\n
Cache-Control: no-cache\r\n
Connection: close\r\n
Content-Type: text/html\r\n
\r\n
[HTTP response 1/1]
File Data: 90 bytes
5 packets later the connection is reset
Wireguard dump of connection reset
Transmission Control Protocol, Src Port: 443, Dst Port: 50085, Seq: 208, Len: 0
Source Port: 443
Destination Port: 50085
[Stream index: 0]
[Conversation completeness: Complete, WITH_DATA (63)]
[TCP Segment Len: 0]
Sequence Number: 208 (relative sequence number)
Sequence Number (raw): 1488205073
[Next Sequence Number: 208 (relative sequence number)]
Acknowledgment Number: 0
Acknowledgment number (raw): 0
0101 .... = Header Length: 20 bytes (5)
Flags: 0x004 (RST)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...0 .... = Acknowledgment: Not set
.... .... 0... = Push: Not set
.... .... .1.. = Reset: Set
[Expert Info (Warning/Sequence): Connection reset (RST)]
[Connection reset (RST)]
[Severity level: Warning]
[Group: Sequence]
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: ·········R··]
Window: 0
[Calculated window size: 0]
[Window size scaling factor: 128]
Checksum: 0x6dc9 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]

TLS with Rabbitmq Docker-Image: handshake_failure

I am running rabbitmq:3.10.7-management for AMQPS on a VM and I am using chained TLS-Certificates.
When I configure Rabbitmq according to the How-To-TLS website from Rabbitmq I get no errors starting the container but I can't connect via TLS and get an immediate Connection-Error. Depending on which Tool I use or where I use it I get a connection reset by peer. But it seems like the handshake does not work at all.
When I enable TCP I can just connect fine with a non-TLS-client.
The Rabbitmq part of the Docker-Compose looks like this:
rabbitmq:
restart: unless-stopped
hostname: rabbitmq
image: rabbitmq:3.10.7-management
networks:
- traefik
ports:
- "5672:5672"
- "5671:5671"
logging:
options:
max-size: "10m"
max-file: "3"
volumes:
- ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf
- ./certs:/certs
- /data/rabbitmq_data:/var/lib/rabbitmq
labels:
- "traefik.enable=true"
- "traefik.http.routers.rabbitmq-secure.priority=150"
- "traefik.http.services.rabbitmq-secure.loadbalancer.server.port=15672"
- "traefik.http.routers.rabbitmq-secure.rule=Host(`<myDomain>`) && PathPrefix(`/`)"
- "traefik.http.routers.rabbitmq-secure.entrypoints=web-secure"
- "traefik.http.routers.rabbitmq-secure.tls=true"
- "traefik.http.routers.rabbitmq-secure.tls.options=myTLSOptions#file"
- "traefik.http.routers.rabbitmq.rule=Host(`<myDomain>`) && PathPrefix(`/`)"
- "traefik.http.routers.rabbitmq.entrypoints=web"
I am using the same certs to serve the management frontend with no problems.
ca_certificate.pem ->
CN=T-TeleSec GlobalRoot Class 2, OU=T-Systems Trust Center, O=T-Systems Enterprise Services GmbH, C=DE
Chain-Cert 1
Chain-Cert 2
server_certificate.pem ->
Wild-Card-Cert
server_key.pem ->
Key for Wild-Card-Cert
I think I tried almost every other configuration as well, for which part goes where.
RabbitMQ-Conf:
log.console.level = debug
listeners.ssl.default = 5671
listeners.tcp = none
ssl_options.cacertfile = /certs/ca_certificate.pem
ssl_options.certfile = /certs/server_certificate.pem
ssl_options.keyfile = /certs/server_key.pem
ssl_options.verify = verify_none
ssl_options.fail_if_no_peer_cert = false
ssl_options.versions.1 = tlsv1.3
ssl_options.versions.2 = tlsv1.2
ssl_options.honor_cipher_order = false
ssl_options.honor_ecc_order = false
ssl_handshake_timeout = 10000
ssl_options.ciphers.1 = ECDHE-ECDSA-AES256-GCM-SHA384
ssl_options.ciphers.2 = ECDHE-RSA-AES256-GCM-SHA384
ssl_options.ciphers.3 = ECDHE-ECDSA-AES128-GCM-SHA256
ssl_options.ciphers.4 = ECDHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.5 = ECDHE-ECDSA-CHACHA20-POLY1305
ssl_options.ciphers.6 = ECDHE-RSA-CHACHA20-POLY1305
ssl_options.ciphers.7 = DHE-RSA-AES128-GCM-SHA256
ssl_options.ciphers.8 = DHE-RSA-AES256-GCM-SHA384
Logs seem to be OK:
...
2022-09-06 14:42:22.024539+00:00 [info] <0.615.0> started TLS (SSL) listener on [::]:5671
...
Going through "Troubleshooting TLS-enabled Connections" I can check all the boxes I can find up until connecting with TLS:
Check Listeners:
rabbitmq#rabbitmq:/$ rabbitmq-diagnostics listeners
Asking node rabbit#rabbitmq to report its protocol listeners ...
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS
Check permissions:
rabbitmq#rabbitmq:/certs$ ls -ll
total 20
-rw-r--r-- 1 rabbitmq rabbitmq 1366 Sep 6 14:39 ca_certificate.pem
-rw-r--r-- 1 rabbitmq rabbitmq 6943 Sep 6 14:39 server_certificate.pem
-rw-r--r-- 1 rabbitmq rabbitmq 3275 Sep 5 13:20 server_key.pem
Check TLS-support in erlang:
rabbitmq#rabbitmq:/certs$ rabbitmq-diagnostics --silent tls_versions
tlsv1.3
tlsv1.2
tlsv1.1
tlsv1
Attempt TLS-Connection with openssl
rabbitmq#rabbitmq:/certs$ openssl s_client -connect localhost:5671
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 293 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
The debug logs don't even show any error when trying to connect.
Trying to connect with TLS to TCP throws an error, though. Somehow the listener does not accept connections at all.
Maybe someone experienced something similar already or I am just dumb overlooking the obvious. But for everything else the VM and the certs work just fine (e.g. Mosquitto-MQTTS).
Help would be really appreciated.

Minimum tls configuration not working in docker private repository

Using below configuration to set minimum tls version in Docker private registry.
http:
addr: 0.0.0.0:5000
net: tcp
relativeurls: false
draintimeout: 60s
host: https://example.com:443
tls:
certificate: /certs/domain.crt
key: /certs/domain.key
minimumtls: tls1.3
ciphersuites:
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
But when u run below command. It is showing tlsv1 is enabled.
openssl s_client -connect example.com:443 -tls1
********
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1
Cipher : ECDHE-RSA-AES128-SHA
Session-ID: A611949FBC6ED8ED322D4360C7B32887C75BE6HJFU5BA9B536C3721924A103F3E
********

dockerhub registery: x509: certificate signed by unknown authority

I've spend hours looking to solve this issue, however I'm unable to find any topics related to this issue, since all I find is custom registeries.
When running any of the docker commands that connect to docker hub, either through https://registry-1.docker.io/v2/ or https://index.docker.io/v1, all requests end up in "x509: certificate signed by unknown authority". However using curl to run query the same endpoints seem to function just fine.
I've reinstalled docker completely, purging all configuration files, however it does not seem to make a difference.
Anything I'm missing?
docker info:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 17.05.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.35-1-lts
Operating System: Arch Linux
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.34GiB
ID: 5Q4D:TLJF:3I3U:O522:VQMK:24BU:H5ND:UPOU:MWYS:WGTB:XFXR:BQES
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Ena
Using docker:
[user#hostname]$ docker search ubunut
Error response from daemon: Get https://index.docker.io/v1/search?q=ubunut&n=25: x509: certificate signed by unknown authority
Using curl:
[user#hostname]$ curl -v https://index.docker.io/v1/search?q=ubunut&n=25
[1] 2152
[user#hostname]$ * Trying 34.200.194.233...
* TCP_NODELAY set
* Connected to index.docker.io (34.200.194.233) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:#STRENGTH
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: OU=GT98568428; OU=See www.rapidssl.com/resources/cps (c)15; OU=Domain Control Validated - RapidSSL(R); CN=*.docker.io
* start date: Mar 19 17:34:32 2015 GMT
* expire date: Apr 21 01:51:52 2018 GMT
* subjectAltName: host "index.docker.io" matched cert's "*.docker.io"
* issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G3
* SSL certificate verify ok.
> GET /v1/search?q=ubunut HTTP/1.1
> Host: index.docker.io
> User-Agent: curl/7.54.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.6.2
< Date: Wed, 05 Jul 2017 12:10:22 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Vary: Cookie
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=31536000
<
{"num_pages": 1, "num_results": 21, "results": [{"is_automated": true, "name": "han4wluc/try-docker-ubunut-node", "is_trusted": true, ... *truncated*
I solved the problem as follows:
I removed the file /etc/ssl/cert/ca-certificates.crt.
I ran the command sudo pacman -S ca-certificates-utils.
I restarted docker with the systemctl restart docker command.
I got this hint from this link:
https://unix.stackexchange.com/questions/339613/arch-linux-ca-certificates-crt-not-found/396169#396169

Resources