Traefik failing TLS handshake with Let's Encrypt Certificate - docker

Question

I am attempting to have Traefik serve as a reverse proxy for services running in Docker containers. I've been following the documentation that Traefik provides and have a small docker environment configured via docker compose that successfully serves data via HTTP. Traefik sits behind HAProxy running in TCP mode forwarding packets received from the Internet to Traefik.
However when I tried to add a new router for serving the same content via HTTPS, I receive the following esoteric (to me) error when I run a curl directed to https://my.domain.tld/: error:1408F10B:SSL routines:ssl3_get_record:wrong version number
Full curl output:
curl -v https://my.domain.tld/
* Trying <IP Address of Domain>...
* TCP_NODELAY set
* Connected to my.domain.tld (<IP Address of Domain>) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
When I attempt to browse to the site via Firefox (web browser) I receive an error code of SSL_ERROR_RX_RECORD_TOO_LONG. When googling this error I was unable to find a post that seemed to have my specific issue.
Below is the docker-compose for the setup I am using to configure the applications
version: "3.9"
secrets:
cloudflare_dns_token:
file: ./secrets/cf_dns_api_token.txt
networks:
socket_proxy:
name: socket_proxy
driver: bridge
ipam:
config:
- subnet: 192.168.0.0/24
container_bridge:
name: container_bridge
driver: bridge
ipam:
config:
- subnet: 192.168.1.0/24
services:
socket-proxy:
image: tecnativa/docker-socket-proxy
container_name: socket-proxy
restart: unless-stopped
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
socket_proxy:
ipv4_address: 192.168.0.2 # Static IP address
environment:
EVENTS: 1
PING: 1
VERSION: 1
CONTAINERS: 1
NETWORKS: 1
traefik:
# The official v2 Traefik docker image
image: traefik:v2.8.1
container_name: traefik-proxy
command:
# Log Level for Traefik
- "--log.level=DEBUG"
# Enables the web UI
- "--api.insecure=true"
# Traefik enables docker as the provider to look for services
- "--providers.docker=true"
# Traefik will use the Docker Socket proxy to communicate with the docker socket
- "--providers.docker.endpoint=tcp://192.168.0.2:2375"
# Traefik will not expose services if they aren't labled for export
- "--providers.docker.exposedByDefault=false"
# Port where Traefik will listen for web (http) traffic for routing
- "--entrypoints.web.address=:80"
# Port where Traefik will listen for web secure (https) traffic for routing
- "--entrypoints.websecure.address=:443"
# Trust Proxy Protocol Packets from only the listed IP address
- "--entryPoints.web.proxyProtocol.trustedIPs=10.0.8.1/32"
# Trust Proxy Protocol Packets from only the listed IP address
- "--entryPoints.websecure.proxyProtocol.trustedIPs=10.0.8.1/32"
# Enable a ACME DNS challenge named "letsencrypt"
- "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
# Tell Traefik which provider to use for DNS Challenge
- "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare"
# Staging environment for let's encrypt for testing
- "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
# Email to provide to let's encrypt
- "--certificatesresolvers.letsencrypt.acme.email=${EMAIL}"
# Tell Traefik to store the certificate on a path under our volume
- "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
networks:
# Tells Traefik to connect to both the socket proxy network and the container bridge network where the other containers will be connected
socket_proxy:
container_bridge:
ports:
# The HTTP port
- "80:80"
# The HTTPS port
- "443:443"
# The Web UI (enabled by --api.insecure=true)
- "8080:8080"
secrets:
- "cloudflare_dns_token"
environment:
# To Be Removed Need Secret Working Properly
- "CF_DNS_API_TOKEN=${CF_DNS_TOKEN}"
#- "CF_DNS_API_TOKEN=/run/secrets/cloudflare_dns_token"
volumes:
# Create a letsencrypt dir within the folder where the docker-compose file is
- "./letsencrypt:/letsencrypt"
whoami:
image: traefik/whoami
container_name: whoami-server
networks:
container_bridge:
labels:
# Tells Traefik to proxy to the service (container)
- "traefik.enable=true"
#####################################################################
#
# Labels for HTTPS Proxying
#
#####################################################################
# Explicitly stating 'whoami-secure' route is HTTPS
- "traefik.http.routers.whoami-secure.tls=true"
# Rule for determing when to route requests to this service for the secure http router
- "traefik.http.routers.whoami-secure.rule=Host(`whoami.${FQDN}`)"
# Entry point for requests to this service for the secure http router
- "traefik.http.routers.whoami-secure.entrypoints=websecure"
# Uses the Host rule to define which certificate to issue
- "traefik.http.routers.whoami-secure.tls.certresolver=letsencrypt"
#####################################################################
#
# Labels for HTTP Proxying
#
#####################################################################
# Rule for determing when to route requests to this service for the unsecure http router
- "traefik.http.routers.whoami.rule=Host(`whoami.${FQDN}`)"
# Entry point for requests to this service for the unsecure http router
- "traefik.http.routers.whoami.entrypoints=web"
My expectation is that Traefik would gracefully handle the request via HTTPS and manage the TLS handshake without issue. I can confirm that Traefik is able to successfully generate a certificate via Let's Encrypt DNS Challenge for Cloudflare. I am using the Let's Encrypt staging environment at the moment so I did expect an error about the certificate being served as invalid, but it seems that TLS handshake errors out before a determination of validity.
EDIT #1: Running OpenSSL and Wireshark
OpenSSL returns the following when run ```openssl s_client -connect my.domain.tld:443``
CONNECTED(00000003)
140330304906560:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../ssl/record/ssl3_record.c:331:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 5 bytes and written 315 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
Wireshark logs show the following:
3 TCP packets preceding the first TLSv1 Client Hello
The Client Hello is acknowledged by the server
Then the server returns a HTTP 400 error - Bad Request
Wireshark dump of error
Transmission Control Protocol, Src Port: 443, Dst Port: 50085, Seq: 1, Ack: 518, Len: 207
Source Port: 443
Destination Port: 50085
[Stream index: 0]
[Conversation completeness: Complete, WITH_DATA (63)]
[TCP Segment Len: 207]
Sequence Number: 1 (relative sequence number)
Sequence Number (raw): 1488204866
[Next Sequence Number: 208 (relative sequence number)]
Acknowledgment Number: 518 (relative ack number)
Acknowledgment number (raw): 500352812
0101 .... = Header Length: 20 bytes (5)
Flags: 0x018 (PSH, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 1... = Push: Set
.... .... .0.. = Reset: Not set
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: ·······AP···]
Window: 501
[Calculated window size: 64128]
[Window size scaling factor: 128]
Checksum: 0x1155 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]
[SEQ/ACK analysis]
TCP payload (207 bytes)
Hypertext Transfer Protocol
[Expert Info (Warning/Security): Unencrypted HTTP protocol detected over encrypted port, could indicate a dangerous misconfiguration.]
[Unencrypted HTTP protocol detected over encrypted port, could indicate a dangerous misconfiguration.]
[Severity level: Warning]
[Group: Security]
HTTP/1.1 400 Bad request\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 400 Bad request\r\n]
[HTTP/1.1 400 Bad request\r\n]
[Severity level: Chat]
[Group: Sequence]
Response Version: HTTP/1.1
Status Code: 400
[Status Code Description: Bad Request]
Response Phrase: Bad request
Content-length: 90\r\n
Cache-Control: no-cache\r\n
Connection: close\r\n
Content-Type: text/html\r\n
\r\n
[HTTP response 1/1]
File Data: 90 bytes
5 packets later the connection is reset
Wireguard dump of connection reset
Transmission Control Protocol, Src Port: 443, Dst Port: 50085, Seq: 208, Len: 0
Source Port: 443
Destination Port: 50085
[Stream index: 0]
[Conversation completeness: Complete, WITH_DATA (63)]
[TCP Segment Len: 0]
Sequence Number: 208 (relative sequence number)
Sequence Number (raw): 1488205073
[Next Sequence Number: 208 (relative sequence number)]
Acknowledgment Number: 0
Acknowledgment number (raw): 0
0101 .... = Header Length: 20 bytes (5)
Flags: 0x004 (RST)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...0 .... = Acknowledgment: Not set
.... .... 0... = Push: Not set
.... .... .1.. = Reset: Set
[Expert Info (Warning/Sequence): Connection reset (RST)]
[Connection reset (RST)]
[Severity level: Warning]
[Group: Sequence]
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: ·········R··]
Window: 0
[Calculated window size: 0]
[Window size scaling factor: 128]
Checksum: 0x6dc9 [unverified]
[Checksum Status: Unverified]
Urgent Pointer: 0
[Timestamps]