Traefik problem setting up an MQTT broker + TLS + LETSENCRYPT certificate - docker

Question

I have been trying to get my mqtt + TLS broker behind Traefik to work for hours but without convincing results.
The broker's administration web server works perfectly in https with letsencrypt but I can't produce the equivalent with the MQTT connection on port 8883
Could someone please help me?
Here is my setup
version: '3.4'
services:
reverse-proxy:
image: traefik:${TRAEFIK_VERSION}
restart: unless-stopped
ports:
- 80:80
- 443:443
- 8883:8883
command:
- --log.level=DEBUG
- --providers.docker=true
- --providers.docker.exposedbydefault=false
# Entrypoints
- --entrypoints.web.address=:80
- --entrypoints.webSecure.address=:443
- --entrypoints.mqtt.address=:8883
# Redirect http to https
- --entrypoints.web.http.redirections.entrypoint.to=webSecure
- --entrypoints.web.http.redirections.entrypoint.scheme=https
# Let's encrypt configuration
- --certificatesresolvers.le.acme.email=contact#hexa-ai.fr
- --certificatesresolvers.le.acme.storage=/letsencrypt/acme.json
- --certificatesresolvers.le.acme.tlschallenge=true
volumes:
- v_traefik:/letsencrypt
- /var/run/docker.sock:/var/run/docker.sock
emqx-service:
image: emqx/emqx:${EMQX_VERSION}
restart: unless-stopped
environment:
- EMQX_LOADED_PLUGINS="emqx_recon,emqx_retainer,emqx_management,emqx_dashboard,emqx_auth_http"
- EMQX_AUTH__HTTP__AUTH_REQ__URL=${EMQX_AUTH__HTTP__AUTH_REQ__URL}
- EMQX_AUTH__HTTP__AUTH_REQ__PARAMS=clientId=%c,username=%u,password=%P
- EMQX_AUTH__HTTP__ACL_REQ__URL=${EMQX_AUTH__HTTP__ACL_REQ__URL}
- EMQX_AUTH__HTTP__ACL_REQ__PARAMS=access=%A,username=%u,clientId=%c,ipaddr=%a,topic=%t,mountpoint=%m
- EMQX_ZONE__EXTERNAL__PUBLISH_LIMIT=${EMQX_ZONE__EXTERNAL__PUBLISH_LIMIT}
- EMQX_MQTT__MAX_PACKET_SIZE=10MB
volumes:
- v_emqx-data:/opt/emqx/data
- v_emqx-etc:/opt/emqx/etc
- v_emqx-log:/opt/emqx/log
#ports:
# - 8883:8883
# - 3000:18083
labels:
- "traefik.enable=true"
# - "traefik.http.routers.emqx-service.rule=Host(`mqtt.hexa-data.fr`)"
# - "traefik.http.routers.emqx-service.tls.certresolver=le"
# - "traefik.http.services.emqx-service.loadbalancer.server.port=18083"
- "traefik.tcp.routers.emqx-service.rule=HostSNI(`*`)"
- "traefik.tcp.routers.emqx-service.tls.certresolver=le"
- "traefik.tcp.services.emqx-service.loadbalancer.server.port=8883"
# Entrypoints
- "traefik.tcp.routers.emqx-service.entrypoints=mqtt"
#- "traefik.http.routers.emqx-service.entrypoints=webSecure"
extra_hosts:
- "host.docker.internal:host-gateway"
Server logs
time="2022-09-13T10:04:02Z" level=debug msg="Handling connection from 91.164.235.89:38383"
time="2022-09-13T10:04:06Z" level=debug msg="Handling connection from 91.164.235.89:33834"
time="2022-09-13T10:04:06Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:44948->172.19.0.15:8883: remote error: tls: expired certificate"
time="2022-09-13T10:04:06Z" level=debug msg="Error while terminating connection: tls: CloseWrite called before handshake complete"
time="2022-09-13T10:04:11Z" level=debug msg="Handling connection from 82.64.242.74:52648"
time="2022-09-13T10:04:11Z" level=debug msg="Handling connection from 91.164.235.89:45091"
time="2022-09-13T10:04:17Z" level=debug msg="Handling connection from 91.164.235.89:40677"
time="2022-09-13T10:04:21Z" level=debug msg="Handling connection from 91.164.235.89:33836"
time="2022-09-13T10:04:21Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:44972->172.19.0.15:8883: remote error: tls: expired certificate"
time="2022-09-13T10:04:21Z" level=debug msg="Error while terminating connection: tls: CloseWrite called before handshake complete"
time="2022-09-13T10:04:26Z" level=debug msg="Handling connection from 82.64.242.74:52710"
time="2022-09-13T10:04:26Z" level=debug msg="Error while setting deadline: set tcp 172.19.0.16:44976: use of closed network connection"
time="2022-09-13T10:04:26Z" level=debug msg="Handling connection from 91.164.235.89:37076"
time="2022-09-13T10:04:33Z" level=debug msg="Handling connection from 91.164.235.89:47424"
time="2022-09-13T10:04:36Z" level=debug msg="Handling connection from 91.164.235.89:33838"
time="2022-09-13T10:04:36Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:44990->172.19.0.15:8883: remote error: tls: expired certificate"
time="2022-09-13T10:04:36Z" level=debug msg="Error while terminating connection: tls: CloseWrite called before handshake complete"
time="2022-09-13T10:04:41Z" level=debug msg="Handling connection from 82.64.242.74:50496"
time="2022-09-13T10:04:42Z" level=debug msg="Handling connection from 91.164.235.89:45524"
time="2022-09-13T10:04:48Z" level=debug msg="Handling connection from 91.164.235.89:43400"
time="2022-09-13T10:04:51Z" level=debug msg="Handling connection from 91.164.235.89:33840"
time="2022-09-13T10:04:51Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:45010->172.19.0.15:8883: remote error: tls: expired certificate"
time="2022-09-13T10:04:51Z" level=debug msg="Error while terminating connection: tls: CloseWrite called before handshake complete"
time="2022-09-13T10:04:54Z" level=debug msg="Handling connection from 91.164.235.89:41826"
time="2022-09-13T10:04:55Z" level=debug msg="Handling connection from 91.164.235.89:38537"
time="2022-09-13T10:04:56Z" level=debug msg="Error while setting deadline: set tcp 172.19.0.16:45016: use of closed network connection"
time="2022-09-13T10:04:56Z" level=debug msg="Handling connection from 82.64.242.74:46692"
time="2022-09-13T10:04:56Z" level=debug msg="Error while setting deadline: set tcp 172.19.0.16:45018: use of closed network connection"
time="2022-09-13T10:04:57Z" level=debug msg="Handling connection from 91.164.235.89:37987"
Client conf
Client logs
[2022-09-13 12:04:56] [INFO] MQTTX client with ID 1fa93978-f254-438e-bc39-e2830403351f assigned
[2022-09-13 12:04:56] [INFO] Connect client PFC2004G, MQTT/SSL connection: mqtts://mqtt.hexa-data.fr:8883
[2022-09-13 12:04:56] [INFO] PFC2004G connect close, MQTT.js onClose trigger

Answer 1

The important part of the log would appear to be this part from server.log
time="2022-09-13T10:04:06Z" level=error msg="Error during connection: readfrom tcp 172.19.0.16:44948->172.19.0.15:8883: remote error: tls: expired certificate"
This implies the client has closed the connection to the broker because the certificate presented has expired.
I would first check the time/timezone for the server and the client.
Also I'm not clear how you've set the hostname (or domain, since it's set up as a wildcard hostname) the for LetsEncrypt certificate for the MQTT connection.
I would use something like openssl s_client to check what certificate is being presented
openssl s_client -connect mqtt.hexa-data.fr:8883