ACME certificates timeout with traefik - docker

Question

Hello!
I am having issues finding out the reason why I am getting this error. Tried googling it.
It seems to be an issue with dns lookup from the container.
Error in traefik log:
time="2020-01-30T12:12:12+01:00" level=error msg="Unable to obtain ACME certificate for domains \"traefik.xyz.se\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:54773->127.0.0.11:53: i/o timeout" providerName=cloudflare.acme routerName=traefik-secure#docker rule="Host(`traefik.xyz.se`)"
time="2020-01-30T12:12:32+01:00" level=error msg="Unable to obtain ACME certificate for domains \"hivemq.xyz.se\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get https://acme-v02.api.letsencrypt.org/directory: dial tcp: lookup acme-v02.api.letsencrypt.org on 127.0.0.11:53: read udp 127.0.0.1:53671->127.0.0.11:53: i/o timeout" rule="Host(`hivemq.xyz.se`)" providerName=cloudflare.acme routerName=hivemq-secure#docker
Unable to lookup google from within traefik container. Don't know if this is working as intended?
/o/a/traefik> docker exec -it traefik /bin/sh
/ # nslookup google.se
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'google.se': Try again
/ #
Traefik docker-compose.yaml
version: '3'
services:
traefik:
image: traefik:v2.1
container_name: traefik
restart: unless-stopped
security_opt:
- no-new-privileges:true
networks:
- proxy
ports:
- 80:80
- 443:443
environment:
- CF_API_EMAIL=redacted
- CF_API_KEY=redacted
volumes:
- /etc/localtime:/etc/localtime:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- ./data/traefik.yml:/traefik.yml:ro
- ./data/acme.json:/acme.json
- ./data/config.yml:/config.yml:ro
labels:
- "traefik.enable=true"
- "traefik.http.routers.traefik.entrypoints=http"
- "traefik.http.routers.traefik.rule=Host(`traefik.xyz.se`)"
- "traefik.http.middlewares.traefik-auth.basicauth.users=redacted"
- "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
- "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
- "traefik.http.routers.traefik-secure.entrypoints=https"
- "traefik.http.routers.traefik-secure.rule=Host(`traefik.xyz.se`)"
- "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
- "traefik.http.routers.traefik-secure.tls=true"
- "traefik.http.routers.traefik-secure.tls.certresolver=cloudflare"
- "traefik.http.routers.traefik-secure.tls.domains[0].main=xyz.se"
- "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.xyz.se"
- "traefik.http.routers.traefik-secure.service=api#internal"
networks:
proxy:
external: true
data/traefik.yml:
api:
dashboard: true
debug: true
entryPoints:
http:
address: ":80"
https:
address: ":443"
providers:
docker:
endpoint: "unix:///var/run/docker.sock"
exposedByDefault: false
file:
filename: /config.yml
certificatesResolvers:
cloudflare:
acme:
email: redacted
storage: acme.json
dnsChallenge:
provider: cloudflare
delayBeforeCheck: 0
resolvers:
- "1.1.1.1:53"
- "8.8.8.8:53"
Service example (hivemq) docker-compose.yml:
version: "3"
services:
hivemq:
image: hivemq/hivemq4
container_name: hivemq
restart: unless-stopped
security_opt:
- no-new-privileges:true
ports:
- 1883:1883
volumes:
- /etc/localtime:/etc/localtime:ro
- /etc/timezone:/etc/timezone:ro
labels:
- "traefik.enable=true"
- "traefik.http.routers.hivemq.entrypoints=http"
- "traefik.http.routers.hivemq.rule=Host(`hivemq.xyz.se`)"
- "traefik.http.routers.hivemq.middlewares=https-redirect#file"
- "traefik.http.routers.hivemq-secure.middlewares=secured#file"
- "traefik.http.routers.hivemq-secure.entrypoints=https"
- "traefik.http.routers.hivemq-secure.rule=Host(`hivemq.xyz.se`)"
- "traefik.http.routers.hivemq-secure.tls=true"
- "traefik.http.routers.hivemq-secure.service=hivemq"
- "traefik.http.services.hivemq.loadbalancer.server.port=8080"
- "traefik.docker.network=proxy"
networks:
- internal
- proxy
networks:
proxy:
external: true
internal:
external: false
I have also tried reinstalling docker-ce, didn't help.

Answer 1

I had a similar issue and it was due to a bug of Docker: all my containers had lost their connection to the internet but they were all already removed for maintenance puprose so I couldn't see it.
In the logs, cannot get ACME client get directory means that Traefik cannot connect to Let's Encrypt url.
I fixed it by:
Removing Traefik stack
Pruning networks so traefik-public was removed
Restarting Docker service
If it's not enough, you can try these:
Try to restart the Docker Engine, which will reset any iptables rules (assuming you are using Docker on Linux)
Try to restart your whole machine
Try to disable (temporary) the firewall of your machine to verify that it fixes the issue
As mentioned here: https://community.containo.us/t/cannot-create-renew-acme-certificate-cannot-get-acme-client-get-directory/2469/2
I gave a rapid look around concerning Docker bugs about loosing connection and seems to be a mess, for years: https://github.com/moby/moby/issues/15172

Answer 2

Not a docker specialist but I had a similar issue and fixed it by activating ipv6 on docker daemon :
% grep ipv6 /etc/docker/daemon.json
"ipv6": true`
You need to reload docker daemon then
% sudo systemctl reload docker