I had (as all of us who used Docker Cloud) to migrate my app to new environment, so i chose Docker Swarm CE. I am using Traefik as reverse proxy, and before migration it workked with segments just as per documentation, but for some reason it can not deal with those anymore in Swarm.
My service expose ports 3000 and 3001 for given path prefixes. Here is the part of docker-compose.yml for problematic service:
my-service:
image: my-service-image
deploy:
restart_policy:
condition: on-failure
labels:
traefik.port: 80
traefik.serviceapi.backend: api
traefik.serviceapi.frontend.entryPoints: "http,https"
traefik.serviceapi.frontend.rule: "PathPrefixStrip:/service/api"
traefik.serviceapi.port: 3000
traefik.servicesocket.backend: socket
traefik.servicesocket.frontend.entryPoints: "http,https,ws,wss"
traefik.servicesocket.frontend.rule: "PathPrefixStrip:/service/socket"
traefik.servicesocket.port: 3001
but for some reason swarm does not recognise these traefik segments, or I am missing something.
Did anyone have same issue?
Thanks!
UPDATE:
traefik:
image: traefik
command:
- "--web"
- "--docker"
- "--docker.swarmMode"
- "--docker.watch"
- "--docker.domain=my-domain"
- "--defaultentrypoints=http,https"
- "--entrypoints=Name:http Address::80 Redirect.EntryPoint:https"
- "--entrypoints=Name:https Address::443 TLS"
- "--acme"
- "--acme.storage=/etc/traefik/acme/acme.json"
- "--acme.entryPoint=https"
- "--acme.domains=my-domain"
- "--acme.httpChallenge.entryPoint=http"
- "--acme.email=email"
- "--logLevel=DEBUG"
deploy:
restart_policy:
condition: on-failure
ports:
- 80:80
- 443:443
- 8080:8080
volumes:
- ./var/run/docker.sock:/var/run/docker.sock
- ./traefik/acme/acme.json:/etc/traefik/acme/acme.json
I have more than 10 services working properly, only issue is that I cannot reach my-service endpoints, seems that swarm do not recognize traefik segments.
I found solution, so it might be helpful to someone:
for Docker Swarm you must set traefik.port, EXCEPT you are using segments - in this case you remove traefik.port and just set up ports of your service. In my case i just had to remove traefik.port: 80, and also both .backend's, traefik will create it's own.
Related
I have a VM which run multiple containers all linked to one docker network.
Traefik (as reverse proxy & load balancer)
cloudflared as tunnel
whoami (for testing purposes)
and some containers like photoprism, nextcloud, node-red,...
I generated an origin cert via Cloudflare which has been added to Traefik.
In Cloudflare, I have a subdomain which points via the tunnel to https://172.16.10.11 (ip from the VM). This causes an unsecure connection (IP SAN applied -> I don't think this is possible on a private ip?). When I disable TLS verification on Cloudflare, it works. However, I am trying to set this up properly. Next,I tried pointing my domain towards https://localhost. the cloudflared service running in a container cannot reach any other services as these are located other containers.
I was thinking, what if I run the cloudflared service within the Traefik container, I believe I can reach Traefik via localhost?
Do you have any advice on how to achieve a secure tunnel with cert verification? Or is this not realistic when self-hosting?
Current docker compose:
version: '3'
services:
traefik:
image: traefik:latest
command:
- --log.level=debug
- --api.insecure=true
- --providers.docker=true
- --providers.docker.exposedbydefault=false
- --entrypoints.web.address=:80
- --entrypoints.websecure.address=:443
- --serverstransport.insecureskipverify
- --providers.file.filename=/etc/traefik/dynamic_conf.yml
- --providers.file.watch=true
ports:
- "8080:8080"
- "443:443"
- "80:80"
networks:
- proxy_network
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- traefik-data:/etc/traefik
labels:
- traefik.enable=true
- traefik.docker.network=proxy_network
- traefik.http.routers.traefik.rule=Host(`${DOMAINNAME_TRAEFIK}`)
- traefik.http.routers.traefik.entrypoints=web
- traefik.http.routers.traefik.service=traefik
- traefik.http.services.traefik.loadbalancer.server.port=8080
tunnel:
container_name: cloudflared-tunnel
image: cloudflare/cloudflared
#restart: unless-stopped
networks:
- proxy_network
command: tunnel --no-autoupdate run --token ${CLOUDFLARED_TOKEN}
whoami:
image: traefik/whoami
container_name: whoami1
command:
# It tells whoami to start listening on 2001 instead of 80
- --port=2000
- --name=iamfoo
networks:
- proxy_network
labels:
- traefik.enable=true
- traefik.http.routers.whoami.rule=Host(`${DOMAINNAME}`)
- traefik.http.routers.whoami.entrypoints=websecure
- traefik.http.routers.whoami.tls=true
- traefik.http.routers.whoami.service=whoami
- traefik.http.services.whoami.loadbalancer.server.port=2000
volumes:
traefik-data:
driver: local
networks:
proxy_network:
name: proxy_network
external: true
I expect a secure tunnel solution and to make sure that this architecture is setup in a good way.
I want to switch from using the docker run-command to a docker-compose file with my nextcloud instance that runs behind a reverse proxy (jwilder/nginx-proxy).
This is the run command I used to use:
sudo docker run -d -p 8080:80 --expose 80 --expose 443 -e VIRTUAL_HOST=nextcloud.example.com -v nextcloud:/var/www/html --restart=always --name=nextcloud nextcloud:24.0.8
I installed the mariaDB later in the container so that I didn't have to struggle with networking. Also I use the Port 8080 only in my internal network for fast up- and downloading.
This worked quite well, but now I want to create a similar environment with docker-compose:
version: '3.8'
volumes:
nextcloud:
db:
services:
db:
image: mariadb:10.5
restart: always
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
volumes:
- db:/var/lib/mysql
environment:
- MYSQL_ROOT_PASSWORD=my-super-strong-password
- MYSQL_PASSWORD=my-other-super-strong-password
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
app:
image: nextcloud:24.0.8
restart: always
ports:
- 8080:80
expose:
- 80
- 443
links:
- db
volumes:
- nextcloud:/var/www/html
environment:
- MYSQL_PASSWORD=my-other-super-strong-password
- MYSQL_DATABASE=nextcloud
- MYSQL_USER=nextcloud
- MYSQL_HOST=db
- PHP_MEMORY_LIMIT=1G
- PHP_UPLOAD_LIMIT=128M
- VIRTUAL_HOST=nextcloud.example.com
The containers are starting successfully and I can use nextcloud in my internal network. But I cannot reach them from my domain. Instead I get a 502 Bad request. The VIRTUAL_HOST redirection seems to work since I'd get a 503 Service Temporarily Unavailable instead.
I think exposing the ports 80 and 443 doesn't work.
I've tried to add a proxy network:
networks:
proxy:
driver: bridge
external: true
and added
networks:
- default
to the db service and
networks:
- default
- proxy
to the app service.
That didn't fixed the problem. Does any of you have an idea what I can try next?
I've tried different ways to expose the ports and tried to create different networks
Nevermind found the problem.
Instead of simply creating an network named proxy, I had to create a new jwilder reverse-proxy service via docker compose with a name, as an example myreverseproxy. In each service I want to make public I needed to name this proxy as:
networks:
- default
- myreverseproxy
Also I had to use the name in the networks service area:
networks:
myreverseproxy:
external: true
I have a traefik environment running in docker. Originally I was running services in standard containers. I am not deploying containers to docker swarm and I have done this for traefik too, where the container is only deployed to my swarm manager.
For some reason, traefik successfully registers the host name I have given it, and can access that fine.
However, when I deploy any other service to the swarm, traefik doesn't pick it up.
There is one other service that has partially worked. I have deployed heimdall to docker swarm which can be access from gateway.docker.swarm:8091 but I don't want the port either.
My traefik compose file is as follows:
version: '3.3'
networks:
swarm-network:
driver: overlay
services:
traefik:
# The official v2 Traefik docker image
image: traefik
deploy:
placement:
constraints:
- node.role == manager
labels:
- "traefik.enable=true"
- "traefik.docker.network=pi_swarm-network"
- "traefik.http.routers.traefik.entrypoints=http"
- "traefik.http.services.traefik.loadbalancer.server.port=8080"
- "traefik.http.routers.traefik.rule=Host(`traefik.docker.swarm`)"
# Enables the web UI and tells Traefik to listen to docker
command:
- '--api.insecure=true'
- '--providers.docker=true'
- '--providers.docker.swarmmode=true'
- '--providers.docker.defaultRule=Host("docker.swarm")'
- '--providers.docker.watch=true'
- '--providers.docker.swarmModeRefreshSeconds=15s'
# Metrics configuration for influx db.
- '--metrics=true'
- '--metrics.influxdb=true'
- '--metrics.influxdb.address=192.168.8.122:8086'
- '--metrics.influxdb.protocol=http'
- '--metrics.influxdb.database=traefik'
- '--metrics.influxdb.addEntryPointsLabels=true'
- '--metrics.influxdb.addServicesLabels=true'
- '--metrics.influxdb.pushInterval=10s'
# Tracing
- '--tracing=true'
- '--tracing.zipkin=true'
- '--tracing.zipkin.httpEndpoint=http://192.168.8.117:9411/api/v2/spans'
- '--log'
- '--accesslog'
ports:
# The HTTP port
- "80:80"
# The Web UI (enabled by --api.insecure=true)
- "8080:8080"
networks:
- swarm-network
volumes:
# So that Traefik can listen to the Docker events
- /var/run/docker.sock:/var/run/docker.sock
An example of another service I am running is heimdall which has the compose file of the following:
version: "3"
networks:
swarm-network:
external:
name: pi_swarm-network
services:
heimdall:
image: ghcr.io/linuxserver/heimdall
environment:
- PUID=1000
- PGID=1000
- TZ=Europe/London
deploy:
placement:
constraints:
- node.labels.tier == web
labels:
- "traefik.enable=true"
- "traefik.docker.network=pi_swarm-network"
- "traefik.http.routers.heimdall.entrypoints=http"
- "traefik.http.services.heimdall.loadbalancer.server.port=8091"
- "traefik.http.routers.heimdall.rule=Host(`gateway.docker.swarm`)"
ports:
- 8091:80
restart: unless-stopped
networks:
- swarm-network
Can anyone see what I'm doing wrong?
I have figured out the problem.
In my compose file, I was using "traefik.http.services.heimdall.loadbalancer.entrypoints=http"
as well as
"traefik.http.routers.heimdall.entrypoints=http"
this was incorrect and needed to just be
"traefik.http.routers.heimdall.entrypoints=http"
For heimdall, I was also targetting the external port of 8091, whereas I actually needed to target the internal port of 80
I’m trying to use traefik with docker swarm but i’m having troubles during service updates. I run a stack deploy or service update the service goes down for some seconds
How to reproduce:
1 - Create a Dockerfile:
FROM jwilder/whoami
RUN echo $(date) > daniel.txt
2 - Build 2 demo images:
$ docker build -t whoami:01 .
$ docker build -t whoami:02 .
3 - Create a docker-compose.yml:
version: '3.5'
services:
app:
image: whoami:01
ports:
- 81:8000
deploy:
replicas: 2
restart_policy:
condition: on-failure
update_config:
parallelism: 1
failure_action: rollback
labels:
- traefik.enable=true
- traefik.backend=app
- traefik.frontend.rule=Host:localhost
- traefik.port=8000
- traefik.docker.network=web
networks:
- web
reverse-proxy:
image: traefik
command:
- "--api"
- "--docker"
- "--docker.swarmMode"
- "--docker.domain=localhost"
- "--docker.watch"
- "--docker.exposedbydefault=false"
- "--docker.network=web"
deploy:
replicas: 1
restart_policy:
condition: on-failure
update_config:
parallelism: 1
failure_action: rollback
placement:
constraints:
- node.role == manager
networks:
- web
ports:
- 80:80
- 8080:8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
web:
external: true
4 - Deploy the stack:
$ docker stack deploy -c docker-compose.yml stack_name
5 - Curl to get the service response:
$ while true ; do sleep .1; curl localhost; done
You should see something like this:
I'm adc1473258e9
I'm bc82ea92b560
I'm adc1473258e9
I'm bc82ea92b560
That means the load balance is working
6 - Update the service
$ docker service update --image whoami:02 got_app
The traefik respond with Bad Gateway when should be zero downtime.
How to fix it?
Bad gateway means traefik is configured to forward requests, but it's not able to reach the container on the ip and port that it's configured to use. Common issues causing this are:
traefik and the service on different docker networks
service exists in multiple networks and traefik picks the wrong one
wrong port being used to connect to the container (use the container port and make sure it's listening on all interfaces, aka 0.0.0.0)
From the comments, this is only happening during the deploy, which means traefik is hitting containers before they are ready to receive requests, or while they are being stopped.
You can configure containers with a healthcheck and send request through swarm mode's VIP using a Dockerfile that looks like:
FROM jwilder/whoami
RUN echo $(date) >/build-date.txt
HEALTHCHECK --start-period=30s --retries=1 CMD wget -O - -q http://localhost:8000
And then in the docker-compose.yml:
labels:
- traefik.enable=true
- traefik.backend=app
- traefik.backend.loadbalancer.swarm=true
...
And I would also configure the traefik service with the following options:
- "--retry.attempts=2"
- "--forwardingTimeouts.dialTimeout=1s"
However, traefik will keep a connection open and the VIP will continue to send all requests to the same backend container over that same connection. What you can do instead is have traefik itself perform the healthcheck:
labels:
- traefik.enable=true
- traefik.backend=app
- traefik.backend.healthcheck.path=/
...
I would still leave the healthcheck on the container itself so Docker gives the container time to start before stopping the other container. And leave the retry option on the traefik service so any request to a stopping container, or just one that hasn't been detected by the healthcheck, has a chance to try try again.
Here's the resulting compose file that I used in my environment:
version: '3.5'
services:
app:
image: test-whoami:1
ports:
- 6081:8000
deploy:
replicas: 2
restart_policy:
condition: on-failure
update_config:
parallelism: 1
failure_action: rollback
labels:
- traefik.enable=true
- traefik.backend=app
- traefik.backend.healthcheck.path=/
- traefik.frontend.rule=Path:/
- traefik.port=8000
- traefik.docker.network=test_web
networks:
- web
reverse-proxy:
image: traefik
command:
- "--api"
- "--retry.attempts=2"
- "--forwardingTimeouts.dialTimeout=1s"
- "--docker"
- "--docker.swarmMode"
- "--docker.domain=localhost"
- "--docker.watch"
- "--docker.exposedbydefault=false"
- "--docker.network=test_web"
deploy:
replicas: 1
restart_policy:
condition: on-failure
update_config:
parallelism: 1
failure_action: rollback
placement:
constraints:
- node.role == manager
networks:
- web
ports:
- 6080:80
- 6880:8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
web:
Dockerfile is as quoted above. Image names, ports, network names, etc were changed to avoid conflicting with other things in my environment.
As of today (jun/2021) Traefik can't drain the connections during update.
To achieve a zero-downtime rolling update you should delegate the load-balancing to docker swarm itself:
# trafik v2
# docker-compose.yml
services:
your_service:
deploy:
labels:
- traefik.docker.lbswarm=true
From the docs:
Enables Swarm's inbuilt load balancer (only relevant in Swarm Mode).
If you enable this option, Traefik will use the virtual IP provided by docker swarm instead of the containers IPs. Which means that Traefik will not perform any kind of load balancing and will delegate this task to swarm.
Further info:
https://github.com/traefik/traefik/issues/41
https://github.com/traefik/traefik/issues/1480
I am trying to use Traefik with Docker Swarm backend, and I am using the stack file below:
version: "3"
services:
traefik:
image: traefik:1.5
command: --web --docker --docker.swarmmode --docker.watch --docker.domain=sample.com --logLevel=DEBUG
deploy:
placement:
constraints: [node.role==manager]
restart_policy:
condition: on-failure
labels:
- "traefik.port=8080"
- "traefik.docker.network=sample-network"
- "traefik.frontend.rule=Host:traefik.sample.com"
ports:
- "80:80"
- "8080:8080"
- "443:443"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /dev/null:/traefik.toml
networks:
- sample-network
portainer:
image: portainer/portainer:latest
command: --no-auth -H unix:///var/run/docker.sock
deploy:
placement:
constraints: [node.role == manager]
labels:
- "traefik.portainer.port=7777"
- "traefik.docker.network=sample-network"
- "traefik.frontend.rule=Host:portainer.sample.com"
ports:
- "7777:9000"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
networks:
- sample-network
networks:
sample-network:
I have 2 questions that I am trying to wrap my head around:
1) (Exposing Traefik dashboard through frontend rule) I can access Traefik's web dashboard on sample.com:8080, yet I cannot access it through traefik.sample.com.
2) (Port Redirection on containers/services) I can access Portainer GUI through sample.com:7777, yet I cannot access it through portainer.sample.com. I am more curious of port redirection, because how will I setup 2 services in a single stack file if I encounter 2 images publishing to the same port? My service label declarations will clash at traefik.port=XXXX
You don´t need the traefik labels on the traefik service itself. It´s accessed from the outside over the specified ports:
ports:
- "80:80"
- "8080:8080"
- "443:443"
On the portainer service you don´t need the port mappings because you probably want to route the request with traefik.
Because traefik and portainer are in the same docker network traefik can access portainer on every port.
Therefore the port for traefik have to match the real portainer port:
labels:
- "traefik.port=9000"
- "traefik.docker.network=sample-network"
- "traefik.frontend.rule=Host:portainer.sample.com"
In the current setup you have to request traefik with Host:portainer.sample.com.
You can test it with
curl --verbose --header 'Host: portainer.sample.com' 'http://<DockerhostIp>:80'
Edit: Updated curl
Edit 2: Reaction to the edit of PO
The portainer.sample.com DNS entry will have to point to your docker host. Then traefik will route it to the correct container.
An alternative is to specifiy a traefik prefix:
"traefik.frontend.rule=Host:site1.org;PathPrefixStrip: /sub/"
With the rule all requests on site1.org/sub will routed to this specific service/container.
Have a look at
Traefik-Docker-Sample
Edit 3:
The self route for the dashboard/webui should work with:
labels:
- "traefik.port=8080"
- "traefik.docker.network=sample-network"
- "traefik.frontend.rule=Host:traefik.sample.com"
Just be sure that you have a DNS entry for traefik.sample.com.
To check if the traefik setup works you can also run
curl --verbose -H Host:traefik.sample.com <DockerHostIp>