Docker Swarm Nginx Service behind Traefik giving a 502 Error - docker

I have a docker swarm configured to use Traefik as reverse proxy. One of the containers in my swarm is running an Nginx server but I am getting a 502 Bad Gateway error when I navigate to that particular endpoint. Traefik is setup as follows:
version: '3.5'
services:
traefik:
image: traefik:alpine
command: |-
--entryPoints="Name:http Address::80 Redirect.EntryPoint:https"
--entryPoints="Name:https Address::443 TLS"
--defaultentrypoints="http,https"
--accesslogsfile="/var/log/access.log"
--acme
--acme.acmelogging="true"
--acme.domains="${SERVER},${SANS1}"
--acme.email="${ACME_EMAIL}"
--acme.entrypoint="https"
--acme.httpchallenge
--acme.httpchallenge.entrypoint="http"
--acme.storage="/opt/traefik/acme.json"
--acme.onhostrule="true"
--docker
--docker.swarmmode
--docker.domain="${SERVER}"
--docker.network="frontend"
--docker.watch
--api
networks:
- frontend
ports:
- target: 80
published: 80
mode: host
- target: 443
published: 443
mode: host
- target: 8080
published: 8080
mode: host
volumes:
- traefik_acme:/opt/traefik
- traefik_logs:/var/log/access.log
- /var/run/docker.sock:/var/run/docker.sock:ro
deploy:
replicas: 1
placement:
constraints: [node.role == manager]
networks:
frontend:
name: "frontend"
driver: overlay
volumes:
traefik_acme:
traefik_logs:
This compose file provides the overlay network and the Traefik service. The rest of my swarm is defined in the following compose file:
version: "3.5"
services:
test:
image: emilevauge/whoami
deploy:
labels:
traefik.enable: "true"
traefik.frontend.rule: "Host:${SERVER};PathPrefixStrip:/test"
traefik.port: 80
networks:
- frontend
octeditor:
image: ${DOCKER_OCTEDITOR_IMAGE_TAG}
deploy:
replicas: 1
labels:
traefik.enable: "true"
traefik.frontend.rule: "Host:${SERVER}"
traefik.port: 3000
networks:
- frontend
ports:
- "3000:80"
octserver:
image: ${DOCKER_OCTSERVER_IMAGE_TAG}
deploy:
replicas: 1
labels:
traefik.enable: "true"
traefik.frontend.rule: "Host:${SERVER};PathPrefixStrip:/api"
traefik.port: 4000
networks:
- frontend
ports:
- "4000:4000"
visualizer:
image: dockersamples/visualizer:stable
deploy:
placement:
constraints:
- 'node.role == manager'
labels:
traefik.enable: "true"
traefik.frontend.rule: "Host:${SERVER};PathPrefixStrip:/visualizer"
traefik.port: 8001
networks:
- frontend
ports:
- "8001:8080"
stop_grace_period: 1m30s
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
networks:
frontend:
external: true
The relevant configuration is for the octeditor service:
octeditor:
image: ${DOCKER_OCTEDITOR_IMAGE_TAG}
deploy:
replicas: 1
labels:
traefik.enable: "true"
traefik.frontend.rule: "Host:${SERVER}"
traefik.port: 3000
networks:
- frontend
ports:
- "3000:80"
I'm mapping port 80 (which Nginx listens to by default) to port 3000 where Traefik is configured to locate this service. This is the Dockerfile for the service running Nginx:
FROM node:latest as builder
WORKDIR /usr/src/app
COPY package.json .
RUN npm install
COPY . .
RUN npm run build
FROM nginx
COPY --from=builder /usr/src/app/build /usr/share/nginx/html
I simply build a react app and copy the build folder to the /usr/share/nginx/html folder. I've tried building and running this Dockerfile as a standalone container and it works, also I've checked the contents of the html folder and everything looks correct. The other services, apart from the visualizer service, are running correctly. Only this octedtior service and the visualizer service are giving me 502 errors. Can anyone suggest a solution or even how to check the traffic being sent to the nginx container? I've tried docker ps servicename but I can't see any errors coming from the service.
Edit:
If I change the configuration of the octeditor to this:
octeditor:
image: ${DOCKER_OCTEDITOR_IMAGE_TAG}
deploy:
replicas: 1
labels:
traefik.enable: "true"
traefik.frontend.rule: "Host:${SERVER}"
traefik.port: 80
networks:
- frontend
ports:
- "80:80"
And remove the test service that was previously listening on port 80 it seems to work. I don't understand what was wrong with the previous configuration however? I thought I was mapping traffic from port 3000 to port 80 of the container before, whereas now I'm mapping from port 80 to 80, but nothing should have changed from the perspective of the container, right?

The relevant configuration is for the octeditor service:
octeditor:
image: ${DOCKER_OCTEDITOR_IMAGE_TAG}
deploy:
replicas: 1
labels:
traefik.enable: "true"
traefik.frontend.rule: "Host:${SERVER}"
traefik.port: 3000
networks:
- frontend
ports:
- "3000:80"
I'm mapping port 80 (which Nginx listens to by default) to port 3000
where Traefik is configured to locate this service.
The traefik port needs to be 80, not 3000. The port mapping will create a forward from the host on 3000 to the container on 80. However traefik talks directly to the container over a shared network (frontend) and you need to provide it the container port.
There is no need to publish a host port for services accessed through traefik or any other reverse proxy unless you need to access them directly without the proxy (which brings into question whether you need a reverse proxy in those scenarios). In other words, this could be written without the ports:
octeditor:
image: ${DOCKER_OCTEDITOR_IMAGE_TAG}
deploy:
replicas: 1
labels:
traefik.enable: "true"
traefik.frontend.rule: "Host:${SERVER}"
traefik.port: 80
networks:
- frontend

Related

Swarm: Traefik returning 404 on compose services

I have two devices running Docker; an Intel NUC and a Raspberry Pi. My NUC is used as a mediaplayer/mediaserver. This also is the manager node. The Pi is being used as Home Assistant and MQTT machine and is set as worker node. I wanted to add them to a swarm so I could use Traefik for reverse proxy and HTTPS on both machines.
NUC:
1 docker-compose file for Traefik, Consul and Portainer.
1 docker-compose file for my media apps (Sabnzbd, Transmission-vpn, Sonarr, Radarr etc).
Pi:
1 docker-compose file for Home Assistant, MQTT etc.
Traefik and Portainer are up and running. I got them setup with `docker stack deploy`. Next I tried to setup my media apps, but they don't need to be connected with the Pi so I tried `docker compose`. Portainer shows the apps are running, but when I go to their subdomain Traefik returns 404 page not found. This makes me conclude that apps running outside the swarm, but connected to Traefik don't work. They also don't show up in the Traefik dashboard.
docker-compose.traefik.yml - 'docker stack deploy'
version: '3.7'
networks:
traefik_proxy:
external: true
agent-network:
attachable: true
volumes:
consul-data-leader:
consul-data-replica:
portainer-data:
services:
consul-leader:
image: consul
command: agent -server -client=0.0.0.0 -bootstrap -ui
volumes:
- consul-data-leader:/consul/data
environment:
- CONSUL_BIND_INTERFACE=eth0
- 'CONSUL_LOCAL_CONFIG={"leave_on_terminate": true}'
networks:
- traefik_proxy
deploy:
labels:
- traefik.frontend.rule=Host:consul.${DOMAINNAME?Variable DOMAINNAME not set}
- traefik.enable=true
- traefik.port=8500
- traefik.tags=${TRAEFIK_PUBLIC_TAG:-traefik-public}
- traefik.docker.network=traefik_proxy
- traefik.frontend.entryPoints=http,https
- traefik.frontend.redirect.entryPoint=https
- traefik.frontend.auth.forward.address=http://oauth:4181
- traefik.frontend.auth.forward.authResponseHeaders=X-Forwarded-User
- traefik.frontend.auth.forward.trustForwardHeader=true
consul-replica:
image: consul
command: agent -server -client=0.0.0.0 -retry-join="consul-leader"
volumes:
- consul-data-replica:/consul/data
environment:
- CONSUL_BIND_INTERFACE=eth0
- 'CONSUL_LOCAL_CONFIG={"leave_on_terminate": true}'
networks:
- traefik_proxy
deploy:
replicas: ${CONSUL_REPLICAS:-3}
placement:
preferences:
- spread: node.id
traefik:
image: traefik:v1.7
hostname: traefik
restart: always
networks:
- traefik_proxy
ports:
- target: 80
published: 80
- target: 443
published: 443
- target: 8080
published: 8145
deploy:
replicas: ${TRAEFIK_REPLICAS:-3}
placement:
constraints:
- node.role == manager
preferences:
- spread: node.id
labels:
traefik.enable: 'true'
traefik.backend: traefik
traefik.protocol: http
traefik.port: 8080
traefik.tags: traefik-public
traefik.frontend.rule: Host:traefik.${DOMAINNAME}
traefik.frontend.headers.SSLHost: traefik.${DOMAINNAME}
traefik.docker.network: traefik_proxy
traefik.frontend.passHostHeader: 'true'
traefik.frontend.headers.SSLForceHost: 'true'
traefik.frontend.headers.SSLRedirect: 'true'
traefik.frontend.headers.browserXSSFilter: 'true'
traefik.frontend.headers.contentTypeNosniff: 'true'
traefik.frontend.headers.forceSTSHeader: 'true'
traefik.frontend.headers.STSSeconds: 315360000
traefik.frontend.headers.STSIncludeSubdomains: 'true'
traefik.frontend.headers.STSPreload: 'true'
traefik.frontend.headers.customResponseHeaders: X-Robots-Tag:noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex
traefik.frontend.headers.customFrameOptionsValue: 'allow-from https:${DOMAINNAME}'
traefik.frontend.auth.forward.address: 'http://oauth:4181'
traefik.frontend.auth.forward.authResponseHeaders: X-Forwarded-User
traefik.frontend.auth.forward.trustForwardHeader: 'true'
domainname: ${DOMAINNAME}
dns:
- 1.1.1.1
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- ${USERDIR}/docker/traefik:/etc/traefik
- ${USERDIR}/docker/shared:/shared
environment:
CF_API_EMAIL: ${CLOUDFLARE_EMAIL}
CF_API_KEY: ${CLOUDFLARE_API_KEY}
command:
#- "storeconfig" #This is the push to consul, secondary traefik must be created and interfaced to this traefik. Remove this traefik's open ports, it shuts down once consul is messaged.
- '--logLevel=INFO'
- '--InsecureSkipVerify=true' #for unifi controller to not throw internal server error message
- '--api'
- '--api.entrypoint=apiport'
- '--defaultentrypoints=http,https'
- '--entrypoints=Name:http Address::80 Redirect.EntryPoint:https'
- '--entrypoints=Name:https Address::443 TLS TLS.SniStrict:true TLS.MinVersion:VersionTLS12 CipherSuites:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256'
- '--entrypoints=Name:apiport Address::8080'
- '--file'
- '--file.directory=/etc/traefik/rules/'
- '--file.watch=true'
- '--acme'
- '--acme.storage=/etc/traefik/acme/acme.json'
- '--acme.entryPoint=https'
# not yet ready?
# - "--acme.TLS-ALPN-01=true"
- '--acme.dnsChallenge=true'
- '--acme.dnsChallenge.provider=cloudflare'
- '--acme.dnsChallenge.delayBeforeCheck=60'
- '--acme.dnsChallenge.resolvers=1.1.1.1,1.0.0.1'
- '--acme.onHostRule=true'
- '--acme.email=admin#${DOMAINNAME}'
- '--acme.acmeLogging=true'
- '--acme.domains=${DOMAINNAME},*.${DOMAINNAME},'
- '--acme.KeyType=RSA4096'
#Let's Encrypt's staging server,
#caServer = "https://acme-staging-v02.api.letsencrypt.org/directory"
- '--docker'
- '--docker.swarmMode'
- '--docker.domain=${DOMAINNAME}'
- '--docker.watch'
- '--docker.exposedbydefault=false'
#- "--consul"
#- "--consul.endpoint=consul:8500"
#- "--consul.prefix=traefik"
- '--retry'
- 'resolvers=[192,168.1.1:53,1.1.1.1:53,]'
depends_on:
- consul-leader
docker-compose.media.yml - 'docker compose'
sabnzbd:
image: linuxserver/sabnzbd
container_name: sabnzbd
restart: always
network_mode: service:transmission-vpn
# depends_on:
# - transmission-vpn
# ports:
# - '${SABNZBD_PORT}:8080'
volumes:
- ${USERDIR}/docker/sabnzbd:/config
- /media/Data/Downloads:/Downloads
# - ${USERDIR}/Downloads/incomplete:/incomplete-downloads
environment:
PUID: ${PUID}
PGID: ${PGID}
TZ: ${TZ}
UMASK_SET: 002
deploy:
replicas: 1
labels:
traefik.enable: 'true'
traefik.backend: sabnzbd
traefik.protocol: http
traefik.port: 8080
traefik.tags: traefik_proxy
traefik.frontend.rule: Host:sabnzbd.${DOMAINNAME}
# traefik.frontend.rule: Host:${DOMAINNAME}; PathPrefix: /sabnzbd
traefik.frontend.headers.SSLHost: sabnzbd.${DOMAINNAME}
traefik.docker.network: traefik_proxy
traefik.frontend.passHostHeader: 'true'
traefik.frontend.headers.SSLForceHost: 'true'
traefik.frontend.headers.SSLRedirect: 'true'
traefik.frontend.headers.browserXSSFilter: 'true'
traefik.frontend.headers.contentTypeNosniff: 'true'
traefik.frontend.headers.forceSTSHeader: 'true'
traefik.frontend.headers.STSSeconds: 315360000
traefik.frontend.headers.STSIncludeSubdomains: 'true'
traefik.frontend.headers.STSPreload: 'true'
traefik.frontend.headers.customResponseHeaders: X-Robots-Tag:noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex
# traefik.frontend.headers.frameDeny: "true" #customFrameOptionsValue overrides this
traefik.frontend.headers.customFrameOptionsValue: 'allow-from https:${DOMAINNAME}'
traefik.frontend.auth.forward.address: 'http://oauth:4181'
traefik.frontend.auth.forward.authResponseHeaders: X-Forwarded-User
traefik.frontend.auth.forward.trustForwardHeader: 'true'
​
​
I already tried multiple things like removing the deploy command and just using labels etc but that didn't help at all. My Traefik logs also don't show anything that might be saying what's going wrong.
Are you running de .env file to set the environment variables? Because the feature .env is not currently supported by docker stack. You must source manually the .env running export $(cat .env) before running docker stack.

Traefik rule need solution for sub domain

I want to run an application behind traefik not as a sub-domain, but something like: xyz.abc.com/m1 or xyz.abc.com/m2 and so on.
Which label will work for it. I have tried with PathPrefix but it's not working. A sample application Joomla is deployed on docker swarm mode . Can I use Nginx or Haproxy for the same ?? If so, How?
When putting multiple applications behind the same name, there's a good chance the app needs to know it's URL prefix isn't at root (as most apps are expecting to be deployed at the root). So, might need to look at that too.
Traefik-wise, a stack file might look like this. This is running in Swarm mode (since you mentioned Swarm), which requires the Traefik labels to be on the service, not on the container. As such, they have to be within the deploy.labels definition, not just labels.
Example using Traefik 1
version: "3.7"
services:
proxy:
image: traefik:1.7
command: --docker --docker.swarmMode
ports:
- 80:80
volumes:
- /var/run/docker.sock:/var/run/docker.sock
app1:
image: your-image-for-app1
deploy:
labels:
traefik.backend: app1
traefik.frontend.rule: PathPrefix:/m1
traefik.port: 80
app2:
image: your-image-for-app2
deploy:
labels:
traefik.backend: app2
traefik.frontend.rule: PathPrefix:/m2
traefik.port: 80
Example using Traefik 2
version: "3.7"
services:
proxy:
image: traefik:2.0
command: --providers.docker --providers.docker.swarmMode
ports:
- 80:80
volumes:
- /var/run/docker.sock:/var/run/docker.sock
app1:
image: your-image-for-app1
deploy:
labels:
traefik.http.routers.app1.rule: PathPrefix(`/m1`)
traefik.http.services.app1.loadbalancer.server.port: 80
app2:
image: your-image-for-app2
deploy:
labels:
traefik.http.routers.app2.rule: PathPrefix(`/m2`)
traefik.http.services.app2.loadbalancer.server.port: 80

Traefik v2 listen on port

I am using Traefik v2 with Docker Swarm. I want to achieve the following routing:
mydomain.com:9000 -> Traefik dashboard
mydomain.com:5000 -> my application
docker-compose-traefik.yml
version: "3.7"
services:
traefik:
image: "traefik:v2.0"
networks:
- traefik-net
command:
- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker.swarmMode=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:5000"
ports:
- "80:80"
- "9000:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
networks:
traefik-net:
external:
name: traefik-net
docker-compose-whoami.yml
version: "3.7"
services:
whoami:
image: "jwilder/whoami"
networks:
- traefik-net
deploy:
replicas: 3
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`mydomain.com`)"
- "traefik.http.routers.whoami.entrypoints=web"
- "traefik.http.services.whoami.loadbalancer.server.port=8000"
networks:
traefik-net:
external:
name: traefik-net
jwilder/whoami exposes port 8000 in its Dockerfile. I want to redirect port 5000 (my entrypoint defined in docker-compose-traefik.yml) to port 8000 in container.
I created network traefik-net with: docker network create -d bridge traefik-net.
I deployed both stacks with:
docker-stack deploy -c docker-compose-traefik.yml Traefik
docker-stack deploy -c docker-compose-whoami.yml Whoami
When I visit mydomain.com:9000 it opens Traefik dashboard as it should. When I visit mydomain.com:5000 it says that "This site can’t be reached".
My question is: How to redirect request to port 5000 (mydomain.com:5000) to port 8000 inside whoami container?
For anyone else having similar problems, I found a solution. I needed to change ports section in docker-compose-traefik.yml from
ports:
- "80:80"
- "9000:8080"
to
ports:
- "80:80"
- "9000:8080"
- "5000:5000" <-- add this
Hope this helps someone. :)

Container on worker node is not accessible using Swarm Mode

I have a Swarm cluster with a Manager and a Worker node.
All the containers running on the manager are accessible through Traefik and working fine.
I just deployed a new Worker node and joined my swarm on the node.
Now I start scaling some services and realized they were timing out on the worker node.
So I setup a simple example using the whoami container, and cannot figure out why I cannot access it. Here are my configs (all deployed on the MANAGER node):
version: '3.6'
networks:
traefik-net:
driver: overlay
attachable: true
external: true
services:
whoami:
image: jwilder/whoami
networks:
- traefik-net
deploy:
labels:
- "traefik.port=8000"
- "traefik.frontend.rule=Host:whoami.myhost.com"
- "traefik.docker.network=traefik-net"
replicas: 2
placement:
constraints: [node.role != manager]
My traefik:
version: '3.6'
networks:
traefik-net:
driver: overlay
attachable: true
external: true
services:
reverse-proxy:
image: traefik # The official Traefik docker image
command: --docker --docker.swarmmode --docker.domain=myhost.com --docker.watch --api
ports:
- "80:80" # The HTTP port
# - "8080:8080" # The Web UI (enabled by --api)
- "443:443"
networks:
- traefik-net
volumes:
- /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen
- /home/ubuntu/docker-configs/traefik/traefik.toml:/traefik.toml
- /home/ubuntu/docker-configs/traefik/acme.json:/acme.json
deploy:
labels:
traefik.port: 8080
traefik.frontend.rule: "Host:traefik.myhost.com"
traefik.docker.network: traefik-net
replicas: 1
placement:
constraints: [node.role == manager]
My worker docker ps output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b825f95b0366 jwilder/whoami:latest "/app/http" 4 hours ago Up 4 hours 8000/tcp whoami_whoami.2.tqbh4csbqxvsu6z5i7vizc312
50cc04b7f0f4 jwilder/whoami:latest "/app/http" 4 hours ago Up 4 hours 8000/tcp whoami_whoami.1.rapnozs650mxtyu970isda3y4
I tried opening firewall ports, disabling it completely, nothing seems to work. Any help is appreciated
I had to use --advertise-addr y.y.y.y to make it work

Exposing a Docker database service only on the internal network with Traefik

Let's say I defined two services "frontend" and "db" in my docker-compose.yml which are deployed to a Docker swarm, i.e. they may also run in different stacks. With this setup Traefik automatically generates the frontend and backend for each stack which is fine.
Now I have another Docker container running temporarily in a Jenkins pipeline which shall be able to access the db service in a specific stack. My first idea was to expose the db service by adding it to the cluster-global-net network so that Traefik can generate a frontend route to the bakend. This basically works.
But I'd like to hide the database service from "the public" while still being able to connect another Docker container to it via its stack or service name using the internal "default" network.
Can this be done somehow?
version: '3.6'
networks:
default: {}
cluster-global-net:
external: true
services:
frontend:
image: frontend_image
ports:
- 8080
networks:
- cluster-global-net
- default
deploy:
labels:
traefik.port: 8080
traefik.docker.network: cluster-global-net
traefik.backend.loadbalancer.swarm: 'true'
traefik.backend.loadbalancer.stickiness: 'true'
replicas: 1
restart_policy:
condition: any
db:
image: db_image
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=false
- MYSQL_DATABASE=db_schema
- MYSQL_USER=db_user
- MYSQL_PASSWORD=db_pass
ports:
- 3306
volumes:
- db_volume:/var/lib/mysql
networks:
- default
restart: on-failure
deploy:
labels:
traefik.port: 3306
traefik.docker.network: default
What you need is a network on which both of them are deployed, but that it's not visible from anyone else.
To do such, create a network , add it to your db service and frontend, and also to your temporary service. And indeed, remove traefik label on db because they are not needed anymore here.
EG :
...
networks:
default: {}
cluster-global-net:
external: true
db-net:
external: true
services:
frontend:
image: frontend_image
networks:
- cluster-global-net
- default
- db-net
deploy:
...
db:
image: db_image
...
networks:
- default
- db-net
restart: on-failure
#no labels
docker network create db-net
docker stack deploy -c <mycompose.yml> <myfront>
docker service create --network db-net <myTemporaryImage> <temporaryService>
Then, the temporaryService as well as the frontend can reach the db through db:3306
BTW : you don't need to open the port for the frontend, since traefik will access it internally (trafik.port).
EDIT : new exemple with network created from compose file.
...
networks:
default: {}
cluster-global-net:
external: true
db-net: {}
services:
frontend:
image: frontend_image
networks:
- cluster-global-net
- default
- db-net
deploy:
...
db:
image: db_image
...
networks:
- default
- db-net
restart: on-failure
#no labels
docker stack deploy -c <mycompose.yml> someStackName
docker service create --network someStackName_db-net <myTemporaryImage> <temporaryService>

Resources