I have a MQTT broker started via docker-compose and managed by traefik:
mqtt:
container_name: mqtt
image: eclipse-mosquitto
restart: always
labels:
- 'traefik.port=1883'
- 'traefik.frontend.rule=Host:mqtt2.ex.com'
- 'traefik.frontend.entryPoints=mqtt'
ports:
- 3883:1883
The relevant part of traefik.toml where I am trying to set up the backend:
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.mqtt]
address = ":2884"
[tcp] # YAY!
[tcp.routers]
[tcp.routers.mqtt]
entrypoints = ["mqtt"]
rule = "HostSNI(`*`)" # Catches every request
service = "mqtt"
[tcp.services]
[tcp.services.mqtt.LoadBalancer]
I can access the broker via port `3883` but this is not what I intend to do (the exposed port above is just to make sure that the container is OK): I would like to proxy it though `traefik` like all my other **HTTP** services.
This however is not a HTTP service, it is a TCP one and my problem is that **I do not know how to configure such a TCP backend**.
The documentation is very light on that feature, it just states that
Currently, LoadBalancer is the only supported kind of TCP Service.
However, since Traefik is an ever evolving project, other kind of TCP
Services will be available in the future, reason why you have to
specify it.
What does this means in terms of configuring the backend? What should I add to either docker-compose.yaml or traefik.toml so that the backend is recognized as a TCP service? For the moment, it is seen as a HTTP one and the proxification does not work:
Related
I have been given a site to work with using https://github.com/dunglas/symfony-docker
All looks good, apart from I usually have 3 or 4 sites at one time in development, so I would usually in my docker-compose files set different ports for various services so they don't clash, for example:
nginx:
build: docker/nginx
ports:
- 97:80
However in this I don't seem to be able to change it, I am not sure if it is because of the caddy communication (first time using it, usually use nginx)
I tried setting different ports in the docker-compose.yml such as
caddy:
ports:
# HTTP
- target: 80
published: 97
protocol: tcp
# HTTPS
- target: 443
published: 8083
protocol: tcp
# HTTP/3
- target: 443
published: 8083
protocol: udp
But this just kept giving server 502 errors.
If I turn off all my other Docker containers I can get it working, so I know it is a port issue.
Does anyone familiar with this Docker setup know what I could be doing wrong? Maybe thinking it doesn't like non 443 port for it's self signed security cert? I did notice if I visit http://localhost:97 it redirected me to https://localhost (no port). However if I visit https://localhost:8083 I would get the 502 error
I also needed to use custom port in my project so i modified the caddy env var from:
caddy:
environment:
SERVER_NAME: ${SERVER_NAME:-localhost, caddy:80}
(comes with the repo) to:
caddy:
environment:
SERVER_NAME: ${SERVER_NAME:-http://localhost:80, https://localhost:443}
in the docker-compose file. Also as Blackban suggested you also need to check outer ports conflict and firewall.
I currently have a Traefik instance that's being run using the following. It works fine forwarding HTTP connections to the appropriate backends.
services_lb:
image: traefik:v2.2
cmd: |
--entrypoints.web.address=:80
--entrypoints.websecure.address=:443
--entrypoints.web.http.redirections.entryPoint.to=websecure
--entrypoints.web.http.redirections.entryPoint.scheme=https
--entrypoints.web.http.redirections.entrypoint.permanent=true
--entrypoints.matrixfederation.address=:8448
--entrypoints.prosodyc2s.address=:5222
--entrypoints.prosodys2s.address=:5269
--providers.docker
--providers.docker.constraints=Label(`lb.net`,`services`)
--providers.docker.network=am-services
--certificatesresolvers.lec.acme.email=notify#battlepenguin.com
--certificatesresolvers.lec.acme.storage=/letsencrypt/acme.json
--certificatesresolvers.lec.acme.tlschallenge=true
--entryPoints.web.forwardedHeaders.trustedIPs=172.50.0.1/24
ports:
- 80
- 443
# Matrix
- 8448
# XMPP
- 5222
- 5269
My web and Matrix federation connections work fine as they're all HTTP. But for Prosody (XMPP) I need to forward 5222 and 5269 directly without any HTTP routing. I configured the container like so:
xmpp:
image: prosody/prosody:0.11
network:
- services
- database
labels:
lb.net: services
traefik.tcp.services.prosodyc2s.loadbalancer.server.port: "5222"
traefik.tcp.services.prosodys2s.loadbalancer.server.port: "5269"
traefik.http.routers.am-app-xmpp.entrypoints: "websecure"
traefik.http.routers.am-app-xmpp.rule: "Host(`xmpp.example.com`)"
traefik.http.routers.am-app-xmpp.tls.certresolver: "lec"
traefik.http.services.am-app-xmpp.loadbalancer.server.port: "5280"
volumes:
- prosody-config:/etc/prosody:rw
- services_certs:/certs:ro
- prosody-logs:/var/log/prosody:rw
- prosody-modules:/usr/lib/prosody-modules:rw
With the tcp services, I still can't get Traefik to forward the raw TCP connections to this container. I've tried removing the --entrypoints from the Traefik instance and of course, Traefik stopped listening on those ports. I assumed the traefik.tcp.service definition would cause that entrypoint to switch to a TCP passthrough mode, but that isn't the case. I couldn't see anything in the Traefik documentation on putting the entrypoint itself into TCP mode instead of HTTP mode.
How do I pass the raw TCP connection from Traefik to this particular container using labels on the container and CLI options for Traefik?
I figured it out. You can't use any standard Traefik TLS offloading due to the differences in how Traefik and Prosidy handle TLS. I had to disable TLS entirely and use the special HostSNI(*) rule below to allow straight pass throughts. I was also missing the routers that connect the Traefik entrypoints to the TCP services.
labels:
lb.net: services
# client to server
traefik.tcp.routers.prosodyc2s.entrypoints: prosodyc2s
traefik.tcp.routers.prosodyc2s.rule: HostSNI(`*`)
traefik.tcp.routers.prosodyc2s.tls: "false"
traefik.tcp.services.prosodyc2s.loadbalancer.server.port: "5222"
traefik.tcp.routers.prosodyc2s.service: prosodyc2s
# server to server
traefik.tcp.routers.prosodys2s.entrypoints: prosodys2s
traefik.tcp.routers.prosodys2s.rule: HostSNI(`*`)
traefik.tcp.routers.prosodys2s.tls: "false"
traefik.tcp.services.prosodys2s.loadbalancer.server.port: "5269"
traefik.tcp.routers.prosodys2s.service: prosodys2s
# web
traefik.http.routers.am-app-xmpp.entrypoints: "websecure"
traefik.http.routers.am-app-xmpp.rule: "Host(`xmpp.example.com`)"
traefik.http.routers.am-app-xmpp.tls.certresolver: "lec"
traefik.http.services.am-app-xmpp.loadbalancer.server.port: "5280"
I have a simple server written in Python that listens on port 8000 inside a private network (HTTP communication). There is now a requirement to switch to HTTPS communications and every client that sends a request to the server should get authenticated with his own cert/key pair.
I have decided to use Traefik v2 for this job. Please see the block diagram.
Traefik runs as a Docker image on a host that has IP 192.168.56.101. First I wanted to simply forward a HTTP request from a client to Traefik and then to the Python server running outside Docker on port 8000. I would add the TLS functionality when the forwarding is running properly.
However, I can not figure out how to configure Traefik to reverse proxy from i.e. 192.168.56.101/notify?wrn=1 to the Python server 127.0.0.1:8000/notify?wrn=1.
When I try to send the above mentioned request to the server (curl "192.168.56.101/notify?wrn=1") I get "Bad Gateway" as an answer. What am I missing here? This is the first time that I am in contact with Docker and reverse proxy/Traefik. I believe it has something to do with ports but I can not figure it out.
Here is my Traefik configuration:
docker-compose.yml
version: "3.3"
services:
traefik:
image: "traefik:v2.1"
container_name: "traefik"
hostname: "traefik"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./traefik.yml:/traefik.yml:ro"
traefik.yml
## STATIC CONFIGURATION
log:
level: INFO
api:
insecure: true
dashboard: true
entryPoints:
web:
address: ":80"
providers:
docker:
watch: true
endpoint: "unix:///var/run/docker.sock"
file:
filename: "traefik.yml"
## DYNAMIC CONFIGURATION
http:
routers:
to-local-ip:
rule: "Host(`192.168.56.101`)"
service: to-local-ip
entryPoints:
- web
services:
to-local-ip:
loadBalancer:
servers:
- url: "http://127.0.0.1:8000"
First, 127.0.0.1 will resolve to the traefik container and not to the docker host. You need to provide a private IP of the node and it needs to be accessible form the traefik container.
There is some workaround to make proxy to localhost:
change 127.0.0.1 to IP of docker0 interface
It should be 172.17.0.1
and then try to listen your python server on all interfaces (0.0.0.0)
if you use simple python http server nothing change... on default it listen on all interfaces
I meet some problems using traefik with docker and I don't know why.
With some containers, it's works like a charm and for other ones, I have an error when I try to access to these ones : Bad gateway (error 502).
Here is my traefik.toml :
# Service logs (here debug mode)
debug = true
logLevel = "DEBUG"
defaultEntryPoints = ["http", "https"]
# Access log
filePath = "/var/log/traefik/access.log"
format = "common"
################################################################
# Web configuration backend
################################################################
[web]
address = ":8080"
################################################################
# Entry-points configuration
################################################################
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
################################################################
# Docker configuration backend
################################################################
[docker]
domain = "domain.tld"
watch = true
exposedbydefault = false
endpoint = "unix:///var/run/docker.sock"
################################################################
# Let's encrypt
################################################################
[acme]
email = "admin#domain.tld"
storageFile = "acme.json"
onDemand = false
onHostRule = true
entryPoint = "https"
[acme.httpChallenge]
entryPoint = "http"
[[acme.domains]]
main = "domain.tld"
sans = ["docker.domain.tld", "traefik.domain.tld", "phpmyadmin.domain.tld", "perso.domain.tld", "muximux.domain.tld", "wekan.domain.tld", "wiki.domain.tld", "cloud.domain.tld", "email.domain.tld"]
Here is my docker-compose.yml (for portainer, which is a container which works) :
version: '2'
services:
portainer:
restart: always
image: portainer/portainer:latest
container_name: "portainer"
#Automatically choose 'Manage the Docker instance where Portainer is running' by adding <--host=unix:///var/run/docker.sock> to the command
ports:
- "9000:9000"
networks:
- traefik-network
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ../portainer:/data
labels:
- traefik.enable=true
- traefik.backend=portainer
- traefik.frontend.rule=Host:docker.domain.tld
- traefik.docker.network=traefik-network
- traefik.port=9000
- traefik.default.protocol=http
networks:
traefik-network:
external : true
If I go to docker.domain.tld, it works ! and in https, with valide let's encrypt certificate :)
Here is my docker-compose.yml (for dokuwiki, which is a container which does not work) :
version: '2'
services:
dokuwiki:
container_name: "dokuwiki"
image: bitnami/dokuwiki:latest
restart: always
volumes:
- ../dokuwiki/data:/bitnami
ports:
- "8085:80"
- "7443:443"
networks:
- traefik-network
labels:
- traefik.backend=dokuwiki
- traefik.docker.network=traefik-network
- traefik.frontend.rule=Host:wiki.domain.tld
- traefik.enable=true
- traefik.port=8085
- traefik.default.protocol=http
networks:
traefik-network:
external: true
If I go to wiki.domain.tld, it does not work ! I have a bad gateway error on the browser. I have tried to change the traefik.port to 7443 and the traefik.default.protocol to https but I have the same error. Of course it works when I try to access the wiki with the IP and the port (in http / https). I have bad gateway only when I type wiki.domain.tld.
So, I don't understand why it works for some containers and not for other ones with the same declaration.
The traefik port should be the http port of the container, not the published port on the host. It communicates over the docker network, so publishing the port is unnecessary and against the goals of only having a single port published with a reverse proxy to access all the containers.
In short, you need:
traefik.port=80
Since this question has gotten lots of views, the other reason lots of people see a 502 from traefik is placing the containers on a different docker network from the traefik instance, or having a container on multiple networks and not telling traefik which network to use. This doesn't apply in your case since you have the following lines in your compose file that match up with the traefik service's network:
services:
dokuwiki:
networks:
- traefik-network
labels:
- traefik.docker.network=traefik-network
networks:
traefik-network:
external : true
Even if you only assign a service to a single network, some actions like publishing a port will result in your service being attached to two different networks (the ingress network being the second). The network name in the label needs to be the external name, which in your case is the same, but for others that do not specify their network as external, it may have a project or stack name prefixed which you can see in the docker network ls output.
traefik.docker.network must also be the fully qualified network name. Either externally defined, or prefixed with the stack name.
You can alternatively define a default network with providers.docker.network=traefik-network which means you don't have to add the label to every container.
Verify Apply:
firewall-cmd --add-masquerade --permanent
FROM: https://www.reddit.com/r/linuxadmin/comments/7iom6e/what_does_firewallcmd_addmasquerade_do/
Masquerading is a fancy term for Source NAT.
firewall-cmd in this instance will be adding an iptables rule,
specifically to the POSTROUTING chain in the nat table.
You can see what it has actually done by running iptables -t nat -nvL
POSTROUTING. A typical command to manually create a masquerading rule
would be iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE, which
translates to "For packets leaving interface eth0 after they have been
routed, change their source address to the interface address of eth0".
This automatically adds a connection tracking entry so that packets
for connections that are masqueraded in this way have their original
address and port information reinstated as they return back through
the system.
None of this makes your Linux system into a router; that is separate
behaviour which is enabled (for IPv4) either by doing sysctl -w
net.ipv4.ip_forward=1 or echo 1 > /proc/sys/net/ipv4/ip_forward.
Routing simply means that the system will dumbly traffic it receives
according to the destination of that traffic; the iptables NAT stuff
allows you to alter the packets which are emitted after that routing
takes place.
This is a really simple overview and there is a lot more complexity
and possibilities available by configuring it in different ways.
I've got some strange issue. I have following setup:
one docker-host running traefik as LB serving multiple sites. sites are most php/apache. HTTPS is managed by traefik.
Each site is started using a docker-compose YAML containing the following:
version: '2.3'
services:
redis:
image: redis:alpine
container_name: ${PROJECT}-redis
networks:
- internal
php:
image: registry.gitlab.com/OUR_NAMESPACE/docker/php:${PHP_IMAGE_TAG}
environment:
- APACHE_DOCUMENT_ROOT=${APACHE_DOCUMENT_ROOT}
container_name: ${PROJECT}-php-fpm
volumes:
- ${PROJECT_PATH}:/var/www/html:cached
- .docker/php/php-ini-overrides.ini:/usr/local/etc/php/conf.d/99-overrides.ini
ports:
- 80
networks:
- proxy
- internal
labels:
- traefik.enable=true
- traefik.port=80
- traefik.frontend.headers.SSLRedirect=false
- traefik.frontend.rule=Host:${PROJECT}
- "traefik.docker.network=proxy"
networks:
proxy:
external:
name: proxy
internal:
(as PHP we use 5.6.33-apache-jessie or 7.1.12-apache f.e.)
Additionally to above, some sites get following labels:
traefik.docker.network=proxy
traefik.enable=true
traefik.frontend.headers.SSLRedirect=true
traefik.frontend.rule=Host:example.com, www.example.com
traefik.port=80
traefik.protocol=http
what we get is that some requests end in 502 Bad Gateway
traefik debug output shows:
time="2018-03-21T12:20:21Z" level=debug msg="vulcand/oxy/forward/http: Round trip: http://172.18.0.8:80, code: 502, Length: 11, duration: 2.516057159s"
can someone help with that?
it's completely random when it happens
our traefik.toml:
debug = true
checkNewVersion = true
logLevel = "DEBUG"
defaultEntryPoints = ["https", "http"]
[accessLog]
[web]
address = ":8080"
[web.auth.digest]
users = ["admin:traefik:some-encoded-pass"]
[entryPoints]
[entryPoints.http]
address = ":80"
# [entryPoints.http.redirect] # had to disable this because HTTPS must be enable manually (not my decission)
# entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[retry]
[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "example.com"
watch = true
exposedbydefault = false
[acme]
email = "info#example.com"
storage = "acme.json"
entryPoint = "https"
onHostRule = true
[acme.httpChallenge]
entryPoint = "http"
Could the issue be related to using the same docker-compose.yml?
Another reason can be that you might be accidentally mapping to the vm's port instead of the container port.
I made a change to my port mapping on the docker-compose file and forgot to update the labeled port so it was trying to map to a port on the machine that was not having any process attached to it
Wrong way:
ports:
- "8080:8081"
labels:
- "traefik.http.services.front-web.loadbalancer.server.port=8080"
Right way
ports:
- "8080:8081"
labels:
- "traefik.http.services.front-web.loadbalancer.server.port=8081"
Also in general don't do this, instead of exposing ports try docker networks they are much better and cleaner. I made my configuration documentation like a year ago and this was more of a beginner mistake on my side but might help someone :)
For anyone getting the same issue:
After recreating the network (proxy) and restarting every site/container it seems to work now.
I still don't know where the issue was from.
If you see Bad Gateway with Traefik chances are you have a Docker networking issue. First have a look at this issue and consider this solution. Then take a look at providers.docker.network (Traefik 2.0) or, in your case, the docker.network setting (Traefik 1.7).
You could add a default network here:
[docker]
endpoint = "unix:///var/run/docker.sock"
domain = "example.com"
watch = true
exposedbydefault = false
network = "proxy"
Or define/override it for a given service using the traefik.docker.network label.
Got the same problem and none of the above mentioned answers solved it for me. In my case a wrong loadbalancer was added. Removing the label or changing it to the correct port made the trick.
- "traefik.http.services.XXX.loadbalancer.server.port=XXX"
In your example you don't have traefik enabled:
traefik.enable=false
Make sure to enable it first and then test your containers.
The error "bad gateway" is returned when the web server in the container doesn't allow traffic from traefik e.g. because of wrong interface binding like localhost instead of 0.0.0.0.
Take Ruby on Rails for example. Its web server puma is configured by default like this (see config/puma.rb):
port ENV.fetch("PORT") { 3000 }
But in order to allow access from traefik puma must bind to 0.0.0.0 like so:
bind "tcp://0.0.0.0:#{ ENV.fetch("PORT") { 3000 } }"
This solved the problem for me.
Another cause can be exposing a container at a port that Traefik already uses.
I forgot to expose the port in my Dockerfile thats why traefik did not find a port to route to. So expose the port BEFORE you start the application like node:
#other stuff before...
EXPOSE 3000
CMD ["node", "dist/main" ]
Or if you have multiple ports open you have to specify which port traefik should route the domain to with:
- "traefik.http.services.myservice.loadbalancer.server.port=3000"
Or see docs
I faced very close issue to this exception my problem was not related to network settings or config, after time we figured out that the exposed port from the backend container is not like the port we mapping to to access form outside the service port was 5000 and we mapped 9000:9000 the solution was to fix the port issue first 9000:5000.