I currently have a Traefik instance that's being run using the following. It works fine forwarding HTTP connections to the appropriate backends.
services_lb:
image: traefik:v2.2
cmd: |
--entrypoints.web.address=:80
--entrypoints.websecure.address=:443
--entrypoints.web.http.redirections.entryPoint.to=websecure
--entrypoints.web.http.redirections.entryPoint.scheme=https
--entrypoints.web.http.redirections.entrypoint.permanent=true
--entrypoints.matrixfederation.address=:8448
--entrypoints.prosodyc2s.address=:5222
--entrypoints.prosodys2s.address=:5269
--providers.docker
--providers.docker.constraints=Label(`lb.net`,`services`)
--providers.docker.network=am-services
--certificatesresolvers.lec.acme.email=notify#battlepenguin.com
--certificatesresolvers.lec.acme.storage=/letsencrypt/acme.json
--certificatesresolvers.lec.acme.tlschallenge=true
--entryPoints.web.forwardedHeaders.trustedIPs=172.50.0.1/24
ports:
- 80
- 443
# Matrix
- 8448
# XMPP
- 5222
- 5269
My web and Matrix federation connections work fine as they're all HTTP. But for Prosody (XMPP) I need to forward 5222 and 5269 directly without any HTTP routing. I configured the container like so:
xmpp:
image: prosody/prosody:0.11
network:
- services
- database
labels:
lb.net: services
traefik.tcp.services.prosodyc2s.loadbalancer.server.port: "5222"
traefik.tcp.services.prosodys2s.loadbalancer.server.port: "5269"
traefik.http.routers.am-app-xmpp.entrypoints: "websecure"
traefik.http.routers.am-app-xmpp.rule: "Host(`xmpp.example.com`)"
traefik.http.routers.am-app-xmpp.tls.certresolver: "lec"
traefik.http.services.am-app-xmpp.loadbalancer.server.port: "5280"
volumes:
- prosody-config:/etc/prosody:rw
- services_certs:/certs:ro
- prosody-logs:/var/log/prosody:rw
- prosody-modules:/usr/lib/prosody-modules:rw
With the tcp services, I still can't get Traefik to forward the raw TCP connections to this container. I've tried removing the --entrypoints from the Traefik instance and of course, Traefik stopped listening on those ports. I assumed the traefik.tcp.service definition would cause that entrypoint to switch to a TCP passthrough mode, but that isn't the case. I couldn't see anything in the Traefik documentation on putting the entrypoint itself into TCP mode instead of HTTP mode.
How do I pass the raw TCP connection from Traefik to this particular container using labels on the container and CLI options for Traefik?
I figured it out. You can't use any standard Traefik TLS offloading due to the differences in how Traefik and Prosidy handle TLS. I had to disable TLS entirely and use the special HostSNI(*) rule below to allow straight pass throughts. I was also missing the routers that connect the Traefik entrypoints to the TCP services.
labels:
lb.net: services
# client to server
traefik.tcp.routers.prosodyc2s.entrypoints: prosodyc2s
traefik.tcp.routers.prosodyc2s.rule: HostSNI(`*`)
traefik.tcp.routers.prosodyc2s.tls: "false"
traefik.tcp.services.prosodyc2s.loadbalancer.server.port: "5222"
traefik.tcp.routers.prosodyc2s.service: prosodyc2s
# server to server
traefik.tcp.routers.prosodys2s.entrypoints: prosodys2s
traefik.tcp.routers.prosodys2s.rule: HostSNI(`*`)
traefik.tcp.routers.prosodys2s.tls: "false"
traefik.tcp.services.prosodys2s.loadbalancer.server.port: "5269"
traefik.tcp.routers.prosodys2s.service: prosodys2s
# web
traefik.http.routers.am-app-xmpp.entrypoints: "websecure"
traefik.http.routers.am-app-xmpp.rule: "Host(`xmpp.example.com`)"
traefik.http.routers.am-app-xmpp.tls.certresolver: "lec"
traefik.http.services.am-app-xmpp.loadbalancer.server.port: "5280"
Related
we have an asp.net core 6 web api inside docker image, and in front of it is haproxy which directs traffic. The problem occurs when we want to change version of net core image we need to stop haproxy to docker-compose up new image. Something along the lines
docker-compose down
systemctl stop haproxy
docker-compose up -d
systemctl start haproxy
because without stopping haproxy we get a docker error Error starting userland proxy: listen tcp4 11.11.0.30:31079: bind: address already in use or everything seems fine but if you curl endpoint on net core api request will keep on running curl -v returns
* Trying 11.11.0.30:31079...
* TCP_NODELAY set
with logs inside docker we saw that some requests are getting inside from outside world, but like 0.1% of all load.
The weird thing is we have sidecar docker image of dotnet-monitor that doesn't have these issues.
-side note main net core image has prometheus .net library inside that uses $env:metrics_port to expose metrics data for internal usage that's why we us ASPNETCORE_URLS in docker-compose.
docker-compose.yml
version: '3.6'
services:
collector:
image: ${COLLECTOR_IMG}
restart: always
command: --urls "http://*:5003;http://*:5004"
container_name: collector
environment:
metrics_port: 5004
ports:
- "11.11.0.30:31079:5003"
- "11.11.0.30:52326:5004"
sysctls:
- "net.ipv6.conf.all.disable_ipv6=1"
networks:
collector-network:
ipv4_address: 162.30.337.10
volumes:
- dotnet-tmp:/tmp
dotnet-monitor:
image: ${MONITOR_IMG}
restart: always
command: --no-auth1 --urls http://*:52324
container_name: dotnet-monitor
ports:
- "11.11.0.30:52323:52324"
networks:
collector-network:
ipv4_address: 162.30.337.20
volumes:
- dotnet-tmp:/tmp
networks:
collector-network:
name: collector-network
driver: bridge
ipam:
driver: default
config:
- subnet: 162.30.337.0/24
volumes:
dotnet-tmp:
external: false
haproxy
global
log /dev/log local0 notice alert
# log /dev/log local1 notice alert
maxconn 400000
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
defaults
log global
option dontlognull
retries 3
timeout connect 10s
timeout client 25s
timeout server 25s
maxconn 400000
frontend collector
bind 145.81.37.211:80
mode tcp
option tcplog
use_backend collector
backend collector
mode tcp
balance roundrobin
server server1 11.11.0.30:31079 check
frontend monitor
bind 145.81.37.211:52323
mode tcp
option tcplog
use_backend monitor
backend monitor
mode tcp
balance roundrobin
server server2 11.11.0.30:52323 check
listen stats
bind 11.11.0.30:1936
option http-use-htx
mode http
option forwardfor
http-request use-service prometheus-exporter if { path /metrics }
stats enable
stats hide-version
stats refresh 30s
stats show-node
stats auth admin:bz74ZGws4eJcAmq
stats uri /stats
I'm trying to deploy a Docker Swarm of three host nodes with a single replicated service and put an HAProxy in front of it. I want the clients to be able to connect via SSL.
My docker-compose.yml:
version: '3.9'
services:
proxy:
image: haproxy
ports:
- 443:8080
volumes:
- haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
deploy:
placement:
constraints: [node.role == manager]
networks:
- servers-network
node-server:
image: glusk/hackathon-2021:latest
ports:
- 8080:8080
command: npm run server
deploy:
mode: replicated
replicas: 2
networks:
- servers-network
networks:
servers-network:
driver: overlay
My haproxy.cfg (based on the official example):
# Simple configuration for an HTTP proxy listening on port 80 on all
# interfaces and forwarding requests to a single backend "servers" with a
# single server "server1" listening on 127.0.0.1:8000
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
default_backend servers
backend servers
server server1 127.0.0.1:8000 maxconn 32
My hosts are Lightsail VPS Ubuntu instances and share the same private network.
node-service runs each https server task inside its own container on: 0.0.0.0:8080.
The way I'm trying to make this work at the moment is to ssh into the manager node (which also has a static and public IP), copy over my configuration files from above, and run:
docker stack deploy --compose-file=docker-compose.yml hackathon-2021
but it doesn't work.
Well, first of all and regarding SSL (since it's the first thing that you mention) you need to configure it using the certificate and listen on the port 443, not port 80.
With that modification, your Proxy configuration would already change to:
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
default_backend servers
frontend https-in
bind *:443 ssl crt /etc/ssl/certs/hackaton2021.pem
default_backend servers
That would be a really simplified configuration for allowing SSL connection.
Now, let's go for the access to the different services.
First of all, you cannot access to the service on localhost, actually you shouldn't even expose the ports of the services you have to the host. The reason? That you already have those applications in the same network than the haproxy, so the ideal would be to take advantage of the Docker DNS to access directly to them
In order to do this, first we need to be able to resolve the service names. For that you need to add the following section to your configuration:
resolvers docker
nameserver dns1 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 10s
hold refused 10s
hold nx 10s
hold timeout 10s
hold valid 10s
hold obsolete 10s
The Docker Swarm DNS service is always available at 127.0.0.11.
Now to your previous existent configuration, we would have to add the server but using the service-name discovery:
backend servers
balance roundrobin
server-template node- 2 node-server:8080 check resolvers docker init-addr libc,none
If you check what we are doing, we are creating a server for each one of the discovered containers in the Swarm within the node-server service (so the replicas) and we will create those adding the prefix node- to each one of them.
Basically, that would be the equivalent to get the actual IPs of each of the replicas and add them stacked as a basic server configuration.
For deployment, you also have some errors, since we aren't interested into actually expose the node-server ports to the host, but to create the two replicas and use HAProxy for the networking.
For that, we should use the following Docker Compose:
version: '3.9'
services:
proxy:
image: haproxy
ports:
- 80:80
- 443:443
volumes:
- hackaton2021.pem:/etc/ssl/certs/hackaton2021.pem
- haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
deploy:
placement:
constraints: [node.role == manager]
node-server:
image: glusk/hackathon-2021:latest
command: npm run server
deploy:
mode: replicated
replicas: 2
Remember to copy your haproxy.cfg and the self-signed (or real) certificate for your application to the instance before deploying the Stack.
Also, when you create that stack it will automatically create a network with the name <STACK_NAME>-default, so you don't need to define a network just for connecting both services.
I have been given a site to work with using https://github.com/dunglas/symfony-docker
All looks good, apart from I usually have 3 or 4 sites at one time in development, so I would usually in my docker-compose files set different ports for various services so they don't clash, for example:
nginx:
build: docker/nginx
ports:
- 97:80
However in this I don't seem to be able to change it, I am not sure if it is because of the caddy communication (first time using it, usually use nginx)
I tried setting different ports in the docker-compose.yml such as
caddy:
ports:
# HTTP
- target: 80
published: 97
protocol: tcp
# HTTPS
- target: 443
published: 8083
protocol: tcp
# HTTP/3
- target: 443
published: 8083
protocol: udp
But this just kept giving server 502 errors.
If I turn off all my other Docker containers I can get it working, so I know it is a port issue.
Does anyone familiar with this Docker setup know what I could be doing wrong? Maybe thinking it doesn't like non 443 port for it's self signed security cert? I did notice if I visit http://localhost:97 it redirected me to https://localhost (no port). However if I visit https://localhost:8083 I would get the 502 error
I also needed to use custom port in my project so i modified the caddy env var from:
caddy:
environment:
SERVER_NAME: ${SERVER_NAME:-localhost, caddy:80}
(comes with the repo) to:
caddy:
environment:
SERVER_NAME: ${SERVER_NAME:-http://localhost:80, https://localhost:443}
in the docker-compose file. Also as Blackban suggested you also need to check outer ports conflict and firewall.
I have a simple server written in Python that listens on port 8000 inside a private network (HTTP communication). There is now a requirement to switch to HTTPS communications and every client that sends a request to the server should get authenticated with his own cert/key pair.
I have decided to use Traefik v2 for this job. Please see the block diagram.
Traefik runs as a Docker image on a host that has IP 192.168.56.101. First I wanted to simply forward a HTTP request from a client to Traefik and then to the Python server running outside Docker on port 8000. I would add the TLS functionality when the forwarding is running properly.
However, I can not figure out how to configure Traefik to reverse proxy from i.e. 192.168.56.101/notify?wrn=1 to the Python server 127.0.0.1:8000/notify?wrn=1.
When I try to send the above mentioned request to the server (curl "192.168.56.101/notify?wrn=1") I get "Bad Gateway" as an answer. What am I missing here? This is the first time that I am in contact with Docker and reverse proxy/Traefik. I believe it has something to do with ports but I can not figure it out.
Here is my Traefik configuration:
docker-compose.yml
version: "3.3"
services:
traefik:
image: "traefik:v2.1"
container_name: "traefik"
hostname: "traefik"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./traefik.yml:/traefik.yml:ro"
traefik.yml
## STATIC CONFIGURATION
log:
level: INFO
api:
insecure: true
dashboard: true
entryPoints:
web:
address: ":80"
providers:
docker:
watch: true
endpoint: "unix:///var/run/docker.sock"
file:
filename: "traefik.yml"
## DYNAMIC CONFIGURATION
http:
routers:
to-local-ip:
rule: "Host(`192.168.56.101`)"
service: to-local-ip
entryPoints:
- web
services:
to-local-ip:
loadBalancer:
servers:
- url: "http://127.0.0.1:8000"
First, 127.0.0.1 will resolve to the traefik container and not to the docker host. You need to provide a private IP of the node and it needs to be accessible form the traefik container.
There is some workaround to make proxy to localhost:
change 127.0.0.1 to IP of docker0 interface
It should be 172.17.0.1
and then try to listen your python server on all interfaces (0.0.0.0)
if you use simple python http server nothing change... on default it listen on all interfaces
I have a MQTT broker started via docker-compose and managed by traefik:
mqtt:
container_name: mqtt
image: eclipse-mosquitto
restart: always
labels:
- 'traefik.port=1883'
- 'traefik.frontend.rule=Host:mqtt2.ex.com'
- 'traefik.frontend.entryPoints=mqtt'
ports:
- 3883:1883
The relevant part of traefik.toml where I am trying to set up the backend:
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.mqtt]
address = ":2884"
[tcp] # YAY!
[tcp.routers]
[tcp.routers.mqtt]
entrypoints = ["mqtt"]
rule = "HostSNI(`*`)" # Catches every request
service = "mqtt"
[tcp.services]
[tcp.services.mqtt.LoadBalancer]
I can access the broker via port `3883` but this is not what I intend to do (the exposed port above is just to make sure that the container is OK): I would like to proxy it though `traefik` like all my other **HTTP** services.
This however is not a HTTP service, it is a TCP one and my problem is that **I do not know how to configure such a TCP backend**.
The documentation is very light on that feature, it just states that
Currently, LoadBalancer is the only supported kind of TCP Service.
However, since Traefik is an ever evolving project, other kind of TCP
Services will be available in the future, reason why you have to
specify it.
What does this means in terms of configuring the backend? What should I add to either docker-compose.yaml or traefik.toml so that the backend is recognized as a TCP service? For the moment, it is seen as a HTTP one and the proxification does not work: