Traefik routes to external proxy instead of local docker services - docker
I am trying the basic example on traefik website on Mac with Docker Desktop. In my docker configuration, I have also mentioned to use my company http and https proxies. I can access the traefik dashboard at localhost:8080 and when I tried accessing http://whoami.localhost locally, traefik routes the request to my company proxy instead of routing it to the whoami service and I receive DNS resolution failure from my companies website.
Below is the example docker-compose.yml
version: "3.3"
services:
traefik:
image: "traefik:v2.9"
container_name: "traefik"
command:
#- "--log.level=DEBUG"
- "--api.insecure=true"
- "--providers.docker=true"
- "--providers.docker.exposedbydefault=false"
- "--entrypoints.web.address=:80"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
whoami:
image: "traefik/whoami"
container_name: "simple-service"
labels:
- "traefik.enable=true"
- "traefik.http.routers.whoami.rule=Host(`whoami.localhost`)"
- "traefik.http.routers.whoami.entrypoints=web"
Traefik Logs when I access the whoami.localhost(73eae85da227 is my traefik container)
traefik | time="2022-11-04T09:50:40Z" level=debug msg="vulcand/oxy/roundrobin/rr: begin ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0\"],\"X-Forwarded-Host\":[\"whoami.localhost\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"73eae85da227\"],\"X-Real-Ip\":[\"172.21.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami.localhost\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.21.0.1:62128\",\"RequestURI\":\"/\",\"TLS\":null}"
traefik | time="2022-11-04T09:50:40Z" level=debug msg="vulcand/oxy/roundrobin/rr: Forwarding this request to URL" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0\"],\"X-Forwarded-Host\":[\"whoami.localhost\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"73eae85da227\"],\"X-Real-Ip\":[\"172.21.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami.localhost\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.21.0.1:62128\",\"RequestURI\":\"/\",\"TLS\":null}" ForwardURL="http://172.21.0.2:80"
traefik | time="2022-11-04T09:50:40Z" level=debug msg="vulcand/oxy/roundrobin/rr: completed ServeHttp on request" Request="{\"Method\":\"GET\",\"URL\":{\"Scheme\":\"\",\"Opaque\":\"\",\"User\":null,\"Host\":\"\",\"Path\":\"/\",\"RawPath\":\"\",\"OmitHost\":false,\"ForceQuery\":false,\"RawQuery\":\"\",\"Fragment\":\"\",\"RawFragment\":\"\"},\"Proto\":\"HTTP/1.1\",\"ProtoMajor\":1,\"ProtoMinor\":1,\"Header\":{\"Accept\":[\"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8\"],\"Accept-Encoding\":[\"gzip, deflate, br\"],\"Accept-Language\":[\"en-US,en;q=0.5\"],\"Connection\":[\"keep-alive\"],\"Dnt\":[\"1\"],\"Sec-Fetch-Dest\":[\"document\"],\"Sec-Fetch-Mode\":[\"navigate\"],\"Sec-Fetch-Site\":[\"none\"],\"Sec-Fetch-User\":[\"?1\"],\"Upgrade-Insecure-Requests\":[\"1\"],\"User-Agent\":[\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0\"],\"X-Forwarded-Host\":[\"whoami.localhost\"],\"X-Forwarded-Port\":[\"80\"],\"X-Forwarded-Proto\":[\"http\"],\"X-Forwarded-Server\":[\"73eae85da227\"],\"X-Real-Ip\":[\"172.21.0.1\"]},\"ContentLength\":0,\"TransferEncoding\":null,\"Host\":\"whoami.localhost\",\"Form\":null,\"PostForm\":null,\"MultipartForm\":null,\"Trailer\":null,\"RemoteAddr\":\"172.21.0.1:62128\",\"RequestURI\":\"/\",\"TLS\":null}"
I checked the traefik container environment, and I can see that http_proxy and https_proxy are set to my company's proxies.
Image of Traefik HTTP Routers
Image of service details
Related
Issue with routing http flow with Traefik
I have a small lab setup on docker lab server where I have several containers. I wanted to set up a proxy with Traefik, but I stuck on one thing. Looks like routing works fine but I am having err_connection_refused when I try to access routed services by the browser. It works fine with the curl command. Using DNSMasq to manage records - traefik for proxy. Below docker-compose files for traefik and test container which I am using for testing the proxy. Heimdall Dashboard: --- version: "2.1" services: heimdall: image: lscr.io/linuxserver/heimdall:latest container_name: heimdall environment: - PUID=1000 - PGID=1000 - TZ=Europe/London volumes: - /path/to/appdata/config:/config restart: unless-stopped labels: - "traefik.http.routers.heimdall-www.rule=Host(`heimdall.lab`)" - "traefik.http.services.heimdall-www.loadbalancer.server.port=80" networks: default: name: traefik_default Traefik container: version: '3' services: reverse-proxy: # The official v2 Traefik docker image image: traefik:v2.9 # Enables the web UI and tells Traefik to listen to Docker command: --api.insecure=true --providers.docker ports: # The HTTP port - "80:80" # The Web UI (enabled by --api.insecure=true) - "8080:8080" volumes: # So that Traefik can listen to the Docker events - /var/run/docker.sock:/var/run/docker.sock whoami: # A container that exposes an API to show its IP address image: traefik/whoami labels: - "traefik.http.routers.whoami.rule=Host(`whoami.lab`)" - "traefik.http.services.whoami.loadbalancer.server.port=80" DNSmasq entry address=/.lab/127.0.0.1 I dont understand why curling works fine e.g.: curl -H Host:whoami.lab http://192.168.0.150 Hostname: 1ad1e42dd818 IP: 127.0.0.1 IP: 172.21.0.4 RemoteAddr: 172.21.0.2:54128 GET / HTTP/1.1 Host: whoami.lab User-Agent: curl/7.84.0 Accept: */* Accept-Encoding: gzip X-Forwarded-For: 192.168.0.100 X-Forwarded-Host: whoami.lab X-Forwarded-Port: 80 X-Forwarded-Proto: http X-Forwarded-Server: ca631c80565a X-Real-Ip: 192.168.0.100
Problem using jwilder/nginx-proxy for multiple domains
I have two apps (app1 and app2) and I would like to run them on my domains (domain1 and domain2). I use docker containers to do this and a docker reverse proxy. If I run each docker app independently, they work well. When I execute docker-compose up -d and I visit my domains, I can see the following message: Welcome to nginx!. I can't see my apps running. My docker-compose.yml is the following: version: "3.7" services: proxy: image: jwilder/nginx-proxy container_name: proxy labels: com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true" ports: - "80:80" - "443:443" volumes: - certs:/etc/nginx/certs - vhostd:/etc/nginx/vhost.d - html:/usr/share/nginx/html - /var/run/docker.sock:/tmp/docker.sock letsencrypt: image: jrcs/letsencrypt-nginx-proxy-companion container_name: letsencrypt environment: - NGINX_PROXY_CONTAINER=proxy volumes: - certs:/etc/nginx/certs - vhostd:/etc/nginx/vhost.d - html:/usr/share/nginx/html - /var/run/docker.sock:/var/run/docker.sock app1: image: app1_image container_name: myapp1 expose: - "9090" environment: - VIRTUAL_PORT=9090 - VIRTUAL_HOST=domain1.com www.domain1.com - LETSENCRYPT_HOST=domain1.com www.domain1.com - LETSENCRYPT_EMAIL=info#domain1.es volumes: - /srv/shiny-server/app1/app/data:/srv/shiny-server/app1/app/data app2: image: app2_image container_name: myapp2 expose: - "8080" environment: - VIRTUAL_PORT=8080 - VIRTUAL_HOST=domain2.com www.domain2.com - LETSENCRYPT_HOST=domain2.com www.domain2.com - LETSENCRYPT_EMAIL=info#domain2.es volumes: - /srv/shiny-server/app2/app/data:/srv/shiny-server/app2/app/data volumes: certs: html: vhostd: If I don't use the detach mode and I execute the docker containers, I get the following error: $ docker-compose up myapp1 | Listening on http://0.0.0.0:9090 myapp2 | myapp2 | Listening on http://0.0.0.0:8080 ... proxy | nginx.1 | myserverIP - - [16/Aug/2022:17:33:01 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://www.domain1.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0" "-" proxy | nginx.1 | 2022/08/16 17:33:01 [error] 25#25: *1 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: myserverIP, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "www.domain1.com", referrer: "http://www.domain1.com/" ... proxy | nginx.1 | myserverIP - - [16/Aug/2022:17:33:46 +0000] "GET /favicon.ico HTTP/1.1" 404 153 "http://www.domain2.com/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0" "-" proxy | nginx.1 | 172.25.0.1 - - [16/Aug/2022:17:33:52 +0000] "GET / HTTP/1.1" 200 615 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:103.0) Gecko/20100101 Firefox/103.0" "-" proxy | nginx.1 | 2022/08/16 17:33:52 [error] 25#25: *6 open() "/usr/share/nginx/html/favicon.ico" failed (2: No such file or directory), client: 172.25.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", host: "domain2.com", referrer: "http://domain2.com/" I don't know if this error is causing the problem, buy I just see on my domains: Welcome to nginx! I can't see my apps running.
I can't see your problem, but I can give some advices. Mount of vhostd You have a mount vhostd:/etc/nginx/vhost.d. If you have settings in there, then you should share them with us. If not, then just remove the mount. Multidomain pattern You specified multiple domains in a different pattern as jwilder/nginx-proxy. Instead of VIRTUAL_HOST=domain1.com www.domain1.com you should write VIRTUAL_HOST=domain1.com,www.domain1.com. See jwilder/nginx-proxy, chapter "Multiple Hosts"
Access client ip inside nextcloud docker with haproxy docker
I have a some docker container running and i want to see the real ip address from the client on the nextcloud docker logs. But currently i only can see the ip address from the haproxy container, i already added option forwardfor but still it does not work. My docker-compose: version: '3' services: db: image: linuxserver/mariadb restart: always environment: - MYSQL_ROOT_PASSWORD= - PUID=100 - PGID=100 - MYSQL_DATABASE=nextcloud - MYSQL_USER=nextcloud - MYSQL_PASSWORD= volumes: - /home/raspi/nextcloud-docker/volumen/mariadb/:/config app: image: nextcloud:apache restart: always volumes: - /home/raspi/nextcloud-docker/volumen/nextcloud/:/var/www/html environment: - VIRTUAL_HOST= - LETSENCRYPT_HOST= - LETSENCRYPT_EMAIL= - MYSQL_HOST=db - OVERWRITEPROTOCOL=https env_file: - db.env depends_on: - db networks: - proxy-tier - default haproxy: restart: always image: haproxy:2.1.7 volumes: - /home/raspi/nextcloud-docker/haproxy/config:/usr/local/etc/haproxy/ - /home/raspi/nextcloud-docker/haproxy/certs/haproxy/:/usr/local/etc/ssl/ ports: - 8080:8080 networks: - proxy-tier depends_on: - app networks: proxy-tier: My haproxy.cfg: global maxconn 50 tune.ssl.default-dh-param 2048 log stdout format raw local0 defaults log global mode http timeout tunnel 1h timeout http-request 1h timeout connect 20s option forwardfor option http-server-close frontend https bind *:8080 ssl crt /usr/local/etc/ssl/website.org http-request redirect scheme https code 301 if !{ ssl_fc } default_backend nextcloud timeout client 1h backend nextcloud server app1 app:80 timeout server 1h The nextcloud logs: {"reqId":"GoyJSJ8Jl1xAUf9xARf6","level":2,"time":"2020-09-11T23:41:54+00:00","remoteAddr":"172.29.0.3","user":"--","app":"no app in context","method":"POST","url":"/login","message":"Login failed: malo (Remote IP: 172.29.0.3)","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36","version":"19.0.0.12"} {"reqId":"A4n1pk3sU8Bsh0pkMjfB","level":2,"time":"2020-09-11T23:43:27+00:00","remoteAddr":"172.29.0.3","user":"--","app":"no app in context","method":"POST","url":"/login","message":"Login failed: malo3 (Remote IP: 172.29.0.3)","userAgent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36","version":"19.0.0.12"} As you can see, in only getting the ip address from the haproxy container. I already did add option forwardfor to haproxy.cfg and did add 'trusted_proxies' => array('172.29.0.3'), 'forwarded_for_headers' => array('HTTP_X_FORWARDED_FOR'), and restarted but nothing worked. I need this to make fail2ban work. What am i missing?
I had to make the proxy-tier network static so my networks look like this: networks: proxy-tier: ipam: config: - subnet: 174.20.0.0/24 and the app and haproxy container i had to put this: networks: proxy-tier: ipv4_address: 174.20.0.x and now add the haproxy container ip address: environment: - TRUSTED_PROXIES=174.20.0.x now it works!
Traefik 2 docker not showing client real ip
I have traefik running in docker (on a windows host). The problem I have is that the X-Real-IP header alawys shows the docker network gateway ip instead of the real client ip. I'm not sure what I'm doing wrong, I know traefik in host network mode should work, but then it would no longer find the docker services on it's own. This is my docker compose file: version: "3.7" networks: t2_proxy: external: name: t2_proxy default: driver: bridge services: traefik: # The official v2 Traefik docker image image: traefik/traefik:latest container_name: traefik restart: unless-stopped # Enables the web UI and tells Traefik to listen to docker command: # CLI arguments - --global.checkNewVersion=true - --global.sendAnonymousUsage=true - --entryPoints.https.address=:443 - --api=true - --api.insecure=true - --api.dashboard=true - --log=true - --log.level=INFO # (Default: error) DEBUG, INFO, WARN, ERROR, FATAL, PANIC - --accessLog=true - --accessLog.filePath=/traefiklog/traefik.log - --accessLog.bufferingSize=100 # Configuring a buffer of 100 lines - --providers.docker=true - --providers.docker.defaultrule=Host(`{{ index .Labels "com.docker.compose.service" }}.xxx.com`) - --providers.docker.endpoint=unix:///var/run/docker.sock - --providers.docker.exposedByDefault=false - --providers.docker.network=t2_proxy - --providers.docker.swarmMode=false - --providers.file.directory=/rules # Load dynamic configuration from one or more .toml or .yml files in a directory. - --providers.file.watch=true # Only works on top level files in the rules folder networks: t2_proxy: ipv4_address: 192.168.90.254 ports: - target: 443 published: 443 protocol: tcp mode: host volumes: - C:\docker/traefik/rules:/rules - /var/run/docker.sock:/var/run/docker.sock:ro - C:\docker/traefik/logs:/traefiklog - C:\docker/shared:/shared labels: - "traefik.enable=true" whoami: image: "containous/whoami" container_name: "whoami" networks: t2_proxy: ipv4_address: 192.168.90.200 labels: - "traefik.enable=true" - "traefik.http.routers.whoami.rule=Host(`whoami.xxx.com`)" - "traefik.http.routers.whoami.entrypoints=https" - "traefik.http.routers.whoami.tls=true" I set up the network with: docker network create --gateway 192.168.90.1 --subnet 192.168.90.0/24 t2_proxy and this is the output of the whoami webrequest in the browser: Hostname: 8752e7b8a5d4 IP: 127.0.0.1 IP: 192.168.90.200 RemoteAddr: 192.168.90.254:36228 GET / HTTP/1.1 Host: whoami.xxx.com:433 User-Agent: Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 Accept-Encoding: gzip, deflate, br Accept-Language: en-US Cache-Control: max-age=0 Dnt: 1 Te: trailers Upgrade-Insecure-Requests: 1 X-Forwarded-For: 192.168.90.1 X-Forwarded-Host: whoami.xxx.com:433 X-Forwarded-Port: 433 X-Forwarded-Proto: https X-Forwarded-Server: 969f601c0c24 X-Real-Ip: 192.168.90.1
The solution seems to be presented in this guide: https://dockerswarm.rocks/traefik/#getting-the-client-ip If you need to read the client IP in your applications/stacks using the X-Forwarded-For or X-Real-IP headers provided by Traefik, you need to make Traefik listen directly, not through Docker Swarm mode, even while being deployed with Docker Swarm mode. ports: - target: 80 published: 80 mode: host - target: 443 published: 443 mode: host
Not seeing the real IP in X-Forwarded-For with the basic setup of Traefik in Docker?
Fell in love with Traefik's presentation on youtube, so I've been trying to set it up. My docker server is on 192.168.0.23. Even when querying from another host from my LAN (192.168.0.144), I am always getting the internal IP in the X-Forwarded-For... curl -H Host:whoami.docker.localhost http://192.168.0.23:8880 shows Hostname: 20f76fbc038e IP: 127.0.0.1 IP: 172.22.0.2 GET / HTTP/1.1 Host: whoami.docker.localhost User-Agent: curl/7.51.0 Accept: */* Accept-Encoding: gzip X-Forwarded-For: 172.22.0.1 X-Forwarded-Host: whoami.docker.localhost X-Forwarded-Port: 80 X-Forwarded-Proto: http X-Forwarded-Server: c40ba22259b6 X-Real-Ip: 172.22.0.1 The same thing happens when calling from another network through the public IP, still seeing X-Forwarded-For: 172.22.0.1 My docker-compose is one of the basic examples... version: '3' services: reverse-proxy: image: traefik command: --api --docker ports: - "8880:80" - "8080:8080" volumes: - /var/run/docker.sock:/var/run/docker.sock whoami: image: containous/whoami labels: - "traefik.frontend.rule=Host:whoami.docker.localhost" - "traefik.frontend.passHostHeader=true" Am I missing something? I'd expect to at least get the IP of the host initiating the request in the list of X-Forwarded-For, as I've been seeing all over examples in the web... The weird thing is that the IP that shows in the forwarded field is the IP of the server in the docker network, which would make sense when the curl is executed from the server itself, not from another host in the same network or through the internet...
We had the same issue in kubernetes. We however solved the problem by adding the following global arguments. globalArguments: - "--api.insecure=true" - "--entryPoints.web.forwardedHeaders.insecure" - "--entryPoints.websecure.forwardedHeaders.insecure"