I have developed several docker-ized full-stack web applications that I am trying to route requests to using Traefik. I want to take advantage of the dynamic configuration via docker-compose labels. I would like to apply the stripPrefix middleware option so I can use the same application routing as if each app were served at the root. However, once these rules are applied it results in a 504 Gateway Timeout response.
Here's my set up:
Traefik 2.0.1
Docker 19.03.2, Compose 1.24.1
NGINX:latest images
A global docker network on which the Traefik container runs
Multiple multi-container applications, each of which includes an NGINX web server
All applications have their own local networks and the NGINX containers are also on the global network.
Each application is configured to be listening at /
Here is the docker-compose.yml definition for the offending NGINX container:
nginx:
image: nginx:latest
container_name: "mps_nginx"
volumes:
- ./nginx/confs/nginx.conf:/etc/nginx/default.conf
- ./static:/www/static
restart: "always"
labels:
- traefik.http.routers.mps.rule=Host(`localhost`) && PathPrefix(`/mps`)
- traefik.http.middlewares.strip-mps.stripprefix.prefixes=/mps
- traefik.http.routers.mps.middlewares=strip-mps#docker
networks:
- default
- mps
The aggravating part is, when I comment out the middlewares labels, it runs just fine but cannot find a matching URL pattern.
Prior to this I tested my approach using the whoami container that is defined in the Traefik Quickstart Tutorial:
# Test service to make sure our local docker-compose network functions
whoami:
image: containous/whoami
labels:
- traefik.http.routers.whoami.rule=Host(`localhost`) && PathPrefix(`/whoami`)
- traefik.http.middlewares.strip-who.stripprefix.prefixes=/whoami
- traefik.http.routers.whoami.middlewares=strip-who#docker
A request to http://localhost/whoami returns (among other things)
GET / HTTP/1.1.
This is exactly how I expected my routing approaches to work for all my other applications. The Traefik dashboard shows green all around for every middleware I register and yet all I see is 504 errors.
If anyone has any clues I would sincerely appreciate it.
To summarize my solution to a similar problem
my-service:
image: some/image
ports: # no need to expose port, just to show that service listens on 8090
- target: 8090
published: 8091
mode: host
labels:
- "traefik.enable=true"
- "traefik.http.routers.myservice-router.rule=PathPrefix(`/myservice/`)"
- "traefik.http.routers.myservice-router.service=my-service"
- "traefik.http.services.myservice-service.loadbalancer.server.port=8090"
- "traefik.http.middlewares.myservice-strip.stripprefix.prefixes=/myservice"
- "traefik.http.middlewares.myservice-strip.stripprefix.forceslash=false"
- "traefik.http.routers.myservice-router.middlewares=myservice-strip"
Now details
Routing rule
... redirect all calls thats path starts with /myservice/
- "traefik.http.routers.myservice-router.rule=PathPrefix(`/myservice/`)"
... to service my-service
- "traefik.http.routers.myservice-router.service=my-service"``
... to port 8090
- "traefik.http.services.myservice-service.loadbalancer.server.port=8090"
... and uses myservice-strip middleware
- "traefik.http.routers.myservice-router.middlewares=myservice-strip"
... this middleware will strip /myservice from path
- "traefik.http.middlewares.myservice-strip.stripprefix.prefixes=/myservice"
... and will not add trailing slash (/) if path becomes an empty string
- "traefik.http.middlewares.myservice-strip.stripprefix.forceslash=false"
There is an issue with prefix without ending with '/'.
Test your config like this:
- "traefik.http.routers.whoami.rule=Host(`localhost`) && (PathPrefix(`//whoami/`) || PathPrefix(`/portainer`))"
- "traefik.http.middlewares.strip-who.stripprefix.prefixes=/whoami"
I had a similar problem that was solved with the addition of
- "traefik.http.middlewares.strip-who.stripprefix.forceslash=true"
It makes sure the strip prefix doesn't also remove the forward slash.
You can read more about the forceslash documentation https://docs.traefik.io/middlewares/stripprefix/
Related
I want to run Grafana in a specific domain, but this problem is encountered
Grafana image version is 9.1.0
docker-compose:
version: "3.8"
services:
grafana:
container_name: ${CONTAINER_GRAFANA}
image: grafana/grafana:9.1.0
restart: unless-stopped
env_file: .env
environment:
TZ: Asia/Tehran
GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER}
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
volumes:
- grafana-data:/var/lib/grafana
- ./docker/grafana/grafana.ini:/etc/grafana/grafana.ini
#- /etc/grafana/provisioning
ports:
- "${GRAFANA_PORT}:3000"
grafana.ini:
[server]
# Protocol (http, https, h2, socket)
;protocol = http
# The ip address to bind to, empty will bind to all interface
;http_addr =
# The http port to use
;http_port = 3000
# The public facing domain name used to access grafana from a browser
domain = ${GRAFANA_DOMAIN}
# Redirect to correct domain if host header does not match domain
# Prevents DNS rebinding attacks
;enforce_domain = false
# The full public facing url you use in browser, used for redirects and emails
# If you use reverse proxy and sub path specify full url (with sub path)
root_url = %(protocol)s://%(domain)s/
If you're seeing this Grafana has failed to load its application files
This could be caused by your reverse proxy settings.
If you host grafana under subpath make sure your grafana.ini root_url setting includes subpath. If not using a reverse proxy make sure to set serve_from_sub_path to true.
If you have a local dev build make sure you build frontend using: yarn start, yarn start:hot, or yarn build
Sometimes restarting grafana-server can help
Check if you are using a non-supported browser. For more information, refer to the list of supported browsers.
I would suspect, that you are missing a GRAFANA_DOMAIN: ${GRAFANA_DOMAIN} in your docker-compose.yml. You need to pass through this environment variable so it is available inside the container.
I have been given a site to work with using https://github.com/dunglas/symfony-docker
All looks good, apart from I usually have 3 or 4 sites at one time in development, so I would usually in my docker-compose files set different ports for various services so they don't clash, for example:
nginx:
build: docker/nginx
ports:
- 97:80
However in this I don't seem to be able to change it, I am not sure if it is because of the caddy communication (first time using it, usually use nginx)
I tried setting different ports in the docker-compose.yml such as
caddy:
ports:
# HTTP
- target: 80
published: 97
protocol: tcp
# HTTPS
- target: 443
published: 8083
protocol: tcp
# HTTP/3
- target: 443
published: 8083
protocol: udp
But this just kept giving server 502 errors.
If I turn off all my other Docker containers I can get it working, so I know it is a port issue.
Does anyone familiar with this Docker setup know what I could be doing wrong? Maybe thinking it doesn't like non 443 port for it's self signed security cert? I did notice if I visit http://localhost:97 it redirected me to https://localhost (no port). However if I visit https://localhost:8083 I would get the 502 error
I also needed to use custom port in my project so i modified the caddy env var from:
caddy:
environment:
SERVER_NAME: ${SERVER_NAME:-localhost, caddy:80}
(comes with the repo) to:
caddy:
environment:
SERVER_NAME: ${SERVER_NAME:-http://localhost:80, https://localhost:443}
in the docker-compose file. Also as Blackban suggested you also need to check outer ports conflict and firewall.
Tracing makes finding parts in code, worthwhile a developers time and attention, much easier. For that reason, I attached Jaeger as tracer to a set of microservices inside Docker containers. I use Traefik as ingress controller/ service-mesh to route and proxy requests.
The problem I am facing is, that something's wrong with the tracing config in Traefik. Jaeger can not find the span context to connect the single/ service-dependend spans to a whole trace.
The following line appears in the logs:
{
"level":"debug",
"middlewareName":"tracing",
"middlewareType":"TracingEntryPoint",
"msg":"Failed to extract the context: opentracing: SpanContext not found in Extract carrier",
"time":"2021-02-02T23:16:51+01:00"
}
What I tried/ searched/ confirmed so far:
I already checked ports (they are open inside the Docker host network) and everything's reachable. So interconnectivity is not the problem here.
The forwarding of headers is set via Docker Compose labels: loadbalancer.passhostheader=true.
The following snippets describe the Docker Compose setup.
Traefik: Ingress Controller
This is a stripped down version of the traefik Container.
# Network
ROOT_DOMAIN=example.test
DEFAULT_NETWORK=traefik
---
version: '3'
services:
image: "traefik:2.4.2"
hostname: "controller"
restart: on-failure
security_opt:
- no-new-privileges:true
ports:
- "443:443"
- "80:80"
# The Web UI (enabled by --api.insecure=true)
- "8080:8080"
- "8082:8082"
- "8083:8083"
networks:
- default
working_dir: /etc/traefik
volumes:
- /private/etc/localtime:/etc/localtime:ro
- ${PWD}/controller/static.yml:/etc/traefik/traefik.yml:ro
- ${PWD}/controller/dynamic.yml:/etc/traefik/dynamic.yml:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- cert-storage:/usr/local/share/ca-certificates:ro
- ${PWD}/logs/traefik:/var/log/traefik
volumes:
cert-storage:
driver_opts:
type: none
o: bind
device: ${PWD}/certs/certs
networks:
default:
external: true
name: ${DEFAULT_NETWORK}
Traefik is set up using the file provider as base and Docker Compose labels on top of it:
# static.yaml (Traefik conf)
debug: true
log:
level: DEBUG
filePath: /var/log/traefik/error.log
format: json
serversTransport:
insecureSkipVerify: true
api:
dashboard: true
insecure: true
debug: true
providers:
docker:
exposedByDefault: false
swarmMode: false
watch: true
defaultRule: "Host(`{{ normalize .Name }}.example.test`)"
endpoint: "unix:///var/run/docker.sock"
network: traefik
file:
filename: /etc/traefik/dynamic.yml
watch: true
tracing:
serviceName: "controller"
spanNameLimit: 250
jaeger:
samplingType: const
samplingParam: 1.0
samplingServerURL: http://tracer:5778/sampling
localAgentHostPort: 127.0.0.1:6831
gen128Bit: true
propagation: jaeger
traceContextHeaderName: "traefik-trace-id"
collector:
endpoint: http://tracer:14268/api/traces?format=jaeger.thrift
Jaeger: Open Tracing/ Open Telemetry
---
version: '3'
services:
tracer:
image: "jaegertracing/all-in-one:1.21.0"
hostname: "tracer"
command:
- "--log-level=info"
- "--admin.http.host-port=:14269"
- "--query.ui-config=/usr/local/share/jaeger/ui/conf.json"
environment:
SPAN_STORAGE_TYPE: memory
restart: on-failure
security_opt:
- no-new-privileges:true
expose:
- 5775/udp
- 6831/udp
- 6832/udp
- 5778
- 14250
- 14268
- 14269
- 14271
- 16686
- 16687
volumes:
- /private/etc/localtime:/etc/localtime:ro
- ${PWD}/tracer/conf:/usr/local/share/jaeger
- ${PWD}/logs/jaeger:/var/log/#TODO
- cert-storage:/usr/local/share/ca-certificates
networks:
- default
labels:
- "traefik.enable=true"
- "traefik.docker.network=${DEFAULT_NETWORK}"
# Admin UI router
- "traefik.http.routers.tracer-router.rule=Host(`tracer.$ROOT_DOMAIN`)"
- "traefik.http.routers.tracer-router.entrypoints=https"
- "traefik.http.routers.tracer-router.tls=true"
- "traefik.http.routers.tracer-router.tls.options=default"
- "traefik.http.routers.tracer-router.service=tracer"
# Service/ Load Balancer
- "traefik.http.services.tracer.loadbalancer.passhostheader=true"
- "traefik.http.services.tracer.loadbalancer.server.port=16686"
- "traefik.http.services.tracer.loadbalancer.server.scheme=http"
I'm not 100% sure what the problem is you're experiencing, but here's some things to consider.
According to this post on the Traefik forums, that message you're seeing is debug level because it's not something you should be worried about. It's just logging that no trace context was found, so a new one will be created. That second part is not in the message, but apparently that's what happens.
You should check to see if you're getting data appearing in Jaeger. If you are, that message is probably nothing to worry.
If you are getting data in Jaeger, but it's not connected, that will be because Traefik can only only work with trace context that is already in inbound requests, but it can't add trace context to outbound requests. Within your application, you'll need to implement trace propagation so that your outbound requests include the trace context that was received as part of the incoming request. Without doing that, every request will be sent without trace context and will start a new trace when it is received at the next Traefik ingress point.
The problem actually was with the traceContextHeaderName. Sadly I can not tell exactly what the problem was as the git diff only shows that nothing changed around traefik and jaeger at the point where I fixed it. I assume config got "stuck" somehow. I tracked down the related lines in source, but as I am no Go-Dev, I can only guess if there's a bug.
What I did was to switch back to uber-trace-id, which magically "fixed" it. After I ran some traces and connected another service (node, npm jaeger-client with process.env.TRACER_STATE_HEADER_NAME set to an equal value), I switched back to traefik-trace-id and things worked.
I'm using a K3S Cluster in a docker(-compose) container in my CI/CD pipeline, to test my application code. However I have problem with the certificate of the cluster. I need to communicate on the cluster using the external addres. My docker-compose script looks as follows
version: '3'
services:
server:
image: rancher/k3s:v0.8.1
command: server --disable-agent
environment:
- K3S_CLUSTER_SECRET=somethingtotallyrandom
- K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
- K3S_KUBECONFIG_MODE=666
volumes:
- k3s-server:/var/lib/rancher/k3s
# get the kubeconfig file
- .:/output
ports:
# - 6443:6443
- 6080:6080
- 192.168.2.110:6443:6443
node:
image: rancher/k3s:v0.8.1
tmpfs:
- /run
- /var/run
privileged: true
environment:
- K3S_URL=https://server:6443
- K3S_CLUSTER_SECRET=somethingtotallyrandom
ports:
- 31000-32000:31000-32000
volumes:
k3s-server: {}
accessing the cluster from python gives me
MaxRetryError: HTTPSConnectionPool(host='192.168.2.110', port=6443): Max retries exceeded with url: /apis/batch/v1/namespaces/mlflow/jobs?pretty=True (Caused by SSLError(SSLCertVerificationError("hostname '192.168.2.110' doesn't match either of 'localhost', '172.19.0.2', '10.43.0.1', '172.23.0.2', '172.18.0.2', '172.23.0.3', '127.0.0.1', '0.0.0.0', '172.18.0.3', '172.20.0.2'")))
Here are my two (three) question
how can I add additional IP adresses to the cert generation? I was hoping the --bind-address in the server command triggers taht
how can I fall back on http providing an --http-listen-port didn't achieve the expected result
any other suggestion how I can enable communication with the cluster
changing the python code is not really an option as I would like o keep the code unaltered for testing. (Fallback on http works via kubeconfig.
The solution is to use the parameter tls-san
server --disable-agent --tls-san 192.168.2.110
I am running Neo4J in Docker behind Traefik. The web interface (7474) is exposed on https, which works as expected.
Through the web-interface, I attempt to connect to bolt://localhost:7687, which I would expect to work, after all it's running in that same container. Instead I receive the message:
"ServiceUnavailable: WebSocket connection failure. Due to security constraints in your web browser, the reason for the failure is not available to this Neo4j Driver. Please use your browsers development console to determine the root cause of the failure. Common reasons include the database being unavailable, using the wrong connection URL or temporary network problems. If you have enabled encryption, ensure your browser is configured to trust the certificate Neo4j is configured to use. WebSocket readyState is: 3"
The reverse proxy uses an internal certificate signed by our corporate CA. The Neo4J container has the root certificate available.
The client connecting also has the root certificate available.
When attempting to connect to localhost:7687 in the container itself, it works as expected; so does Cypher shell.
There were also some posts on GitHub regarding the Java version which could cause these problems - I have tried with Oracle JDK 8, OpenJDK 8 (HotSpot & J9).
Browser also seems unrelated - the described issue reproduces with FF, Chrome & Safari.
I was in the same situation and here is my solution in order to run a Neo4j instance (for Bloodhound in my case) in a docker swarm with Traefik 2.0.2. Due to Traefik's new router concept, it's totally verbose and difficult to read, because you have to define five routers. But at least it works and enables you to access both the webinterface and the actual database via wss and bolt. Maybe it's possible to simplify my configuration.
Furthermore, I specified the redirectScheme middleware and certificates signed by an internal PKI in dynamic config files delivered by Traefik's file provider.
HTH!
version: '3.7'
services:
neo4j:
image: neo4j:latest
networks:
- traefiknet
volumes:
- type: volume
source: data
target: /data
- type: volume
source: logs
target: /logs
- type: volume
source: conf
target: /conf
environment:
- NEO4J_AUTH=neo4j/somepassword
- NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
deploy:
replicas: 1
restart_policy:
condition: any
placement:
constraints:
- node.hostname == node-02
labels:
- traefik.http.services.neo4j.loadbalancer.server.port=7474
- traefik.http.routers.neo4j.rule=Host(`neo4j.dockerswarm.domain.com`)
- traefik.http.routers.neo4j-secure.rule=Host(`neo4j.dockerswarm.domain.com`)
- traefik.http.routers.neo4j.service=neo4j
- traefik.http.routers.neo4j-secure.service=neo4j
- traefik.http.routers.neo4j.middlewares=httpsredirection#file
- traefik.http.routers.neo4j-secure.tls=true
- traefik.http.routers.neo4j.entrypoints=web
- traefik.http.routers.neo4j-secure.entrypoints=web-secure
- traefik.http.services.neo4jdb.loadbalancer.server.port=7687
- traefik.http.routers.neo4jdb.rule=Host(`neo4jdb.dockerswarm.domain.com`)
- traefik.http.routers.neo4jdb-secure.rule=Host(`neo4jdb.dockerswarm.domain.com`)
- traefik.http.routers.neo4jdb.service=neo4jdb
- traefik.http.routers.neo4jdb-secure.service=neo4jdb
- traefik.http.routers.neo4jdb.middlewares=httpsredirection#file
- traefik.http.routers.neo4jdb-secure.tls=true
- traefik.http.routers.neo4jdb.entrypoints=web
- traefik.http.routers.neo4jdb-secure.entrypoints=web-secure
- traefik.tcp.services.neo4jdb.loadbalancer.server.port=7687
- traefik.tcp.routers.neo4jdb.rule=HostSNI(`neo4jdb.dockerswarm.domain.com`)
- traefik.tcp.routers.neo4jdb.service=neo4jdb
- traefik.tcp.routers.neo4jdb.tls=true
networks:
traefiknet:
external: true
volumes:
data:
logs:
conf: