Keycloak docker Administration Console shows blank page with reverse proxy - docker

I am using HAProxy with Keycloak, welcome page is showing fine but each time I enter Administration Console it shows me a blank page with no info with status code 200.
I am using let's encrypt SSL certificate and here is my HAProxy config and docker-compose.
Screenshot of the page:-
link to screenshot
HAProxy config:-
global
log stdout local0 debug
daemon
maxconn 4096
defaults
mode http
option httplog
option dontlognull
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
log global
log-format {"type":"haproxy","timestamp":%Ts,"http_status":%ST,"http_request":"%r","remote_addr":"%ci","bytes_read":%B,"upstream_addr":"%si","backend_name":"%b","retries":%rc,"bytes_uploaded":%U,"upstream_response_time":"%Tr","upstream_connect_time":"%Tc","session_duration":"%Tt","termination_state":"%ts"}
frontend public
bind *:80
bind *:443 ssl crt /usr/local/etc/haproxy/haproxy.pem alpn h2,http/1.1
http-request redirect scheme https unless { ssl_fc }
default_backend web_servers
backend web_servers
option forwardfor
server auth1 auth:8080
docker-compose.yaml :-
version: "3"
networks:
internal-network:
services:
reverse-proxy:
build: ./reverse-proxy/.
image: reverseproxy:latest
ports:
- "80:80"
- "443:443"
networks:
- internal-network
depends_on:
- auth
auth:
image: quay.io/keycloak/keycloak:latest
networks:
internal-network:
environment:
PROXY_ADDRESS_FORWARDING: "true"
KEYCLOAK_USER: ***
KEYCLOAK_PASSWORD: ***
# Uncomment the line below if you want to specify JDBC parameters. The parameter below is just an example, and it shouldn't be used in production without knowledge. It is highly recommended that you read the PostgreSQL JDBC driver documentation in order to use it.
#JDBC_PARAMS: "ssl=false"
the URL to the page I am trying to access is https:///auth/admin/master/console/
Notes: when trying to remove SSL from HAProxy, the Keycloak opens a page with error "https required"

One obvious issue (there can be more issues, so fix of this one may still not fix everything):
https://www.keycloak.org/docs/latest/server_installation/index.html#_setting-up-a-load-balancer-or-proxy
Configure your reverse proxy or loadbalancer to properly set X-Forwarded-For and X-Forwarded-Proto HTTP headers.
You didn't configure this part in your haproxy frontend section. You need that:
http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Proto https
Https protocol is required for OIDC, so "https required" is correct response, when you reach Keycloak via http protocol.

Related

How to fix 502 Bad Gateway error in nginx?

I have a server running docker-compose. Docker-compose has 2 services: nginx (as a reverse proxy) and back (as an api that handles 2 requests). In addition, there is a database that is not located on the server, but separately (database as a service).
Requests processed by back:
get('/api') - the back service simply replies "API is running" to it
get('/db') - the back service sends a simple query to an external database ('SELECT random() as random, current_database() as db')
request 1 - works fine, request 2 - the back service crashes, nginx continues to work and a 502 Bad Gateway error appears in the console.
An error occurs in the nginx service Logs: upstream prematurely
closed connection while reading response header from upstream.
The back service Logs: connection terminated due to connection timeout.
These are both rather vague errors. And I don’t know how else to get close to them, given that the code is not in a container, without Nginx and with the same database, it works as it should.
What I have tried:
increase the number of cores and RAM (now 2 cores and 4 GB of Ram);
add/remove/change proxy_read_timeout, proxy_send_timeout and proxy_connect_timeout parameters;
test the www.test.com/db request via postman and curl (fails with the same error);
run the code on your local machine without a container and compose and connect to the same database using the same pool and the same ip (everything is ok, both requests work and send what you need);
change the parameter worker_processes (tested with a value of 1 and auto);
add/remove attribute proxy_set_header Host $http_host, replace $http_host with "www.test.com".
Question:
What else can I try to fix the error and make the db request work?
My nginx.conf:
worker_processes 1;
events {
worker_connections 1024;
}
http{
upstream back-stream {
server back:8080;
}
server {
listen 80;
listen [::]:80;
server_name test.com www.test.com;
location / {
root /usr/share/nginx/html;
resolver 121.0.0.11;
proxy_pass http://back-stream;
}
}
}
My docker-compose.yml:
version: '3.9'
services:
nginx-proxy:
image: nginx:stable-alpine
container_name: nginx-proxy
ports:
- 80:80
- 443:443
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
networks:
- network
back:
image: "mycustomimage"
container_name: back
restart: unless-stopped
ports:
- '81:8080'
networks:
- network
networks:
network:
driver: bridge
I can upload other files if needed. Just taking into account the fact that the code does not work correctly in the container, the problem is rather in setting up the container.
I will be grateful for any help.
Code of the back: here
The reason for the error is this: I forgot to add my server's ip to the list of allowed addresses in the database cluster.

Docker Registry behind HAProxy not working properly

I have configured Docker Registry using htpasswd authentication as docker service. I use a port mapping 443:5000 and it works perfectly. "docker login :443" works, "docker pull <mydomain:433/myimage" works.
The container service looks like this:
myhub:
image: myhub
ports:
- "443:5000"
environment:
- REGISTRY_HTTP_ADDR=0.0.0.0:5000
- REGISTRY_HTTP_TLS_KEY=/certs/efimeridopolis.privkey.pem
- REGISTRY_HTTP_TLS_CERTIFICATE=/certs/efimeridopolis.fullchain.pem
- REGISTRY_AUTH=htpasswd
- REGISTRY_AUTH_HTPASSWD_REALM='Registry Realm'
- REGISTRY_AUTH_HTPASSWD_PATH=/auth/registry.htpasswd
volumes:
- /mnt/volume1/hub:/var/lib/registry
networks:
- myoverlay
Now, I try to put it behind HAProxy. SSL is terminated at HAProxy, so the service looks like:
myhub:
image: myhub
environment:
- REGISTRY_HTTP_ADDR=0.0.0.0:5000
- REGISTRY_AUTH=htpasswd
- REGISTRY_AUTH_HTPASSWD_REALM='Registry Realm'
- REGISTRY_AUTH_HTPASSWD_PATH=/auth/registry.htpasswd
volumes:
- /mnt/volume1/hub:/var/lib/registry
networks:
- myoverlay
and the relevant part of HAProxy configuration is:
frontend fe_443
mode http
bind *:443 ssl crt /etc/ssl/private/mydomain.pem
http-request add-header X-Forwarded-Proto https if { ssl_fc }
acl host_registry hdr(host) -i hub.mydomain
use_backend be_registry_443 if host_registry
backend be_registry_443
mode http
option forwardfor
server hub1 myhub:5000 check
It seems something is going wrong here. While accessing
https://hub.mydomain/v2/_catalog
through browser works, which means I am asked for username/password and then I get the list or repositories, when I try to use the console to:
$ docker pull hub.mydomain:443/v2/myhaproxy
it gives me:
Using default tag: latest
Error response from daemon: received unexpected HTTP status: 503 Service Unavailable
The same when I try:
$ docker login mydomain:443
I am asked username and password but then I get the same 503 message.
Since browser can list the repositories, I know the registry is online and accessible.
What is wrong ?
https (443 port) should be tcp mode and using req.ssl_sni instead hdr(host).

Forwarding TCP traffic from Traefik to a Docker container

I currently have a Traefik instance that's being run using the following. It works fine forwarding HTTP connections to the appropriate backends.
services_lb:
image: traefik:v2.2
cmd: |
--entrypoints.web.address=:80
--entrypoints.websecure.address=:443
--entrypoints.web.http.redirections.entryPoint.to=websecure
--entrypoints.web.http.redirections.entryPoint.scheme=https
--entrypoints.web.http.redirections.entrypoint.permanent=true
--entrypoints.matrixfederation.address=:8448
--entrypoints.prosodyc2s.address=:5222
--entrypoints.prosodys2s.address=:5269
--providers.docker
--providers.docker.constraints=Label(`lb.net`,`services`)
--providers.docker.network=am-services
--certificatesresolvers.lec.acme.email=notify#battlepenguin.com
--certificatesresolvers.lec.acme.storage=/letsencrypt/acme.json
--certificatesresolvers.lec.acme.tlschallenge=true
--entryPoints.web.forwardedHeaders.trustedIPs=172.50.0.1/24
ports:
- 80
- 443
# Matrix
- 8448
# XMPP
- 5222
- 5269
My web and Matrix federation connections work fine as they're all HTTP. But for Prosody (XMPP) I need to forward 5222 and 5269 directly without any HTTP routing. I configured the container like so:
xmpp:
image: prosody/prosody:0.11
network:
- services
- database
labels:
lb.net: services
traefik.tcp.services.prosodyc2s.loadbalancer.server.port: "5222"
traefik.tcp.services.prosodys2s.loadbalancer.server.port: "5269"
traefik.http.routers.am-app-xmpp.entrypoints: "websecure"
traefik.http.routers.am-app-xmpp.rule: "Host(`xmpp.example.com`)"
traefik.http.routers.am-app-xmpp.tls.certresolver: "lec"
traefik.http.services.am-app-xmpp.loadbalancer.server.port: "5280"
volumes:
- prosody-config:/etc/prosody:rw
- services_certs:/certs:ro
- prosody-logs:/var/log/prosody:rw
- prosody-modules:/usr/lib/prosody-modules:rw
With the tcp services, I still can't get Traefik to forward the raw TCP connections to this container. I've tried removing the --entrypoints from the Traefik instance and of course, Traefik stopped listening on those ports. I assumed the traefik.tcp.service definition would cause that entrypoint to switch to a TCP passthrough mode, but that isn't the case. I couldn't see anything in the Traefik documentation on putting the entrypoint itself into TCP mode instead of HTTP mode.
How do I pass the raw TCP connection from Traefik to this particular container using labels on the container and CLI options for Traefik?
I figured it out. You can't use any standard Traefik TLS offloading due to the differences in how Traefik and Prosidy handle TLS. I had to disable TLS entirely and use the special HostSNI(*) rule below to allow straight pass throughts. I was also missing the routers that connect the Traefik entrypoints to the TCP services.
labels:
lb.net: services
# client to server
traefik.tcp.routers.prosodyc2s.entrypoints: prosodyc2s
traefik.tcp.routers.prosodyc2s.rule: HostSNI(`*`)
traefik.tcp.routers.prosodyc2s.tls: "false"
traefik.tcp.services.prosodyc2s.loadbalancer.server.port: "5222"
traefik.tcp.routers.prosodyc2s.service: prosodyc2s
# server to server
traefik.tcp.routers.prosodys2s.entrypoints: prosodys2s
traefik.tcp.routers.prosodys2s.rule: HostSNI(`*`)
traefik.tcp.routers.prosodys2s.tls: "false"
traefik.tcp.services.prosodys2s.loadbalancer.server.port: "5269"
traefik.tcp.routers.prosodys2s.service: prosodys2s
# web
traefik.http.routers.am-app-xmpp.entrypoints: "websecure"
traefik.http.routers.am-app-xmpp.rule: "Host(`xmpp.example.com`)"
traefik.http.routers.am-app-xmpp.tls.certresolver: "lec"
traefik.http.services.am-app-xmpp.loadbalancer.server.port: "5280"

Traefik v2 reverse proxy to a local server outside Docker

I have a simple server written in Python that listens on port 8000 inside a private network (HTTP communication). There is now a requirement to switch to HTTPS communications and every client that sends a request to the server should get authenticated with his own cert/key pair.
I have decided to use Traefik v2 for this job. Please see the block diagram.
Traefik runs as a Docker image on a host that has IP 192.168.56.101. First I wanted to simply forward a HTTP request from a client to Traefik and then to the Python server running outside Docker on port 8000. I would add the TLS functionality when the forwarding is running properly.
However, I can not figure out how to configure Traefik to reverse proxy from i.e. 192.168.56.101/notify?wrn=1 to the Python server 127.0.0.1:8000/notify?wrn=1.
When I try to send the above mentioned request to the server (curl "192.168.56.101/notify?wrn=1") I get "Bad Gateway" as an answer. What am I missing here? This is the first time that I am in contact with Docker and reverse proxy/Traefik. I believe it has something to do with ports but I can not figure it out.
Here is my Traefik configuration:
docker-compose.yml
version: "3.3"
services:
traefik:
image: "traefik:v2.1"
container_name: "traefik"
hostname: "traefik"
ports:
- "80:80"
- "8080:8080"
volumes:
- "/var/run/docker.sock:/var/run/docker.sock:ro"
- "./traefik.yml:/traefik.yml:ro"
traefik.yml
## STATIC CONFIGURATION
log:
level: INFO
api:
insecure: true
dashboard: true
entryPoints:
web:
address: ":80"
providers:
docker:
watch: true
endpoint: "unix:///var/run/docker.sock"
file:
filename: "traefik.yml"
## DYNAMIC CONFIGURATION
http:
routers:
to-local-ip:
rule: "Host(`192.168.56.101`)"
service: to-local-ip
entryPoints:
- web
services:
to-local-ip:
loadBalancer:
servers:
- url: "http://127.0.0.1:8000"
First, 127.0.0.1 will resolve to the traefik container and not to the docker host. You need to provide a private IP of the node and it needs to be accessible form the traefik container.
There is some workaround to make proxy to localhost:
change 127.0.0.1 to IP of docker0 interface
It should be 172.17.0.1
and then try to listen your python server on all interfaces (0.0.0.0)
if you use simple python http server nothing change... on default it listen on all interfaces

Traefik and Nginx with HTTPS on Docker / 400 Bad Request

I'm trying to build stack with Traefik and Nginx based on Docker. Without HTTPS is everything fine, but I get an error as soon as I put on HTTPS configuration.
I'm getting this error from Nginx on example.com: 400 Bad Request / The plain HTTP request was sent to HTTPS port. In the address bar I can see the green lock saying connection is secure.
Certbot works fine so I have real SSL certificate inside the proper folder.
I can get to the Traefik dasboard when I visit traefik.example.com but I have to accept no SSL browser warning and dasboard is also working without HTTPS.
docker-compose.yml
version: '3.4'
services:
traefik:
image: traefik:latest
ports:
- "80:80"
- "443:443"
- "8080:8080"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- ./traefik/traefik.toml:/etc/traefik/traefik.toml
- ../letsencrypt:/etc/letsencrypt
labels:
- traefik.backend=traefik
- traefik.frontend.rule=Host:traefik.example.com
- traefik.port=8080
networks:
- traefik
nginx:
image: nginx:latest
volumes:
- ../www:/var/www
- ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf
- ../letsencrypt:/etc/letsencrypt
labels:
- traefik.backend=nginx
- traefik.frontend.rule=Host:example.com
- traefik.port=80
- traefik.port=443
networks:
- traefik
networks:
traefik:
driver: overlay
external: true
attachable: true
traefik.toml
defaultEntryPoints = ["http", "https"]
[web]
address = ":8080"
[entryPoints]
[entryPoints.http]
address = ":80"
[entryPoints.http.redirect]
entryPoint = "https"
[entryPoints.https]
address = ":443"
[entryPoints.https.tls]
[[entryPoints.https.tls.certificates]]
certFile = "/etc/letsencrypt/live/example.com/fullchain.pem"
keyFile = "/etc/letsencrypt/live/example.com/privkey.pem"
[docker]
domain="example.com"
watch = true
exposedByDefault = true
swarmMode = false
nginx.conf
server {
listen 80;
server_name example.com www.example.com;
return 301 https://www.example.com$request_uri;
}
server {
listen 443 ssl http2;
server_name example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
return 301 https://www.example.com$request_uri;
}
server {
listen 443 ssl http2;
server_name www.example.com;
ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
root /var/www/public;
index index.html;
}
Thanks for your help.
First there is no need to have SSL redirection configured in both Traefik and Nginx. Also Traefik frontend is matching only non www variant but backend app expects www. Finally Traefik web provider is deprecated so there should be newer api provider.
As I just stumbled upon a similar problem with Traefik v2
400 Bad Request / The plain HTTP request was sent to HTTPS port
with an Nginx error log stating
400 client sent plain HTTP request to HTTPS port while reading client request headers
and scratching my head over it, I finally found the source of that error. It's not that the TLS certs were invalid or something in the transport broken, but that the wiring between routers, services and port mappings were off.
Previously I did not see, that the Docker Compose stack had an Nginx container only listening on 80/tcp. I assumed everything's ok as I attached the ports to Traefik load balancers attached to a separate service per http/https endpoints with separated routers. This somehow did not work:
- "traefik.http.services.proxy.loadbalancer.server.port=80"
- "traefik.http.services.proxy-secure.loadbalancer.server.port=443"
Intermediary I now opened port: - "8008:80" - "8443:443" and got it working. Investigating further what's wrong with Traefik ports as those should get exposed per default. This is not a solution as those ports are now available to the outside world, but I am leaving this explanation here as I could not find anything on this topic that would point me in the right direction, so hopefully it's helpful for someone else later on.

Resources