NGINX, Docker-Compose, SSL Reverse Proxy Has failed (111: Connection refused) - docker

I'm trying to get a docker-compose to use a nginx reverse proxy for ssl. I've looked at several different tutorials online, and the below is the best approximation of the answer. However, I am getting a 502 Bad Gateway Error and the following error in nginx. I'm not sure why. https seems to work (as it routes to the Error page), but I don't know what else is happening here. Any ideas?
production_nginx | 2021/01/12 02:54:34 [error] 29#29: *1 connect() failed (111: Connection refused) while connecting to upstream, client: <IP_ADDRESS_HERE>, server: www.websiteunderdevelopment.com, request: "GET / HTTP/1.1", upstream: "http://172.21.0.4:3001/", host: "www.websiteunderdevelopment.net"
Here is the docker container -
version: "3.3"
services:
nginx:
image: nginx:latest
container_name: production_nginx
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
- ./error_log:/etc/nginx/error_log.log
- ./nginx/cache/:/etc/nginx/cache
- /etc/letsencrypt/:/etc/letsencrypt/
ports:
- 80:80
- 443:443
depends_on:
- blog
- api
- db
blog:
container_name: blog
build: ./blog
ports:
- 3001:3000
expose:
- "3000"
- "3001"
- "80"
depends_on:
- api
api:
container_name: api
build: ./api
restart: always
ports:
- 4001:4000
expose:
- "4000"
- "4001"
depends_on:
- db
command: ["./wait-for-it.sh", "http://localhost:3306", "--", "npm", "start"]
volumes:
- ./api:/var/lib/api
db:
container_name: db
build: ./db
command: --default-authentication-plugin=mysql_native_password
restart: always
ports:
- "3306:3306"
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=True
- MYSQL_DATABASE=blog
- MYSQL_USER=SUFUPDFD
- MYSQL_ROOT_PASSWORD=NEST
- MYSQL_PASSWORD=SUPERSECCRETT
volumes:
- db_data:/var/lib/mysql
volumes:
db_data: {}
Here is my nginx file -
events{}
http{
server {
listen 80;
listen 443 ssl;
server_name www.websiteunderdevelopment.com websiteunderdevelopment.com;
ssl_certificate /etc/letsencrypt/live/www.websiteunderdevelopment.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/www.websiteunderdevelopment.net/privkey.pem;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:443;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Proto https;
proxy_pass http://blog:3001/;
}
location /api {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host:443;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-Proto https;
proxy_pass http://api:4001/;
}
}
}

Related

certbot challenge fails with jellyfin as it returns 404

jellyfin container runs behind nginx reverse proxy.
When I try to get an ssl certificate, jellyfin unfortunately returns a 404 error. Anyone know what I need to change in the configuration to make it work?
my docker-compose.yml
services:
nginx:
container_name: nginx
image: nginx:1.23.3-alpine
restart: unless-stopped
ports:
- 80:80
- 443:443
volumes:
- ./nginx/conf.d/:/etc/nginx/conf.d/
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
- ./nginx/certs/:/etc/nginx/certs/
networks:
- jellyfin
jellyfin:
container_name: jellyfin
image: jellyfin/jellyfin
restart: unless-stopped
user: 1000:1000
volumes:
- ./jellyfin/config:/config
- ./jellyfin/cache:/cache
- ./jellyfin/media/:/media
networks:
- jellyfin
networks:
jellyfin:
driver: bridge
my nginx .conf file
upstream jellyfin {
server jellyfin:8096;
}
server {
listen 80;
server_name jellyfin.mydomain.com;
location / {
proxy_pass http://jellyfin/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#upgrade to WebSocket protocol when requested
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
}
}
certbot response
Type: unauthorized
Detail: Invalid response from
http://jellyfin.mydomain.com/.well-known/acme-challenge/C8YTfjbIku65D_Hb2BCTkWEzdcwBqk4g8Wks0umq4Hw:
404

Containerized Keycloak behind Nginx not working (502 Bad Gateway)

I need to serve containerized keycloak behind Nginx. Keycloak runs without any problem at 'localhost:8080' but when I try to access it through the reverse proxy at 'localhost/auth' I get '502 Bad Gateway'.
Here's the details of the error taken from Nginx logs:
[error] 8#8: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.80.1, server: , request: "GET /auth/ HTTP/1.1", upstream: "http://127.0.0.1:8080/auth/", host: "localhost"
Please find below my docker-compose file (I haven't pasted the other containers):
version: '3'
services:
keycloak:
image: jboss/keycloak
container_name: keycloak
ports:
- "8080:8080"
environment:
KEYCLOAK_USER: ${KEYCLOAK_USER}
KEYCLOAK_PASSWORD: ${KEYCLOAK_PASSWORD}
KEYCLOAK_DB_VENDOR: ${KEYCLOAK_DB_VENDOR}
KEYCLOAK_DB_ADDR: ${KEYCLOAK_DB_ADDR}
KEYCLOAK_DB_DATABASE: ${KEYCLOAK_DB_DATABASE}
KEYCLOAK_DB_USER: ${KEYCLOAK_DB_USER}
KEYCLOAK_DB_PASSWORD: ${KEYCLOAK_DB_PASSWORD}
PROXY_ADDRESS_FORWARDING: 'true'
depends_on:
- keycloak-db
keycloak-db:
image: postgres
container_name: keycloak-db
restart: unless-stopped
environment:
POSTGRES_USER: ${POSTGRES_USERNAME}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
POSTGRES_DB: keycloak
volumes:
- ./keycloak/data:/var/lib/postgresql/data/
networks:
- my-app
nginx:
image: nginx:1.15-alpine
container_name: nginx
build:
context: ./nginx
restart: unless-stopped
ports:
- "80:80"
volumes:
- ./nginx/conf:/etc/nginx/conf.d
networks:
- my-app
networks:
my-app:
This is the Nginx upstream.conf:
# path: /etc/nginx/conf.d/upstream.conf
# Keycloak server
upstream keycloak {
server localhost:8080;
}
Nginx default.conf:
server {
listen 80;
root /usr/share/nginx/html;
include /etc/nginx/mime.types;
# keycloak
location /auth {
proxy_pass http://keycloak/auth;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
location /auth/admin {
proxy_pass http://keycloak/auth/admin;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
}
Does anyone have any idea what is wrong in my configuration?
Obvious problem:
upstream keycloak {
server localhost:8080;
}
Each container has own "localhost", so you are connecting to nginx's localhost != keycloak's localhost != host's localhost. Use service name there, e.g.:
upstream keycloak {
server keycloak:8080;
}

Docker nginx reverse proxy error - 502 bad gateway - connection refused

I'm having a problem with getting to work my NGINX reverse proxy on Docker.
When I access:
local.lab - NGINX responds with expected index.html page
127.0.0.1:2000 or 127.0.0.1:2001 or 127.0.0.1:2002 - service works and I get expected results
local.lab/a1 or local.lab/a2 or local.lab/a3 - I get "502 Bad Gateway" error.
Detailed error from nginx log:
2021/02/25 18:20:48 [error] 30#30: *4 connect() failed (111: Connection refused) while connecting to upstream, client: 172.19.0.1, server: local.lab, request: "GET /a2 HTTP/2.0", upstream: "http://127.0.0.1:2006/", host: "www.local.lab"
I tried to add network_mode: host to nginx service in docker compose without success.
I'm using docker compose:
version: '3.7'
services:
nginx:
container_name: lab-nginx
image: nginx:latest
restart: always
depends_on:
- http1
- http2
- http3
volumes:
- ./html:/usr/share/nginx/html/
- ./nginx.conf:/etc/nginx/nginx.conf
- ./error_log/error.log:/var/log/nginx/error.log
- ./cert:/var/log/nginx/cert/
ports:
- 80:80
- 443:443
http1:
container_name: lab-http1
image: httpd:latest
restart: always
# build:
# context: ./apache_service
ports:
- 2000:80
- 2005:443
volumes:
- ./apache/index1.html:/usr/local/apache2/htdocs/index.html
http2:
container_name: lab-http2
image: httpd:latest
restart: always
ports:
- 2001:80
- 2006:443
volumes:
- ./apache/index2.html:/usr/local/apache2/htdocs/index.html
http3:
container_name: lab-http3
image: httpd:latest
restart: always
ports:
- 2002:80
- 2007:443
volumes:
- ./apache/index3.html:/usr/local/apache2/htdocs/index.html
My nginx config:
worker_processes auto;
events { worker_connections 1024;}
error_log /var/log/nginx/error.log error;
http{
server {
listen 443 ssl http2;
server_name local.lab;
ssl_certificate /var/log/nginx/cert/local.lab.crt;
ssl_certificate_key /var/log/nginx/cert/local.lab.key;
ssl_protocols TLSv1.3;
location / {
root /usr/share/nginx/html;
index index.html;
}
location /a1 {
proxy_pass http://127.0.0.1:2000/;
proxy_set_header X-Forwarded-For $remote_addr;
}
location /a2 {
proxy_pass http://127.0.0.1:2001/;
proxy_set_header X-Forwarded-For $remote_addr;
}
location /a3 {
proxy_pass http://127.0.0.1:2002/;
proxy_set_header X-Forwarded-For $remote_addr;
}
}
}
How can I fix this?
The reverse proxy configuration in NGINX should reference the internal ports of your services, not the external ports they are mapped to in the docker-compose.yml. The services all have different names running in different containers so they can run on the same port (80 in this case) and use the service name, not the loopback address. You need to map them to different ports externally though because you can't have more than one service per port on your host.
For example:
location /a1 {
proxy_pass http://http1:80/;
proxy_set_header X-Forwarded-For $remote_addr;
}
location /a2 {
proxy_pass http://http2:80/;
proxy_set_header X-Forwarded-For $remote_addr;
}
location /a3 {
proxy_pass http://http3:80/;
proxy_set_header X-Forwarded-For $remote_addr;
}

NginX reverse proxy for Kibana over Docker

I have a Docker Compose setup with NginX, ElasticSearch and Kibana like the following:
web:
build:
context: .
dockerfile: ./system/docker/development/web.Dockerfile
depends_on:
- app
volumes:
- './system/ssl:/etc/ssl/certs'
networks:
- mynet
ports:
- 80:80
- 443:443
elasticsearch_1:
image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
container_name: "${COMPOSE_PROJECT_NAME:-service}_elasticsearch_1"
environment:
- node.name=elasticsearch_1
- cluster.name=es-docker-cluster
- discovery.seed_hosts=elasticsearch_2,elasticsearch_3
- cluster.initial_master_nodes=elasticsearch_1,elasticsearch_2,elasticsearch_3
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es_volume_1:/usr/share/elasticsearch/data
ports:
- 9200:9200
networks:
- mynet
elasticsearch_2:
image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
container_name: "${COMPOSE_PROJECT_NAME:-service}_elasticsearch_2"
environment:
- node.name=elasticsearch_2
- cluster.name=es-docker-cluster
- discovery.seed_hosts=elasticsearch_1,elasticsearch_3
- cluster.initial_master_nodes=elasticsearch_1,elasticsearch_2,elasticsearch_3
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es_volume_2:/usr/share/elasticsearch/data
ports:
- 9201:9201
networks:
- mynet
elasticsearch_3:
image: docker.elastic.co/elasticsearch/elasticsearch:7.7.0
container_name: "${COMPOSE_PROJECT_NAME:-service}_elasticsearch_3"
environment:
- node.name=elasticsearch_3
- cluster.name=es-docker-cluster
- discovery.seed_hosts=elasticsearch_1,elasticsearch_2
- cluster.initial_master_nodes=elasticsearch_1,elasticsearch_2,elasticsearch_3
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
volumes:
- es_volume_3:/usr/share/elasticsearch/data
ports:
- 9202:9202
networks:
- mynet
kibana:
image: docker.elastic.co/kibana/kibana:7.7.0
container_name: "${COMPOSE_PROJECT_NAME:-service}_kibana"
ports:
- 5601:5601
environment:
ELASTICSEARCH_URL: http://elasticsearch_1:9200
ELASTICSEARCH_HOSTS: http://elasticsearch_1:9200
networks:
- mynet
volumes:
es_volume_1: null
es_volume_2: null
es_volume_3: null
networks:
mynet:
driver: bridge
ipam:
config:
- subnet: 172.18.0.0/24
gateway: 172.18.0.1
When I (build and) run this using docker-compose up I'm able to access Kibana through URL http://localhost:5601/ but when I try to setup a reverse proxy for the same using NginX, I get a 502 Bad Gateway error. Here's my NginX config file:
server {
listen 80;
listen 443 ssl http2;
ssl_certificate /ssl/localhost.crt;
ssl_certificate_key /ssl/localhost.key;
...
location /app/kibana {
proxy_pass http://localhost:5601;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
location ~ /\. {
deny all;
}
...
}
What I'm trying to do here is be able to access Kibana like http://localhost/app/kibana. The articles I've gone through (like this) seem to be focused more on securing Kibana access through NginX (using Basic Auth) rather than the ability to access on a particular path on port 80.
Update
So, I changed localhost to kibana (as suggested by #mikezter) and now it seems to be able to at least find the Kibana service (so there's no more 502 error).
However, then I encountered a blank page with a few errors in browser debug console. Upon searching, I came across this location directive:
location ~ (/app|/translations|/node_modules|/built_assets/|/bundles|/es_admin|/plugins|/api|/ui|/elasticsearch|/spaces/enter) {
proxy_pass http://kibana:5601;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $http_host;
proxy_set_header Authorization "";
proxy_hide_header Authorization;
}
Now the page loads and there is some UI, but there's still some issue with the scripting, so the page is not available for user interaction.
You are connecting all the containers in this config via container network. Look at the environment variables set in the Kibana config:
ELASTICSEARCH_URL: http://elasticsearch_1:9200
Here you can see, that the hostname of the other container running ElasticSearch is elasticsearch_1. In a similar manner, the hostname of the container running Kibana woud be kibana. These hostnames are only availiable inside the container network.
So in your Nginx config, you'll have to proxy_pass to http://kibana:5601 instead of localhost.
I understand this might not be a total fix for your problem for the second part of the problem but using the following
ELK_VERSION=7.12.0
Kibana seems to work well on the default route '/'
The below worked for me.
server {
listen 80 default_server;
server_name $hostname;
location / {
proxy_pass http://kibana:5601;
# kindly add your header config that works for you
}
}
I think it has to do with the way you're configuring your nginx location regex match.
The configuration I eventually went with was to enable nginx listen on multiple ports.
so I isolated by port exposed by kibanna which listen on the default route.
E.g. in my nginx.conf
server {
listen 80 default_server;
server_name $hostname;
location / {
proxy_pass http://identity-api;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
}
}
server {
listen 81;
server_name $hostname;
location / {
proxy_pass http://kibana:5601;
# kindly add your header config that works for you
}
}
Lastly I update my nginx port in docker-compose
nginx-reverseproxy:
ports:
- "80:80"
- "81:81"
First create a site-file for Nginx:
$ sudo nano /etc/nginx/sites-available/kibana.example.com
$ sudo ln -s /etc/nginx/sites-available/kibana.example.com /etc/nginx/sites-enabled/
Put the following into it:
server {
listen 80;
client_max_body_size 4G;
server_name kibana.example.com;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_redirect off;
proxy_buffering off;
proxy_pass http://kibana_server;
}
}
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream kibana_server {
server 127.0.0.1:5601;
}
In your docker-compose.yml, serve Kibana only locally on the host:
services:
...
kibana:
ports:
- "127.0.0.1:5601:5601"
...
Execute docker compose up -d
Run sanity check for your nginx configuration: sudo nginx -t.
Reload nginx: sudo systemctl restart nginx.
Access your kibana server at http://kibana.example.org.
PS: Its implied that kibana.example.org is just a placeholder domain name.

Nginx + Docker Compose - connect() failed (111: Connection refused) while connecting to upstream

thanks for taking the time to read this. I am trying to deploy my application to an AWS EC2 Instance using docker-compose. When i run the command docker-compose up and visit the site, I get an error from nginx saying the below error. I understand that nginx is receiving the request but is unable to find an upstream connection to my react app, and would appreciate any help in correctly configuring the ports/settings.
Error
2 connect() failed (111: Connection refused) while connecting to upstream, client: 108.212.77.70 server: example.com, request: "GET / HTTP/1.1", upstream: "http://172.29.0.4:8003/", host: "example.com"
Here is my nginx default config
upstream meetup_ws {
server channels:8001;
}
upstream meetup_backend {
server backend:8000;
}
upstream meetup_frontend {
server frontend:8003;
}
server {
listen 0.0.0.0:80;
server_name example.com www.example.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name example.com example.com;
root /var/www/frontend;
index index.html;
ssl_certificate /etc/nginx/certs/fullchain.pem;
ssl_certificate_key /etc/nginx/certs/privkey.pem;
add_header Strict-Transport-Security "max-age=31536000";
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://meetup_frontend;
}
location /api {
try_files $uri #proxy_api;
}
location #proxy_api {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://meetup_backend;
}
location /ws {
try_files $uri #proxy_websocket;
}
location #proxy_websocket {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://meetup_ws;
}
}
And this is my docker-compose.yml
version: '3'
services:
nginx:
build: ./nginx
restart: always
ports:
- 80:80
- 443:443
volumes:
- ./nginx/default.conf:/etc/nginx/conf.d/default.conf
- ./frontend/build:/var/www/frontend
- ./nginx/certs:/etc/nginx/certs
depends_on:
- channels
db:
image: postgres:12.0-alpine
ports:
- 5432:5432
environment:
- POSTGRES_USER=postgres
- POSTGRES_HOST=db
- POSTGRES_PASSWORD=password
volumes:
- postgres_data:/var/lib/postgresql/data/
backend: &backend
build: ./backend
volumes:
- ./backend:/app
ports:
- 8000:8000
command: ["python", "manage.py", "runserver"]
env_file:
- ./.env
depends_on:
- db
- redis
frontend:
build: ./frontend
volumes:
- ./frontend:/app
- node_modules:/app/node_modules
ports:
- 8003:8003
command: npm start
stdin_open: true
redis:
image: "redis:5.0.7"
worker_channels:
<<: *backend
command: ["python", "manage.py", "runworker", "channels"]
depends_on:
- db
- redis
ports:
- 8002:8002
channels:
<<: *backend
command: daphne -b 0.0.0.0 -p 8001 backend.asgi:application
ports:
- 8001:8001
depends_on:
- db
- redis
volumes:
node_modules:
postgres_data:
It is a bit embarrassing why the issue existed but I was able to solve the issue. I did
ping frontend in my nginx container and it was successfully pinging the frontend container. Next I did curl -L http://frontend:8003 and it said curl: (7) Failed to connect to frontend port 8003: Connection refused. I went to the frontend container and did netstat -tulpn and it listed 3000 as the port that was exposed. I check my .env file and it was missing port=8003. Nginx was able to connect upstream afterwards.

Resources