nginx reverse proxy with Docker - Connection refused while connecting to upstream - docker

Environment setup
I have an application which is composed by some services:
jenkins server
nginx server with angular
nginx server as a proxy
Those services are defined in the docker-compose file:
version: '3'
services:
reverse:
container_name: reverse-proxy
build:
context: /app/mywallet/MyWalletFe/reverse-proxy
ports:
- "80:80"
networks:
- net
jenkins:
container_name: jenkins
image: jenkins/jenkins
volumes:
- "$PWD/jenkins_home:/var/jenkins_home"
- "/var/run/docker.sock:/var/run/docker.sock"
networks:
- net
angular:
container_name: mywallet_fe
build:
context: /app/mywallet/MyWalletFe
networks:
- net
networks:
net:
I defined the following configuration file for the reverse-proxy:
upstream client {
# angular is the name of the service in docker-compose file
server angular:4200;
}
upstream jenkins {
server jenkins:8080;
}
server {
listen 80;
location / {
proxy_pass http://client;
}
location /jenkins {
proxy_pass http://jenkins;
}
}
Finally, here is the Dockerfile for the reverse-proxy service, which copies the configuration file in the nginx container:
FROM nginx
# override default files if present
COPY ./default.conf /etc/nginx/conf.d/default.conf
Goal
My goal is to access Jenkins with SERVER_IP/jenkins
Output
When I run the whole application and try to access to SERVER_IP/jenkins, I get the following error in the reverse-proxy logs:
2020/04/15 21:44:55 [error] 6#6: *10 connect() failed (111: Connection
refused) while connecting to upstream, client: MY_CLIENT_IP, server: ,
request: "GET /login?from=%2Fjenkins HTTP/1.1", upstream:
"http://172.18.0.5:4200/login?from=%2Fjenkins", host: "SERVER_IP",
referrer: "http://SERVER_IP/jenkins" MY_CLIENT_IP
[15/Apr/2020:21:44:55 +0000] "GET /favicon.ico HTTP/1.1" 502 559
"http://SERVER_IP/login?from=%2Fjenkins" "Mozilla/5.0 (Windows NT
10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36" "-" 2020/04/15 21:44:55 [error]
6#6: *10 connect() failed (111: Connection refused) while connecting
to upstream, client: MY_CLIENT_IP, server: , request: "GET
/favicon.ico HTTP/1.1", upstream:
"http://172.18.0.5:4200/favicon.ico", host: "SERVER_IP",
referrer: "http://SERVER_IP/login?from=%2Fjenkins"
where MY_CLIENT_IP is my laptop IP and SERVER_IP is the IP of the server where the application runs.
What's wrong in the configuration of the reverse proxy? If I expose the jenkins and angular services I can reach them, while through the proxy I can't.
Similar question, which doesn't help me (or I don't understand how would those help me)
Docker nginx reverse proxy returns 502 bad gateway "connection refused while connecting to upstream"
Connection refused while connection to upstream - Docker
connect() failed (111: Connection refused) while connecting to upstream for nginx+php-fpm docker
docker nginx connection refused while connecting to upstream

Related

docker nginx load balancer connect() failed (111: Connection refused) while connecting to upstream

I'm trying to load balance a simple Nodejs app with 3 instances using docker-compose & nginx. This configuration works on my local machine (windows laptop) but doesn't seem to work on EC2 server.
nginx.conf
http {
upstream all {
server nodeapp1:4100;
server nodeapp2:4200;
server nodeapp3:4300;
}
server {
listen 8080;
location / {
proxy_pass http://all/;
}
}
}
events { }
docker-compose.yml
version: '3'
services:
lb:
image: nginx
volumes:
- ./nginxproxy/nginx.conf:/etc/nginx/nginx.conf
ports:
- "3000:8080"
nodeapp1:
image: nodeapp
environment:
- PORT=4100
ports:
- "4100:4100"
nodeapp2:
image: nodeapp
environment:
- PORT=4200
ports:
- "4200:4200"
nodeapp3:
image: nodeapp
environment:
- PORT=4300
ports:
- "4300:4300"
I'm new to docker. I'm surprised why this works locally but does not work on EC2 instance. The load balancer was able to resolve the url correctly but it still says connection refused.
Error:
2022/02/28 20:00:22 [error] 33#33: *9 connect() failed (111: Connection refused) while
connecting to upstream, client: 62.113.237.40, server: , request: "GET / HTTP/1.1",
upstream: "http://172.121.0.5:4100/", host: "18.121.121.23:3000"
For me service name or ip address not worked, only work put the gateway IP of network, for default bridge is 172.17.0.1.
In the servers put the (gateway ip):(port of container) and with this haproxy connects with success.
My example of custom network with fixed ips and gateway:
---- nginx config
upstream loadbalancer {
server 172.17.0.1:8001 weight=5;
server 172.17.0.1:8002 weight=5;
}
----- haproxy config similar
backend be_pe_8545
mode http
balance roundrobin
server p1 172.20.0.254:18545 check inter 10s
server p2 172.20.0.254:28545 check inter 10s
----- docker app / network
docker_app: ...
networks:
public_network:
ipv4_address: 172.20.0.50
public_network:
name: public_network
driver: bridge
ipam:
driver: default
config:
- subnet: 172.20.0.0/24
gateway: 172.20.0.254

Nginx Reverse Proxy cannot communicate with container

I've been trying to teach myself Nginx. Naturally I figured I should use docker. I'm trying to do this with docker for windows. Would eventually move to Linux server. I feel like I'm so close, but I'm stuck on this last issue.
reverseproxy_1 | 2021/07/14 22:37:31 [error] 31#31: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://172.18.0.2:5000/favicon.ico", host: "localhost:4000", referrer: "http://localhost:4000/"
Anyone have any suggestions? I'm new to this, so it's probably something stupid. I've gone through several tutorials and I really feel like this should work.
version: '3.7'
services:
web:
image: 'anatomy-lab2'
container_name: 'AnatomyLabWeb'
ports:
- "5000:80"
restart: always
reverseproxy:
image: nginx:alpine
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
ports:
- '4000:4000'
depends_on:
- web
restart: always
user nginx;
events {
worker_connections 1000;
}
http {
upstream web-api {
server web:5000;
}
server {
listen 4000;
location / {
proxy_pass http://web-api;
}
}
}
λ docker-compose up
Starting AnatomyLabWeb ... done
Starting anatomy-lab_reverseproxy_1 ... done
Attaching to AnatomyLabWeb, anatomy-lab_reverseproxy_1
reverseproxy_1 | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
reverseproxy_1 | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
reverseproxy_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
reverseproxy_1 | 10-listen-on-ipv6-by-default.sh: info: IPv6 listen already enabled
reverseproxy_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
reverseproxy_1 | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
reverseproxy_1 | /docker-entrypoint.sh: Configuration complete; ready for start up
AnatomyLabWeb | [04:56:26 WRN] Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed
.
AnatomyLabWeb | [04:56:26 INF] User profile is available. Using '/root/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.
AnatomyLabWeb | Hosting environment: Production
AnatomyLabWeb | Content root path: /app
AnatomyLabWeb | Now listening on: http://[::]:80
AnatomyLabWeb | Application started. Press Ctrl+C to shut down.
reverseproxy_1 | 2021/07/15 04:56:33 [error] 23#23: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://172.18.
0.2:5000/", host: "localhost:4000"
reverseproxy_1 | 172.18.0.1 - - [15/Jul/2021:04:56:33 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
reverseproxy_1 | 2021/07/15 04:56:33 [error] 23#23: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "htt
p://172.18.0.2:5000/favicon.ico", host: "localhost:4000", referrer: "http://localhost:4000/"
reverseproxy_1 | 172.18.0.1 - - [15/Jul/2021:04:56:33 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://localhost:4000/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome
/91.0.4472.124 Safari/537.36"
I get the web app to work just fine by itself (asp.net/kestrel). But I can't seem to hook it to Nginx.
Any thoughts on this would be great. I've been stuck for quite a bit of time.
The problem came from
upstream web-api {
server web:5000;
}
In the dockerized environment the web container listens :80 so you need to change the config like
upstream web-api {
server web:80;
}

Reverse proxying containerized angular app

We have an angular_app that upon running docker-compose up is accessible from
under localhost.
Now, we have another dockerized Nginx app that reverses proxies several apps. /app_A /app_B /agular_app. Now A and B work already but the angular_app gives 502.
A snippet from Nginx
location / {
proxy_pass http://agular_app:5001;
}
A snippet from the docker file
agular_app:
image: docker_repo/agular_app:latest
ports:
- 5001:80
The angular_app is accessible from localhost:5001
but not accessible from localhost (gives 502)
I have tried the following things:
Yes, changing 5001:5001, the dame 502
Different Nginx config such as adding proxy_set_header etc
Tried not using redirects over proxy_pass
For me... The missing puzzle piece is... Why reverse proxy does not work. Again, the dockerized angular image running manually (not containerized) on Nginx works fine.
Here is the docker file for angular_app
# Stage 1
FROM node:12-alpine3.12 as build-step
RUN mkdir -p /app
WORKDIR /app
COPY package.json /app
RUN npm install
COPY . /app
RUN npm run build --prod
# Stage 2
FROM nginx:1.14.2-alpine
COPY --from=build-step /app/frontend/dist /usr/share/nginx/html
COPY ./nginx_docker.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
Error from Nginx log upon calling localhost:
gateway | 2021/05/27 06:52:07 [error] 28#28: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: localhost, request: "GET / HTTP/1.1", upstream: "http://172.18.0.2:5
001/", host: "localhost"
gateway | 2021/05/27 06:52:07 [error] 28#28: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: localhost, request: "GET / HTTP/1.1", upstream: "http://172.18.0.2:5
001/502.html", host: "localhost"
gateway | 172.18.0.1 - - [27/May/2021:06:52:07 +0000] "GET / HTTP/1.1" 502 552 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
gateway | 2021/05/27 06:52:07 [error] 28#28: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", upstream: "http://1
72.18.0.2:5001/favicon.ico", host: "localhost", referrer: "http://localhost/"
gateway | 2021/05/27 06:52:07 [error] 28#28: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", upstream: "http://1
72.18.0.2:5001/502.html", host: "localhost", referrer: "http://localhost/"

I got problem while using Nginx to direct requests to services defined in docker-compose.yml

I'm setting up an app with multiple containers, and use nginx to redirect requests to correct container. However, I got stuck with the 502 Bad Gateway error.
Actually, the code is from a course on Udemy: Docker and Kubernetes.
I just copy and paste the code, it ran on instructor machine, but not mine. I tried on my windows and my macbook, restart docker, but still no hope. I looked for solutions on other stackoverflow posts, some other articles, but none of them tell me why it works on others' machines, but not mine.
Here is the repo of the code.
docker-compose.yml (full code):
version: "3"
services:
postgres:
...
redis:
...
nginx:
restart: always
build:
dockerfile: Dockerfile.dev
context: ./nginx
ports:
- "3050:80"
api:
...
client:
...
worker:
...
nginx/Dockerfile.dev
FROM nginx
COPY ./default.conf /etc/nginx/conf.d/default.conf
nginx/default.conf
upstream client {
server client:3000;
}
upstream api {
server api:5000;
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://client;
}
location /api {
rewrite /api/(.*) /$1 break;
proxy_pass http://api;
}
}
It runs just fine on instructor's machine and other learners', but not on my machines. I got error when connecting http://localhost:3050 and http://localhost:3050/api:
nginx_1 | 2019/07/08 02:52:35 [error] 6#6: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.25.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://125.235.4.59:3000/", host: "localhost:3050"
nginx_1 | 172.25.0.1 - - [08/Jul/2019:02:52:35 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-"
nginx_1 | 2019/07/08 02:52:57 [error] 6#6: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.25.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://125.235.4.59:3000/favicon.ico", host: "localhost:3050", referrer: "http://localhost:3050/"
nginx_1 | 172.25.0.1 - - [08/Jul/2019:02:52:57 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://localhost:3050/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-"
Any help is appreciated.
Needed to amend the docker-compose.yml:
the upstream services should expose their ports so that the nginx service can connect i.e.:
api:
expose:
- '5000'
client:
expose:
- '3000'
the nginx service depends_on the upstream services:
nginx:
depends_on:
- 'client'
- 'api'

Roundcube & Dovecot inside Docker Containers

I have a Docker stack for my mail server.
My docker-compose.xml contains
version: '3.7'
services:
postfix:
...
dovecot:
....
ports:
- "110:110"
- "995:995"
- "143:143"
- "993:993"
networks:
- mail
....
roundcube:
image: roundcube/roundcubemail
container_name: roundcube
environment:
- ROUNDCUBEMAIL_DEFAULT_HOST=dovecot
# - ROUNDCUBEMAIL_DEFAULT_PORT=993
networks:
- proxy
- mail
I also have a Nginx container running as a proxy for all my web applications. For roundcube I have
set $roundcube_upstream http://roundcube;
location /roundcube/ {
rewrite ^/roundcube/(.*) /$1 break;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_pass $roundcube_upstream;
}
With that configuration it's working. I can go to https://www.mydomain.be/rouncube/ and I can login. The default port is 143. So roundcube si connecting to dovecot with imap.
Now, I'd like to use port 993 and ssl/tls.
I tried decommenting the ROUNDCUBEMAIL_DEFAULT_PORT=993, but also using ssl://dovecot or tls://dovecot or ssl://mail.mydomain.be, ... but nothing is working.
When I click on the connextion button, after a while I receive an nginx error page. In my proxy logs I can see
2019/01/31 09:29:25 [error] 460#460: *82483 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 194.197.210.75, server: www.mydomain.be, request: "POST /roundcube/?_task=login HTTP/1.1", upstream: "http://172.18.0.9:80/?_task=login", host: "www.mydomain.be", referrer: "https://www.mydomain.be/roundcube/"
And I don't understand where the http://172.18.0.9:80/?_task=login is coming from ?
With Thunderbird client I can connect on that port.
What's the problem ?
Edit
Using
- ROUNDCUBEMAIL_DEFAULT_HOST=ssl://dovecot
- ROUNDCUBEMAIL_DEFAULT_PORT=993
I now have a response : connection error to the storage server.
In my roundcube logs :
errors: <1db522a3> IMAP Error: Login failed for me#mydomain.be from 172.18.0.8(X-Real-IP: ...,X-Forwarded-For: ...). Could not connect to ssl://dovecot:993: Unknown reason in /var/www/html/program/lib/Roundcube/rcube_imap.php on line 196 (POST /?_task=login&_action=login)172.18.0.8 - - [31/Jan/2019:13:57:37 +0100] "POST /?_task=login HTTP/1.1" 200 3089 "https://www.mydomain.be/roundcube/?_task=login" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0"
and in dovecot logs
2019-01-31T13:57:38.002653+01:00 536ff3507263 dovecot: auth: Debug: auth client connected (pid=35),
2019-01-31T13:57:38.010096+01:00 536ff3507263 dovecot: imap-login: Disconnected (no auth attempts in 0 secs): user=<>, rip=192.168.240.3, lip=192.168.240.2, TLS, session=<nVssksCAT7LAqPAD>
So dovecot is well contacted but ... ? Don't know whats the problem.
Your issue is that roundcube requires TLS or SSL certificates to be verified by default. Either copy the certificate from the mail server, use letsencrypt to validate your certificate or turn off peer verification in your roundcube configuration.

Resources