nginx reverse proxy upstream fails in docker-compose with connection refused message - docker

I have a docker-compose.yaml similar to this (shortened for simplicity):
# ...
services:
my-ui:
# ...
ports:
- 5402:8080
networks:
- my-net
networks:
my-net:
external:
name: my-net
and I'm trying to set up nginx as a reverse proxy with this configuration:
upstream client {
server my-ui:5402;
}
server {
listen 80;
location / {
proxy_pass http://client;
}
}
and this is the docker-compose.yaml I have for nginx:
# ...
services:
reverse-proxy:
# ...
ports:
- 5500:80
networks:
- my-net
networks:
my-net:
external:
name: my-net
What happens now is that when I run my-ui and reverse-proxy (each using its own docker-compose up), and I go to http://localhost:5500, I get a Bad Gateway message, and my nginx logs says this:
connect() failed (111: Connection refused) while connecting to
upstream, client: 172.19.0.1, server: , request: "GET / HTTP/1.1",
upstream: "http://172.19.0.5:5402/", host: "localhost:5500"
If I exec into my nginx container and use ping:
ping my-ui
ping 172.19.0.5
Both are successful, but if I want to, for example, curl:
curl -L http://my-ui
curl -L http://my-ui:5402
curl -L http://172.19.0.1
All of them fail with connection refused message. What am I missing here?
PS: I'm not sure, but it might be useful to add that my-ui is a basic vuejs application, running on Webpack dev server.
PS2: I also tried passing host headers etc. but same result

The name of the container (my-ui) resolves to the IP of the container. Therefor you have to provide in upstream the port of the container and not the port you have mapped to the host.
upstream client {
server my-ui:8080;
}
server {
listen 80;
location / {
proxy_pass http://client;
}
}
You could also configure your upstream with the name of your host machine and use the mapped port. (server <name of host>:5402) But this could get quite messy and you would lose the advantage of isolating services with docker networks.
Furthermore you could also remove the port mapping unless you need to access the webservice without reverse proxy:
# ...
services:
reverse-proxy:
# ...
# ports:
# - 5500:80

Related

docker nginx load balancer connect() failed (111: Connection refused) while connecting to upstream

I'm trying to load balance a simple Nodejs app with 3 instances using docker-compose & nginx. This configuration works on my local machine (windows laptop) but doesn't seem to work on EC2 server.
nginx.conf
http {
upstream all {
server nodeapp1:4100;
server nodeapp2:4200;
server nodeapp3:4300;
}
server {
listen 8080;
location / {
proxy_pass http://all/;
}
}
}
events { }
docker-compose.yml
version: '3'
services:
lb:
image: nginx
volumes:
- ./nginxproxy/nginx.conf:/etc/nginx/nginx.conf
ports:
- "3000:8080"
nodeapp1:
image: nodeapp
environment:
- PORT=4100
ports:
- "4100:4100"
nodeapp2:
image: nodeapp
environment:
- PORT=4200
ports:
- "4200:4200"
nodeapp3:
image: nodeapp
environment:
- PORT=4300
ports:
- "4300:4300"
I'm new to docker. I'm surprised why this works locally but does not work on EC2 instance. The load balancer was able to resolve the url correctly but it still says connection refused.
Error:
2022/02/28 20:00:22 [error] 33#33: *9 connect() failed (111: Connection refused) while
connecting to upstream, client: 62.113.237.40, server: , request: "GET / HTTP/1.1",
upstream: "http://172.121.0.5:4100/", host: "18.121.121.23:3000"
For me service name or ip address not worked, only work put the gateway IP of network, for default bridge is 172.17.0.1.
In the servers put the (gateway ip):(port of container) and with this haproxy connects with success.
My example of custom network with fixed ips and gateway:
---- nginx config
upstream loadbalancer {
server 172.17.0.1:8001 weight=5;
server 172.17.0.1:8002 weight=5;
}
----- haproxy config similar
backend be_pe_8545
mode http
balance roundrobin
server p1 172.20.0.254:18545 check inter 10s
server p2 172.20.0.254:28545 check inter 10s
----- docker app / network
docker_app: ...
networks:
public_network:
ipv4_address: 172.20.0.50
public_network:
name: public_network
driver: bridge
ipam:
driver: default
config:
- subnet: 172.20.0.0/24
gateway: 172.20.0.254

How to reverse proxy a docker image at domain root and another image at a subdomain with swag?

I'm using "linuxserver"'s swag image to reverse-proxy my docker-compose images. I want to serve one of my images as my domain root and the other one as a subdomain (e.g. root-image # mysite.com and subdomain-image # staging.mysite.com). Here are the steps I went through:
redirected my domain and subdomain names to my server in Cloudflare (pinging them shows my server's IP and this is step OK! )
Configured Cloudflare DNS config for swag (this working OK!)
Configured docker-compose file:
swag:
image: linuxserver/swag:version-1.14.0
container_name: swag
cap_add:
- NET_ADMIN
environment:
- PUID=1000
- PGID=1000
- URL=mysite.com
- SUBDOMAINS=www,staging
- VALIDATION=dns
- DNSPLUGIN=cloudflare #optional
- EMAIL=me#mail.com #optional
- ONLY_SUBDOMAINS=false #optional
- STAGING=false #optional
volumes:
- /docker-confs/swag/config:/config
ports:
- 443:443
- 80:80 #optional
restart: unless-stopped
root-image:
image: ghcr.io/me/root-image
container_name: root-image
restart: unless-stopped
subdomain-image:
image: ghcr.io/me/subdomain-image
container_name: subdomain-image
restart: unless-stopped
Defined a proxy conf for my root-image (at swag/config/nginx/proxy-confs/root-image.subfolder.conf)
location / {
include /config/nginx/proxy.conf;
resolver 127.0.0.11 valid=30s;
set $upstream_app root-image;
set $upstream_port 3000;
set $upstream_proto http;
proxy_pass $upstream_proto://$upstream_app:$upstream_port;
}
Commented out the nginx's default location / {} block (at swag/config/nginx/site-confs/default)
Defined a proxy conf for my subdomain-image (at swag/config/nginx/proxy-confs/subdomain-image.subdomain.conf)
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name staging.*;
include /config/nginx/ssl.conf;
client_max_body_size 0;
location / {
include /config/nginx/proxy.conf;
resolver 127.0.0.11 valid=30s;
set $upstream_app subdomain-image;
set $upstream_port 443;
set $upstream_proto https;
proxy_pass $upstream_proto://$upstream_app:$upstream_port;
}
}
Both images expose port 3000. Now my root image is working fine (OK!), but I'm getting 502 error for my subdomain image. I checked the nginx error log, but it doesn't show anything meaningful for me:
*35 connect() failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxxx.xxx, server: staging.*, request: "GET /favicon.ico HTTP/2.0", upstream: "https://xxx.xxx.xxx:443/favicon.ico", host: "staging.mysite.com", referrer: "https://staging.mysite.com"
Docker logs for all the 3 containers are also fine (without showing any warnings or errors).
Which step I'm going wrong or is there anything I missed? Thanks for helping

Docker: nginx reverse proxy and how to configure multiple services (i.e on 8080 &8081)

I am a bit stuck with configuring multiple services where nginx is the proxy server.
running :
docker -v
Docker version 19.03.8, build afacb8b7f0
docker-compose -v
docker-compose version 1.23.2, build 1110ad01
I want to start with this test, everything in the same docker-compose.yml-file :
link to jwilder/nginx
proxy : nginx-server (jwilder/nginx-proxy:0.7.0 , which is nginx 1.17.6)
container1 : httpd:2.4
container2 : httpd:2.4
updating my /etc/hosts before I start
127.0.0.1 container1.com
127.0.0.1 container2.com
Here is my docker-compose.yml-file (obs -> version 3.7)
version: '3.7'
services:
proxy:
image: jwilder/nginx-proxy:0.7.0
container_name: proxy-test
ports:
- "80:80"
- "443:443"
volumes:
- /var/run/docker.sock:/tmp/docker.sock:ro
- ./nginx-proxy.conf:/etc/nginx/conf.d/nginx-proxy.conf:ro
container1:
image: httpd:2.4
container_name: container-1
environment:
- VIRTUAL_HOST:container1.com
ports:
- 8080:80
container2:
image: httpd:2.4
container_name: container-2
environment:
- VIRTUAL_HOST:container2.com
ports:
- 8081:80
here is my nginx-proxy.conf:
server {
listen 80;
server_name container1.com;
location / {
proxy_pass http://localhost:8080;
}
}
server {
listen 80;
server_name container2.com;
location / {
proxy_pass http://localhost:8081;
}
}
After this I run the
docker exec container-1 sed -i 's/It works!/Container 1/' /usr/local/apache2/htdocs/index.html AND docker exec container-2 sed -i 's/It works!/Container 2/' /usr/local/apache2/htdocs/index.html
Test 1 : with curl to the port 8080 and port 8081
curl localhost:8080
response -> Container 1
curl localhost:8081
response -> Container 2
Test 2 : with curl to container1.com AND container2.com
curl container1.com
status 502
curl container2.com
status 502
Are the settings in my conf wrong ?
Troubleshooting 1:
docker exec -it proxy-test bash
I can see that the nginx-proxy.conf is in the directory (/etc/nginx/conf.d)
/etc/nginx/conf.d/default.conf is there as well
Troubleshooting 2: The proxy-log (Connection refused - while connecting to upstream)
proxy-test | nginx.1 | 2020/04/03 10:52:08 [error] 61#61: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 172.29.0.1, server: container1.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "container1.com"
proxy-test | nginx.1 | 2020/04/03 10:52:08 [error] 61#61: *9 connect() failed (111: Connection refused) while connecting to upstream, client: 172.29.0.1, server: container1.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "container1.com"
Found 2 solutions to this.
(1)
the first is to update the nginx-proxy.conf with the name of the containers instead of pointing to http://localhost:8080; and http://localhost:8081; :
new config-file
server {
listen 80;
server_name container1.com;
location / {
proxy_pass http://container-1;
}
}
server {
listen 80;
server_name container2.com;
location / {
proxy_pass http://container-2;
}
}
(2)
Leaving out the nginx-proxy.conf-file , docker-compose.yml will map things correctly.

Docker: "Cannot assign requested address while connecting to upstream"

I have docker running on a machine with the IP address fd42:1337::31. One container is a nginx reverse proxy with the port mapping 443:443, in its configuration file it proxy_pass-es depending on the server name to other ports on the same machine, e.g.
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name plex.mydomain.tld;
location / {
proxy_pass http://[fd42:1337::31]:32400;
}
}
server {
listen 443 ssl;
listen [::]:443 ssl;
server_name file.mydomain.tld;
location / {
proxy_pass http://[fd42:1337::31]:2020;
}
}
These other ports refer to bottle py servers or other containers with mapped ports.
I've started this container with the command
docker run -d -p 443:443 (volume mappings) --name reverseproxy nginx
and it has served me well for a year.
I've now decided to work with docker-compose and have the following configuration file:
version: '3'
services:
reverseproxy:
image: "nginx"
ports:
- "443:443"
volumes:
(volume mappings)
When I shut down the original container and start my new one with docker-compose up, it starts, but every request gives me something like this:
2019/02/13 17:04:43 [crit] 6#6: *1 connect() to [fd42:1337::31]:32400 failed (99: Cannot assign requested address) while connecting to upstream, client: 192.168.178.126, server: plex.mydomain.tld, request: "GET / HTTP/1.1", upstream: "http://[fd42:1337::31]:32400/", host: "plex.mydomain.tld"
Why is the new container behaving differently? What do I have to change?
(I know I can have a virtual network mode to connect to other containers directly, but my proxy is supposed to connect to some services that are not inside containers (but on the same metal).)

docker-compose --scale X nginx.conf configuration

My nginx.conf file currently has the routes defined directly:
worker_processes auto;
events { worker_connections 1024; }
http {
upstream wordSearcherApi {
least_conn;
server api1:61370 max_fails=3 fail_timeout=30s;
server api2:61370 max_fails=3 fail_timeout=30s;
server api3:61370 max_fails=3 fail_timeout=30s;
}
server {
listen 80;
server_name server_name 0.0.0.0;
location / {
proxy_pass http://wordSearcherApi;
}
}
}
Is there any way to create just one service in docker-compose.yml and when docker-compose up --scale api=3, does nginx do automatic load balance?
Nginx
Dynamic upstreams are possible in Nginx (normal, sans Plus) but with tricks and limitations.
You give up on upstream directive and use plain proxy_pass.
It gives round robin load balancing and failover, but no extra feature of the directive like weights, failure modes, timeout, etc.
Your upstream hostname must be passed to proxy_pass by a variable and you must provide a resolver.
It forces Nginx to re-resolve the hostname (against Docker networks' DNS).
You lose location/proxy_pass behaviour related to trailing slash.
In the case of reverse-proxying to bare / like in the question, it does not matter. Otherwise you have to manually rewrite the path (see the references below).
Let's see how it works.
docker-compose.yml
version: '2.2'
services:
reverse-proxy:
image: nginx:1.15-alpine
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf
ports:
- 8080:8080
app:
# A container that exposes an API to show its IP address
image: containous/whoami
scale: 4
nginx.conf
worker_processes 1;
events {
worker_connections 1024;
}
http {
access_log /dev/stdout;
error_log /dev/stderr;
server {
listen 8080;
server_name localhost;
resolver 127.0.0.11 valid=5s;
set $upstream app;
location / {
proxy_pass http://$upstream:80;
}
}
}
Then...
docker-compose up -d
seq 10 | xargs -I -- curl -s localhost:8080 | grep "IP: 172"
...produces something like the following which indicates the requests are distributed across 4 app containers:
IP: 172.30.0.2
IP: 172.30.0.2
IP: 172.30.0.3
IP: 172.30.0.3
IP: 172.30.0.6
IP: 172.30.0.5
IP: 172.30.0.3
IP: 172.30.0.6
IP: 172.30.0.5
IP: 172.30.0.5
References:
Nginx with dynamic upstreams
Using Containers to Learn Nginx Reverse Proxy
Dynamic Nginx configuration for Docker with Python
Traefik
Traefik relies on Docker API directly and may be a simpler and more configurable option. Let's see it in action.
docker-compose.yml
version: '2.2'
services:
reverse-proxy:
image: traefik
# Enables the web UI and tells Traefik to listen to docker
command: --api --docker
ports:
- 8080:80
- 8081:8080 # Traefik's web UI, enabled by --api
volumes:
# So that Traefik can listen to the Docker events
- /var/run/docker.sock:/var/run/docker.sock
app:
image: containous/whoami
scale: 4
labels:
- "traefik.frontend.rule=Host:localhost"
Then...
docker-compose up -d
seq 10 | xargs -I -- curl -s localhost:8080 | grep "IP: 172"
...also produces something the output that indicates the requests are distributed across 4 app containers:
IP: 172.31.0.2
IP: 172.31.0.5
IP: 172.31.0.6
IP: 172.31.0.4
IP: 172.31.0.2
IP: 172.31.0.5
IP: 172.31.0.6
IP: 172.31.0.4
IP: 172.31.0.2
IP: 172.31.0.5
In the Traefik UI (http://localhost:8081/dashboard/ in the example) you can see it recognised the 4 app containers:
References:
The Traefik Quickstart (Using Docker)
It's not possible with your current config since it's static. You have two options -
1. Use docker engine swarm mode - You can define replicas & swarm internal DNS will automatically balance the load across those replicas.
Ref - https://docs.docker.com/engine/swarm/
2. Use famous Jwilder nginx proxy - This image listens to the docker sockets, uses templates in GO to dynamically change your nginx configs when you scale your containers up or down.
Ref - https://github.com/jwilder/nginx-proxy

Resources