Cannot Connect Container on Overlay Network: [emerg] invalid host - docker

I am trying to reverse proxy a request through an nginx container in a swarm to a standalone container which shares the same overlay network.
tldr; I receive the following error:
2018/03/15 19:00:35 [emerg] 1#1: invalid host in upstream "http://nginx" in /etc/nginx/nginx.conf:96
nginx: [emerg] invalid host in upstream "http://nginx" in /etc/nginx/nginx.conf:96
The standalone container contains an app which has another nginx frontend:
version: "3"
services:
nginx:
restart: always
container_name: my.nginx
build: ./nginx
networks:
- default
- my-overlay-network
depends_on:
- another-service
... other services
networks:
my-overlay-network:
external: true
I start this app with docker-compose up -d.
My swarm contains the reverse proxy:
version: "3"
services:
reverseproxy:
build: ./reverseproxy
image: reverse_proxy
networks:
- my-overlay-network
ports:
- "80:80"
- "443:443"
volumes:
- /etc/letsencrypt:/etc/letsencrypt
deploy:
replicas: 10
restart_policy:
condition: on-failure
networks:
my-overlay-network:
external: true
If I startup the nginx swarm without specifying a proxy_pass for the standalone app, I can successfully ping the other host like so:
ping http://nginx/
I can confirm the other host receives this request based on the nginx logs.
However, if I specify the docker standalone app in the reverse proxy:
upstream standalone {
server http://nginx/;
}
and
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
... other stuff ...
location / {
proxy_pass http://standalone/;
}
}
I get the following errors:
2018/03/15 19:00:35 [emerg] 1#1: invalid host in upstream "http://nginx" in /etc/nginx/nginx.conf:96
nginx: [emerg] invalid host in upstream "http://nginx" in /etc/nginx/nginx.conf:96

Try to add resolver before upstream:
resolver 127.0.0.11;
upstream standalone {
server http://nginx/;
}
127.0.0.11 is an address of embedded docker DNS server

The issue is that the syntax for upstream is incorrect: it takes a host and port, so it should be:
upstream standalone {
server nginx;
}

Related

502 Bad Gateway when using reverse proxy with Docker and Nginx

I have searched StackOverflow for my problem but I always seem to be hitting the 502 Bad Gateway with my Nginx Docker configuration. I am trying to access pgadmin4 using my domain mydomain.com/pgadmin instead of mydomain.com:8060 where 8060 is the port exposed by it's docker container. My docker-compose.yml file looks like this:
version: '3.5'
services:
reverse-proxy:
image: nginx:1.19.6
restart: always
ports:
- "80:80"
- "443:443"
postgres:
image: postgres:12
ports:
- "5432:5432"
pgadmin:
image: dpage/pgadmin4
depends_on:
- postgres
ports:
- "8060:80"
networks:
default:
external:
name: defaultnetwork
The default.conf file of my nginx container looks like this:
upstream pgadmin {
server 127.0.0.1:8060;
}
server {
listen 80;
listen [::]:80;
server_name mydomain.com;
root /usr/share/nginx/html;
index index.html index.htm;
location /pgadmin {
proxy_pass http://pgadmin;
}
}
With this configuration, I keep getting the 502 Bad Gateway error. Could someone kindly point to me where I am going wrong. I would really appreciate it.
Thanks.
[EDIT]
This is from the docker logs:
2021/02/03 08:07:42 [error] 23#23: *2 connect() failed (111: Connection refused) while connecting to upstream, client: ***.***.***.***, server: mydomain.com, request: "GET /pgadmin HTTP/1.1", upstream: "http://127.0.0.1:8082/pgadmin", host: "mydomain.com"
The 502 problem comes from the loopback IP here:
upstream pgadmin {
server 127.0.0.1:8060;
}
127.0.0.1 or localhost for the NGINX container is the NGINX container itself. You should use the name of the service instead:
upstream pgadmin {
server pgadmin:8060;
}
Name of the service comes from the docker-compose.yml:
services:
pgadmin: # <- this
image: dpage/pgadmin4
If you hit 404 after these changes, this is because you have to change base path of the application. Try using this config:
location /pgadmin/ {
proxy_set_header X-Script-Name /pgadmin;
proxy_set_header Host $host;
proxy_pass http://pgadmin;
proxy_redirect off;
}
Since your containers are working in the same network, you should access the Pgadmin container via 80th port from your Nginx container.
You should replace this line server 127.0.0.1:8060 with server pgadmin:80 in your Nginx config.

Nginx can't find upstream host in multi-container docker compose setup and also host can't be found on client:3000

Here is github repo: https://github.com/irahulsah/mutlicontainerapp
please,visit it for more info,please help me to fix this error.
I'm running a multi-docker container locally with docker-compose, the containers are React front-end 'client', a Nodejs app 'api', and a Nginx proxy in sits in front of two. I have been using the docker-compose setup as follow for a while.
version: '3'
services:
client:
build:
dockerfile: Dockerfile.dev
context: ./client
volumes:
- /usr/app/node_modules
- ./client:/usr/app
api:
build:
dockerfile: Dockerfile.dev
context: ./server
volumes:
- /usr/app/node_modules
- ./server:/usr/app
nginx:
restart: always
build:
dockerfile: Dockerfile.dev
context: ./nginx
ports:
- '8080:80'
and my Nginx setup is as follows
upstream client {
server client:3000;
}
upstream api {
server api:5000;
}
server {
listen 80;
server_name _;
location / {
if ($http_x_forwarded_proto != 'https') {
return 301 https://$host$request_uri;
}
proxy_pass http://client;
}
location /api {
if ($http_x_forwarded_proto != 'https') {
return 301 https://$host$request_uri;
}
rewrite /api/(.*) /$1 break;
proxy_pass http://api;
}
}
Recently when I tried to start up the containers, i got following error:
nginx_1 | 2019/08/08 18:11:12 [emerg] 1#1: host not found in upstream "client:3000" in /etc/nginx/conf.d/default.conf:2
nginx_1 | nginx: [emerg] host not found in upstream "client:3000" in /etc/nginx/conf.d/default.conf:2
Any idea why nginx not able to find upstream?
I have tried to add links to nginx setup blocks as follows:
nginx:
restart: always
build:
dockerfile: Dockerfile.dev
context: ./nginx
links:
- client:client
- api:api
ports:
- '8080:80'
I also tried 'depends_on' ,but also am getting host can be found on client:300 error,any idea on how to fix this error,will be deeply appreciated.
Any help or direction are greatly appreciated!!!
I had this exactly same issue today. I solved it by attaching all containers that nginx upstream is referring to in the same docker virtual network.
Also, make sure to explicitly define the containers name. If I am not wrong, docker-compose prefix your services name with the yaml file name.
service_one:
networks:
- app-network # define the network
container_name: backend # the "backend" name must be used in the upstream directive in nginx configuration

docker nginx proxy to linked container not working

I have tried this:
NGINX reverse proxy not working to other docker container
and this:
Docker nginx-proxy : proxy between containers
and followed nginx config from here:
nginx proxy_pass to a linked docker container
I am simply trying to tell nginx to proxy to a linked api service on port 4000. I do not want to expose 4000 to host machine because there will be multiple services running on this port.
This is my docker-compose.yml:
version: '3'
services:
api:
build: ./api
image: myapi:latest
container_nameE: api
api_nginx:
image: nginx:latest
container_name: api_nginx
depends_on:
- api
links:
- api
ports:
- "80:80"
environment:
- NGINX_SERVER_NAME:localhost
volumes:
- ./nginx:/etc/nginx/conf.d
...
...
and my nginx server is super minimal:
upstream backend {
server api;
}
server {
listen 80;
listen [::]:80;
server_name ${NGINX_SERVEE_NAME};
location / {
resolver 127.0.0.1;
proxy_pass http://backend/$1;
}
}
This is the error is throwing:
...[error] 20#20: *1 no resolver defined to resolve api, client: 172.23.0.1, server: ${nginx_server_name}....
and the page shows a 502 Bad Gateway
What is going on? I've followed other people's nginx configs and it's not working, I have no idea.

How to connect from nginx on docker host to application in container

The situation is: we have service with nginx, which also acts as docker server.
Also we have Java application in docker container, which listens at 8080.
The problem is permissions to connect from nginx to container's published port.
Nginx.error.log shows:
2017/11/23 13:44:12 [crit] 3599#0: *1 connect() to 127.0.0.1:8080 failed (13: Permission denied) while connecting to upstream,
2017/11/23 13:44:13 [error] 3599#0: *1 no live upstreams while connecting to upstream
Site config is:
server {
listen 80 default_server;
server_name _;
location / {
proxy_pass http://localhost:8080;
include proxy.conf;
}
}
Container config:
version: '3'
services:
app:
image: user/appX
restart: always
container_name: appX
ports:
- "8080:8080"
env_file:
- ./appX.env
extra_hosts:
- "host:172.101.0.1"
networks:
mynet:
ipv4_address: 172.101.0.2
networks:
mynet:
external:
name: mynet
Permission happens because nginx user is nginx and docker user is root.
How to fix this problem without moving nginx to container ?
Or maybe there is some work around exists ?
Problem was in SELinux configuration.
Solved by running:
setsebool -P httpd_can_network_connect 1

Connection refused while connecting to upstream when using nginx as reverse proxy

The setup is as follows: I have a Gunicorn/Django app running on 0.0.0.0:8000 that is accessible via the browser. To serve static files I am running nginx as a reverse proxy. /etc/nginx/nginx.conf is configured to forward requests as follows:
server {
location /static/ {
alias /data/www/;
}
# Proxying the connections
location / {
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_pass http://0.0.0.0:8000;
}
}
and my docker-compose.yml file is as follows:
version: '3.3'
services:
web:
restart: always
build: ./web
expose:
- "8000"
ports:
- "8000:8000"
volumes:
- staticdata:/usr/src/app/static_files
command: gunicorn wsgi:application --workers 2 --bind 0.0.0.0:8000
depends_on:
- postgres
nginx:
restart: always
build: ./nginx
ports:
- "80:80"
- "443:443"
volumes:
- staticdata:/data/www
depends_on:
- web
postgres:
image: postgres:9.2
restart: always
volumes:
- pgdata:/var/lib/postgresql/data
ports:
- "5432:5432"
volumes:
staticdata:
pgdata:
When I visit 0.0.0.0:8000 via the browser the application works fine (albeit without serving static files), but when I visit 127.0.0.1:80 I get the following error:
nginx_1 | 2017/09/17 13:59:46 [error] 6#6: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://0.0.0.0:8000/", host: "127.0.0.1"
I know that this error indicates that the server running on 0.0.0.0:8000 is not accepting requests, but since I can visit it via the browser I am a bit confused.
Thank you in advance.
Change your proxy_pass from
proxy_pass http://0.0.0.0:8000;
to
proxy_pass http://web:8000;
Your nginx needs to forward to request the web container
Edit-1: Explanation
0.0.0.0 is a special IP address which is used to refer to any available interface on the machine. So if your machine has a loopback (lo), ethernet (eth0), Wifi (wlan0) with respective IP as 127.0.0.1, 192.168.0.100, 10.0.0.100.
So now while listening for incoming connection you can choose any of the above IP
gunicorn wsgi:application --workers 2 --bind 10.0.0.100:8000
This will only be reachable from your Wifi network. But other machines on Lan network can't visit it. So if you want your app to listen on any available network on the machine you use a special IP 0.0.0.0. This means bind on all network interfaces
gunicorn wsgi:application --workers 2 --bind 0.0.0.0:8000
Now when you access the app using http://0.0.0.0 it is equivalent to using 127.0.0.1. So your proxy_pass http://0.0.0.0:8000; is equivalent to proxy_pass http://127.0.0.1:8000;
So when you run that in nginx container, it passes on the request on port 8000 of the same container and there is nothing running on 8000 in your nginx container. So you need to send that request to the your gunicorn container. This is reachable using the service name web in docker-compose.
See the below article for more details https://docs.docker.com/compose/networking/

Resources