We have an angular_app that upon running docker-compose up is accessible from
under localhost.
Now, we have another dockerized Nginx app that reverses proxies several apps. /app_A /app_B /agular_app. Now A and B work already but the angular_app gives 502.
A snippet from Nginx
location / {
proxy_pass http://agular_app:5001;
}
A snippet from the docker file
agular_app:
image: docker_repo/agular_app:latest
ports:
- 5001:80
The angular_app is accessible from localhost:5001
but not accessible from localhost (gives 502)
I have tried the following things:
Yes, changing 5001:5001, the dame 502
Different Nginx config such as adding proxy_set_header etc
Tried not using redirects over proxy_pass
For me... The missing puzzle piece is... Why reverse proxy does not work. Again, the dockerized angular image running manually (not containerized) on Nginx works fine.
Here is the docker file for angular_app
# Stage 1
FROM node:12-alpine3.12 as build-step
RUN mkdir -p /app
WORKDIR /app
COPY package.json /app
RUN npm install
COPY . /app
RUN npm run build --prod
# Stage 2
FROM nginx:1.14.2-alpine
COPY --from=build-step /app/frontend/dist /usr/share/nginx/html
COPY ./nginx_docker.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
Error from Nginx log upon calling localhost:
gateway | 2021/05/27 06:52:07 [error] 28#28: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: localhost, request: "GET / HTTP/1.1", upstream: "http://172.18.0.2:5
001/", host: "localhost"
gateway | 2021/05/27 06:52:07 [error] 28#28: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: localhost, request: "GET / HTTP/1.1", upstream: "http://172.18.0.2:5
001/502.html", host: "localhost"
gateway | 172.18.0.1 - - [27/May/2021:06:52:07 +0000] "GET / HTTP/1.1" 502 552 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36" "-"
gateway | 2021/05/27 06:52:07 [error] 28#28: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", upstream: "http://1
72.18.0.2:5001/favicon.ico", host: "localhost", referrer: "http://localhost/"
gateway | 2021/05/27 06:52:07 [error] 28#28: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.18.0.1, server: localhost, request: "GET /favicon.ico HTTP/1.1", upstream: "http://1
72.18.0.2:5001/502.html", host: "localhost", referrer: "http://localhost/"
Environment setup
I have an application which is composed by some services:
jenkins server
nginx server with angular
nginx server as a proxy
Those services are defined in the docker-compose file:
version: '3'
services:
reverse:
container_name: reverse-proxy
build:
context: /app/mywallet/MyWalletFe/reverse-proxy
ports:
- "80:80"
networks:
- net
jenkins:
container_name: jenkins
image: jenkins/jenkins
volumes:
- "$PWD/jenkins_home:/var/jenkins_home"
- "/var/run/docker.sock:/var/run/docker.sock"
networks:
- net
angular:
container_name: mywallet_fe
build:
context: /app/mywallet/MyWalletFe
networks:
- net
networks:
net:
I defined the following configuration file for the reverse-proxy:
upstream client {
# angular is the name of the service in docker-compose file
server angular:4200;
}
upstream jenkins {
server jenkins:8080;
}
server {
listen 80;
location / {
proxy_pass http://client;
}
location /jenkins {
proxy_pass http://jenkins;
}
}
Finally, here is the Dockerfile for the reverse-proxy service, which copies the configuration file in the nginx container:
FROM nginx
# override default files if present
COPY ./default.conf /etc/nginx/conf.d/default.conf
Goal
My goal is to access Jenkins with SERVER_IP/jenkins
Output
When I run the whole application and try to access to SERVER_IP/jenkins, I get the following error in the reverse-proxy logs:
2020/04/15 21:44:55 [error] 6#6: *10 connect() failed (111: Connection
refused) while connecting to upstream, client: MY_CLIENT_IP, server: ,
request: "GET /login?from=%2Fjenkins HTTP/1.1", upstream:
"http://172.18.0.5:4200/login?from=%2Fjenkins", host: "SERVER_IP",
referrer: "http://SERVER_IP/jenkins" MY_CLIENT_IP
[15/Apr/2020:21:44:55 +0000] "GET /favicon.ico HTTP/1.1" 502 559
"http://SERVER_IP/login?from=%2Fjenkins" "Mozilla/5.0 (Windows NT
10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36" "-" 2020/04/15 21:44:55 [error]
6#6: *10 connect() failed (111: Connection refused) while connecting
to upstream, client: MY_CLIENT_IP, server: , request: "GET
/favicon.ico HTTP/1.1", upstream:
"http://172.18.0.5:4200/favicon.ico", host: "SERVER_IP",
referrer: "http://SERVER_IP/login?from=%2Fjenkins"
where MY_CLIENT_IP is my laptop IP and SERVER_IP is the IP of the server where the application runs.
What's wrong in the configuration of the reverse proxy? If I expose the jenkins and angular services I can reach them, while through the proxy I can't.
Similar question, which doesn't help me (or I don't understand how would those help me)
Docker nginx reverse proxy returns 502 bad gateway "connection refused while connecting to upstream"
Connection refused while connection to upstream - Docker
connect() failed (111: Connection refused) while connecting to upstream for nginx+php-fpm docker
docker nginx connection refused while connecting to upstream
I installed NGINX Ingress in the cluster, but when i am trying to access it i am getting 502
These are the steps that i have followed to install NGINX Ingress
kubectl apply -f common/ns-and-sa.yaml
kubectl apply -f common/default-server-secret.yaml
kubectl apply -f common/nginx-config.yaml
kubectl apply -f common/vs-definition.yaml
kubectl apply -f common/vsr-definition.yaml
kubectl apply -f rbac/rbac.yaml
kubectl apply -f daemon-set/nginx-ingress.yaml
This is the Deployment which i am trying to access from Ingress Controller
apiVersion: apps/v1
kind: Deployment
metadata:
name: ngtest
namespace: practice
labels:
app: ngtest
spec:
replicas: 1
selector:
matchLabels:
app: ngtest
template:
metadata:
labels:
app: ngtest
spec:
imagePullSecrets:
- name: testkuldeepsecret
containers:
- name: ngtest
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: ngtest
namespace: practice
labels:
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: ngtest
With the Cluster IP i am able to access the nginx
Now i configured Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: practice-ingress
namespace: practice
spec:
rules:
- host: practice.example.com
http:
paths:
- path: /
backend:
serviceName: ngtest
servicePort: 80
Ingress Description
Name: practice-ingress
Namespace: practice
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
practice.example.com
/ ngtest:80 (10.32.0.7:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"practice-ingress","namespace":"practice"},"spec":{"rules":[{"host":"practice.example.com","http":{"paths":[{"backend":{"serviceName":"ngtest","servicePort":80},"path":"/"}]}}]}}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 46m nginx-ingress-controller Configuration for practice/practice-ingress was added or updated
Normal AddedOrUpdated 46m nginx-ingress-controller Configuration for practice/practice-ingress was added or updated
Its clear that ingress will forward it to Port 80 and ngtest service also exposed on port 80
I added IP address and practice.example.com in my host file.
So when i try to access http://practice.example.com i am getting 502
Ingress NGINX Pod Logs
10.32.0.1 - - [14/Mar/2020:07:37:53 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:37:56 [error] 30#30: *13 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
2020/03/14 07:37:56 [error] 30#30: *15 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:37:56 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
10.32.0.1 - - [14/Mar/2020:07:37:56 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:37:59 [error] 30#30: *13 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:07:37:59 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:03 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:43:03 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:06 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:07:43:06 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:09 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:43:09 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:12 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:07:43:12 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:52:07 [error] 30#30: *25 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /practice HTTP/1.1", upstream: "http://10.32.0.7:80/practice", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:52:07 +0000] "GET /practice HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:52:11 [error] 30#30: *25 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/practice"
10.32.0.1 - - [14/Mar/2020:07:52:11 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/practice" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:54:48 [error] 30#30: *29 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/practice"
10.32.0.1 - - [14/Mar/2020:07:54:48 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/practice" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:54:58 [error] 30#30: *29 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/practice"
10.32.0.1 - - [14/Mar/2020:07:54:58 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/practice" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 08:13:36 [error] 30#30: *32 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:08:13:36 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 08:13:39 [error] 30#30: *32 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:08:13:39 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
Resources that were running
root#ip-172-31-12-0:/home/ubuntu# kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-6955765f44-7fxkh 1/1 Running 0 6d13h
kube-system pod/coredns-6955765f44-l7dkf 1/1 Running 0 6d13h
kube-system pod/etcd-ip-172-31-12-0 1/1 Running 0 6d13h
kube-system pod/kube-apiserver-ip-172-31-12-0 1/1 Running 0 3h2m
kube-system pod/kube-controller-manager-ip-172-31-12-0 1/1 Running 1 6d13h
kube-system pod/kube-proxy-2h672 1/1 Running 1 6d13h
kube-system pod/kube-proxy-7zc4j 1/1 Running 0 6d13h
kube-system pod/kube-scheduler-ip-172-31-12-0 1/1 Running 1 6d13h
kube-system pod/weave-net-wrrml 2/2 Running 0 6d13h
kube-system pod/weave-net-wxjv4 2/2 Running 4 6d13h
nginx-ingress pod/nginx-ingress-9hsck 1/1 Running 0 58m
nginx-ingress pod/nginx-ingress-xvhfg 1/1 Running 0 58m
practice pod/customer-855c6d9bc8-8pq8v 1/1 Running 0 14h
practice pod/ngtest-6c5ff8c7df-2867b 1/1 Running 0 46m
practice pod/some-mysql-7896cfbb4c-zlpl8 1/1 Running 0 38h
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d13h
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 6d13h
practice service/customer-service ClusterIP 10.110.222.12 <none> 80/TCP 14h
practice service/ngtest ClusterIP 10.105.115.222 <none> 80/TCP 46m
practice service/some-mysql ClusterIP None <none> 3306/TCP 38h
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/kube-proxy 2 2 2 2 2 beta.kubernetes.io/os=linux 6d13h
kube-system daemonset.apps/weave-net 2 2 2 2 2 <none> 6d13h
nginx-ingress daemonset.apps/nginx-ingress 2 2 2 2 2 <none> 58m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 2/2 2 2 6d13h
practice deployment.apps/customer 1/1 1 1 14h
practice deployment.apps/ngtest 1/1 1 1 46m
practice deployment.apps/some-mysql 1/1 1 1 38h
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-6955765f44 2 2 2 6d13h
practice replicaset.apps/customer-855c6d9bc8 1 1 1 14h
practice replicaset.apps/ngtest-6c5ff8c7df 1 1 1 46m
practice replicaset.apps/some-mysql-7896cfbb4c 1 1 1 38h
No route to host means the nginx ingress controller was not able to connect to pod IP. Check if you have any firewall between the node where nginx ingress controller is running and the node where the pod is running.
Did try using https://github.com/kubernetes/ingress-nginx? Most of the time we use wrong ingress.
I'm setting up an app with multiple containers, and use nginx to redirect requests to correct container. However, I got stuck with the 502 Bad Gateway error.
Actually, the code is from a course on Udemy: Docker and Kubernetes.
I just copy and paste the code, it ran on instructor machine, but not mine. I tried on my windows and my macbook, restart docker, but still no hope. I looked for solutions on other stackoverflow posts, some other articles, but none of them tell me why it works on others' machines, but not mine.
Here is the repo of the code.
docker-compose.yml (full code):
version: "3"
services:
postgres:
...
redis:
...
nginx:
restart: always
build:
dockerfile: Dockerfile.dev
context: ./nginx
ports:
- "3050:80"
api:
...
client:
...
worker:
...
nginx/Dockerfile.dev
FROM nginx
COPY ./default.conf /etc/nginx/conf.d/default.conf
nginx/default.conf
upstream client {
server client:3000;
}
upstream api {
server api:5000;
}
server {
listen 80;
server_name localhost;
location / {
proxy_pass http://client;
}
location /api {
rewrite /api/(.*) /$1 break;
proxy_pass http://api;
}
}
It runs just fine on instructor's machine and other learners', but not on my machines. I got error when connecting http://localhost:3050 and http://localhost:3050/api:
nginx_1 | 2019/07/08 02:52:35 [error] 6#6: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.25.0.1, server: , request: "GET / HTTP/1.1", upstream: "http://125.235.4.59:3000/", host: "localhost:3050"
nginx_1 | 172.25.0.1 - - [08/Jul/2019:02:52:35 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-"
nginx_1 | 2019/07/08 02:52:57 [error] 6#6: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 172.25.0.1, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://125.235.4.59:3000/favicon.ico", host: "localhost:3050", referrer: "http://localhost:3050/"
nginx_1 | 172.25.0.1 - - [08/Jul/2019:02:52:57 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://localhost:3050/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36" "-"
Any help is appreciated.
Needed to amend the docker-compose.yml:
the upstream services should expose their ports so that the nginx service can connect i.e.:
api:
expose:
- '5000'
client:
expose:
- '3000'
the nginx service depends_on the upstream services:
nginx:
depends_on:
- 'client'
- 'api'
Im running docker with nginx, PHP and MariaDb in my company.
But if my laptop is offline or Im connected with vpn the nginx responds with an 403 error. Docker is starting normally. Do you have any idea why I get the 403 from nginx?
Maybe Its because of the mounted drives?
2018/06/05 13:23:20 [error] 5#5: *3 directory index of "/var/www/html/" is
forbidden, client: 172.21.0.1, server: _, request: "GET / HTTP/1.1", host:
"localhost"
172.21.0.1 - - [05/Jun/2018:13:23:20 +0000] "GET / HTTP/1.1" 403 572 "-"
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/66.0.3359.139 Safari/537.36" "-"
Edit: I figured out that I cant mount the shared drives. I got an error that the firewall blocks it.