502 Bad Gateway issue with some containers - docker

Question

I installed NGINX Ingress in the cluster, but when i am trying to access it i am getting 502
These are the steps that i have followed to install NGINX Ingress
kubectl apply -f common/ns-and-sa.yaml
kubectl apply -f common/default-server-secret.yaml
kubectl apply -f common/nginx-config.yaml
kubectl apply -f common/vs-definition.yaml
kubectl apply -f common/vsr-definition.yaml
kubectl apply -f rbac/rbac.yaml
kubectl apply -f daemon-set/nginx-ingress.yaml
This is the Deployment which i am trying to access from Ingress Controller
apiVersion: apps/v1
kind: Deployment
metadata:
name: ngtest
namespace: practice
labels:
app: ngtest
spec:
replicas: 1
selector:
matchLabels:
app: ngtest
template:
metadata:
labels:
app: ngtest
spec:
imagePullSecrets:
- name: testkuldeepsecret
containers:
- name: ngtest
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: ngtest
namespace: practice
labels:
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: ngtest
With the Cluster IP i am able to access the nginx
Now i configured Ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: practice-ingress
namespace: practice
spec:
rules:
- host: practice.example.com
http:
paths:
- path: /
backend:
serviceName: ngtest
servicePort: 80
Ingress Description
Name: practice-ingress
Namespace: practice
Address:
Default backend: default-http-backend:80 (<none>)
Rules:
Host Path Backends
---- ---- --------
practice.example.com
/ ngtest:80 (10.32.0.7:80)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"practice-ingress","namespace":"practice"},"spec":{"rules":[{"host":"practice.example.com","http":{"paths":[{"backend":{"serviceName":"ngtest","servicePort":80},"path":"/"}]}}]}}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 46m nginx-ingress-controller Configuration for practice/practice-ingress was added or updated
Normal AddedOrUpdated 46m nginx-ingress-controller Configuration for practice/practice-ingress was added or updated
Its clear that ingress will forward it to Port 80 and ngtest service also exposed on port 80
I added IP address and practice.example.com in my host file.
So when i try to access http://practice.example.com i am getting 502
Ingress NGINX Pod Logs
10.32.0.1 - - [14/Mar/2020:07:37:53 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:37:56 [error] 30#30: *13 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
2020/03/14 07:37:56 [error] 30#30: *15 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:37:56 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
10.32.0.1 - - [14/Mar/2020:07:37:56 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:37:59 [error] 30#30: *13 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:07:37:59 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:03 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:43:03 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:06 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:07:43:06 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:09 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:43:09 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:43:12 [error] 30#30: *19 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:07:43:12 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:52:07 [error] 30#30: *25 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /practice HTTP/1.1", upstream: "http://10.32.0.7:80/practice", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:07:52:07 +0000] "GET /practice HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:52:11 [error] 30#30: *25 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/practice"
10.32.0.1 - - [14/Mar/2020:07:52:11 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/practice" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:54:48 [error] 30#30: *29 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/practice"
10.32.0.1 - - [14/Mar/2020:07:54:48 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/practice" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 07:54:58 [error] 30#30: *29 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/practice"
10.32.0.1 - - [14/Mar/2020:07:54:58 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/practice" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 08:13:36 [error] 30#30: *32 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET / HTTP/1.1", upstream: "http://10.32.0.7:80/", host: "practice.example.com"
10.32.0.1 - - [14/Mar/2020:08:13:36 +0000] "GET / HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
2020/03/14 08:13:39 [error] 30#30: *32 connect() failed (113: No route to host) while connecting to upstream, client: 10.32.0.1, server: practice.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://10.32.0.7:80/favicon.ico", host: "practice.example.com", referrer: "http://practice.example.com/"
10.32.0.1 - - [14/Mar/2020:08:13:39 +0000] "GET /favicon.ico HTTP/1.1" 502 559 "http://practice.example.com/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" "-"
Resources that were running
root#ip-172-31-12-0:/home/ubuntu# kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-6955765f44-7fxkh 1/1 Running 0 6d13h
kube-system pod/coredns-6955765f44-l7dkf 1/1 Running 0 6d13h
kube-system pod/etcd-ip-172-31-12-0 1/1 Running 0 6d13h
kube-system pod/kube-apiserver-ip-172-31-12-0 1/1 Running 0 3h2m
kube-system pod/kube-controller-manager-ip-172-31-12-0 1/1 Running 1 6d13h
kube-system pod/kube-proxy-2h672 1/1 Running 1 6d13h
kube-system pod/kube-proxy-7zc4j 1/1 Running 0 6d13h
kube-system pod/kube-scheduler-ip-172-31-12-0 1/1 Running 1 6d13h
kube-system pod/weave-net-wrrml 2/2 Running 0 6d13h
kube-system pod/weave-net-wxjv4 2/2 Running 4 6d13h
nginx-ingress pod/nginx-ingress-9hsck 1/1 Running 0 58m
nginx-ingress pod/nginx-ingress-xvhfg 1/1 Running 0 58m
practice pod/customer-855c6d9bc8-8pq8v 1/1 Running 0 14h
practice pod/ngtest-6c5ff8c7df-2867b 1/1 Running 0 46m
practice pod/some-mysql-7896cfbb4c-zlpl8 1/1 Running 0 38h
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 6d13h
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 6d13h
practice service/customer-service ClusterIP 10.110.222.12 <none> 80/TCP 14h
practice service/ngtest ClusterIP 10.105.115.222 <none> 80/TCP 46m
practice service/some-mysql ClusterIP None <none> 3306/TCP 38h
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/kube-proxy 2 2 2 2 2 beta.kubernetes.io/os=linux 6d13h
kube-system daemonset.apps/weave-net 2 2 2 2 2 <none> 6d13h
nginx-ingress daemonset.apps/nginx-ingress 2 2 2 2 2 <none> 58m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/coredns 2/2 2 2 6d13h
practice deployment.apps/customer 1/1 1 1 14h
practice deployment.apps/ngtest 1/1 1 1 46m
practice deployment.apps/some-mysql 1/1 1 1 38h
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/coredns-6955765f44 2 2 2 6d13h
practice replicaset.apps/customer-855c6d9bc8 1 1 1 14h
practice replicaset.apps/ngtest-6c5ff8c7df 1 1 1 46m
practice replicaset.apps/some-mysql-7896cfbb4c 1 1 1 38h

Answer 1

No route to host means the nginx ingress controller was not able to connect to pod IP. Check if you have any firewall between the node where nginx ingress controller is running and the node where the pod is running.

Answer 2

Did try using https://github.com/kubernetes/ingress-nginx? Most of the time we use wrong ingress.