Service/pod starts up without any issues but cannot connect to service - docker

Question

So, I have a service (Python Flask/uwsgi app) which I deploy via helm on Kubernetes which locally runs on my Docker Desktop. I have successfully deployed the application, the pods are running fine, no issues with application, app logs also look good. But when I hit the url in the browser/via curl, I get a connection refused error.
Here's what my kubectl describe pod looks like:
Name: rhs-servicebase-6f676c458c-s6b8j
Namespace: default
Priority: 0
Node: docker-desktop/192.168.65.4
Start Time: Mon, 29 Aug 2022 07:54:30 -0700
Labels: app.kubernetes.io/instance=rhs
app.kubernetes.io/name=servicebase
pod-template-hash=6f676c458c
Annotations: <none>
Status: Running
IP: 10.1.0.44
IPs:
IP: 10.1.0.44
Controlled By: ReplicaSet/rhs-servicebase-6f676c458c
Containers:
servicebase:
Container ID: docker://7e7c3dca5a465deeba3312e54b789233a5c652b92dd9017cb2e422bd202130f6
Image: localhost:5000/referral-health-signal:latest
Image ID: docker-pullable://localhost:5000/referral-health-signal#sha256:72af6911f60def6af987d8b5f611ab8553f5771278b3821e911f437f93c73743
Port: 9000/TCP
Host Port: 0/TCP
State: Running
Started: Mon, 29 Aug 2022 07:54:36 -0700
Ready: True
Restart Count: 0
Environment:
TZ: UTC
LOG_FILE: application.log
S3_BUCKET: livongo-int-healthsignal
AWS_ACCESS_KEY_ID: <set to the key 'aws-access-key-id' in secret 'referral-health-signal'> Optional: false
AWS_SECRET_ACCESS_KEY: <set to the key 'aws-secret-access-key' in secret 'referral-health-signal'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5vtmd (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
secret-referral-health-signal:
Type: Secret (a volume populated by a Secret)
SecretName: referral-health-signal
Optional: false
kube-api-access-5vtmd:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 15m default-scheduler Successfully assigned default/rhs-servicebase-6f676c458c-s6b8j to docker-desktop
Normal Pulling 15m kubelet Pulling image "localhost:5000/referral-health-signal:latest"
Normal Pulled 15m kubelet Successfully pulled image "localhost:5000/referral-health-signal:latest" in 347.31179ms
Normal Created 15m kubelet Created container servicebase
Normal Started 15m kubelet Started container servicebase
My deployment status:
$ kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
rhs-servicebase 1/1 1 1 24m
My service status:
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d22h
rhs-servicebase NodePort 10.97.229.137 <none> 9000:30001/TCP 29m
But when I do nc -zv localhost 30001 I see:
nc: connectx to localhost port 30001 (tcp) failed: Connection refused
nc: connectx to localhost port 30001 (tcp) failed: Connection refused
An lsof -i tcp:30001 also produces an empty result.
Pod spec:
apiVersion: v1
kind: Pod
metadata:
creationTimestamp: "2022-08-29T14:54:30Z"
generateName: rhs-servicebase-6f676c458c-
labels:
app.kubernetes.io/instance: rhs
app.kubernetes.io/name: servicebase
pod-template-hash: 6f676c458c
name: rhs-servicebase-6f676c458c-s6b8j
namespace: default
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: ReplicaSet
name: rhs-servicebase-6f676c458c
uid: 0d076b86-44e8-41fa-b6ee-b73a066f20e0
resourceVersion: "23554"
uid: 6d251ab1-c240-4241-9041-7c03b4d0d9c8
spec:
containers:
- env:
- name: TZ
value: UTC
- name: LOG_FILE
value: application.log
- name: S3_BUCKET
value: livongo-int-healthsignal
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: aws-access-key-id
name: referral-health-signal
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: aws-secret-access-key
name: referral-health-signal
image: localhost:5000/referral-health-signal:latest
imagePullPolicy: Always
name: servicebase
ports:
- containerPort: 9000
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-5vtmd
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
nodeName: docker-desktop
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: default
serviceAccountName: default
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: secret-referral-health-signal
secret:
defaultMode: 420
items:
- key: aws-access-key-id
path: AWS_ACCESS_KEY_ID
- key: aws-secret-access-key
path: AWS_SECRET_ACCESS_KEY
secretName: referral-health-signal
- name: kube-api-access-5vtmd
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2022-08-29T14:54:30Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2022-08-29T14:54:36Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2022-08-29T14:54:36Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2022-08-29T14:54:30Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: docker://7e7c3dca5a465deeba3312e54b789233a5c652b92dd9017cb2e422bd202130f6
image: referral-health-signal:latest
imageID: docker-pullable://localhost:5000/referral-health-signal#sha256:72af6911f60def6af987d8b5f611ab8553f5771278b3821e911f437f93c73743
lastState: {}
name: servicebase
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2022-08-29T14:54:36Z"
hostIP: 192.168.65.4
phase: Running
podIP: 10.1.0.44
podIPs:
- ip: 10.1.0.44
qosClass: BestEffort
startTime: "2022-08-29T14:54:30Z"
Deployment spec:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: "2022-08-29T14:54:30Z"
generation: 1
labels:
app.kubernetes.io/instance: rhs
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: servicebase
helm.sh/chart: servicebase-0.1.5
name: rhs-servicebase
namespace: default
resourceVersion: "23558"
uid: aae0086a-ecfd-4c80-b633-92307e2f059d
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: rhs
app.kubernetes.io/name: servicebase
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app.kubernetes.io/instance: rhs
app.kubernetes.io/name: servicebase
spec:
containers:
- env:
- name: TZ
value: UTC
- name: LOG_FILE
value: application.log
- name: S3_BUCKET
value: livongo-int-healthsignal
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: aws-access-key-id
name: referral-health-signal
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: aws-secret-access-key
name: referral-health-signal
image: localhost:5000/referral-health-signal:latest
imagePullPolicy: Always
name: servicebase
ports:
- containerPort: 9000
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: secret-referral-health-signal
secret:
defaultMode: 420
items:
- key: aws-access-key-id
path: AWS_ACCESS_KEY_ID
- key: aws-secret-access-key
path: AWS_SECRET_ACCESS_KEY
secretName: referral-health-signal
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2022-08-29T14:54:36Z"
lastUpdateTime: "2022-08-29T14:54:36Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2022-08-29T14:54:30Z"
lastUpdateTime: "2022-08-29T14:54:36Z"
message: ReplicaSet "rhs-servicebase-6f676c458c" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
observedGeneration: 1
readyReplicas: 1
replicas: 1
updatedReplicas: 1
I'm so lost, I don't even know where to start debugging. I'm on MacOS (Big Sur).