Helm kubernetes on AKS Pod CrashLoopBackOff - docker

I'm trying to deploy a simple nodejs application via helm kubernetes on Azure Kubernetes Service, but after pulling my image it says CrashLoopBackOff.
Here's what I have tried so far:
My Dockerfile:
FROM node:6
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 32000
CMD [ "npm", "start" ]
My server.js:
'use strict';
const express = require('express');
const PORT = 32000;
const HOST = '0.0.0.0';
const app = express();
app.get('/', (req, res) => {
res.send('Hello world from container.\n');
});
app.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);
I have pushed this image to ACR.
New Update: Here's the complete output of kubectl describe pod POD_NAME:
Name: myrel02-mychart06-5dc9d4b86c-kqg4n
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: aks-nodepool1-19665249-0/10.240.0.6
Start Time: Tue, 05 Feb 2019 11:31:27 +0500
Labels: app.kubernetes.io/instance=myrel02
app.kubernetes.io/name=mychart06
pod-template-hash=5dc9d4b86c
Annotations: <none>
Status: Running
IP: 10.244.2.5
Controlled By: ReplicaSet/myrel02-mychart06-5dc9d4b86c
Containers:
mychart06:
Container ID: docker://c239a2b9c38974098bbb1646a272504edd2d199afa50f61d02a0ce335fe60660
Image: registry-1.docker.io/arycloud/docker-web-app:0.5
Image ID: docker-pullable://registry-1.docker.io/arycloud/docker-web-app#sha256:4faab280d161b727e0a6a6d9dfb52b22cf9c6cd7dd07916d6fe164d9af5737a7
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 05 Feb 2019 11:39:56 +0500
Finished: Tue, 05 Feb 2019 11:40:22 +0500
Ready: False
Restart Count: 7
Liveness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: http-get http://:http/ delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
KUBERNETES_PORT_443_TCP_ADDR: cluster06-ary-2a187a-dc393b82.hcp.centralus.azmk8s.io
KUBERNETES_PORT: tcp://cluster06-ary-2a187a-dc393b82.hcp.centralus.azmk8s.io:443
KUBERNETES_PORT_443_TCP: tcp://cluster06-ary-2a187a-dc393b82.hcp.centralus.azmk8s.io:443
KUBERNETES_SERVICE_HOST: cluster06-ary-2a187a-dc393b82.hcp.centralus.azmk8s.io
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-gm49w (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-gm49w:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-gm49w
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned default/myrel02-mychart06-5dc9d4b86c-kqg4n to aks-nodepool1-19665249-0
Normal Pulling 10m kubelet, aks-nodepool1-19665249-0 pulling image "registry-1.docker.io/arycloud/docker-web-app:0.5"
Normal Pulled 10m kubelet, aks-nodepool1-19665249-0 Successfully pulled image "registry-1.docker.io/arycloud/docker-web-app:0.5"
Warning Unhealthy 9m30s (x6 over 10m) kubelet, aks-nodepool1-19665249-0 Liveness probe failed: Get http://10.244.2.5:80/: dial tcp 10.244.2.5:80: connect: connection refused
Normal Created 9m29s (x3 over 10m) kubelet, aks-nodepool1-19665249-0 Created container
Normal Started 9m29s (x3 over 10m) kubelet, aks-nodepool1-19665249-0 Started container
Normal Killing 9m29s (x2 over 9m59s) kubelet, aks-nodepool1-19665249-0 Killing container with id docker://mychart06:Container failed liveness probe.. Container will be killed and recreated.
Warning Unhealthy 9m23s (x7 over 10m) kubelet, aks-nodepool1-19665249-0 Readiness probe failed: Get http://10.244.2.5:80/: dial tcp 10.244.2.5:80: connect: connection refused
Normal Pulled 5m29s (x6 over 9m59s) kubelet, aks-nodepool1-19665249-0 Container image "registry-1.docker.io/arycloud/docker-web-app:0.5" already present on machine
Warning BackOff 22s (x33 over 7m59s) kubelet, aks-nodepool1-19665249-0 Back-off restarting failed container
Update: docker logs CONTAINER_ID output:
> nodejs#1.0.0 start /usr/src/app
> node server.js
Running on http://0.0.0.0:32000
How can I avoid this issue?
Thanks in advance!

As I can see from kubectl describe pod command output, your Container inside the Pod has been already completed with exit code 0 (#4c74356b41 mentioned about it in the comments). Reason: Completed, which states about successful completion without any errors/problems. However the life cycle of the Pod was very short, therefore Kubernetes continuously schedules new Pods but Liveness and Readiness probes are still failing on the container healthiness.
To keep the Pod running, you must specify a task (process) inside to the container for the ability of continuously running. There are lots of discussions and solutions available on how to solve that kind of issue, more hints can be found here.

kubectl logs command only works if the pod is up and running. If they are not, you can use the kubectl events command. It will give you some log of events and sometimes (in my experience) will give you also clues on what is going on.
kubectl get events -n <your_app_namespace> --sort-by='.metadata.creationTimestamp'
By default it does not sort the events, hence the --sort-by flag.

Related

CrashLoopBackOff while deploying pod using image from private registry

I am trying to create a pod using my own docker image on localhost.
This is the dockerfile used to create the image :
FROM centos:8
RUN yum install -y gdb
RUN yum group install -y "Development Tools"
CMD ["/usr/bin/bash"]
The yaml file used to create the pod is this :
---
apiVersion: v1
kind: Pod
metadata:
name: server
labels:
app: server
spec:
containers:
- name: server
imagePullPolicy: Never
image: localhost:5000/server
ports:
- containerPort: 80
root#node1:~/test/server# docker images | grep server
server latest 82c5228a553d 3 hours ago 948MB
localhost.localdomain:5000/server latest 82c5228a553d 3 hours ago 948MB
localhost:5000/server latest 82c5228a553d 3 hours ago 948MB
The image has been pushed to localhost registry.
Following is the error I receive.
root#node1:~/test/server# kubectl get pods
NAME READY STATUS RESTARTS AGE
server 0/1 CrashLoopBackOff 5 5m18s
The output of describe pod :
root#node1:~/test/server# kubectl describe pod server
Name: server
Namespace: default
Priority: 0
Node: node1/10.0.2.15
Start Time: Mon, 07 Dec 2020 15:35:49 +0530
Labels: app=server
Annotations: cni.projectcalico.org/podIP: 10.233.90.192/32
cni.projectcalico.org/podIPs: 10.233.90.192/32
Status: Running
IP: 10.233.90.192
IPs:
IP: 10.233.90.192
Containers:
server:
Container ID: docker://c2982e677bf37ff11272f9ea3f68565e0120fb8ccfb1595393794746ee29b821
Image: localhost:5000/server
Image ID: docker-pullable://localhost.localdomain:5000/server#sha256:6bc8193296d46e1e6fa4cb849fa83cb49e5accc8b0c89a14d95928982ec9d8e9
Port: 80/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 07 Dec 2020 15:41:33 +0530
Finished: Mon, 07 Dec 2020 15:41:33 +0530
Ready: False
Restart Count: 6
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-tb7wb (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-tb7wb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-tb7wb
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m default-scheduler Successfully assigned default/server to node1
Normal Pulled 4m34s (x5 over 5m59s) kubelet Container image "localhost:5000/server" already present on machine
Normal Created 4m34s (x5 over 5m59s) kubelet Created container server
Normal Started 4m34s (x5 over 5m59s) kubelet Started container server
Warning BackOff 56s (x25 over 5m58s) kubelet Back-off restarting failed container
I get no logs :
root#node1:~/test/server# kubectl logs -f server
root#node1:~/test/server#
I am unable to figure out whether the issue is with the container or yaml file for creating pod. Any help would be appreciated.
Posting this as Community Wiki.
As pointed by #David Maze in comment section.
If docker run exits immediately, a Kubernetes Pod will always go into CrashLoopBackOff state. Your Dockerfile needs to COPY in or otherwise install and application and set its CMD to run it.
Root cause can be also determined by Exit Code. In 3) Check the exit code article, you can find a few exit codes like 0, 1, 128, 137 with description.
3.1) Exit Code 0
This exit code implies that the specified container command completed ‘sucessfully’, but too often for Kubernetes to accept as working.
In short story, your container was created, all action mentioned was executed and as there was nothing else to do, it exit with Exit Code 0.
A CrashLoopBackOff error occurs when a pod startup fails repeatedly in Kubernetes.`
Your image based on centos with few additional installations did not have any process in backgroud left, so it was categorized as Completed. As this happen so fast, kubernetes restarted it and it fall in loop.
$ kubectl run centos --image=centos
$ kubectl get po -w
NAME READY STATUS RESTARTS AGE
centos 0/1 CrashLoopBackOff 1 5s
centos 0/1 Completed 2 17s
centos 0/1 CrashLoopBackOff 2 31s
centos 0/1 Completed 3 46s
centos 0/1 CrashLoopBackOff 3 58s
centos 1/1 Running 4 88s
centos 0/1 Completed 4 89s
centos 0/1 CrashLoopBackOff 4 102s
$ kubectl describe po centos | grep 'Exit Code'
Exit Code: 0
But when you have used sleep 3600, in your container, command sleep was executing for hour. After this time it would also exit with Exit Code 0.
Hope it clarified.

Kind kubernetes cluster failed to pull docker images

I tried to use KinD as an alternative of Minikube to bootstrap a K8S cluster in my local machine.
The cluster is created successfully.
But when I tried to create some pods/deployments from images, it failed.
$ kubectl run nginx --image=nginx
$ kubectl run hello --image=hello-world
After some minutes, use get pods to get a failed status.
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hello 0/1 ImagePullBackOff 0 11m
nginx 0/1 ImagePullBackOff 0 22m
I am afraid this is another Global Firewall problem in China.
kubectl describe pods/nginx
Name: nginx
Namespace: default
Priority: 0
Node: dev-control-plane/172.19.0.2
Start Time: Sun, 30 Aug 2020 19:46:06 +0800
Labels: run=nginx
Annotations: <none>
Status: Pending
IP: 10.244.0.5
IPs:
IP: 10.244.0.5
Containers:
nginx:
Container ID:
Image: nginx
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ErrImagePull
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mgq96 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-mgq96:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mgq96
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 56m default-scheduler Successfully assigned default/nginx to dev-control-plane
Normal BackOff 40m kubelet, dev-control-plane Back-off pulling image "nginx"
Warning Failed 40m kubelet, dev-control-plane Error: ImagePullBackOff
Warning Failed 13m (x3 over 40m) kubelet, dev-control-plane Failed to pull image "nginx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:latest": failed to copy: unexpected EOF
Warning Failed 13m (x3 over 40m) kubelet, dev-control-plane Error: ErrImagePull
Normal Pulling 13m (x4 over 56m) kubelet, dev-control-plane Pulling image "nginx"
When I entered to the kindest/node container, but there is no docker in it. Not sure how KIND works, originally I understand it deploys a K8S cluster into a Docker container.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a644f8b61314 kindest/node:v1.19.0 "/usr/local/bin/entr…" About an hour ago Up About an hour 127.0.0.1:52301->6443/tcp dev-control-plane
$ docker exec -it a644f8b61314 /bin/bash
root#dev-control-plane:/# docker -v
bash: docker: command not found
After reading the Kind docs, I can not find an option to set a repository mirror there like that in Minikube.
BTW, I am using the latest Docker Desktop beta on a Windows 10.
First pull the image in your local system using docker pull nginx and then use below command to load that image to the kind cluster
kind load docker-image nginx --name kind-cluster-name
Kind uses containerd instead of docker as runtime, that's why docker is not installed on the nodes.
Alternatively you can use crictl tool to pull and check images inside the kind node.
crictl pull nginx
crictl images
I've run into same issue because I've exported http_proxy and https_proxy before creating cluster to a local proxy (127.0.0.1), which is unrechable in the cluster. After unset http(s)_proxy and recreate cluster, everything runs fine.

Kubectl create deployment fails to pull image from local docker repo connection refused

I am encountering a very basic error:
I have docker-desktop and minikube setup on my windows 10 machine.
Further I setup a local docker registry using the steps here.
Here is what I have when I run docker-ps :
c:\>docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6675d2c57a74 registry:2 "/entrypoint.sh /etc…" 7 hours ago Up 2 hours 0.0.0.0:5000->5000/tcp registry
d05edc8f05b0 gcr.io/k8s-minikube/kicbase:v0.0.10 "/usr/local/bin/entr…" 8 hours ago Up 8 hours 127.0.0.1:32771->22/tcp, 127.0.0.1:32770->2376/tcp, 127.0.0.1:32769->5000/tcp, 127.0.0.1:32768->8443/tcp minikube
I pushed my local image to my local registry using docker push, and I deleted the local image using docker image remove command to avoid any confusion.
I now tried pulling the local registry image to see if it works, and it does
docker pull localhost:5000/dev/my-web:v1
v1: Pulling from dev/my-web
Digest: sha256:b3a0cf5c66ade8d39709c0cbbd0e08c9cc5f5e1c97f039a2bd1afed657dc8b74
Status: Downloaded newer image for localhost:5000/dev/my-web:v1
localhost:5000/dev/my-web:v1
Now I run my kubectl create commands and they fail with the error connection refused.
C:\>kubectl create deployment myweb --image=localhost:5000/dev/my-web:v1
deployment.apps/myweb created
C:\>kubectl get pods
NAME READY STATUS RESTARTS AGE
myweb-7d467f4bc4-r9xhc 0/1 ErrImagePull 0 12s
C:\>kubectl describe pod/myweb-7d467f4bc4-r9xhc
Name: myweb-7d467f4bc4-r9xhc
Namespace: default
Priority: 0
Node: minikube/172.17.0.3
Start Time: Sun, 09 Aug 2020 23:30:07 -0400
Labels: app=myweb
pod-template-hash=7d467f4bc4
Annotations: <none>
Status: Pending
IP: 172.18.0.3
IPs:
IP: 172.18.0.3
Controlled By: ReplicaSet/myweb-7d467f4bc4
Containers:
my-web:
Container ID:
Image: localhost:5000/dev/my-web:v1
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ImagePullBackOff
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-nr7vj (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-nr7vj:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-nr7vj
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 93s default-scheduler Successfully assigned default/myweb-7d467f4bc4-r9xhc to minikube
Normal Pulling 53s (x3 over 92s) kubelet, minikube Pulling image "localhost:5000/dev/my-web:v1"
Warning Failed 53s (x3 over 92s) kubelet, minikube Failed to pull image "localhost:5000/dev/my-web:v1": rpc error: code = Unknown desc = Error response from daemon: Get http://localhost:5000/v2/: dial tcp 127.0.0.1:5000: connect: connection refused
Warning Failed 53s (x3 over 92s) kubelet, minikube Error: ErrImagePull
Normal BackOff 13s (x5 over 91s) kubelet, minikube Back-off pulling image "localhost:5000/dev/my-web:v1"
Warning Failed 13s (x5 over 91s) kubelet, minikube Error: ImagePullBackOff
I am not able to figure out why am I getting connection refused error when I am trying through kubectl command as against docker pull command.
Please help.
Additional notes: (1) I am using in-built windows hypervisor (2) Using default networking
Well, Kubernetes cannot find your registry. It depends on the setup you have, but Docker for Windows runs in a VM and you didn't specify how you are running minikube, but most likely in another VM. So potentially here you have two VMs that can or cannot talk to each depending on how you set them up.
And localhost is almost never going to work with Kubernetes because that always resolves to the local IP of your Kubernetes node when it comes to pulling the image. That means that you would have to have your registry and the kubelet pulling the image on the exact same VM.
I would just focus on making it work with the VM IP address where your registry is running and make sure that your Kubernetes VM can reach the registry VM IP address. Also, remember that when you run your registry you also have to expose you container port in this case 5000
P.S. You didn't specify what Hypervisor you are running? VirtualBox? VMware? are you using bridged networking? host only❓
✌️

Docker image works, Kubernetes Pod not working. Ubuntu. Log: /bin/sh: [npm,start]: not found

I'm taking a course that uses Kubernetes and am running into an error when I try to create a pod in Kubernetes.
I'm using Ubuntu, AMD64
I installed microk8s.kubectl following these instructions (https://ubuntu.com/kubernetes/install)
Here's my Dockerfile which runs correctly when I use only Docker.
FROM node:alpine
WORKDIR /app
COPY package.json ./
RUN npm install
COPY ./ ./
CMD ["npm", "start"]
Here's my posts.yaml file, verbatim to the course I'm taking
apiVersion: v1
kind: Pod
metadata:
name: posts
spec:
containers:
- name: posts
image: emendoza1986/blog_posts:0.0.1
output from kubectl get pods
NAME READY STATUS RESTARTS AGE
posts 0/1 CrashLoopBackOff 6 10m
output from kubectl logs posts
/bin/sh: [npm,start]: not found
output from kubectl describe pod posts
Name: posts
Namespace: default
Priority: 0
Node: desktope/192.168.0.18
Start Time: Thu, 23 Jul 2020 10:58:40 -0700
Labels: <none>
Annotations: Status: Running
IP: 10.1.87.20
IPs:
IP: 10.1.87.20
Containers:
posts:
Container ID: containerd://acb403c53759670370959cfa2cc0939f53126aee889e1f6dc2e831bc4dc22c3c
Image: emendoza1986/blog_posts:0.0.1
Image ID: docker.io/emendoza1986/blog_posts#sha256:f69b30cf0382d4c273643ac11c505378854b966063974cc57d187718cc0b0fd5
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 127
Started: Thu, 23 Jul 2020 10:58:59 -0700
Finished: Thu, 23 Jul 2020 10:58:59 -0700
Ready: False
Restart Count: 2
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-2fm2c (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-2fm2c:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-2fm2c
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 48s default-scheduler Successfully assigned default/posts to desktope
Normal Pulled 29s (x3 over 47s) kubelet, desktope Container image "emendoza1986/blog_posts:0.0.1" already present on machine
Normal Created 29s (x3 over 47s) kubelet, desktope Created container posts
Normal Started 29s (x3 over 47s) kubelet, desktope Started container posts
Warning BackOff 12s (x4 over 45s) kubelet, desktope Back-off restarting failed container
output from microk8s.status
microk8s is running
addons:
dashboard: enabled
dns: enabled
metrics-server: enabled
cilium: disabled
fluentd: disabled
gpu: disabled
helm: disabled
helm3: disabled
host-access: disabled
ingress: disabled
istio: disabled
jaeger: disabled
knative: disabled
kubeflow: disabled
linkerd: disabled
metallb: disabled
prometheus: disabled
rbac: disabled
registry: disabled
storage: disabled
output from microk8s inspect
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Service snap.microk8s.daemon-flanneld is running
Service snap.microk8s.daemon-etcd is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Building the report tarball
Report tarball is at /var/snap/microk8s/1503/inspection-report-20200723_112646.tar.gz
I see the error coming from the log but I haven't been able to find a solution. Thank you for your help!
Thank you for the helpful comments. Originally I had my Dockerfile as
CMD ['npm', 'start'].
I had fixed it locally to
CMD ["npm", "start"]
but I didn't push the new version to docker hub.
Pushing the new version fixed the problem.

Pod gets into status of CrashLoopBackOff and gets restarted repeatedly - Exit code is 0

I have a docker container that is running fine when I run it using docker run. I am trying to put that container inside a pod but I am facing issues. The first run of the pod shows status as "Completed". And then the pod keeps restarting with CrashLoopBackoff status. The exit code however is 0.
Here is the result of kubectl describe pod :
Name: messagingclientuiui-6bf95598db-5znfh
Namespace: mgmt
Node: db1mgr0deploy01/172.16.32.68
Start Time: Fri, 03 Aug 2018 09:46:20 -0400
Labels: app=messagingclientuiui
pod-template-hash=2695115486
Annotations: <none>
Status: Running
IP: 10.244.0.7
Controlled By: ReplicaSet/messagingclientuiui-6bf95598db
Containers:
messagingclientuiui:
Container ID: docker://a41db3bcb584582e9eacf26b02c7ef26f57c2d43b813f44e4fd1ba63347d3fc3
Image: 172.32.1.4/messagingclientuiui:667-I20180802-0202
Image ID: docker-pullable://172.32.1.4/messagingclientuiui#sha256:89a002448660e25492bed1956cfb8fff447569e80ac8b7f7e0fa4d44e8abee82
Port: 9087/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Fri, 03 Aug 2018 09:50:06 -0400
Finished: Fri, 03 Aug 2018 09:50:16 -0400
Ready: False
Restart Count: 5
Environment Variables from:
mesg-config ConfigMap Optional: false
Environment: <none>
Mounts:
/docker-mount from messuimount (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-2pthw (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
messuimount:
Type: HostPath (bare host directory volume)
Path: /mon/monitoring-messui/docker-mount
HostPathType:
default-token-2pthw:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-2pthw
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m default-scheduler Successfully assigned messagingclientuiui-6bf95598db-5znfh to db1mgr0deploy01
Normal SuccessfulMountVolume 4m kubelet, db1mgr0deploy01 MountVolume.SetUp succeeded for volume "messuimount"
Normal SuccessfulMountVolume 4m kubelet, db1mgr0deploy01 MountVolume.SetUp succeeded for volume "default-token-2pthw"
Normal Pulled 2m (x5 over 4m) kubelet, db1mgr0deploy01 Container image "172.32.1.4/messagingclientuiui:667-I20180802-0202" already present on machine
Normal Created 2m (x5 over 4m) kubelet, db1mgr0deploy01 Created container
Normal Started 2m (x5 over 4m) kubelet, db1mgr0deploy01 Started container
Warning BackOff 1m (x8 over 4m) kubelet, db1mgr0deploy01 Back-off restarting failed container
kubectl get pods
NAME READY STATUS RESTARTS AGE
messagingclientuiui-6bf95598db-5znfh 0/1 CrashLoopBackOff 9 23m
I am assuming we need a loop to keep the container running in this case. But I dont understand why it worked when it ran using docker and not working when it is inside a pod. Shouldnt it behave the same ?
How do we henerally debug CrashLoopBackOff status apart from running kubectl describe pod and kubectl logs
The container would terminate with exit code 0 if there isn't at least one process running in the background. To keep the container running, add these to the deployment configuration:
command: ["sh"]
stdin: true
Replace sh with bash on any other shell that the image may have.
Then you can drop inside the container with exec:
kubectl exec -it <pod-name> sh
Add -c <container-name> argument if the pod has more than one container.
are you sure you run your software as docker run ... -d ... <command> and it kept running and you use the same exact command in your pod ? In some cases, if you compare things that run on docker with -it and no -d you might find your self in a pinch as they expect terminal to communicate with user and exit if tty is not available (hint: pod/container can be run with tty: true)
It is very unlikely that you have software that runs in a detached docker and does not in kube.

Resources