I'm trying to deploy a Docker image from ECR to my EKS. When attempting to deploy my docker image to a pod, I get the following events from a CrashLoopBackOff:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 62s default-scheduler Successfully assigned default/mlflow-tracking-server to <EC2 IP>.internal
Normal SuccessfulAttachVolume 60s attachdetach-controller AttachVolume.Attach succeeded for volume "<PVC>"
Normal Pulling 56s kubelet, <IP>.ec2.internal Pulling image "<ECR Image UI>"
Normal Pulled 56s kubelet, <IP>.ec2.internal Successfully pulled image "<ECR Image UI>"
Normal Created 7s (x4 over 56s) kubelet, <IP>.ec2.internal Created container mlflow-tracking-server
Normal Pulled 7s (x3 over 54s) kubelet, <IP>.ec2.internal Container image "<ECR Image UI>" already present on machine
Normal Started 6s (x4 over 56s) kubelet, <IP>.ec2.internal Started container mlflow-tracking-server
Warning BackOff 4s (x5 over 52s) kubelet, <IP>.ec2.internal Back-off restarting failed container
I don't understand why it keeps looping like this and failing. Would anyone know why this is happening?
CrashLoopBackError can be related to these possible reasons:
the application inside your pod is not starting due to an error;
the image your pod is based on is not present in the registry, or the
node where your pod has been scheduled cannot pull from the registry;
some parameters of the pod has not been configured correctly.
In your case it seems an application error, inside the container.
Try to view the logs with:
kubectl logs <your_pod> -n <namespace>
For more info on how to troubleshoot this kind of error refer to:
https://pillsfromtheweb.blogspot.com/2020/05/troubleshooting-kubernetes.html
The process inside container is crashing. Could be reason of entrypoint on docker base images.
You can try something like this to check the logs of container
kubectl logs -f <pod_name>
Related
How do you deploy from within Kubernetes container - using CI/CD?
Senario:
I am building within Kubernetes using Kaniko
Now how to run kubectl within Kuberneters. And I do have the right serviceAccount for it. First problem is to have a container ready for executing kubectl.
Note: - /bin/cat
I found this, but it give errors:
apiVersion: v1
kind: Pod
metadata:
name: kubectl-deploy
spec:
containers:
- name: kubectl
image: bitnami/kubectl:latest
imagePullPolicy: Always
command:
- /bin/cat
tty: true
Errors:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 78s default-scheduler Successfully assigned default/kubectl-deploy to master
Normal Pulled 76s kubelet Successfully pulled image "bitnami/kubectl:latest" in 874.059036ms
Normal Pulled 74s kubelet Successfully pulled image "bitnami/kubectl:latest" in 860.59161ms
Normal Pulled 60s kubelet Successfully pulled image "bitnami/kubectl:latest" in 859.31958ms
Normal Pulling 33s (x4 over 77s) kubelet Pulling image "bitnami/kubectl:latest"
Normal Created 32s (x4 over 76s) kubelet Created container kubectl
Normal Started 32s (x4 over 76s) kubelet Started container kubectl
Normal Pulled 32s kubelet Successfully pulled image "bitnami/kubectl:latest" in 849.398179ms
Warning BackOff 7s (x7 over 73s) kubelet Back-off restarting failed container
I found this, but it give errors
When you run a Pod in Kubernetes, by default, it expect it to be a long running service. But in your case, you run a one-off command that terminates immediately. To run one-off commands in Kubernetes, it is easiest to run them as Kubernetes Jobs.
First problem is to have a container ready for executing kubectl.
Since you are using Tekton, have a look at the "deploy task" from Tekton Hub, it is configured with an image that includes kubectl.
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 51s default-scheduler Successfully assigned jenkins/jenkins-0 to minikube
Normal BackOff 31s kubelet, minikube Back-off pulling image "jenkins/jenkins:2.303.3-jdk11"
Warning Failed 31s kubelet, minikube Error: ImagePullBackOff
Normal Pulling 17s (x2 over 47s) kubelet, minikube Pulling image "jenkins/jenkins:2.303.3-jdk11"
Warning Failed 1s (x2 over 32s) kubelet, minikube Failed to pull image "jenkins/jenkins:2.303.3-jdk11": rpc error: code = Unknown desc = Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 1s (x2 over 32s) kubelet, minikube Error: ErrImagePull
The above error is what I am seeing, when trying to install Jenkins on a minikube cluster. I am using this link and following along: https://www.jenkins.io/doc/book/installing/kubernetes/
appreciate any ideas.
I tried with minikube with virtualbox and that worked out of the box.
But, wanted to get docker working, which I wasn't able to.
Finally, I deleted everything (even reinstalled ubuntu) and re-setup k8s cluster with latest k8s version (before i tried with --version=1.19.0 of k8s)
I used: minikube start --driver=docker. and then followed the official jenkins install with helm3, that too latest, lts –
I am trying to deploy Windows Container image on the following software stack
Windows 10 Pro + Docker Desktop + Embedded Kubernetes in docker desktop
Due to some reason 'embedded kubernetes' does not recognize 'local images' no matter whatever --image-pull-policy was set
Docker images
PS C:\WINDOWS\system32> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
myimg final 90c09acbfc59 15 hours ago 5.45GB
Kubectl run
PS C:\WINDOWS\system32> kubectl run --image=myimg:final tskuberun
Pod output
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 25s default-scheduler Successfully assigned default/tskuberun to docker-desktop
Normal BackOff 23s (x2 over 24s) kubelet Back-off pulling image "myimg:final"
Warning Failed 23s (x2 over 24s) kubelet Error: ImagePullBackOff
Normal Pulling 9s (x2 over 25s) kubelet Pulling image "myimg:final"
Warning Failed 8s (x2 over 25s) kubelet Failed to pull image "myimg:final": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io on 192.168.65.5:53: no such host
Warning Failed 8s (x2 over 25s) kubelet Error: ErrImagePull
However, when I execute docker run it pulled the local image. Following worked as expected
PS C:\WINDOWS\system32> docker run myimg:final
I googled for the answer but most of the links were related to Unix flavors and Minikube.
Only few links were related to Docker desktop + embedded kubernetes, but unfortunately none resolved the issue
I am struggling to get rid of this issue. Any help is highly appreciated
EDIT
On further investigation, I observed that 'Docker desktop' refers to local images in case had I selected option "Switch to Linux Containers"
Kubectl run for Linux image
PS C:\WINDOWS\system32> kubectl run --image=wphp --image-pull-policy=IfNotPresent lntest
PS C:\WINDOWS\system32> kubectl describe pod/lntest
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 40s default-scheduler Successfully assigned default/lntest to docker-desktop
Normal Pulled 2s (x4 over 39s) kubelet Container image "wphp" already present on machine
Normal Created 2s (x4 over 39s) kubelet Created container lntest
Normal Started 2s (x4 over 39s) kubelet Started container lntest
It appears that this issue occurs only for 'Windows containers' ie Docker desktop does NOT refer local images had I selected option 'Switch to Windows Containers'
Allthough imagePullPolicy: never should do the trick for you, there could be some certificate related issues.
Personally I avoided using locally built Docker images because of those issues.
You can try to integrate docker push to docker hub in your workflow or build a docker registry in your kubernetes cluster e.g. using https://www.linuxtechi.com/setup-private-docker-registry-kubernetes/
The VM used by Docker Desktop is unable to access the internet. You'll have to sort out that networking.
I am using kubernetes cluster to deploy an image using kubectl create -f dummy.yaml . my image is public in docker hub, the size of the image is 1.3 GB.
the image pull successfully but it is not running it is " CrashLoopBackOff".
when i run creation deployment command "kubectl create -f dummy.yaml" i got:
Name READY STATUS RESTARTS AGE
dummy-ser-5459bf444d-9b7sz 0/1 CrashLoopBackOff 118 10h
I tried to used
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ] in my yaml file, it is work with image size 700 MB but it show CrashLoopBackOff when i use it with other image 1.3 GB, it seems the container after pulling cannot run because the image successful pulled.
The describe pods show :
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12m default-scheduler Successfully assigned dummy-ser-779 7db4cd4-djqdz to node02
Normal SuccessfulMountVolume 12m kubelet, node02 MountVolume.SetUp succeeded for vol ume "default-token-8p9lq"
Normal Created 1m (x4 over 2m) kubelet, node02 Created container
Normal Started 1m (x4 over 2m) kubelet, node02 Started container
Warning BackOff 53s (x8 over 2m) kubelet, node02 Back-off restarting failed containe r
Normal Pulling 41s (x5 over 12m) kubelet, node02 pulling image "xxx/dummyenc:ba ni"
Normal Pulled 40s (x5 over 2m) kubelet, node02 Successfully pulled image "xxx
Thank you in advanced
I fixed this problem. I got this error because the image was not compatible with the hardware that I tried to run on (ARM7)RPi. I create the image on ubuntu 64bit using docker build for Dockerfile so that image cannot run on Raspberry pi.
I am getting below error when i deploy nginxplus container:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1m default-scheduler Successfully assigned nginx-ingress-744f8c7d8-mtx5c to aks-nodepool1-39499429-1
Normal SuccessfulMountVolume 1m kubelet, aks-nodepool1-39499429-1 MountVolume.SetUp succeeded for volume "nginx-ingress-token-lzp54"
Normal Pulling 1m kubelet, aks-nodepool1-39499429-1 pulling image "**.azurecr.io/nginxplus:v2"
Normal Pulled 1m kubelet, aks-nodepool1-39499429-1 Successfully pulled image "**.azurecr.io/nginxplus:v2"
Warning BackOff 56s (x3 over 1m) kubelet, aks-nodepool1-39499429-1 Back-off restarting failed container
Normal Pulled 41s (x3 over 1m) kubelet, aks-nodepool1-39499429-1 Container image "**.azurecr.io/nginxplus:v2" already present on machine
Normal Created 40s (x4 over 1m) kubelet, aks-nodepool1-39499429-1 Created container
Warning Failed 40s (x4 over 1m) kubelet, aks-nodepool1-39499429-1 Error: failed to start container "nginxplus": executable not found in $PATH
Warning FailedSync 28s (x8 over 1m) kubelet, aks-nodepool1-39499429-1 Error syncing pod
I have used nginxplus dockerfile from https://www.nginx.com/blog/deploying-nginx-nginx-plus-docker/
I have used ingress controller file from https://github.com/nginxinc/kubernetes-ingress/blob/master/install/deployment/nginx-plus-ingress.yaml
Pod Logs updated:
This is what i get when i try to retrieve pod logs.
$ kubectl get pods -n nginx-ingress
NAME READY STATUS RESTARTS AGE
nginx-ingress-744f8c7d8-mtx5c 0/1 CrashLoopBackOff 39 2h
$ kubectl logs nginx-ingress-744f8c7d8-mtx5c -n nginx-ingress
failed to open log file "/var/log/pods/e6ed4c91-0fb7-11e8-b1bc-0a58ac1f1223/nginxplus_39.log": open /var/log/pods/e6ed4c91-0fb7-11e8-b1bc-0a58ac1f1223/nginxplus_39.log: no such file or directory
Please help!!
The nginx plus dockerfile you referenced cannot be used for the nginx ingress controller.
You should use the nginx plus ingress controller docker image instead.