After building a docker image named my-http I can create a deployment from it with
kubectl create deploy http-deployment --image=my-http
This will not pull the image because imagePullPolicy is Always.
So then run
kubectl edit deploy http-deployment
and change the imagePullPolicy to Never, then it runs.
But for automation purposes I've created a yaml to create the deployment and set the imagePullPolicy at the same time.
apiVersion: apps/v1
kind: Deployment
metadata:
name: http-deployment
spec:
replicas: 3
selector:
matchLabels:
app: http
template:
metadata:
labels:
app: http
spec:
containers:
- name: my-http
image: my-http
imagePullPolicy: Never
ports:
- containerPort: 8080
Then apply -f and the pods start running but after a while a Crashloopbackoff starts with the message
container image my-http already present on machine
Apparently it has something to do with the container port but what to use for that port to get it running? There is no container running...
edit: the image already present is just informational, this is the last line in the pod description
Warning BackOff 7s (x8 over 91s) kubelet, minikube Back-off
restarting failed container
If you using kubernetes cluster your images only available on the nodes that you build the images.
You have to push images to container registries then the kubernetes will try to pull the image to node that will running the container.
If you want to run the container in the nodes that you build the images you have to use NodeSelector, or PodAffinity.
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
Your image is probably private image which Kubernetes can't pull if you didn't specify imagePullSecrets.
This shouldn't be the problem however, because imagePullPolicy: Never would just use the image on the nodes. You can diagnose real problem by either kubectl describe pod pod_name or getting logs of the previous pod with --previous flag because newer pod may not have encountered the problem.
Related
Been learning kubernetes in the past several weeks. I've recently built a bare-metal kubernetes cluster with (3) master nodes and (3) worker nodes (containerd runtime). Installed an another stand-alone bare-metal gitlab server with container registry enabled.
I was successful in building a simple nginx container with a custom index.html using docker build and pushed it to the registry; up until this point everything works great.
Now I wanted to create a simple pod using the image built above.
So, did the following steps.
Created a deploy token with read_registry access
Created a secret in kubernetes with username and the token as the password
Inserted imagePullSecrets to the deployment yaml file.
kubectl apply -f nginx.yaml.
Kubernetes pod status stays in ImagePullBackOff.
Failed to pull image "<gitlab-host>:5050/<user>/<project>/nginx:v1": rpc error: code = FailedPrecondition desc = failed to pull and unpack image
"<gitlab-host>:5050/<user>/<project>/nginx:v1": failed commit on ref "unknown-sha256:4ca40a571e91ac4c425500a504490a65852ce49c1f56d7e642c0ec44d13be252": unexpected commit digest sha256:0d899af03c0398a85e36d5cd7ee9a8828e5618db255770a4a96331785ff26d9c, expected sha256:4ca40a571e91ac4c425500a504490a65852ce49c1f56d7e642c0ec44d13be252: failed precondition.
Troubleshooting steps followed.
docker login from another server works.
docker pull works
In one of the worker nodes where kubernetes was scheduling the pod, I did ctr image pull which works
Did some googling but couldn't find any solutions. So, here I am as a last resort to figure this out.
Appreciate any help that I get.
My Deployment nginx.yml file
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: <gitlab-host>:5050/<username>/<project>/nginx:v1
imagePullPolicy: IfNotPresent
name: nginx
imagePullSecrets:
- name: regcred
I found the problem. I made a silly mistake in /etc/containerd/config.toml in the registry section and not mentioning the endpoint with port number <gitlab-host>:5050.
And also adding the private registries in config.toml is not necessary unless you want to run ctr command on the k8s nodes.
I am trying to create a Kubernetes deployment from local docker images. And using imagePullPolicy as Never such that Kubernetes would pick it up from local docker image imported via tar.
Environment
SingleNodeMaster # one node deployment
But Kubernetes always trying to fetch the private repository although local docker images are present.
Any pointers on how to debug and resolve the issue such that Kubernetes would pick the images from the local docker registry? Thank you.
Steps performed
docker load -i images.tar
docker images # displays images from myprivatehub.com/nginx/nginx-custom:v1.1.8
kubectl create -f local-test.yaml with imagepullPolicy as Never
Error
Pulling pod/nginx-custom-6499765dbc-2fts2 Pulling image "myprivatehub.com/nginx/nginx-custom:v1.1.8"
Failed pod/nginx-custom-6499765dbc-2fts2 Error: ErrImagePull
Failed pod/nginx-custom-6499765dbc-2fts2 Failed to pull image "myprivatehub.com/nginx/nginx-custom:v1.1.8": rpc error: code = Unknown desc = failed to pull and unpack image "myprivatehub.com/nginx/nginx-custom:v1.1.8": failed to resolve reference "myprivatehub.com/nginx/nginx-custom:v1.1.8": failed to do request: Head "https://myprivatehub.com/v2/nginx/nginx-custom/manifests/v1.1.8": dial tcp: lookup myprivatehub.com: no such host
docker pull <imagename>
Error response from daemon: Get https://myprivatehub.com/v2/: dial tcp: lookup myprivatehub.com on 172.31.0.2:53: no such host
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-custom
namespace: default
spec:
selector:
matchLabels:
run: nginx-custom
replicas: 5
template:
metadata:
labels:
run: nginx-custom
spec:
containers:
- image: myprivatehub.com/nginx/nginx-custom:v1.1.8
imagePullPolicy: Never
name: nginx-custom
ports:
- containerPort: 80
This happens due to container runtime being different than docker. I am using containerd , after switching container runtime to docker , it started working.
This is to update another approach that can be taken to achieve the similar result. In this case, one can use Docker Registry. Docker Registry Doc
We can create a Docker registry on the machine where Kubernetes is running and docker too is installed. One of the easiest way to achieve the same can be done as following:
Create a local private docker registry. If the registry:2 image is not present, then it would download it and run.
sudo docker run -d -p 5000:5000 --restart=always --name registry registry:2
Build the image or load the image from a tar as required. For my example, i am creating it to add it to the local repository.
sudo docker build -t coolapp:v1 .
Once the build is done, create a tag with this image such that it represents a host and a port.
sudo docker tag coolapp:v1 localhost:5000/coolapp:v1
Push the new tag to the local private registry
sudo docker push localhost:5000/coolapp:v1
Now in the Kubernetes YAML, we can specify the deployment as following:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mycoolapp
spec:
replicas: 1
selector:
matchLabels:
app: mycoolapp
template:
metadata:
labels:
app: mycoolapp
spec:
containers:
- name: mycoolapp
image: localhost:5000/coolapp:v1
ports:
- containerPort: 3000
and we apply the YAML
sudo kubectl apply -f deployment.yaml
Once this is done, we will be able to see that Kubernetes has pulled the image from the local private repository and is running it.
I have this that is working...
# Set docker env
eval $(minikube docker-env)
# Build image
docker build -t jrg/hw .
# Run in minikube
kubectl run hello-world --image=jrg/hw:latest --image-pull-policy=Never --port=8080
kubectl expose deployment hello-world --type=NodePort --name=hello-service
I can access the endpoint just as I expect. Now I am trying to use a .yml file to deploy like this...
apiVersion: v1
kind: Pod
metadata:
name: hello-world-dev
labels:
purpose: simple
spec:
containers:
- name: hello-world-dev-container
image: jrg/hw:latest
env:
- name: WORKING
value: "Yup Working"
But when I run kubectl apply -f k8s/ineject/dev.envvars.yml I get...
NAME READY STATUS RESTARTS AGE
hello-world-7d87b8ddd5-gqr8k 1/1 Running 1 2d22h
hello-world-dev 0/1 ErrImagePull 0 6s
So why can one see my local docker to get the image and 1 has an issue?
In the docs regarding Pre-pulling Images we can read:
By default, the kubelet will try to pull each image from the specified registry. However, if the imagePullPolicy property of the container is set to IfNotPresent or Never, then a local image is used (preferentially or exclusively, respectively).
Also please see other options with imagePullPolicy in Container Images docs.
The imagePullPolicy and the tag of the image affect when the kubelet attempts to pull the specified image.
imagePullPolicy: IfNotPresent: the image is pulled only if it is not already present locally.
imagePullPolicy: Always: the image is pulled every time the pod is started.
imagePullPolicy is omitted and either the image tag is :latest or it is omitted: Always is applied.
imagePullPolicy is omitted and the image tag is present but not :latest: IfNotPresent is applied.
imagePullPolicy: Never: the image is assumed to exist locally. No attempt is made to pull the image.
I have successfully built Docker images and ran them in a Docker swarm. When I attempt to build an image and run it with Docker Desktop's Kubernetes cluster:
docker build -t myimage -f myDockerFile .
(the above successfully creates an image in the docker local registry)
kubectl run myapp --image=myimage:latest
(as far as I understand, this is the same as using the kubectl create deployment command)
The above command successfully creates a deployment, but when it makes a pod, the pod status always shows:
NAME READY STATUS RESTARTS AGE
myapp-<a random alphanumeric string> 0/1 ImagePullBackoff 0 <age>
I am not sure why it is having trouble pulling the image - does it maybe not know where the docker local images are?
I just had the exact same problem. Boils down to the imagePullPolicy:
PC:~$ kubectl explain deployment.spec.template.spec.containers.imagePullPolicy
KIND: Deployment
VERSION: extensions/v1beta1
FIELD: imagePullPolicy <string>
DESCRIPTION:
Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always
if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated.
More info:
https://kubernetes.io/docs/concepts/containers/images#updating-images
Specifically, the part that says: Defaults to Always if :latest tag is specified.
That means, you created a local image, but, because you use the :latest it will try to find it in whatever remote repository you configured (by default docker hub) rather than using your local. Simply change your command to:
kubectl run myapp --image=myimage:latest --image-pull-policy Never
or
kubectl run myapp --image=myimage:latest --image-pull-policy IfNotPresent
I had this same ImagePullBack error while running a pod deployment with a YAML file, also on Docker Desktop.
For anyone else that finds this via Google (like I did), the imagePullPolicy that Lucas mentions above can also be set in the deployment yaml file. See the spec.templage.spec.containers.imagePullPolicy in the yaml snippet below (3 lines from the bottom).
I added that and my app deployed successfully into my local kube cluser, using the kubectl yaml deploy command: kubectl apply -f .\Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app-deployment
labels:
app: web-app
spec:
replicas: 3
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: web-app
image: node-web-app:latest
imagePullPolicy: Never
ports:
- containerPort: 3000
You didn't specify where myimage:latest is hosted, but essentially ImagePullBackoff means that I cannot pull the image because either:
You don't have networking setup in your Docker VM that can get to your Docker registry (Docker Hub?)
myimage:latest doesn't exist in your registry or is misspelled.
myimage:latest requires credentials (you are pulling from a private registry). You can take a look at this to configure container credentials in a Pod.
I am running kubeadm alpha version to set up my kubernates cluster.
From kubernates , I am trying to pull docker images which is hosted in nexus repository.
When ever I am trying to create a pods , It is giving "ImagePullBackOff" every time. Can anybody help me on this ?
Detail for this are present in https://github.com/kubernetes/kubernetes/issues/41536
Pod definition :
apiVersion: v1
kind: Pod
metadata:
name: test-pod
labels:
name: test
spec:
containers:
- image: 123.456.789.0:9595/test
name: test
ports:
- containerPort: 8443
imagePullSecrets:
- name: my-secret
You need to refer to the secret you have just created from the Pod definition.
When you create the secret with kubectl create secret docker-registry my-secret --docker-server=123.456.789.0 ... the server must exactly match what's in your Pod definition - including the port number (and if it's a secure one then it also must match up with the docker command line in systemd).
Also, the secret must be in the same namespace where you are creating your Pod, but that seems to be in order.
I received similar error while launching containers from the amazon ECR registry. The issue was that I didn;t mention the exact "Image URI" location in deployment file.