I have this basic Dockerfile:
FROM nginx
RUN apt-get -y update && apt install -y curl
In the master node of my Kubernetes cluster I build that image:
docker build -t cnginx:v1 .
docker images shows that the image has been correctly generated:
REPOSITORY TAG IMAGE ID CREATED SIZE
cgninx v1 d3b1b19d069e 39 minutes ago 141MB
I use this deployment referencing this custom image:
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: nginx
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: cnginx
image: cnginx:v1
imagePullPolicy: Never
ports:
- containerPort: 80
nodeSelector:
nodetype: webserver
However the image is not found:
NAME READY STATUS RESTARTS AGE
nginx-deployment-7dd98bd746-lw6tp 0/1 ErrImageNeverPull 0 4s
nginx-deployment-7dd98bd746-szr9n 0/1 ErrImageNeverPull 0 4s
Describe pod info:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1m default-scheduler Successfully assigned nginx-deployment-7dd98bd746-szr9n to kubenode2
Normal SuccessfulMountVolume 1m kubelet, kubenode2 MountVolume.SetUp succeeded for volume "default-token-bpbpl"
Warning ErrImageNeverPull 9s (x9 over 1m) kubelet, kubenode2 Container image "cnginx:v1" is not present with pull policy of Never
Warning Failed 9s (x9 over 1m) kubelet, kubenode2 Error: ErrImageNeverPull
I have also tried using the default imagePullPolicy, and some other things such as tagging the image with latest...
So, how can I make Kubernetes use a locally generated docker image?
Your PODs are scheduled on your worker nodes. Since you set imagePullPolicy to Never you need to make your image available to both nodes. In other words, you need to build it on both nodes as you did on the master.
As a sidenote, it would be probably easier in the long term if you setup a custom docker registry and push your images there.
Related
I have pushed some docker images to my private repo on dockerhub, which I am now trying to use to create deployments in a Minikube Kubernetes cluster.
I have done the following:
docker login -u [username] -p [password]
docker tag [mslearn-microservices-pizzabackend] [username]/[mslearn-microservices-pizzabackend]
docker tag [mslearn-microservices-pizzafrontend] [username]/[mslearn-microservices-pizzafrontend]
I can see both the images in my private dockerhub repo. To be able to use them in a deployment, I have done the following:
kubectl create secret docker-registry dockerhub-credentials --docker-server="docker.io" --docker-username="[username]" --docker-password="[password]" --docker-email="[email]"
After that, I try to create a deployment for the first image using the following manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mslearn-microservices-pizzabackend
spec:
replicas: 1
selector:
matchLabels:
app: mslearn-microservices-pizzabackend
template:
metadata:
labels:
app: mslearn-microservices-pizzabackend
spec:
imagePullSecrets:
- name: dockerhub-credentials
containers:
- name: mslearn-microservices-pizzabackend
image: [username]/mslearn-microservices-pizzabackend:latest
ports:
- containerPort: 80
env:
- name: ASPNETCORE_URLS
value: http://*:80
---
apiVersion: v1
kind: Service
metadata:
name: mslearn-microservices-pizzabackend
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: mslearn-microservices-pizzabackend
But when I check the events of the pod that gets created by the deployment, I can see the following:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18s default-scheduler Successfully assigned default/mslearn-microservices-pizzabackend-79dcd6677d-cgh7z to minikube
Normal BackOff 15s kubelet Back-off pulling image "[username]/mslearn-microservices-pizzabackend:latest"
Warning Failed 15s kubelet Error: ImagePullBackOff
Normal Pulling 3s (x2 over 18s) kubelet Pulling image "[username]/mslearn-microservices-pizzabackend:latest"
Warning Failed 1s (x2 over 16s) kubelet Failed to pull image "[username]/mslearn-microservices-pizzabackend:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for [username]/mslearn-microservices-pizzabackend, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning Failed 1s (x2 over 16s) kubelet Error: ErrImagePull
I have tried searching for solutions on the web and I can see that other people have had similiar issues, but none of their solutions have worked for me.
Any suggestions?
Good day!
I am facing a strange problem. I have a standard deployment that uses a public image. But when I create it, I get the error ImagePullBackOff
$ kubectl get pods
result
api-gateway-deployment-74968fbf5c-cvqwj 0/1 ImagePullBackOff 0 6h23m
api-gateway-gateway-deployment-74968fbf5c-hpdxb 0/1 ImagePullBackOff 0 6h23m
api-gateway-gateway-deployment-74968fbf5c-rctv6 0/1 ImagePullBackOff 0 6h23m
my deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-gateway-deployment
labels:
app: api-gateway-deployment
spec:
replicas: 3
selector:
matchLabels:
app: api-gateway-deployment
template:
metadata:
labels:
app: api-gateway-deployment
spec:
containers:
- name: api-gateway-node
image: creatorsprodhouse/api-gateway:latest
imagePullPolicy: Always
ports:
- containerPort: 80
I am using the docker driver, is there anything I can do wrong?
minikube start --driver=docker
I think your internet connection is slow. The timeout to pull an image is 120 seconds, so kubectl could not pull the image in under 120 seconds.
First, pull the image via Docker
docker image pull creatorsprodhouse/api-gateway:latest
Then load the downloaded image to minikube
minikube image load creatorsprodhouse/api-gateway:latest
And then everything will work because now kubectl will use the image that is stored locally.
I've a K8s cluster on Linode and another VM for operating.
I've installed Docker & K8s on operating VM to build images and do deployment on cluster.
Note: I haven't installed minikube on this VM.
I'm able to build my image but not able to pull that from local registry to k8s pod.
Below are the things I've already done & tried to solve the problem.
Create and push docker image to local registry.
Run docker container from the image, but not getting pulled in K8s.
Created "regcred" secret and used it in deployment yaml.
create image and push with VM's IP(10.128.234.123:5000/app-frontend) and use the same in deployment image reference.
Change image pull policy to IfNotPresent
I get the following error in pod description:
Warning ErrImageNeverPull 11s (x4 over 13s) kubelet Container image "localhost:5000/app-frontend" is not present with pull policy of Never
Warning Failed 11s (x4 over 13s) kubelet Error: ErrImageNeverPull
Below is my deployment yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-frontend
labels:
app: app-frontend
spec:
replicas: 1
selector:
matchLabels:
app: app-frontend
template:
metadata:
labels:
app: app-frontend
spec:
containers:
- name: app-frontend
image: localhost:5000/docker-image
imagePullPolicy: Never
ports:
- containerPort: 80
imagePullSecrets:
- name: regcred
Any help or guidance will be grateful.
In the Docs I see this
While with imagePullPolicy set to Never, never pull the image.
Try this instead
imagePullPolicy: IfNotPresent
Also
image: localhost:5000/docker-image
But in point 4. you specify an IP
I am new to kubernetes. I am running my kubernetes cluster inside my Docker Desktop VM. Below are the versions
Docker Desktop Community : 2.3.0.4 (Stable)
Engine: 19.03.12
kubernetes: 1.16.5
i created a simple react app. below is the Docker file.
FROM node:13.12.0-alpine
WORKDIR /app
ENV PATH /app/node_modules/.bin:$PATH
COPY package*.json ./
RUN npm install
# add app files
COPY . ./
# start app
CMD ["npm", "start"]
I built a docker image and ran it. it works fine. I added the image in the below deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-deployment
labels:
app: test-react-app
namespace: dev
spec:
replicas: 1
selector:
matchLabels:
app: test-react-app
template:
metadata:
labels:
app: test-react-app
spec:
containers:
- name: test-react
image: myrepo/test-react:v2
imagePullPolicy: Never
ports:
- containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
name: test-service
namespace: dev
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
nodePort: 31000
selector:
app: test-react-app
The pod never starts. Below is the events from describe.
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned dev/test-deployment-7766949554-m2fbz to docker-desktop
Normal Pulled 8m38s (x5 over 10m) kubelet, docker-desktop Container image "myrepo/test-react:v2" already present on machine
Normal Created 8m38s (x5 over 10m) kubelet, docker-desktop Created container test-react
Normal Started 8m38s (x5 over 10m) kubelet, docker-desktop Started container test-react
Warning BackOff 26s (x44 over 10m) kubelet, docker-desktop Back-off restarting failed container
Below is the logs from the container. It looks as if the container is running..
> react-cart#0.1.0 start /app
> react-scripts start
[34mℹ[39m [90m「wds」[39m: Project is running at http://10.1.0.33/
[34mℹ[39m [90m「wds」[39m: webpack output is served from
[34mℹ[39m [90m「wds」[39m: Content not from webpack is served from /app/public
[34mℹ[39m [90m「wds」[39m: 404s will fallback to /
Starting the development server...
It worked!!!
I build the react app into a production app and then copied the docker file. I followed the technique given in this link https://dev.to/rieckpil/deploy-a-react-application-to-kubernetes-in-5-easy-steps-516j.
I set up a Harbor registry which worked successfully for a couple of weeks now. For each deployment and namespace I a have a secret with the credentials from my ~/.docker/config.json file to get access to the registry. Since last weekend I was not able to pull images from that registry anymore and I didn't change anything! The cluster is running on GKE v1.12.5 btw.
What works?
I can pull and push images from my local machine witch docker.
What does not work?
My Kubernetes cluster cannot pull images anymore and runs in a timeout.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 13m default-scheduler Successfully assigned k8s-test7/nginx-k8s-test7-6f7b8fdd79-2ffmp to gke-k8s-cloudops-test-default-pool-72fccd21-hrhk
Normal SandboxChanged 12m kubelet, gke-k8s-cloudops-test-default-pool-72fccd21-hrhk Pod sandbox changed, it will be killed and re-created.
Warning Failed 11m (x3 over 12m) kubelet, gke-k8s-cloudops-test-default-pool-72fccd21-hrhk Failed to pull image "core.k8s-harbor-test.my-domain.com/nginx-test/nginx:1.15.10": rpc error: code = Unknown desc = Error response from daemon: Get https://core.k8s-harbor-test.my-domain.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Failed 11m (x3 over 12m) kubelet, gke-k8s-cloudops-test-default-pool-72fccd21-hrhk Error: ErrImagePull
Normal BackOff 11m (x7 over 12m) kubelet, gke-k8s-cloudops-test-default-pool-72fccd21-hrhk Back-off pulling image "core.k8s-harbor-test.my-domain.com/nginx-test/nginx:1.15.10"
Normal Pulling 10m (x4 over 13m) kubelet, gke-k8s-cloudops-test-default-pool-72fccd21-hrhk pulling image "core.k8s-harbor-test.my-domain.com/nginx-test/nginx:1.15.10"
Warning Failed 3m2s (x38 over 12m) kubelet, gke-k8s-cloudops-test-default-pool-72fccd21-hrhk Error: ImagePullBackOff
deployment.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-k8s-test7
namespace: k8s-test7
spec:
replicas: 1
template:
metadata:
labels:
app: nginx-k8s-test7
spec:
containers:
- name: nginx-k8s-test7
image: core.k8s-harbor-test.my-domain.com/nginx-test/nginx:1.15.10
volumeMounts:
- name: webcontent
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: webcontent
configMap:
name: webcontent
imagePullSecrets:
- name: harborcred
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: webcontent
namespace: k8s-test7
annotations:
volume.alpha.kubernetes.io/storage-class: default
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 5Gi
The secret "harborcred" is part of every namespace so that the deployment can access it. The secret was created per kubernetes documentation:
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
kubectl create secret generic harborcred \
--from-file=.dockerconfigjson=~/.docker/config.json \
--type=kubernetes.io/dockerconfigjson \
--namespace=k8s-test7
Any help would be appreciated!
Hi at first look could you please:
Change image source and use some public one f.e. nginx to verify your deployment doesn't have other issues.
https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/ provide also more details about inspecting the "Secrets".
Please also perform additional tests related to connectivity directly from your node as described within this post [How to debug "ImagePullBackOff"?
Additional steps to find the root cause:
1. Convert your secrets data:
kubectl get secret harborcred -n k8s-test7
--output="jsonpath={.data.\.dockerconfigjson}" | base64 --decode
2. Compare the result of decoding your "auth" field from the 1 step with your docker credentials using:
echo "your auth data" | base64 --decode
3. To find the root cause please use also:
kubectl get events -n k8s-test7 | grep pull
Please share with your logs.