I've a K8s cluster on Linode and another VM for operating.
I've installed Docker & K8s on operating VM to build images and do deployment on cluster.
Note: I haven't installed minikube on this VM.
I'm able to build my image but not able to pull that from local registry to k8s pod.
Below are the things I've already done & tried to solve the problem.
Create and push docker image to local registry.
Run docker container from the image, but not getting pulled in K8s.
Created "regcred" secret and used it in deployment yaml.
create image and push with VM's IP(10.128.234.123:5000/app-frontend) and use the same in deployment image reference.
Change image pull policy to IfNotPresent
I get the following error in pod description:
Warning ErrImageNeverPull 11s (x4 over 13s) kubelet Container image "localhost:5000/app-frontend" is not present with pull policy of Never
Warning Failed 11s (x4 over 13s) kubelet Error: ErrImageNeverPull
Below is my deployment yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-frontend
labels:
app: app-frontend
spec:
replicas: 1
selector:
matchLabels:
app: app-frontend
template:
metadata:
labels:
app: app-frontend
spec:
containers:
- name: app-frontend
image: localhost:5000/docker-image
imagePullPolicy: Never
ports:
- containerPort: 80
imagePullSecrets:
- name: regcred
Any help or guidance will be grateful.
In the Docs I see this
While with imagePullPolicy set to Never, never pull the image.
Try this instead
imagePullPolicy: IfNotPresent
Also
image: localhost:5000/docker-image
But in point 4. you specify an IP
Related
Good day!
I am facing a strange problem. I have a standard deployment that uses a public image. But when I create it, I get the error ImagePullBackOff
$ kubectl get pods
result
api-gateway-deployment-74968fbf5c-cvqwj 0/1 ImagePullBackOff 0 6h23m
api-gateway-gateway-deployment-74968fbf5c-hpdxb 0/1 ImagePullBackOff 0 6h23m
api-gateway-gateway-deployment-74968fbf5c-rctv6 0/1 ImagePullBackOff 0 6h23m
my deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-gateway-deployment
labels:
app: api-gateway-deployment
spec:
replicas: 3
selector:
matchLabels:
app: api-gateway-deployment
template:
metadata:
labels:
app: api-gateway-deployment
spec:
containers:
- name: api-gateway-node
image: creatorsprodhouse/api-gateway:latest
imagePullPolicy: Always
ports:
- containerPort: 80
I am using the docker driver, is there anything I can do wrong?
minikube start --driver=docker
I think your internet connection is slow. The timeout to pull an image is 120 seconds, so kubectl could not pull the image in under 120 seconds.
First, pull the image via Docker
docker image pull creatorsprodhouse/api-gateway:latest
Then load the downloaded image to minikube
minikube image load creatorsprodhouse/api-gateway:latest
And then everything will work because now kubectl will use the image that is stored locally.
I have an unsecured private docker registry hosted on a vm server (vm1). I am trying to create a k8s deployment from an image pushed on to this registry. Surprising the docker pull command works fine since I have configured /etc/docker/daemon.json with insecure-registries.
The detailed error through the kubectl describe command is as below. Any idea what could be going wrong?
Thanks.
Failed to pull image "vm1:5000/temp/leads:latest": rpc error: code = Unknown desc = failed to pull and unpack image "vm1:5000/temp/leads:latest": failed to resolve reference "vm1:5000/temp/leads:latest": failed to do request: Head "https://vm1:5000/v2/temp/leads/manifests/latest": http: server gave HTTP response to HTTPS client
The docker pull command is
docker pull vm1:5000/temp/leads:latest
The k8s manifest file is as follows
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
namespace: oleads
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: vm1:5000/temp/leads:latest
resources:
requests:
memory: "64Mi"
cpu: 0.5
limits:
memory: "512Mi"
cpu: 0.5
ports:
- containerPort: 8980
imagePullPolicy: Always
I realised that the kubernetes engine I am using k3s uses a different container runtime. It uses containerd instead of docker.
With k3s the config for using private registries is different. It is mentioned here.
The config I had to add in /etc/rancher/k3s/registries.yaml file is
mirrors:
vm1:5000:
endpoint:
- "http://vm1:5000"
Restarting the k3s service after adding this file resolved the issue and k8s was able to pull the image from my private insecured docker registry.
we had the same issue , the solution could be adding the insecure registry with docker deamon.
Activity on all nodes
create a file in : /etc/docker/daemon.json and add the insecure registry details :
{ "insecure-registries":["vm1:5000"] }
and restart docker on all nodes .
I've been looking at different references on how to enable k3s (running on my pi) to pull docker images from a private registry on my home network (server laptop on my network). If someone can please point my head in the right direction? This is my approach:
Created the docker registry on my server (and making accessible via port 10000):
docker run -d -p 10000:5000 --restart=always --local-docker-registry registry:2
This worked, and was able to push-pull images to it from the "server pc". I didn't add authentication TLS etc. yet...
(viewing the images via docker plugin on VS Code).
Added the inbound firewall rule on my laptop server, and tested that the registry can be 'seen' from my pi (so this also works):
$ curl -ks http://<server IP>:10000/v2/_catalog
{"repositories":["tcpserialpassthrough"]}
Added the registry link to k3s (k3s running on my pi) in registries.yaml file, and restarted k3s and the pi
$ cat /etc/rancher/k3s/registries.yaml
mirrors:
pwlaptopregistry:
endpoint:
- "http://<host IP here>:10000"
Putting the registry prefix to my image endpoint on a deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tcpserialpassthrough
spec:
selector:
matchLabels:
app: tcpserialpassthrough
replicas: 1
template:
metadata:
labels:
app: tcpserialpassthrough
spec:
containers:
- name: tcpserialpassthrough
image: pwlaptopregistry/tcpserialpassthrough:vers1.3-arm
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8001
hostPort: 8001
protocol: TCP
command: ["dotnet", "/app/TcpConnector.dll"]
However, when I check the deployment startup sequence, it's still not able to pull the image (and possibly also still referencing docker hub?):
kubectl get events -w
LAST SEEN TYPE REASON OBJECT MESSAGE
8m24s Normal SuccessfulCreate replicaset/tcpserialpassthrough-88fb974d9 Created pod: tcpserialpassthrough-88fb974d9-b88fc
8m23s Warning FailedScheduling pod/tcpserialpassthrough-88fb974d9-b88fc 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
8m23s Warning FailedScheduling pod/tcpserialpassthrough-88fb974d9-b88fc 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
8m21s Normal Scheduled pod/tcpserialpassthrough-88fb974d9-b88fc Successfully assigned default/tcpserialpassthrough-88fb974d9-b88fc to raspberrypi
6m52s Normal Pulling pod/tcpserialpassthrough-88fb974d9-b88fc Pulling image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm"
6m50s Warning Failed pod/tcpserialpassthrough-88fb974d9-b88fc Error: ErrImagePull
6m50s Warning Failed pod/tcpserialpassthrough-88fb974d9-b88fc Failed to pull image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": failed to resolve reference "docker.io/pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
6m3s Normal BackOff pod/tcpserialpassthrough-88fb974d9-b88fc Back-off pulling image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm"
3m15s Warning Failed pod/tcpserialpassthrough-88fb974d9-b88fc Error: ImagePullBackOff
Wondered if the issue is with authorization, and added based on basic auth, following this youtube guide, but the same issue persists.
Also noted that that /etc/docker/daemon.json must be edited to allow unauthorized, non-TLS connections, via:
{
"Insecure-registries": [ "<host IP>:10000" ]
}
but seemed that this needs to be done on node side, whereas nodes don't have docker cli installed??
... this is so stupid, have no idea why a domain name and port needs to be specified as the "name" of your referred registry, but anyway this solved my issue (for reference):
$cat /etc/rancher/k3s/registries.yaml
mirrors:
"<host IP>:10000":
endpoint:
- "http://<host IP>:10000"
and restarting k3s:
systemctl restart k3s
Then in your deployment, referring to that in your image path as:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tcpserialpassthrough
spec:
selector:
matchLabels:
app: tcpserialpassthrough
replicas: 1
template:
metadata:
labels:
app: tcpserialpassthrough
spec:
containers:
- name: tcpserialpassthrough
image: <host IP>:10000/tcpserialpassthrough:vers1.3-arm
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8001
hostPort: 8001
protocol: TCP
command: ["dotnet", "/app/TcpConnector.dll"]
imagePullSecrets:
- name: mydockercredentials
referring to registry's basic auth details saved as a secret:
$ kubectl create secret docker-registry mydockercredentials --docker-server host IP:10000 --docker-username username --docker-password password
You'll be able to verify the pull process via
$ kubectl get events -w
I've build docker image locally:
docker build -t backend -f backend.docker
Now I want to create deployment with it:
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-deployment
spec:
selector:
matchLabels:
tier: backend
replicas: 2
template:
metadata:
labels:
tier: backend
spec:
containers:
- name: backend
image: backend
imagePullPolicy: IfNotPresent # This should be by default so
ports:
- containerPort: 80
kubectl apply -f file_provided_above.yaml works, but then I have following pods statuses:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
backend-deployment-66cff7d4c6-gwbzf 0/1 ImagePullBackOff 0 18s
Before that it was ErrImagePull. So, my question is, how to tell it to use local docker images? Somewhere on the internet I read that I need to build images using microk8s.docker but it seems to be removed.
Found docs on how to use private registry: https://microk8s.io/docs/working
First it needs to be enabled:
microk8s.enable registry
Then images pushed to registry:
docker tag backend localhost:32000/backend
docker push localhost:32000/backend
And then in above config image: backend needs to be replaced with image: localhost:32000/backend
Is possible to set a local image in a kubernetes pod yml file?
This is my pod yml file, and the question is if I can use a local image to containers tag (in local, I have all files to my api project, dockerfile, etc).
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: api-service
spec:
selector:
matchLabels:
api-name: api-service
replicas: 2
template:
metadata:
labels:
api-name: api-service
spec:
containers:
- name: api-service
image: #HERE
By local you mean it doesn't pull from dockerhub or any of the public registry. Yes it's possible if you run a single node kubernetes. You will utlize the docker cache where your kubernetes/kubelet is running.
First thing is, you need to set your imagePullPolicy: IfNotPresent. Then, when you build your image, you need to point to the docker instance your kubernetes is using.
I do this mostly with minikube, so the dev iteration is faster without pushing to my registry.