I have an unsecured private docker registry hosted on a vm server (vm1). I am trying to create a k8s deployment from an image pushed on to this registry. Surprising the docker pull command works fine since I have configured /etc/docker/daemon.json with insecure-registries.
The detailed error through the kubectl describe command is as below. Any idea what could be going wrong?
Thanks.
Failed to pull image "vm1:5000/temp/leads:latest": rpc error: code = Unknown desc = failed to pull and unpack image "vm1:5000/temp/leads:latest": failed to resolve reference "vm1:5000/temp/leads:latest": failed to do request: Head "https://vm1:5000/v2/temp/leads/manifests/latest": http: server gave HTTP response to HTTPS client
The docker pull command is
docker pull vm1:5000/temp/leads:latest
The k8s manifest file is as follows
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
namespace: oleads
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: vm1:5000/temp/leads:latest
resources:
requests:
memory: "64Mi"
cpu: 0.5
limits:
memory: "512Mi"
cpu: 0.5
ports:
- containerPort: 8980
imagePullPolicy: Always
I realised that the kubernetes engine I am using k3s uses a different container runtime. It uses containerd instead of docker.
With k3s the config for using private registries is different. It is mentioned here.
The config I had to add in /etc/rancher/k3s/registries.yaml file is
mirrors:
vm1:5000:
endpoint:
- "http://vm1:5000"
Restarting the k3s service after adding this file resolved the issue and k8s was able to pull the image from my private insecured docker registry.
we had the same issue , the solution could be adding the insecure registry with docker deamon.
Activity on all nodes
create a file in : /etc/docker/daemon.json and add the insecure registry details :
{ "insecure-registries":["vm1:5000"] }
and restart docker on all nodes .
Related
I've a K8s cluster on Linode and another VM for operating.
I've installed Docker & K8s on operating VM to build images and do deployment on cluster.
Note: I haven't installed minikube on this VM.
I'm able to build my image but not able to pull that from local registry to k8s pod.
Below are the things I've already done & tried to solve the problem.
Create and push docker image to local registry.
Run docker container from the image, but not getting pulled in K8s.
Created "regcred" secret and used it in deployment yaml.
create image and push with VM's IP(10.128.234.123:5000/app-frontend) and use the same in deployment image reference.
Change image pull policy to IfNotPresent
I get the following error in pod description:
Warning ErrImageNeverPull 11s (x4 over 13s) kubelet Container image "localhost:5000/app-frontend" is not present with pull policy of Never
Warning Failed 11s (x4 over 13s) kubelet Error: ErrImageNeverPull
Below is my deployment yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-frontend
labels:
app: app-frontend
spec:
replicas: 1
selector:
matchLabels:
app: app-frontend
template:
metadata:
labels:
app: app-frontend
spec:
containers:
- name: app-frontend
image: localhost:5000/docker-image
imagePullPolicy: Never
ports:
- containerPort: 80
imagePullSecrets:
- name: regcred
Any help or guidance will be grateful.
In the Docs I see this
While with imagePullPolicy set to Never, never pull the image.
Try this instead
imagePullPolicy: IfNotPresent
Also
image: localhost:5000/docker-image
But in point 4. you specify an IP
I've been looking at different references on how to enable k3s (running on my pi) to pull docker images from a private registry on my home network (server laptop on my network). If someone can please point my head in the right direction? This is my approach:
Created the docker registry on my server (and making accessible via port 10000):
docker run -d -p 10000:5000 --restart=always --local-docker-registry registry:2
This worked, and was able to push-pull images to it from the "server pc". I didn't add authentication TLS etc. yet...
(viewing the images via docker plugin on VS Code).
Added the inbound firewall rule on my laptop server, and tested that the registry can be 'seen' from my pi (so this also works):
$ curl -ks http://<server IP>:10000/v2/_catalog
{"repositories":["tcpserialpassthrough"]}
Added the registry link to k3s (k3s running on my pi) in registries.yaml file, and restarted k3s and the pi
$ cat /etc/rancher/k3s/registries.yaml
mirrors:
pwlaptopregistry:
endpoint:
- "http://<host IP here>:10000"
Putting the registry prefix to my image endpoint on a deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tcpserialpassthrough
spec:
selector:
matchLabels:
app: tcpserialpassthrough
replicas: 1
template:
metadata:
labels:
app: tcpserialpassthrough
spec:
containers:
- name: tcpserialpassthrough
image: pwlaptopregistry/tcpserialpassthrough:vers1.3-arm
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8001
hostPort: 8001
protocol: TCP
command: ["dotnet", "/app/TcpConnector.dll"]
However, when I check the deployment startup sequence, it's still not able to pull the image (and possibly also still referencing docker hub?):
kubectl get events -w
LAST SEEN TYPE REASON OBJECT MESSAGE
8m24s Normal SuccessfulCreate replicaset/tcpserialpassthrough-88fb974d9 Created pod: tcpserialpassthrough-88fb974d9-b88fc
8m23s Warning FailedScheduling pod/tcpserialpassthrough-88fb974d9-b88fc 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
8m23s Warning FailedScheduling pod/tcpserialpassthrough-88fb974d9-b88fc 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
8m21s Normal Scheduled pod/tcpserialpassthrough-88fb974d9-b88fc Successfully assigned default/tcpserialpassthrough-88fb974d9-b88fc to raspberrypi
6m52s Normal Pulling pod/tcpserialpassthrough-88fb974d9-b88fc Pulling image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm"
6m50s Warning Failed pod/tcpserialpassthrough-88fb974d9-b88fc Error: ErrImagePull
6m50s Warning Failed pod/tcpserialpassthrough-88fb974d9-b88fc Failed to pull image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": failed to resolve reference "docker.io/pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
6m3s Normal BackOff pod/tcpserialpassthrough-88fb974d9-b88fc Back-off pulling image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm"
3m15s Warning Failed pod/tcpserialpassthrough-88fb974d9-b88fc Error: ImagePullBackOff
Wondered if the issue is with authorization, and added based on basic auth, following this youtube guide, but the same issue persists.
Also noted that that /etc/docker/daemon.json must be edited to allow unauthorized, non-TLS connections, via:
{
"Insecure-registries": [ "<host IP>:10000" ]
}
but seemed that this needs to be done on node side, whereas nodes don't have docker cli installed??
... this is so stupid, have no idea why a domain name and port needs to be specified as the "name" of your referred registry, but anyway this solved my issue (for reference):
$cat /etc/rancher/k3s/registries.yaml
mirrors:
"<host IP>:10000":
endpoint:
- "http://<host IP>:10000"
and restarting k3s:
systemctl restart k3s
Then in your deployment, referring to that in your image path as:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tcpserialpassthrough
spec:
selector:
matchLabels:
app: tcpserialpassthrough
replicas: 1
template:
metadata:
labels:
app: tcpserialpassthrough
spec:
containers:
- name: tcpserialpassthrough
image: <host IP>:10000/tcpserialpassthrough:vers1.3-arm
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8001
hostPort: 8001
protocol: TCP
command: ["dotnet", "/app/TcpConnector.dll"]
imagePullSecrets:
- name: mydockercredentials
referring to registry's basic auth details saved as a secret:
$ kubectl create secret docker-registry mydockercredentials --docker-server host IP:10000 --docker-username username --docker-password password
You'll be able to verify the pull process via
$ kubectl get events -w
I here for hours every day, reading and learning, but this is my first question, so bear with me.
I'm simply trying to get my Kubernetes cluster to start up.
Below is my skaffold.yaml file in the root of the project:
apiVersion: skaffold/v2alpha3
kind: Config
deploy:
kubectl:
manifests:
- ./infra/k8s/*
build:
local:
push: false
artifacts:
- image: omesadev/auth
context: auth
docker:
dockerfile: Dockerfile
sync:
manual:
- src: 'src/**/*.ts'
dest: .
Below is my auth-depl.yaml file in the infra/k8s/ directory:
apiVersion: apps/v1
kind: Deployment
metadata:
name: auth-depl
spec:
replicas: 1
selector:
matchLabels:
app: auth
template:
metadata:
labels:
app: auth
spec:
containers:
- name: auth
image: omesadev/auth
---
apiVersion: v1
kind: Service
metadata:
name: auth-srv
spec:
selector:
app: auth
ports:
- name: auth
protocol: TCP
port: 3000
targetPort: 3000
Below is the error message I'm receiving in the cli:
exiting dev mode because first deploy failed: unable to connect to Kubernetes: getting client config for Kubernetes client: error creating REST client config for kubeContext "": invalid configuration: [unable to read client-cert C:\Users\omesa\.minikube\profiles\minikube\client.crt for minikube due to open C:\Users\omesa\.minikube\profiles\minikube\client.crt: The system cannot find the path specified., unable to read client-key C:\Users\omesa\.minikube\profiles\minikube\client.key for minikube due to open C:\Users\omesa\.minikube\profiles\minikube\client.key: The system cannot find the path specified., unable to read certificate-authority C:\Users\omesa\.minikube\ca.crt for minikube due to open C:\Users\omesa\.minikube\ca.crt: The system cannot find the file specified.
I've tried to install kubernetes, minikube, and kubectl. I've added them to the path and removed them a few times in different ways because I thought my configuration or usage could have been incorrect.
Then, I read that if I'm using the Docker GUI that Kubernetes should be running in that, so I checked the settings in the Docker GUI to ensure Kubernetes was running through Docker and it is.
I have Hyper-V set up. I've used it in the past successfully with Docker and with Virtualbox, so I know my Hyper-V is not the issue.
I've also attached an image of my file directory, but I'm pretty sure everything is good to go here too.
src tree
Thanks in advance!
Enable Kubernetes!
The reason why you are getting is that Kubernetes is not enabled.
Posting #Jim solution from comments as community wiki for better visibility:
The problem was, I had two different contexts inside of my kubectl
config and the project I was trying to launch was using the wrong
cluster/context. I don't know how the minikube cluster and context
were created, but I deleted them and set the new context to
docker-desktop with "kubectl config use-context docker-desktop"
Helpful links:
Organizing Cluster Access Using kubeconfig Files
Configure Access to Multiple Clusters
Im trying to follow the get started docker's tutorials, but I get stuck when you have to work with kuberetes. I'm using microk8s to create the clusters.
My Dockerfile:
FROM node:6.11.5WORKDIR /usr/src/app
COPY package.json .
RUN npm install
COPY . .
CMD [ "npm", "start" ]
My bb.yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: bb-demo
namespace: default
spec:
replicas: 1
selector:
matchLabels:
bb: web
template:
metadata:
labels:
bb: web
spec:
containers:
- name: bb-site
image: bulletinboard:1.0
---
apiVersion: v1
kind: Service
metadata:
name: bb-entrypoint
namespace: default
spec:
type: NodePort
selector:
bb: web
ports:
- port: 8080
targetPort: 8080
nodePort: 30001
I create the image with
docker image build -t bulletinboard:1.0 .
And I create the pod and the service with:
microk8s.kubectl apply -f bb.yaml
The pod is created, but, when I look for the state of my pods with
microk8s.kubectl get all
It says:
NAME READY STATUS RESTARTS AGE
pod/bb-demo-7ffb568776-6njfg 0/1 ImagePullBackOff 0 11m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/bb-entrypoint NodePort 10.152.183.2 <none> 8080:30001/TCP 11m
service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 4d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/bb-demo 0/1 1 0 11m
NAME DESIRED CURRENT READY AGE
replicaset.apps/bb-demo-7ffb568776 1 1 0 11m
Also, when I look for it at the kubernetes dashboard it says:
Failed to pull image "bulletinboard:1.0": rpc error: code = Unknown desc = failed to resolve image "docker.io/library/bulletinboard:1.0": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Q: Why do I get this error? Im just following the tutorial without skipping anything.
Im already logged with docker.
You need to push this locally built image to the Docker Hub registry. For that, you need to create a Docker Hub account if you do not have one already.
Once you do that, you need to login to Docker Hub from your command line.
docker login
Tag your image so it goes to your Docker Hub repository.
docker tag bulletinboard:1.0 <your docker hub user>/bulletinboard:1.0
Push your image to Docker Hub
docker push <your docker hub user>/bulletinboard:1.0
Update the yaml file to reflect the new image repo on Docker Hub.
spec:
containers:
- name: bb-site
image: <your docker hub user>/bulletinboard:1.0
re-apply the yaml file
microk8s.kubectl apply -f bb.yaml
You can host a local registry server if you do not wish to use Docker hub.
Start a local registry server:
docker run -d -p 5000:5000 --restart=always --name registry registry:2
Tag your image:
sudo docker tag bulletinboard:1.0 localhost:5000/bulletinboard
Push it to a local registry:
sudo docker push localhost:5000/bulletinboard
Change the yaml file:
spec:
containers:
- name: bb-site
image: localhost:5000/bulletinboard
Start deployment
kubectl apply -f bb.yaml
A suggested solution is to add imagePullPolicy: Never to your Deployment as per the answer here but this didn't work for me, so I followed this guide since I was working in local development.
After following the link below, I can successfully pull my private images in Docker Hub from my Pods: Pull from Private repo
However, attempting to pull a Docker Store image doesn't seem to work.
I am able to pull this store image locally on my deskop using docker pull store/oracle/database-instantclient:12.2.0.1 and the same credentials that have been stored in Kubernetes as a secret.
What is the correct way to pull a Docker Store image from Kubernetes Pods?
Working pod config for my private repo/image:
image: index.docker.io/<privaterepo>/<privateimage>
I have tried the following in my pod config, none work:
image: store/oracle/database-instantclient:12.2.0.1
image: oracle/database-instantclient:12.2.0.1
image: index.docker.io/oracle/database-instantclient:12.2.0.1
image: index.docker.io/store/oracle/database-instantclient:12.2.0.1
All of the above attempts return the same error (with different image paths):
Failed to pull image "store/oracle/database-instantclient:12.2.0.1": rpc error: code = Unknown desc = Error response from daemon: repository store/oracle/database-instantclient not found: does not exist or no pull access
I managed to run this in minikube by setting up a secret with my docker login:
kubectl create secret docker-registry dockerstore \
--docker-server=index.docker.io/v1/ \
--docker-username={docker store username} \
--docker-password={docker store password} \
--docker-email={your email}
Then kubectl create -f testreplicaset.yaml
on
#testreplicaset.yaml
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: oracle-instantclient
labels:
app: oracle-instantclient
spec:
replicas: 1
selector:
matchLabels:
app: oracle-instantclient
template:
metadata:
labels:
app: oracle-instantclient
spec:
containers:
- name: oracle-instantclient-container
image: store/oracle/database-instantclient:12.2.0.1
env:
ports:
imagePullSecrets:
- name: dockerstore
I can't tell exactly why it doesn't work for you, but it might give more clues if you ssh into your kubernetes node and try docker pull in there.