I just started learning Docker and Kubernetes. Installed minikube and docker on my windows machine. I am able to pull image from docker using docker pull command but getting below error with kubectl. Please help.
Warning Failed 18s (x2 over 53s) kubelet Failed to pull image "nginx:alpine": rpc error: code = Unknown desc = error pulling image configuration: download failed after attempts=6: x509: certificate signed by unknown authority Warning Failed 18s (
This is my yml file.
apiVersion: v1
kind: Pod
metadata:
name: nginx1
spec:
containers:
name: nginx1
image: nginx:alpine
ports:
containerPort: 80
containerPort: 443
Thanks in advance.
enter image description here
Your yaml file looks messed up. Please ensure you write the yaml file properly. Can you test if this yaml file works for you? I have used a different image in the below mention yaml file:
apiVersion: v1
kind: Pod
metadata:
name: nginx1
namespace: test
spec:
containers:
- name: webserver
image: nginx:latest
ports:
- containerPort: 80
resources:
requests:
memory: "64Mi"
cpu: "200m"
limits:
memory: "128Mi"
cpu: "350m"
Related
I have pushed some docker images to my private repo on dockerhub, which I am now trying to use to create deployments in a Minikube Kubernetes cluster.
I have done the following:
docker login -u [username] -p [password]
docker tag [mslearn-microservices-pizzabackend] [username]/[mslearn-microservices-pizzabackend]
docker tag [mslearn-microservices-pizzafrontend] [username]/[mslearn-microservices-pizzafrontend]
I can see both the images in my private dockerhub repo. To be able to use them in a deployment, I have done the following:
kubectl create secret docker-registry dockerhub-credentials --docker-server="docker.io" --docker-username="[username]" --docker-password="[password]" --docker-email="[email]"
After that, I try to create a deployment for the first image using the following manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: mslearn-microservices-pizzabackend
spec:
replicas: 1
selector:
matchLabels:
app: mslearn-microservices-pizzabackend
template:
metadata:
labels:
app: mslearn-microservices-pizzabackend
spec:
imagePullSecrets:
- name: dockerhub-credentials
containers:
- name: mslearn-microservices-pizzabackend
image: [username]/mslearn-microservices-pizzabackend:latest
ports:
- containerPort: 80
env:
- name: ASPNETCORE_URLS
value: http://*:80
---
apiVersion: v1
kind: Service
metadata:
name: mslearn-microservices-pizzabackend
spec:
type: ClusterIP
ports:
- port: 80
selector:
app: mslearn-microservices-pizzabackend
But when I check the events of the pod that gets created by the deployment, I can see the following:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 18s default-scheduler Successfully assigned default/mslearn-microservices-pizzabackend-79dcd6677d-cgh7z to minikube
Normal BackOff 15s kubelet Back-off pulling image "[username]/mslearn-microservices-pizzabackend:latest"
Warning Failed 15s kubelet Error: ImagePullBackOff
Normal Pulling 3s (x2 over 18s) kubelet Pulling image "[username]/mslearn-microservices-pizzabackend:latest"
Warning Failed 1s (x2 over 16s) kubelet Failed to pull image "[username]/mslearn-microservices-pizzabackend:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for [username]/mslearn-microservices-pizzabackend, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning Failed 1s (x2 over 16s) kubelet Error: ErrImagePull
I have tried searching for solutions on the web and I can see that other people have had similiar issues, but none of their solutions have worked for me.
Any suggestions?
I am trying to build a sidecar image from skaffold and then push it onto my minikube cluster.
My skaffold.yaml file looks like this :
apiVersion: skaffold/v2beta28
kind: Config
metadata:
name: sidecar
build:
artifacts:
- image: amolgautam25/sidecar
docker:
dockerfile: Dockerfile
deploy:
kubectl:
manifests:
- pg-example.yaml
My pod deployment file (pg_example.yaml) looks like this :
apiVersion: acid.zalan.do/v1
kind: postgresql
metadata:
name: vmw-test
spec:
databases:
foo: zalando
numberOfInstances: 1
podAnnotations:
prometheus.io/port: "9187"
prometheus.io/scrape: "true"
postgresql:
parameters:
log_filename: postgresql.log
log_rotation_age: "0"
log_rotation_size: "0"
version: "14"
preparedDatabases:
bar: {}
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: "0"
memory: "0"
spiloFSGroup: 103
spiloRunAsGroup: 103
spiloRunAsUser: 101
teamId: vmw
users:
foo_user: []
zalando:
- superuser
- createdb
volume:
size: 1Gi
sidecars:
- name: "postgres-exporter"
image: quay.io/prometheuscommunity/postgres-exporter
env:
# The default "host all all 127.0.0.1/32 md5" rule in pg_hba.conf
# allows us to connect over 127.0.0.1 without TLS as long as we have the password
- name: DATA_SOURCE_URI
value: "localhost:5432/postgres?sslmode=disable"
- name: DATA_SOURCE_USER
value: postgres
- name: DATA_SOURCE_PASS
valueFrom:
secretKeyRef:
key: password
name: postgres.vmw-test.credentials.postgresql.acid.zalan.do
ports:
- name: exporter
containerPort: 9187
protocol: TCP
resources:
requests:
cpu: 500m
memory: 500Mi
limits:
cpu: 1000m
memory: 1Gi
- name: "metrics-sidecar"
image: amolgautam25/sidecar
In the minikube images i can see the image built by skaffold
<usename-hidden>$ minikube image ls -p my-profile
docker.io/amolgautam25/sidecar:6b1af6a1fd25825dc63fd843f951e10c98bd9eb87d80cd8cf81da5641dc041e2
However, minikube refuses to use that image. Here is the error i get when i do describe pod:
Normal Pulling 7s kubelet Pulling image "amolgautam25/sidecar"
Warning Failed 6s kubelet Failed to pull image "amolgautam25/sidecar": rpc error: code = Unknown desc = Error response from daemon: pull access denied for amolgautam25/sidecar, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
Warning Failed 6s kubelet Error: ErrImagePull
Normal BackOff 3s (x2 over 5s) kubelet Back-off pulling image "amolgautam25/sidecar"
Warning Failed 3s (x2 over 5s) kubelet Error: ImagePullBackOff
But it seems that minikube already has that image. I have tried variations of docker.io/amolgautam25/sidecar etc but it does not work.
Any help would be appreciated.
Edit:
On further investigation i have found out that the skaffold is not modifying the 'pg-example.yaml' file. For some reason it does not change the 'image' tag to the one that is built by skaffold. I think the answer lies in : https://skaffold.dev/docs/tutorials/skaffold-resource-selector/ ( still investigating )
I have an unsecured private docker registry hosted on a vm server (vm1). I am trying to create a k8s deployment from an image pushed on to this registry. Surprising the docker pull command works fine since I have configured /etc/docker/daemon.json with insecure-registries.
The detailed error through the kubectl describe command is as below. Any idea what could be going wrong?
Thanks.
Failed to pull image "vm1:5000/temp/leads:latest": rpc error: code = Unknown desc = failed to pull and unpack image "vm1:5000/temp/leads:latest": failed to resolve reference "vm1:5000/temp/leads:latest": failed to do request: Head "https://vm1:5000/v2/temp/leads/manifests/latest": http: server gave HTTP response to HTTPS client
The docker pull command is
docker pull vm1:5000/temp/leads:latest
The k8s manifest file is as follows
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-deployment
namespace: oleads
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: my-app
image: vm1:5000/temp/leads:latest
resources:
requests:
memory: "64Mi"
cpu: 0.5
limits:
memory: "512Mi"
cpu: 0.5
ports:
- containerPort: 8980
imagePullPolicy: Always
I realised that the kubernetes engine I am using k3s uses a different container runtime. It uses containerd instead of docker.
With k3s the config for using private registries is different. It is mentioned here.
The config I had to add in /etc/rancher/k3s/registries.yaml file is
mirrors:
vm1:5000:
endpoint:
- "http://vm1:5000"
Restarting the k3s service after adding this file resolved the issue and k8s was able to pull the image from my private insecured docker registry.
we had the same issue , the solution could be adding the insecure registry with docker deamon.
Activity on all nodes
create a file in : /etc/docker/daemon.json and add the insecure registry details :
{ "insecure-registries":["vm1:5000"] }
and restart docker on all nodes .
I've been looking at different references on how to enable k3s (running on my pi) to pull docker images from a private registry on my home network (server laptop on my network). If someone can please point my head in the right direction? This is my approach:
Created the docker registry on my server (and making accessible via port 10000):
docker run -d -p 10000:5000 --restart=always --local-docker-registry registry:2
This worked, and was able to push-pull images to it from the "server pc". I didn't add authentication TLS etc. yet...
(viewing the images via docker plugin on VS Code).
Added the inbound firewall rule on my laptop server, and tested that the registry can be 'seen' from my pi (so this also works):
$ curl -ks http://<server IP>:10000/v2/_catalog
{"repositories":["tcpserialpassthrough"]}
Added the registry link to k3s (k3s running on my pi) in registries.yaml file, and restarted k3s and the pi
$ cat /etc/rancher/k3s/registries.yaml
mirrors:
pwlaptopregistry:
endpoint:
- "http://<host IP here>:10000"
Putting the registry prefix to my image endpoint on a deployment manifest:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tcpserialpassthrough
spec:
selector:
matchLabels:
app: tcpserialpassthrough
replicas: 1
template:
metadata:
labels:
app: tcpserialpassthrough
spec:
containers:
- name: tcpserialpassthrough
image: pwlaptopregistry/tcpserialpassthrough:vers1.3-arm
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8001
hostPort: 8001
protocol: TCP
command: ["dotnet", "/app/TcpConnector.dll"]
However, when I check the deployment startup sequence, it's still not able to pull the image (and possibly also still referencing docker hub?):
kubectl get events -w
LAST SEEN TYPE REASON OBJECT MESSAGE
8m24s Normal SuccessfulCreate replicaset/tcpserialpassthrough-88fb974d9 Created pod: tcpserialpassthrough-88fb974d9-b88fc
8m23s Warning FailedScheduling pod/tcpserialpassthrough-88fb974d9-b88fc 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
8m23s Warning FailedScheduling pod/tcpserialpassthrough-88fb974d9-b88fc 0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports.
8m21s Normal Scheduled pod/tcpserialpassthrough-88fb974d9-b88fc Successfully assigned default/tcpserialpassthrough-88fb974d9-b88fc to raspberrypi
6m52s Normal Pulling pod/tcpserialpassthrough-88fb974d9-b88fc Pulling image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm"
6m50s Warning Failed pod/tcpserialpassthrough-88fb974d9-b88fc Error: ErrImagePull
6m50s Warning Failed pod/tcpserialpassthrough-88fb974d9-b88fc Failed to pull image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": failed to resolve reference "docker.io/pwlaptopregistry/tcpserialpassthrough:vers1.3-arm": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
6m3s Normal BackOff pod/tcpserialpassthrough-88fb974d9-b88fc Back-off pulling image "pwlaptopregistry/tcpserialpassthrough:vers1.3-arm"
3m15s Warning Failed pod/tcpserialpassthrough-88fb974d9-b88fc Error: ImagePullBackOff
Wondered if the issue is with authorization, and added based on basic auth, following this youtube guide, but the same issue persists.
Also noted that that /etc/docker/daemon.json must be edited to allow unauthorized, non-TLS connections, via:
{
"Insecure-registries": [ "<host IP>:10000" ]
}
but seemed that this needs to be done on node side, whereas nodes don't have docker cli installed??
... this is so stupid, have no idea why a domain name and port needs to be specified as the "name" of your referred registry, but anyway this solved my issue (for reference):
$cat /etc/rancher/k3s/registries.yaml
mirrors:
"<host IP>:10000":
endpoint:
- "http://<host IP>:10000"
and restarting k3s:
systemctl restart k3s
Then in your deployment, referring to that in your image path as:
apiVersion: apps/v1
kind: Deployment
metadata:
name: tcpserialpassthrough
spec:
selector:
matchLabels:
app: tcpserialpassthrough
replicas: 1
template:
metadata:
labels:
app: tcpserialpassthrough
spec:
containers:
- name: tcpserialpassthrough
image: <host IP>:10000/tcpserialpassthrough:vers1.3-arm
resources:
limits:
memory: "128Mi"
cpu: "500m"
ports:
- containerPort: 8001
hostPort: 8001
protocol: TCP
command: ["dotnet", "/app/TcpConnector.dll"]
imagePullSecrets:
- name: mydockercredentials
referring to registry's basic auth details saved as a secret:
$ kubectl create secret docker-registry mydockercredentials --docker-server host IP:10000 --docker-username username --docker-password password
You'll be able to verify the pull process via
$ kubectl get events -w
I'm trying to create a deployment on GKE (running 1.6.0) which is looking like this:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: my-api
spec:
replicas: 1
template:
metadata:
labels:
app: api
spec:
containers:
- name: api
image: eu.gcr.io/<PROJECT>/<IMAGE>:latest
imagePullPolicy: Always
ports:
- containerPort: 3000
env:
- name: NODE_ENV
value: production
resources:
requests:
cpu: 100m
Creating this fails with the following error message:
Failed to pull image "eu.gcr.io/<PROJECT>/<IMAGE>:latest": rpc error: code = 2 desc = failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/layer-629814250 /var/lib/docker/image/overlay/layerdb/sha256/bd2793152ee77e9d503e981352ff16122b220968ce9df1cc3b49b9704d7dfe28: directory not empty
Error syncing pod, skipping: failed to "StartContainer" for "api" with ErrImagePull: "rpc error: code = 2 desc = failed to register layer: rename /var/lib/docker/image/overlay/layerdb/tmp/layer-629814250 /var/lib/docker/image/overlay/layerdb/sha256/bd2793152ee77e9d503e981352ff16122b220968ce9df1cc3b49b9704d7dfe28: directory not empty"
Other deployments that look almost identical but use a different image are working as expected. What is wrong with the image I'm trying to pull? And how can I debug/fix this?
This may be caused by a known docker bug where shutdown occurs before the content is synced to disk on layer creation. The fix is included in docker v1.13.
One temporary workaround sugguested is to remove the empty files in the directory, and re-pull the image.