I've build docker image locally:
docker build -t backend -f backend.docker
Now I want to create deployment with it:
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-deployment
spec:
selector:
matchLabels:
tier: backend
replicas: 2
template:
metadata:
labels:
tier: backend
spec:
containers:
- name: backend
image: backend
imagePullPolicy: IfNotPresent # This should be by default so
ports:
- containerPort: 80
kubectl apply -f file_provided_above.yaml works, but then I have following pods statuses:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
backend-deployment-66cff7d4c6-gwbzf 0/1 ImagePullBackOff 0 18s
Before that it was ErrImagePull. So, my question is, how to tell it to use local docker images? Somewhere on the internet I read that I need to build images using microk8s.docker but it seems to be removed.
Found docs on how to use private registry: https://microk8s.io/docs/working
First it needs to be enabled:
microk8s.enable registry
Then images pushed to registry:
docker tag backend localhost:32000/backend
docker push localhost:32000/backend
And then in above config image: backend needs to be replaced with image: localhost:32000/backend
Related
I've a K8s cluster on Linode and another VM for operating.
I've installed Docker & K8s on operating VM to build images and do deployment on cluster.
Note: I haven't installed minikube on this VM.
I'm able to build my image but not able to pull that from local registry to k8s pod.
Below are the things I've already done & tried to solve the problem.
Create and push docker image to local registry.
Run docker container from the image, but not getting pulled in K8s.
Created "regcred" secret and used it in deployment yaml.
create image and push with VM's IP(10.128.234.123:5000/app-frontend) and use the same in deployment image reference.
Change image pull policy to IfNotPresent
I get the following error in pod description:
Warning ErrImageNeverPull 11s (x4 over 13s) kubelet Container image "localhost:5000/app-frontend" is not present with pull policy of Never
Warning Failed 11s (x4 over 13s) kubelet Error: ErrImageNeverPull
Below is my deployment yaml:
apiVersion: apps/v1
kind: Deployment
metadata:
name: app-frontend
labels:
app: app-frontend
spec:
replicas: 1
selector:
matchLabels:
app: app-frontend
template:
metadata:
labels:
app: app-frontend
spec:
containers:
- name: app-frontend
image: localhost:5000/docker-image
imagePullPolicy: Never
ports:
- containerPort: 80
imagePullSecrets:
- name: regcred
Any help or guidance will be grateful.
In the Docs I see this
While with imagePullPolicy set to Never, never pull the image.
Try this instead
imagePullPolicy: IfNotPresent
Also
image: localhost:5000/docker-image
But in point 4. you specify an IP
I hope it's ok to ask for your advice.
The problem in a nutshell: my pipeline cannot pull private images from GHCR.IO into Okteto Kubernetes, but public images from the same private repo work.
I'm on Windows 10 and use WSL2-Ubuntu 20.04 LTS with kinD for development and tried minikube too.
I get an error in Okteto which says that the image pull is “unauthorized” -> “imagePullBackOff”.
Things I did:browsed Stack Overflow, RTFM, Okteto FAQ, download the Okteto kubeconfig, pulled my hair out and spent more hours than I would like to admit – still no success yet.
For whatever reason I cannot create a “kubectl secret” that works. When logged-in to ghcr.io via “docker login --username” I can pull private images locally.
No matter what I’ve tried I still get the error “unauthorized” when trying to pull a private image in Okteto.
My Setup with latest updates:
Windows 10 Pro
JetBrains Rider IDE
WSL2-Ubuntu 20.04 LTS
ASP.NET Core MVC app
.NET 6 SDK
Docker
kinD
minikube
Chocolatey
Homebrew
Setup kinD
kind create cluster --name my-name
kubectl create my-namespace
// create a secret to pull images from ghcr.io
kubectl create secret docker-registry my-secret -n my-namespace --docker-username="my-username" --docker-password="my-password" --docker-email="my-email" --docker-server="https://ghcr.io"
// patch local service account
kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "my-secret"}]}'
kubernetes.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: okteto-repo
namespace: my-namespace
spec:
replicas: 1
selector:
matchLabels:
app: okteto-repo
template:
metadata:
labels:
app: okteto-repo
spec:
containers:
- name: okteto-repo
image: ghcr.io/user/okteto-repo:latest
ports:
- containerPort: 80
imagePullSecrets:
- name: my-secret
---
apiVersion: v1
kind: Service
metadata:
name: okteto-repo
annotations:
dev.okteto.com/auto-ingress: "true"
spec:
type: ClusterIP
selector:
app: okteto-repo
ports:
- protocol: TCP
port: 8080
targetPort: 80
Do you have an idea why it doesn't work and what I could do?
Thanks a lot my dear friends, every input is highly appreciated!
Hope you guys have great holidays.
Cheers,
Michael
I was able to pull a private image by doing the following:
Create a personal token in GitHub with repo access.
Build and push the image to GitHub's Container registry (I used okteto build -t ghcr.io/rberrelleza/go-getting-started:0.0.1)
Download my kubeconfig credentials from Okteto Cloud by running okteto context update-kubeconfig.
Create a secret with my credentials: kubectl create secret docker-registry gh-regcred --docker-server=ghcr.io --docker-username=rberrelleza --docker-password=ghp_XXXXXX
Patched the default account to include the secret as an image pull secret: kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "gh-regcred"}]}'
Updated the image name in the kubernetes manifest
Created the deployment (kubectl apply -f k8s.yaml)
These is what my kubernetes resources looks like, in case it helps:
# k8s.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-world
spec:
replicas: 1
selector:
matchLabels:
app: hello-world
template:
metadata:
labels:
app: hello-world
spec:
containers:
- image: ghcr.io/rberrelleza/go-getting-started:0.0.1
name: hello-world
---
apiVersion: v1
kind: Service
metadata:
name: hello-world
annotations:
dev.okteto.com/auto-ingress: "true"
spec:
type: ClusterIP
ports:
- name: "hello-world"
port: 8080
selector:
app: hello-world
# default SA
apiVersion: v1
imagePullSecrets:
- name: gh-regcred
- name: okteto-regcred
kind: ServiceAccount
metadata:
creationTimestamp: "2021-05-21T22:26:38Z"
name: default
namespace: rberrelleza
resourceVersion: "405042662"
uid: 2b6a6eef-2ce7-40d3-841a-c0a5497279f7
secrets:
- name: default-token-7tm42
A colleague created a K8s cluster for me. I can run services in that cluster without any problem. However, I cannot run services that depend on an image from Amazon ECR, which I really do not understand. Probably, I made a small mistake in my deployment file and thus caused this problem.
Here is my deployment file:
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-deployment
labels:
app: hello
spec:
replicas: 3
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: xxxxxxxxx.yyy.ecr.eu-zzzzz.amazonaws.com/test:latest
ports:
- containerPort: 5000
Here is my service file:
apiVersion: v1
kind: Service
metadata:
name: hello-svc
labels:
app: hello
spec:
type: NodePort
ports:
- port: 5000
nodePort: 30002
protocol: TCP
selector:
app: hello
On the master node, I have run this to ensure kubernetes knows about the deployment and the service.
kubectl create -f dep.yml
kubectl create -f service.yml
I used the K8s extension in vscode to check the logs of my pods.
This is the error I get:
Error from server (BadRequest): container "hello" in pod
"hello-deployment-xxxx-49pbs" is waiting to start: trying and failing
to pull image.
Apparently, pulling is an issue..... This is not happening when using a public image from the public docker hub. Logically, this would be a rights issue. But looks like it is not. I get no error message when running this command on the master node:
docker pull xxxxxxxxx.yyy.ecr.eu-zzzzz.amazonaws.com/test:latest
This command just pulls my image.
I am confused now. I can pull my image with docker pull on the master node . But K8s fails doing the pull. Am I missing something in my deployment file? Some property that says: "repositoryIsPrivateButDoNotComplain"? I just do not get it.
How to fix this so K8s can easily use my image from Amazon ECR?
You should create and use secretes for the ECR authorization.
This is what you need to do.
Create a secrete for the Kubernetes cluster, execute the below-given shell script from a machine from where you can access the AWS account in which ECR registry is hosted. Please change the placeholders as per your setup. Please ensure that the machine on which you execute this shell script should have aws cli installed and aws credential configured. If you are using a windows machine then execute this script in Cygwin or git bash console.
#!/bin/bash
ACCOUNT=<AWS_ACCOUNT_ID>
REGION=<REGION>
SECRET_NAME=<SECRETE_NAME>
EMAIL=<SOME_DUMMY_EMAIL>
TOKEN=`/usr/local/bin/aws ecr --region=$REGION --profile <AWS_PROFILE> get-authorization-token --output text --query authorizationData[].authorizationToken | base64 -d | cut -d: -f2`
kubectl delete secret --ignore-not-found $SECRET_NAME
kubectl create secret docker-registry $SECRET_NAME \
--docker-server=https://${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com \
--docker-username=AWS \
--docker-password="${TOKEN}" \
--docker-email="${EMAIL}"
Change the deployment and add a section for secrete which you're pods will be using while downloading the image from ECR.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hello-deployment
labels:
app: hello
spec:
replicas: 3
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: xxxxxxxxx.yyy.ecr.eu-zzzzz.amazonaws.com/test:latest
ports:
- containerPort: 5000
imagePullSecrets:
- name: SECRET_NAME
Create the pods and service.
IF it succeeds, then still the secret will expire in 12 hours, to overcome that setup a crone ( for recreating the secretes on the Kubernetes cluster periodically. For setting up crone use the same script which is given above.
For the complete picture of how it is happening under the hood please refer to below diagram.
Regards
Amit Meena
For 12 Hour problem, If you are using Kubernetes 1.20, Please configure and use Kubelet image credential provider
https://kubernetes.io/docs/tasks/kubelet-credential-provider/kubelet-credential-provider/
You need to enable alpha feature gate KubeletCredentialProviders in your kubelet
If using Lower Kubernetes Version and this feature is not available then use https://medium.com/#damitj07/how-to-configure-and-use-aws-ecr-with-kubernetes-rancher2-0-6144c626d42c
I want to run a private docker image on my minikube k8s .
But the pod is never able to pull my image from docker .
How can i pull private image in k8s and use it?
This my yaml for pod
{apiVersion: v1
kind: Pod
metadata:
name: privaterepo
spec:
containers:
- name: private-reg-container
image: raveena1/test
imagePullSecrets:
- name: regsecret}
The log is:-
container "private-reg-container" in pod "privaterepo" is waiting to start: trying and failing to pull image
You need to create a secret & use it in your YAML/JSON deployment file -
Create secret (Like for Docker registry, you can change the registry server URL) -
$ kubectl create secret docker-registry regsecret --docker-server=https://index.docker.io/v1/ --docker-username=$USERNM --docker-password=$PASSWD --docker-email=vivekyad4v#gmail.com
deployment.yaml (use regsecret)-
---
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: local-simple-python
spec:
replicas: 2
selector:
matchLabels:
app: local-simple-python
template:
metadata:
labels:
app: local-simple-python
spec:
containers:
- name: python
image: vivekyad4v/local-simple-python:latest
ports:
- containerPort: 8080
imagePullSecrets:
- name: regsecret
Deploy -
$ kubectl create -f deployment.yml
Your pods should now be able to fetch docker images on private registry.
You can find more info on -
https://github.com/vivekyad4v/kubernetes/tree/master/kubernetes-for-beginners
Official doc - https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
I am very new to Kubernetes and I setup Kubernetes Registry addons just copy and pasting the yaml from Kubernetes Registry Addon just a small change in ReplicationController with emptyDir
apiVersion: v1
kind: ReplicationController
metadata:
name: kube-registry-v0
namespace: kube-system
labels:
k8s-app: kube-registry-upstream
version: v0
kubernetes.io/cluster-service: "true"
spec:
replicas: 1
selector:
k8s-app: kube-registry-upstream
version: v0
template:
metadata:
labels:
k8s-app: kube-registry-upstream
version: v0
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: registry
image: registry:2
resources:
limits:
cpu: 100m
memory: 100Mi
env:
- name: REGISTRY_HTTP_ADDR
value: :5000
- name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
value: /var/lib/registry
volumeMounts:
- name: image-store
mountPath: /var/lib/registry
ports:
- containerPort: 5000
name: registry
protocol: TCP
volumes:
- name: image-store
emptyDir: {}
Then I forward the 5000 port as follows
$POD=$(kubectl get pods --namespace kube-system -l k8s-app=kube-registry-upstream \
-o template --template '{{range .items}}{{.metadata.name}} {{.status.phase}}{{"\n"}}{{end}}' \
| grep Running | head -1 | cut -f1 -d' ')
$kubectl port-forward --namespace kube-system $POD 5000:5000 &
I can push my images fine as follows
$docker tag alpine localhost:5000/nurrony/alpine
$docker push localhost:5000/nurrony/alpine
Then I write a Pod to test it like below
Version: v1
kind: Pod
metadata:
name: registry-demo
labels:
purpose: registry-demo
spec:
containers:
- name: registry-demo-container
image: localhost:5000/nurrony/alpine
command: ["printenv"]
args: ["HOSTNAME", "KUBERNETES_PORT"]
env:
- name: MESSAGE
value: "hello world"
command: ["/bin/echo"]
args: ["$(MESSAGE)"]
It is throwing an error
Failed to pull image "localhost:5000/nurrony/alpine": image pull failed for localhost:5000/nurrony/alpine:latest, this may be because there are no credentials on this request. details: (net/http: request canceled)
Any idea why is this happening? Thanks in advance.
Most likely your proxy is not working.
The Docker Registry K8S addon comes with DaemonSet which defines registry proxy for every node which runes your kubelets. What I would suggest you is to inspect those proxies since they will map Docker Registry (K8S) Service to localhost:5000 on every node.
Please note, that even if you have green check mark on your registry proxies that does not mean they work correctly. Open the logs of them and make sure that everything is working.
If your proxy is configured and you are still getting this error then most likely environment variable REGISTRY_HOST inside kube-registry-proxy is wrong. Are you using DNS here like in example? Is your DNS configured correctely? Is it working if you put this variable to ClusterIP of your service?
Also, please be aware that your RC labels need to match SVC selectors, otherwise service cannot discover your pods.
Hope it helps.