I'm new with k8s and I've deployed a simple nginx deployment using eks and ecr. The problem only present when I'm using ecr image, it's not happens with nginx public image from docker hub.
Here is my deployment file.
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
imagePullSecrets:
- name: *-ecr-registry
containers:
- name: nginx
# image: nginx:latest
image: *.dkr.ecr.*.amazonaws.com/nginx:dev
Here is output of a pod.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 12s default-scheduler Successfully assigned *
Normal Pulled 10s (x2 over 12s) kubelet Container image * already present on machine
Normal Created 10s (x2 over 12s) kubelet Created container nginx
Normal Started 10s (x2 over 11s) kubelet Started container nginx
Warning BackOff 8s (x2 over 9s) kubelet Back-off restarting failed container
Here is result log of the pod
exec /docker-entrypoint.sh: exec format error
I also ran the ecr image with docker in my local machine without any error.
Can someone give me some hints, how can I overcome this issue, thanks in advance.
The requested image's platform (linux/amd64) does not match the
detected host platform (linux/arm64/v8) and no specific platform was
requested
it's due to you have built the Docker image on an ARM M1 chip either on Mac and try to run it on AMD or the opposite due to that it's throwing an error.
You can build the docker image again by passing ARG --platform
--platform linux/amd64
Or you can also set the Environment variable export DOCKER_DEFAULT_PLATFORM=linux/amd64
You can pass ARG with FROM in Dockerfile or while running command docker build --platform
Dockerfile
FROM --platform=linux/amd64 node
ARG with build command
docker build -t <image-name> --platform linux/amd64
ref doc: https://docs.docker.com/engine/reference/builder/#from
You can also run the image with docker like
docker run --platform linux/amd64 node
After hours of trying many ways, I realize that I'm using base-image keymetrics/pm2:18-alpine only supports amd64 kernel. Even I tried to rebuild my image in arm64, that doesn't not work in my work nodes with arm64 kernel type. So I see 2 ways to overcome this issue is:
recreate my eks cluster using amd64 worker nodes.
Or switch to other base-image like node:18.14-alpine that supports both arm64 and amd64 kernel.
Related
I'm trying to run some docker images on Kubernetes.
docker images
master* $ docker images [15:16:49]
REPOSITORY TAG IMAGE ID CREATED SIZE
usm latest 4dd5245393bf About an hour ago 158MB
kuard latest 497961f486c7 4 days ago 22.9MB
docker container
master* $ docker ps [15:21:40]
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a46850d28303 usm "/docker-entrypoint.…" About an hour ago Up About an hour 0.0.0.0:6061->80/tcp, :::6061->80/tcp usm
88471e086486 gcr.io/k8s-minikube/kicbase:v0.0.32 "/usr/local/bin/entr…" 2 days ago Up 2 hours 127.0.0.1:49157->22/tcp, 127.0.0.1:49156->2376/tcp, 127.0.0.1:49155->5000/tcp, 127.0.0.1:49154->8443/tcp, 127.0.0.1:49153->32443/tcp minikube
Dockerfile
FROM nginx
COPY ./dist /usr/share/nginx/html
EXPOSE 80
kube version
master* $ minikube version [15:37:13]
minikube version: v1.26.0
commit: f4b412861bb746be73053c9f6d2895f12cf78565
When I run kubectl run mypod --image=usm, I get ErrImagePull
How to run the pod with the local docker image?
master* $ kubectl run mypod --image=usm
pod/mypod created
master* $ kubectl get pods [15:07:49]
NAME READY STATUS RESTARTS AGE
mypod 0/1 ErrImagePull 0 6s
I'm trying to set the imagePullPolicy to never
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
- image: usm
imagePullPolicy: Never
name: mypod
ports:
- containerPort: 80
name: http
protocol: TCP
master* $ kubectl apply -f kube-pod-usm.yaml [15:55:39]
pod/mypod created
master* $ kubectl get pods [15:55:54]
NAME READY STATUS RESTARTS AGE
mypod 0/1 ErrImageNeverPull 0 42s
You need that image to be available in someplace. Normally this is done through a registry, but for local development, you can just upload your local image to your minikukbe cluster with the following command:
minikube image load image:tag
You may also want to check minikube docker-env which allows pointing your terminal docker-cli to the docker inside minikube in a simple way.
When a Kubernetes cluster creates a new deployment or updates an existing deployment, it needs to pull an image. This is done through the kubelet process on each user node. In order for kubelets to pull this image successfully, they must be accessible from all nodes in the cluster that match the scheduling request.
Edit pod specification and provide the correct registry
If you set the Image pull Policy to Never :
the kubelet does not try fetching the image. If the image is somehow already present locally (in the local registry of Kubernetes), the kubelet attempts to start the container; otherwise, startup fails.
After building a docker image named my-http I can create a deployment from it with
kubectl create deploy http-deployment --image=my-http
This will not pull the image because imagePullPolicy is Always.
So then run
kubectl edit deploy http-deployment
and change the imagePullPolicy to Never, then it runs.
But for automation purposes I've created a yaml to create the deployment and set the imagePullPolicy at the same time.
apiVersion: apps/v1
kind: Deployment
metadata:
name: http-deployment
spec:
replicas: 3
selector:
matchLabels:
app: http
template:
metadata:
labels:
app: http
spec:
containers:
- name: my-http
image: my-http
imagePullPolicy: Never
ports:
- containerPort: 8080
Then apply -f and the pods start running but after a while a Crashloopbackoff starts with the message
container image my-http already present on machine
Apparently it has something to do with the container port but what to use for that port to get it running? There is no container running...
edit: the image already present is just informational, this is the last line in the pod description
Warning BackOff 7s (x8 over 91s) kubelet, minikube Back-off
restarting failed container
If you using kubernetes cluster your images only available on the nodes that you build the images.
You have to push images to container registries then the kubernetes will try to pull the image to node that will running the container.
If you want to run the container in the nodes that you build the images you have to use NodeSelector, or PodAffinity.
https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
Your image is probably private image which Kubernetes can't pull if you didn't specify imagePullSecrets.
This shouldn't be the problem however, because imagePullPolicy: Never would just use the image on the nodes. You can diagnose real problem by either kubectl describe pod pod_name or getting logs of the previous pod with --previous flag because newer pod may not have encountered the problem.
I have a problem to deploy docker image via kubernetes.
One issue is that, we cannot use any docker image registry service e.g. docker hub or any cloud services. But, yes I have docker images as .tar file.
However, it always fails with following message
Warning Failed 1s kubelet, dell20
Failed to pull image "test:latest": rpc
error: code = Unknown
desc = failed to resolve image "docker.io/library/test:latest":
failed to do request: Head https://registry-1.docker.io/v2/library/test/manifests/latest: dial tcp i/o timeout
I also change deployment description with IfNotPresent or Never. In this case it will fail anyway with ErrImageNeverPull.
My guess is: kubernetes tries to use Docker Hub anyway, since it https://registry-1.docker.io in order to pull the image. I just want to use tar docker image in local disk, rather than pulling from some services.
And yes the image is in docker:
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
test latest 9f4916a0780c 6 days ago 1.72GB
Can anyone give me any advices on this problem?
I was successful with using local image with Kubernetes cluster. I provided the explanation with example below:
The only prerequisite is that you need to make sure you have access to upload this image directly to nodes.
Create the image
Pull the default nginx image from docker registry with below command:
$ docker pull nginx:1.17.5
Nginx image is used only for demonstration purposes.
Tag this image with new name as nginx-local with command:
$ docker tag nginx:1.17.5 nginx-local:1.17.5
Save this image as nginx-local.tar executing command:
$ docker save nginx-local:1.17.5 > nginx-local.tar
Link to documentation: docker save
File nginx-local.tar is used as your image.
Copy the image to all of the nodes
The problem with this technique is that you need to ensure all of the nodes have this image.
Lack of image will result in failed pod creation.
To copy it you can use scp. It's secure way to transer files between machines.
Example command for scp:
$ scp /path/to/your/file/nginx-local.tar user#ip_adddress:/where/you/want/it/nginx-local.tar
If image is already on the node, you will need to load it into local docker image repository with command:
$ docker load -i nginx-local.tar
To ensure that image is loaded invoke command
$ docker images | grep nginx-local
Link to documentation: docker load:
It should show something like that:
docker images | grep nginx
nginx-local 1.17.5 540a289bab6c 3 weeks ago 126MB
Creating deployment with local image
The last part is to create deployment with use of nginx-local image.
Please note that:
The image version is explicitly typed inside yaml file.
ImagePullPolicy is set to Never. ImagePullPolicy
Without this options the pod creation will fail.
Below is example deployment which uses exactly that image:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-local
namespace: default
spec:
selector:
matchLabels:
run: nginx-local
replicas: 5
template:
metadata:
labels:
run: nginx-local
spec:
containers:
- image: nginx-local:1.17.5
imagePullPolicy: Never
name: nginx-local
ports:
- containerPort: 80
Create this deployment with command:
$ kubectl create -f local-test.yaml
The result was that pods were created successfully as shown below:
NAME READY STATUS RESTARTS AGE
nginx-local-84ddb99b55-7vpvd 1/1 Running 0 2m15s
nginx-local-84ddb99b55-fgb2n 1/1 Running 0 2m15s
nginx-local-84ddb99b55-jlpz8 1/1 Running 0 2m15s
nginx-local-84ddb99b55-kzgw5 1/1 Running 0 2m15s
nginx-local-84ddb99b55-mc7rw 1/1 Running 0 2m15s
This operation was successful but I would recommend you to use local docker repository. It will easier management process with images and will be inside your infrastructure.
Link to documentation about it: Local Docker Registry
Im trying to create a pod using my local docker image as follow.
1.First I run this command in terminal
eval $(minikube docker-env)
2.I created a docker image as follow
sudo docker image build -t my-first-image:3.0.0 .
3.I created the pod.yml as shown below and I run this command
kubectl -f create pod.yml.
4.then i tried to run this command
kubectl get pods
but it shows following error
NAME READY STATUS RESTARTS AGE
multiplication-b47499db9-phpb7 0/1 ImagePullBackOff 0 23h
my-first-pod 0/1 ErrImagePull 0 7s
5.i get the pods logs
kubectl describe pod my-first-pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 99s default-scheduler Successfully assigned default/my-first-pod to minikube
Warning Failed 41s (x3 over 94s) kubelet, minikube Failed to pull image "my-first-image:3.0.0": rpc error: code = Unknown desc = Error response from daemon: pull access denied for my-first-image, repository does not exist or may require 'docker login'
Warning Failed 41s (x3 over 94s) kubelet, minikube Error: ErrImagePull
Normal BackOff 12s (x4 over 93s) kubelet, minikube Back-off pulling image "my-first-image:3.0.0"
Warning Failed 12s (x4 over 93s) kubelet, minikube Error: ImagePullBackOff
Normal Pulling 0s (x4 over 98s) kubelet, minikube pulling image "my-first-image:3.0.0"
Dockerfile
FROM node:carbon
WORKDIR /app
COPY . .
CMD [ "node", "index.js" ]
pods.yml
kind: Pod
apiVersion: v1
metadata:
name: my-first-pod
spec:
containers:
- name: my-first-container
image: my-first-image:3.0.0
index.js
var http = require('http');
var server = http.createServer(function(request, response) {
response.statusCode = 200;
response.setHeader('Content-Type', 'text/plain');
response.end('Welcome to the Golden Guide to Kubernetes
Application Development!');
});
server.listen(3000, function() {
console.log('Server running on port 3000');
});
Reason
This is because it can't download the docker image defined in your pod definition file. By default it downloads required images from DockerHub.
Way 1
So after creating your my-first-image:3.0.0 image you have to publish it at DockerHub. For that create an account at DockerHub and login from terminal using login command
sudo docker login
After successful login, rebuild your docker image with your DockerHub username in tag and push it to DockerHub (more details)
sudo docker image build -t YOUR_DOCKERHUB_USERNAME/my-first-image:3.0.0 .
sudo docker push YOUR_DOCKERHUB_USERNAME/my-first-image:3.0.0
Update your image in pod.yml as YOUR_DOCKERHUB_USERNAME/my-first-image:3.0.0 and create your pods as before.
Way 2
You can instruct to find required docker image from your local machine instead of downloading from DockerHub. To do so you have to add imagePullPolicy: Never in your pod.yml file under specific container description. Below is an example of your pod.yml file to show where to define image pull policy
kind: Pod
apiVersion: v1
metadata:
name: my-first-pod
spec:
containers:
- name: my-first-container
image: YOUR_DOCKERHUB_USERNAME/my-first-image:3.0.0
imagePullPolicy: Never
Since you are using an image without uploading it .You will have to set the imagePullPolicy to Never, otherwise Kubernetes will try to download the image.
Start minikube
minikube start
Set docker env
eval $(minikube docker-env)
Build image
docker build -t my-first-image:3.0.0 .
Run in minikube
kubectl run my-first-container --image=my-first-image:3.0.0 --image-pull-policy=Never
Check that it's running
kubectl get pods
Your pod spec should be like below
pods.yml
kind: Pod
apiVersion: v1
metadata:
name: my-first-pod
spec:
containers:
- name: my-first-container
image: my-first-image:3.0.0
imagePullPolicy: Never
You have build the image but you need to push to docker repository.
In case you are building docker image on kubernetes node, it can work but better to push to repository so docker will pull from central repo, as the number of nodes increases you cant control where pod will invoke and image will be unavailable.
docker push -t my-first-image:3.0.0
In case you have private docker repositiory, push to private repository and use full qualified image name.
repository name is missing.
create an account in dockerHub.
then build and push the image to your repo in dockerhub
sudo docker image build -t niranga/my-first-image:3.0.0 .
sudo docker login
sudo docker push niranga/my-first-image:3.0.0
update the image name in pod.yaml file as niranga/my-first-image:3.0.0
it should work
When you build image to local Minikube registry you might want to set imagePullPolicy: Never so that it will not try pulling the image from remote registry.
The pod spec would look something like that:
kind: Pod
apiVersion: v1
metadata:
name: my-first-pod
spec:
containers:
- name: my-first-container
image: my-first-image:3.0.0
imagePullPolicy: Never
I have successfully built Docker images and ran them in a Docker swarm. When I attempt to build an image and run it with Docker Desktop's Kubernetes cluster:
docker build -t myimage -f myDockerFile .
(the above successfully creates an image in the docker local registry)
kubectl run myapp --image=myimage:latest
(as far as I understand, this is the same as using the kubectl create deployment command)
The above command successfully creates a deployment, but when it makes a pod, the pod status always shows:
NAME READY STATUS RESTARTS AGE
myapp-<a random alphanumeric string> 0/1 ImagePullBackoff 0 <age>
I am not sure why it is having trouble pulling the image - does it maybe not know where the docker local images are?
I just had the exact same problem. Boils down to the imagePullPolicy:
PC:~$ kubectl explain deployment.spec.template.spec.containers.imagePullPolicy
KIND: Deployment
VERSION: extensions/v1beta1
FIELD: imagePullPolicy <string>
DESCRIPTION:
Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always
if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated.
More info:
https://kubernetes.io/docs/concepts/containers/images#updating-images
Specifically, the part that says: Defaults to Always if :latest tag is specified.
That means, you created a local image, but, because you use the :latest it will try to find it in whatever remote repository you configured (by default docker hub) rather than using your local. Simply change your command to:
kubectl run myapp --image=myimage:latest --image-pull-policy Never
or
kubectl run myapp --image=myimage:latest --image-pull-policy IfNotPresent
I had this same ImagePullBack error while running a pod deployment with a YAML file, also on Docker Desktop.
For anyone else that finds this via Google (like I did), the imagePullPolicy that Lucas mentions above can also be set in the deployment yaml file. See the spec.templage.spec.containers.imagePullPolicy in the yaml snippet below (3 lines from the bottom).
I added that and my app deployed successfully into my local kube cluser, using the kubectl yaml deploy command: kubectl apply -f .\Deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app-deployment
labels:
app: web-app
spec:
replicas: 3
selector:
matchLabels:
app: web-app
template:
metadata:
labels:
app: web-app
spec:
containers:
- name: web-app
image: node-web-app:latest
imagePullPolicy: Never
ports:
- containerPort: 3000
You didn't specify where myimage:latest is hosted, but essentially ImagePullBackoff means that I cannot pull the image because either:
You don't have networking setup in your Docker VM that can get to your Docker registry (Docker Hub?)
myimage:latest doesn't exist in your registry or is misspelled.
myimage:latest requires credentials (you are pulling from a private registry). You can take a look at this to configure container credentials in a Pod.