A node of my k8s cluster has GC trying to remove images used by a container.
This behaviour seems strange to me.
Here the logs:
kubelet: I1218 12:44:19.925831 11177 image_gc_manager.go:334] [imageGCManager]: Removing image "sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2" to free 746888 bytes
kubelet: E1218 12:44:19.928742 11177 remote_image.go:130] RemoveImage "sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container 6f236a385a8e
kubelet: E1218 12:44:19.928793 11177 kuberuntime_image.go:126] Remove image "sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container 6f236a385a8e
kubelet: W1218 12:44:19.928821 11177 eviction_manager.go:435] eviction manager: unexpected error when attempting to reduce nodefs pressure: wanted to free 9223372036854775807 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container 6f236a385a8e
Any suggestions?
May a manual remove of docker images and stopped containers on a node cause such a problem?
Thank you in advance.
What you've encountered is not the regular Kubernetes garbage collection that deleted orphaned API resource objects, but the kubelet's Image collection.
Whenever a node experiences Disk pressure, the Kubelet daemon will desperately try to reclaim disk space by deleting (supposedly) unused images. Reading the source code shows that the Kubelet sorts the images to remove by the time since they have last been used for creating a Pod -- if all images are in use, the Kubelet will try to delete them anyways and fail (which is probably what happened to you).
You can use the Kubelet's --minimum-image-ttl-duration flag to specify a minimum age that an image needs to have before the Kubelet will ever try to remove it (although this will not prevent the Kubelet from trying to remove used images altogether). Alternatively, see if you can provision your nodes with more disk space for images (or build smaller images).
As I understand, Kubelet has a garbage collector and the purpose of it to remove unnecessary k8s objects for utilising resources.
If the object does not belong to any owner it means its orphaned. There is a pattern in Kubernetes which is known as ownership in kubernetes.
For Instance, If you apply the deployment object then it will create a replicaSet object, further ResplicaSet will create pods objects.
So Ownership flow
Deployment <== RepicaSet <=== Pod
Now if you delete Deployment object which means ReplicaSet does not have an owner then Garbage collector will try to remove ReplicaSet and now Pods do not have owner, therefore, GC will try to remove pods.
There is a field called ownerReferences which describe the relationship among all of these Objects such as Deployment, ReplicaSet, Pods etc.
There are 3 ways to delete objects in Kubernetes.
Foreground: If you try to delete Deployment, First pods will be deleted then replicaset after that deployment will be removed.
Background: If you try to delete Deployment, First Deployment will be deleted Now GC will remove replicasets and pods.
Orphan: If you remove Deployment, then Repicaset will be Orphaned and GC will remove all of these orphaned objects.
Solutions to your issues
It seems to me that your pod (containers) is orphaned, therefore, GC is making sure that it is removed from the cluster.
If you want to check ownerRererences status :
kubectl get pod $PODNAME -o yaml
In the metadata sections, there will be an adequate information.
I have attached references for further research.
garbage-collection
garbage-collection-k8s
Related
I am deploying an app on k3s with containerd (not docker)
am trying to deploy a local image
the image exists but the given error
said image doesn't
docker.io/kubernetesui/metrics-scraper v1.0.8 115053965e86b 19.7MB
**docker.io/library/laravel-example-project-fpm latest** ac83c8e9c86fd 378MB
docker.io/library/laravel-example-project-nginx latest 6403f5d7120c6 24.2MB
rn-manager v1.1.0 6a6ea3dbf71b9 116MB
docker.io/monachus/rancher-demo latest 65053fcc5f37a 7.01MB
docker.io/rancher/klipper-lb v0.3.5 dbd43b6716a08 3.33MB
docker.io/rancher/local-path-provisioner v0.0.21 fb9b574e03c34 11.4MB
docker.io/rancher/mirrored-coredns-coredns 1.9.1 99376d8f35e0a 14.1MB
docker.io/rancher/mirrored-library-traefik 2.6.2 72463d8000a35 30.3MB
docker.io/rancher/mirrored-metrics-server v0.5.2 f73640fb50619 26MB
docker.io/rancher/mirrored-pause 3.6 6270bb605e12e 301kB
root#u-server-master:~#
kubectl describe pod laravel-example-project-5597cdcc97-bv6fs
kubectl describe pod laravel-example-project-5597cdcc97-bv6fs
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 45m default-scheduler Successfully assigned default/laravel-example-project-5597cdcc97-bv6fs to zakham
Warning Failed 42m (x12 over 45m) kubelet Error: ErrImageNeverPull
Warning ErrImageNeverPull 4m57s (x187 over 45m) kubelet Container image "**laravel-example-project-fpm:latest**" is not present with pull policy of Never
kubectl get pods
laravel-example-project-5597cdcc97-26scj........ 0/2...........**Init:ErrImageNeverPull**.......0................ 26m
SOLVED
i am using a local image with containerd i dont need to pull it, i solved it by specifing the node name nodeName because i noticed that when the pod is scheduled to the master node it run without error, so i specified that nodeName in the deployment
You get this error because the image is never pulled to the node, and you are using a never imagePullPolicy, which means that K8S will not pull the image if it doesn't exist or if there is a new version on the docker registry.
To solve this problem, you can use imagePullPolicy IfNotPresent to pull the image if it doesn't exist or Always to pull the image at every pod creation.
FIRST I SOLVED it like this i am using a local image with containerd i dont need to pull it, i solved it by specifing the node name nodeName because i noticed that when the pod is scheduled to the master node it run without error, so i specified that nodeName in the deployment
FINALLY I SOLVED IT LIKE THIS
i just imported my image to all workers nodes and delete that nodeName parameter from the pod and it worked like a charm
Since your image is local and/or you can be using a local registry the pull policy Never is fine. Also double check if there is any valid reason to not get the image runnning on any other node.
Try to not use a nodeName unless it has for example dedicated hardware among other things. Theres a big chance to have a taint on your Master.
Check the taints on master node if you want to schedule tasks there.
I would recommend to take a look here for further info: Taints and Tolerations
Pod Priority and Preemption
Node-pressure Eviction
A while ago i solved something really similar using antiaffinity, this video explains perfectly how to achieve it.
Background:
I have a GKE cluster which has suddenly stopped being able to pull my docker images from GCR; both are in the same GCP project. It has been working well for several months, no issues pulling images, and has now started throwing errors without having made any changes.
(NB: I'm generally the only one on my team who accesses Google Cloud, though it's entirely possible that someone else on my team may have made changes / inadvertently made changes without realising).
I've seen a few other posts on this topic, but the solutions offered in others haven't helped. Two of these posts stood out to me in particular, as they were both posted around the same day my issues started ~13/14 days ago. Whether this is coincidence or not who knows..
This post has the same issue as me; unsure whether the posted comments helped them resolve, but it hasn't fixed for me. This post seemed to also be the same issue, but the poster says it resolved by itself after waiting some time.
The Issue:
I first noticed the issue on the cluster a few days ago. Went to deploy a new image by pushing image to GCR and then bouncing the pods kubectl rollout restart deployment.
The pods all then came back with ImagePullBackOff, saying that they couldn't get the image from GCR:
kubectl get pods:
XXX-XXX-XXX 0/1 ImagePullBackOff 0 13d
XXX-XXX-XXX 0/1 ImagePullBackOff 0 13d
XXX-XXX-XXX 0/1 ImagePullBackOff 0 13d
...
kubectl describe pod XXX-XXX-XXX:
Normal BackOff 20s kubelet Back-off pulling image "gcr.io/<GCP_PROJECT>/XXX:dev-latest"
Warning Failed 20s kubelet Error: ImagePullBackOff
Normal Pulling 8s (x2 over 21s) kubelet Pulling image "gcr.io/<GCP_PROJECT>/XXX:dev-latest"
Warning Failed 7s (x2 over 20s) kubelet Failed to pull image "gcr.io/<GCP_PROJECT>/XXX:dev-latest": rpc error: code = Unknown desc = failed to pull and unpack image "gcr.io/<GCP_PROJECT>/XXX:dev-latest": failed to resolve reference "gcr.io/<GCR_PROJECT>/XXX:dev-latest": unexpected status code [manifests dev-latest]: 403 Forbidden
Warning Failed 7s (x2 over 20s) kubelet Error: ErrImagePull
Troubleshooting steps followed from other posts:
I know that the image definitely exists in GCR -
I can pull the image to my own machine (also removed all docker images from my machine to confirm it was really pulling)
I can see the tagged image if I look on the GCR UI on chrome.
I've SSH'd into one of the cluster nodes and tried to docker pull manually, with no success:
docker pull gcr.io/<GCP_PROJECT>/XXX:dev-latest
Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication
(Also did a docker pull of a public mongodb image to confirm that was working, and it's specific to GCR).
So this leads me to believe it's an issue with the service account not having the correct permissions, as in the cloud docs under the 'Error 400/403' section. This seems to suggest that the service account has either been deleted, or edited manually.
During my troubleshooting, I tried to find out exactly which service account GKE was using to pull from GCR. In the steps outlined in the docs, it says that: The name of your Google Kubernetes Engine service account is as follows, where PROJECT_NUMBER is your project number:
service-PROJECT_NUMBER#container-engine-robot.iam.gserviceaccount.com
I found the service account and checked the polices - it did have one for roles/container.serviceAgent, but nothing specifically mentioning kubernetes as I would expect from the description in the docs.. 'the Kubernetes Engine Service Agent role' (unless that is the one they're describing, in which case I'm no better off that before anyway..).
Must not have had the correct roles, so I then followed the steps to re-enable (disable then enable the Kubernetes API). Running cloud projects get-iam-policy <GCP_PROJECT> again and diffing the two outputs (before/after), the only difference is that a service account for '#cloud-filer...' has been deleted.
Thinking maybe the error was something else, I thought I would try spinning up a new cluster. Same error - can't pull images.
Send help..
I've been racking my brains to try to troubleshoot, but I'm now out of ideas! Any and all help much appreciated!
I believe the correct solution is to add the "roles/artifactregistry.reader" role to the service account that the node pool is configured to use.
In terraform that can be done by
resource "google_project_iam_member" "allow_image_pull" {
project = var.project_id
role = "roles/artifactregistry.reader"
member = "serviceAccount:${var.service_account_email}"
}
I don't know if it still helps, but I had the same issue and managed to fix it.
In my case I was deploying GKE trough terraform and did not specify oauth_scope property for node pool as show in example. As I understand you need to make gcp APIs available here to make nodes able to use them.
Have now solved this.
The service account had the correct roles/permissions, but for whatever reason stopped working.
I manually created a key for that service account, added that secret into the kube cluster, and set the service account to use that key.
Still at a loss as to why it wasn't already doing this, or why it stopped working in the first place all of a sudden, but it's working...
Fix was from this guide, from the section starting 'Create & use GCR credentials'.
From the docs compute engine default service account accesses container registry for pulling image not the kubernetes engine service account.You can go to node pool and check the service account name in the security section.Check the access logs of the service account to see errors and then provide necessary permission to the service account.
In my case worked re-add (i.e. deletion and then addition) role "Artifact registry reader" for serviceaccount used by cluster.
I tried simple PVC example from here with nginx claiming azure-managed-disk and I getting 'unable to mount' error, see below. Also I can't remove the created PV with 'kubectl delete pv pvc-3f3c3c78-9779-11e9-a7eb-1aafd0e2f988'.
$kubectl get events
LAST SEEN TYPE REASON KIND MESSAGE
10m Warning FailedMount Pod MountVolume.WaitForAttach failed for volume "pvc-3f3c3c78-9779-11e9-a7eb-1aafd0e2f988" : azureDisk - WaitForAttach failed within timeout node (aks-agentpool-10844952-2) diskId:(kubernetes-dynamic-pvc-3f3c3c78-9779-11e9-a7eb-1aafd0e2f988) lun:(1)
22s Warning FailedMount Pod Unable to mount volumes for pod "nginx_default(bd16b9c8-97b2-11e9-9018-eaa2ea1705c5)": timeout expired waiting for volumes to attach or mount for pod "default"/"nginx". list of unmounted volumes=[volume]. list of unattached volumes=[volume default-token-92rj6]
My managed aks cluster is using v1.12.8 , SP has contributor role (owner role doesn't not help too). There is storage class 'managed-premium', in the yaml from my simple nginx example (link provided).
For your issue, there are no more details to judge the exact reason. Just list the possible reason here.
It's just a simple error that it's failed when the API call to Azure. If so, you just need to delete them and recreate again.
The node that the pod run in already has too many Azure disks attached. If so, you need to schedule the pod run in another node which does not attach to many disks.
The Azure disk cannot be unmounted or detached from the old node. It means that the PV is in use and attach to another node. If so, you need to create another dynamic PV that does not in use for your pod.
You can check carefully again according to these reasons. In my opinion, the third reason is the most possible one. Of curse, it all dependents on the actual situation. For more details about the similar errors, see How to Understand & Resolve “Warning Failed Attach Volume” and “Warning Failed Mount” Errors in Kubernetes on Azure.
During image upgrade of pods few of the pods are stuck in ContainerCreating state.
kubectl get events has below error: FailedSync kubelet,
10.102.10.34 Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod
"default"/"ob-service-1124355621-1th47". list of unattached/unmounted
volumes=[timezone default-token-3x1x9]
Docker Logs :
^[[31mERRO^[[0m[240242] Handler for DELETE /v1.22/containers/749d05b355e2b80bffb90d207232d37e3ebc5ff57942c46ce0a2b4ca5950ed0e returned error: Driver devicemapper failed to remove root filesystem 749d05b355e2b80bffb90d207232d37e3ebc5ff57942c46ce0a2b4ca5950ed0e: Device is Busy
^[[31mERRO^[[0m[240242] Error saving dying container to disk: open /var/lib/docker/containers/5d01db2c31a3073cc7fb68f2be5acc45c34583d5f2ae0c0879ec064f90da6943/config.v2.json: no such file or directory
^[[31mERRO^[[0m[240263] Error removing mounted layer 5d01db2c31a3073cc7fb68f2be5acc45c34583d5f2ae0c0879ec064f90da6943: Device is Busy
it's a bit hard to debug with just the information you provided, but the general direction you should be looking into is resources of your cluster.
failed to sync usually means the pods can't be fit into any of the workers (maybe adding more will help) or from your error seems like you're trying to "connect" to volumes that are busy and can't accept the connection which fails the pod.
Again lacking details, but let's assume you're on AWS and you have a volume that didn't dismount and now you're trying to connect to it again - the above would be the result pretty much, so you'll need to detach the volume so the new pod can connect to it.
if you say there are some pods that are okay with the same image it means you don't have enough volumes and/or some of the current volumes are not available to accept new connection (maybe during the deletion of the old pods they didn't dismount properly)
Apparently the GC of my Kubernetes cluster is failing to delete any image and the server is getting to full-disk.
Can you please guide me on where to find the logs for the ImageGC with the error trying to delete the images or to a reason of why this is happening?
3m 5d 1591 ip-xxx.internal Node Warning FreeDiskSpaceFailed {kubelet ip-xxx.internal} failed to garbage collect required amount of images. Wanted to free 6312950988, but freed 0
3m 5d 1591 ip-xxx.internal Node Warning ImageGCFailed {kubelet ip-xxx.internal} failed to garbage collect required amount of images. Wanted to free 6312950988, but freed 0
Thanks!
There may not be much in the way of logs (see this issue) but there may be Kubernetes event data. Look for events of type ImageGCFailed.
Alternatively you could check the cadvisor Prometheus metrics to see if it exposes any information about container garbage collecton.
Docs on the GC feature in general: https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/
most likely your host file system is full, can you check /var file system usage.
you can use docker-gc to cleanup old image.
https://github.com/spotify/docker-gc
Run it like this
docker run --rm --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc:ro spotify/docker-gc