I loaded some docker images running
docker load --input <file>
I can then see these images when executing
docker image ls
After a while images start disappearing. Every few minutes there are less and less images listed. I did not run any of images yet. What could be the cause of this issue?
EDIT: This issue arises with docker inside minikube VM.
Since you've mentioned that Docker daemon runs inside minikube VM, I assume that you might hit K8s Garbage collection mechanism, which keeps system utilization on appropriate level and reduce amount of unused containers(built from images) by adjusting the specific thresholds.
These eviction thresholds are fully managed by Kubelet k8s node agent, cleaning uncertain images and containers according to the parameters(flags) propagated in kubelet configuration file.
Therefore, I guess you can investigate K8s eviction behavior looking at the certain thresholds, adjusted in kubelet config file which is generated by minikube bootstrapper in the following path /var/lib/kubelet/config.yaml.
As mention in #mk_sta answer to fix issue you need:
Create or edit /var/lib/kubelet/config.yaml with
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
imagefs.available: "5%"
Default value is 15%
minikube stop
minikube start --extra-config=kubelet.config=/var/lib/kubelet/config.yaml
Or free more space on docker partition.
https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/#create-the-config-file
https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#hard-eviction-thresholds
Related
I've been having loads of issues with kubectl not having enough space. How can I increase the default storage size allocated to docker?
None of minikube recommendations worked.
1. Run "docker system prune" to remove unused Docker data (optionally with "-a")
2. Increase the storage allocated to Docker for Desktop by clicking on:
Docker icon > Preferences > Resources > Disk Image Size
3. Run "minikube ssh -- docker system prune" if using the Docker container runtime
And the second is not possible from command line...
Taking your comment into consideration
I get ImagePullBackOff when I try to deploy nginx on the cluster –
Caterina
You can specify minikube's disk allocations separately:
minikube start --memory=8192 --cpus=4 --disk-size=50g
Which can help you to work around the disk space issues as the default is significantly smaller: --disk-size string Disk size allocated to the minikube VM (format: <number>[<unit>], where unit = b, k, m or g). (default "20000mb")
I use minikube with Docker driver on Linux. For a manual workflow I can enable registry addon in minikube, push there my images and refer to them in deployment config file simply as localhost:5000/anything. Then they are pulled to a minikube's environment by its Docker daemon and deployments successfully start in here. As a result I get all the base images saved only on my local device (as I build my images using my local Docker daemon) and minikube's environment gets cluttered only by images that are pulled by its Docker daemon.
Can I implement the same workflow when use Skaffold? By default Skaffold uses minikube's environment for both building images and running containers out of them, and also it duplicates (sometimes even triplicates) my images inside minikube (don't know why).
Skaffold builds directly to Minikube's Docker daemon as an optimization so as to avoid the additional retrieve-and-unpack required when pushing to a registry.
I believe your duplicates are like the following:
$ (eval $(minikube docker-env); docker images node-example)
REPOSITORY TAG IMAGE ID CREATED SIZE
node-example bb9830940d8803b9ad60dfe92d4abcbaf3eb8701c5672c785ee0189178d815bf bb9830940d88 3 days ago 92.9MB
node-example v1.17.1-38-g1c6517887 bb9830940d88 3 days ago 92.9MB
Although these images have different tags, those tags are just pointers to the same Image ID so there is a single image being retained.
Skaffold normally cleans up left-over images from previous runs. So you shouldn't see the minikube daemon's space continuously growing.
An aside: even if those Image IDs were different, an image is made up of multiple layers, and those layers are shared across the images. So Docker's reported image sizes may not actually match the actual disk space consumed.
An error occurred because there is not enough disk space
I decided to check how much is free and came across this miracle
Cleaned up via docker system prune -a and
docker container prune -f
docker image prune -f
docker system prune -f
But only 9GB was cleared
Prune removes containers/images that have not been used for a while/stopped. I would suggest you do a docker ps -a and then remove/stop all the containers that you don't want with docker stop <container-id>, and then move on to remove docker images by docker images ps and then remove them docker rmi <image-name>
Once you have stooped/removed all the unwanted containers run docker system prune --volumes to remove all the volumes/cache and dangling images.
Don't forget to prune the volumes! Those almost always take up way more space than images and containers. docker volume prune. Be careful if you have anything of value stored in them though.
It could be your logging of the running containers. I've seen Docker logging writing disks full with logging. By default Docker containers can write unlimited logs.
I always add logging configuration to my docker-compose restrict total size. Docker Logging Configuration.
From the screenshot I think there's some confusion on what's taking up how much space. Overlay filesystems are assembled from directories on the parent filesystem. In this case, that parent filesystem is within /var/lib/docker which is part of / in your example. So the df output for each overlay filesystem is showing how much disk space is used/available within /dev/vda2. Each container isn't using 84G, that's just how much is used in your entire / filesystem.
To see how much space is being used by docker containers, images, volumes, etc, you can use docker system df. If you have running containers (likely from the above screenshot), docker will not clean those up, you need to stop them before they are eligible for pruning. Once containers have been deleted, you can then prune images and volumes. Note that deleting volumes deletes any data you were storing in that volume, so be sure it's not data you wanted to save.
It's not uncommon for docker to use a lot of disk space from downloading lots of images (docker makes it easy to try new things) and those are the easiest to prune when the containers are stopped. However what's harder to see are the logs that containers are outputting which will slowly fill the drive within the containers directory. For more details on how to clean the logs automatically, see this answer.
If you want, you could dig deep at granular level and pinpoint the file(s) which are the cause of this much disk usage.
du -sh /var/lib/docker/overley2/<container hash>/merged/* | sort -h
this would help you coming to a conclusion much easily.
I need to understand how we can delete a docker image using Kubernetes. I'm using Jenkins for pipeline automation and Jenkins only has access to the master node, not the slaves. So when I generate the deploy everything works fine, the deploy makes the slaves pull from the repository and I get everything going.
But if Jenkins kills the deploy and tries to remove the image, it only deletes the image on the master node and not on other slaves. So I don't want to manually delete the image.
Is there a way to delete images on slave nodes from the master node?
Kubernetes is responsible for deleting images, It is kubelet that makes garbage collection on nodes, including image deletion, it is customizable.
Deleting images by external methods is not recomended as these tools can potentially break the behavior of kubelet by removing containers expected to exist.
Kubelet verify if the storage available for the images is more than 85% full, in that case it delete some images to make room. Min and Max threshold can be customized in the file
/var/lib/kubelet/config.yaml
imageGCHighThresholdPercent is the percent of disk usage after which image garbage collection is always run.
ImageGCHighThresholdPercent is the percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to.
Default values are :
ImageGCHighThresholdPercent 85
ImageGCLowThresholdPercent 80
Kubelet does eventually garbage collect old unused images when disk usage gets beyond 85% (default setting). Otherwise kubernetes doesn't provide a remote interface for deleting images.
To do it would require something like an SSH to each node or to run a Daemonset that has permissions to manage the underlying container runtime. See Mr.Axe answer for the Docker/SSH variant.
From master node,
You can write shell script/ansible playbook to ssh into slave nodes and remove images-
docker images -f "dangling=true" -q | xargs --no-run-if-empty docker rmi
I am getting my hands dirty with Containers and Docker. In the world of Docker (or the world of Containers in general), I am trying to understand what "removing a container" means. The mental model I have of a container is that it is a running instance of an image. If a container has stopped executing, what resources is it consuming that need to be freed? I can understand removing the associated image(s) as they consume disk space. Maybe my mental model of a container as "a running instance of an image" is not correct, and there is more to it. If someone could shed some light, I would greatly appreciate it.
You are nearly there.
In Docker when a container is run, the docker daemon mounts the images layers using a unionfs. It adds a layer on top of the images layers to track changes that are happening inside of that container. Say a log file is written that is tracked as part of this layer.
By removing a container you remove the resources use to track this layer of changes that the container has done on top of the image.
When a container is running it consumes CPU, memory and the HDD space associated with this layer.
When a container is stopped, the CPU and memory are released but the HDD space is still used. (You can check stopped containers using docker ps -a; this will show you all the containers across all states)
When a stopped container is removed the HDD space is also freed up.
Docker also removes all the meta-data(when it was started, where its files are etc) associated with the container when you remove the container.
To have some more fun with this do this:
docker inspect <image_name>:<tag> and see its output.
docker inspect <containerid> and see its output, here containerid should be of a container running off the above image. And see if you can figure out the layers and cross relate them across the image and container.