Kubernetes garbage collection clean docker components - docker

Question

Currently running a k8s cluster however occasionally I get memory issues. The following error will pop up,
Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "<web app>": Error response from daemon: devmapper: Thin Pool has 6500 free data blocks which is less than minimum required 7781 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior
I can resolve this by manually running docker ps -a -f status=exited -q | xargs -r docker rm -v
However I want Kubernetes to do this work itself. Currently in my kublet config I have:
evictionHard:
imagefs.available: 15%
memory.available: "100Mi"
nodefs.available: 10%
nodefs.inodesFree: 5%
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
What am i doing wrong?

Answer 1

Reading the error you've posted seems to me you are using "devicemapper" as storage driver.
The devicemapper storage driver is deprecated in Docker Engine 18.09, and will be removed in a future release. It is recommended that users of the devicemapper storage driver migrate to overlay2.
I should suggest you use "overlay2" as storage drive, unless you are running a non-support OS. See here the support OS versions.
You can check your actual storage drive using docker info command, you will get an output like this:
Client:
Debug Mode: false
Server:
Containers: 21
Running: 18
Paused: 0
Stopped: 3
Images: 11
Server Version: 19.03.5
Storage Driver: devicemapper <<== See here
Pool Name: docker-8:1-7999625-pool
Pool Blocksize: 65.54kB
...
>
Supposing you want to change the storage drive from devicemapper to overlay2, you need to following this steps:
Changing the storage driver makes existing containers and images inaccessible on the local system. Use docker save to save any images you have built or push them to Docker Hub or a private registry before changing the storage driver, so that you do not need to re-create them later.
Before following this procedure, you must first meet all the prerequisites.
Stop Docker.
$ sudo systemctl stop docker
Copy the contents of /var/lib/docker to a temporary location.
$ cp -au /var/lib/docker /var/lib/docker.bk
If you want to use a separate backing filesystem from the one used by /var/lib/, format the filesystem and mount it into /var/lib/docker. Make sure add this mount to /etc/fstab to make it permanent.
Edit /etc/docker/daemon.json. If it does not yet exist, create it. Assuming that the file was empty, add the following contents.
{
"storage-driver": "overlay2"
}
Docker does not start if the daemon.json file contains badly-formed JSON.
Start Docker.
$ sudo systemctl start docker
Verify that the daemon is using the overlay2 storage driver. Use the docker info command and look for Storage Driver and Backing filesystem.
Client:
Debug Mode: false
Server:
Containers: 35
Running: 15
Paused: 0
Stopped: 20
Images: 11
Server Version: 19.03.5
Storage Driver: overlay2 <=== HERE
Backing Filesystem: extfs <== HERE
Supports d_type: true
Extracted from Docker Documentation.