COS is running out of inodes for /var/lib/docker volume - docker

Question

I'm trying to us cos to run some services on GCP.
One of the issues I'm seeing currently is that the VMs I've started very quickly seem to run out of inodes for the /var/lib/docker filesystem. I'd have expected this to be one of the things tuned in a container optimized os?
wouter#nbwm-cron ~ $ df -hi
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 78K 13K 65K 17% /
devtmpfs 463K 204 463K 1% /dev
tmpfs 464K 1 464K 1% /dev/shm
tmpfs 464K 500 463K 1% /run
tmpfs 464K 13 464K 1% /sys/fs/cgroup
tmpfs 464K 9 464K 1% /mnt/disks
tmpfs 464K 16K 448K 4% /tmp
/dev/sda8 4.0K 11 4.0K 1% /usr/share/oem
/dev/sda1 1013K 998K 15K 99% /var
tmpfs 464K 45 464K 1% /var/lib/cloud
overlayfs 464K 39 464K 1% /etc
wouter#nbwm-cron ~ $ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
<name>/stackdriver-agent latest 0c4b075e7550 3 days ago 1.423 GB
<none> <none> 96d027d3feea 4 days ago 905.2 MB
gcr.io/<project>/nbwm-ops/docker-php5 latest 5d2c59c7dd7a 2 weeks ago 1.788 GB
nbwm-cron wouter # tune2fs -l /dev/sda1
tune2fs 1.43.3 (04-Sep-2016)
Filesystem volume name: STATE
Last mounted on: /var
Filesystem UUID: ca44779b-ffd5-405a-bd3e-528071b45f73
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Remount read-only
Filesystem OS type: Linux
Inode count: 1036320
Block count: 4158971
Reserved block count: 0
Free blocks: 4062454
Free inodes: 1030756
First block: 0
Block size: 4096
Fragment size: 4096
Group descriptor size: 64
Reserved GDT blocks: 747
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 8160
Inode blocks per group: 510
Flex block group size: 16
Filesystem created: Thu Jun 15 22:39:33 2017
Last mount time: Wed Jun 28 13:51:31 2017
Last write time: Wed Jun 28 13:51:31 2017
Mount count: 5
Maximum mount count: -1
Last checked: Thu Nov 19 19:00:00 2009
Check interval: 0 (<none>)
Lifetime writes: 67 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 32
Desired extra isize: 32
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: 66aa0e7f-57da-41d0-86f7-d93270e53030
Journal backup: inode blocks
How do I tune the filesystem to have more inodes available?

Answer 1

This is a known issue with the overlay storage driver in docker and is addressed by the overlay2 driver.
The new cos-61 releases use docker 17.03 with overlay2 storage driver. Could you please give it a try and see if the issue happens again?
Thanks!

Answer 2

I have witnessed the same issue with all COS versions from 57.9202.64.0 (docker 1.11.2) on GKE 1.5 to 65.10323.85.0 (docker 17.03.2) on GKE 1.8.12-gke.1. Older version were certainly affected too.
Those all use the overlay driver:
pdecat#gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ docker info 2>&1 | grep "Storage Driver"
Storage Driver: overlay
pdecat#gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ grep "\(CHROMEOS_RELEASE_VERSION\|CHROMEOS_RELEASE_CHROME_MILESTONE\)" /etc/lsb-release
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_VERSION=10323.85.0
The overlay2 driver is only used for GKE 1.9+ clusters (fresh or upgraded) with the same COS version:
pdecat#gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ docker info 2>&1 | grep "Storage Driver"
Storage Driver: overlay2
pdecat#gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ grep "\(CHROMEOS_RELEASE_VERSION\|CHROMEOS_RELEASE_CHROME_MILESTONE\)" /etc/lsb-release
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_VERSION=10323.85.0
When the free space/inodes issue occurs with the overlay driver, I resolve it using spotify's docker-gc:
# docker run --rm --userns host -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc
Before:
# df -hi /var/lib/docker/
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6.0M 5.0M 1.1M 83% /var
# df -h /var/lib/docker/
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 95G 84G 11G 89% /var
# du --inodes -s /var/lib/docker/*
180 /var/lib/docker/containers
4093 /var/lib/docker/image
4 /var/lib/docker/network
4906733 /var/lib/docker/overlay
1 /var/lib/docker/tmp
1 /var/lib/docker/trust
25 /var/lib/docker/volumes
After:
# df -hi /var/lib/docker/
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 6.0M 327K 5.7M 6% /var/lib/docker
# df -h /var/lib/docker/
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 95G 6.6G 88G 7% /var/lib/docker
# du --inodes -s /var/lib/docker/*
218 /var/lib/docker/containers
1792 /var/lib/docker/image
4 /var/lib/docker/network
279002 /var/lib/docker/overlay
1 /var/lib/docker/tmp
1 /var/lib/docker/trust
25 /var/lib/docker/volumes
Note: using the usual docker rmi $(docker images --filter "dangling=true" -q --no-trunc) and docker rm $(docker ps -qa --no-trunc --filter "status=exited") did not help to recover resources in /var/lib/docker/overlay.