Why is Docker filling up /var/lib/docker/overlay2? - docker

Question

Even though I have successfully (?) removed all Docker images and containers, the folder /var/lib/docker/overlay2 still is vast (152 GB). Why? How do I reduce the used disk size?
I have tried to rename the folder (in preparation for a possible removal of the folder) but that caused subsequent pull requests to fail.
To me it appears pretty unbelievable that Docker would need this amount of vast disk space just for later being able to pull an image again. Please enlighten me what is wrong or why it has to be this way.
List of commands run which should show what I have tried and the current status:
$ docker image prune --force
Total reclaimed space: 0B
$ docker system prune --force
Total reclaimed space: 0B
$ docker image prune -a --force
Total reclaimed space: 0B
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
$ du -h --max-depth=1 /var/lib/docker/overlay2 | sort -rh | head -25
152G /var/lib/docker/overlay2
1.7G /var/lib/docker/overlay2/ys1nmeu2aewhduj0dfykrnw8m
1.7G /var/lib/docker/overlay2/ydqchhcaqokdokxzbh6htqa49
1.7G /var/lib/docker/overlay2/xmffou5nk3zkrldlfllopxcab
1.7G /var/lib/docker/overlay2/tjz58rjkote2c79veonb3s6qa
1.7G /var/lib/docker/overlay2/rlnr04hlcudgoh6ujobtsu2ck
1.7G /var/lib/docker/overlay2/r4ubwsmrorpr08k8o5rko9n98
1.7G /var/lib/docker/overlay2/q8x21c9enjhpitt365smkmn4e
1.7G /var/lib/docker/overlay2/ntr973uef37oweqlxr4kmaxps
1.7G /var/lib/docker/overlay2/mcyasqzo2gry5dvjxoao1opws
1.7G /var/lib/docker/overlay2/m2k4u58mob6e2db86qqu1e1f8
1.7G /var/lib/docker/overlay2/lizesless03kch8j7kpk89rcf
1.7G /var/lib/docker/overlay2/kmu7mjvsopr8o63onbsijb98j
1.7G /var/lib/docker/overlay2/khgjwqry5drdy0jbwf47gr2lb
1.7G /var/lib/docker/overlay2/gt70ur50vw3whq265vmpep7ay
1.7G /var/lib/docker/overlay2/c3tm1fcuekmdreowrfcso7nd4
1.7G /var/lib/docker/overlay2/7j93t64mt63arj6sewyyejwyo
1.7G /var/lib/docker/overlay2/3ftxvvg2xg02xuwcb3ut3dq89
1.7G /var/lib/docker/overlay2/0m3o3lw6b1ggs8m6z4uv6ueqf
1.4G /var/lib/docker/overlay2/r82rfxme096cq5pg1xz1z5arg
1.4G /var/lib/docker/overlay2/qric73hv1z3nx4k0zop3fvcm6
1.4G /var/lib/docker/overlay2/oyb0a5ab5h642y30s6hawj4r9
1.4G /var/lib/docker/overlay2/oqf9ltfoy36evnkuo8ga2uepl
1.4G /var/lib/docker/overlay2/ntuwvljxxzqs2oxhgg3enyo7x
1.4G /var/lib/docker/overlay2/l0oi2lxdrtg42hk2rznknqk0r
$ ls -l /var/lib/docker/overlay2
total 136
drwx------ 4 root root 72 Nov 20 13:03 00ep8i7v5bdmhqsxdoikslr19
drwx------ 4 root root 72 Feb 28 09:47 026x5e2xns6ui2acym19qfvl7
drwx------ 4 root root 72 Apr 2 19:20 032y8d31damevtfymq6yzkyi4
drwx------ 4 root root 72 Apr 23 13:42 03wwbyd4uge9u0auk94wwdlig
drwx------ 4 root root 72 Jan 15 12:46 04cy91a19owwqu9hyw6vruhzo
drwx------ 4 root root 72 Apr 2 14:44 051625a0f856b63ed67a3bc9c19f09fb1c90303b9536791dc88717cb7379ceeb
drwx------ 4 root root 72 Dec 3 19:56 059fk19uw70p6fqzei6wnj8s2
drwx------ 4 root root 72 Apr 21 15:03 059mddrhqegqhxv1ockejw9gs
drwx------ 4 root root 72 Nov 28 11:26 069dwkz92m8fao6whxnj4x9vp
drwx------ 4 root root 72 Feb 28 09:47 06h7qo5f70oyzaqgn1elbx5u8
drwx------ 4 root root 72 Dec 18 13:27 0756fd640036fa92499cfdcf4bcc3081d9ec16c25eebe5964d5e12d22beb9991
drwx------ 4 root root 72 Apr 20 11:32 09rk4gm6x2mcquc5cz0yvbawq
drwx------ 4 root root 72 Apr 2 19:55 09scfio3qvtewzgc5bdwgw4f6
drwx------ 4 root root 72 May 4 14:00 0ac2a09aa4a038981d37730e56dece4a3d28e80f261b68587c072b4012dc044a
drwx------ 4 root root 72 Feb 25 14:19 0c399f5c349ec61ac175525c52533b069a52028354c1055894466e9b5430fbc3
drwx------ 4 root root 72 May 4 14:00 0cac39b1382986a2d9a6690985792c04d03192337ea37ee06cb74f2f457b7bb7
drwx------ 4 root root 72 Mar 5 08:41 0czco1xx3148slgwf8imdrk33
drwx------ 4 root root 72 Apr 21 08:30 0gb2iqev9e7kr587l09u19eff
drwx------ 4 root root 72 Feb 20 18:03 0gknqh4pyg46uzi6asskbf8xk
drwx------ 4 root root 72 Jan 8 11:43 0gugiou3wqu53os4dageh77ty
drwx------ 4 root root 72 Jan 7 11:31 0i8fd5jet6ieajyl2uo1xj2ai
.
.
.
$ docker version
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:27:04 2020
OS/Arch: linux/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:25:42 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

Answer 1

You might have switched storage drivers somewhere along the way, so maybe docker is just cleaning out those drivers but leaving overlay2 as is (I still can't understand why would pulling images would fail).
Let's try this, run docker info and check what is your storage driver:
$ docker info
Containers: 0
Images: 0
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
<output truncated>
If it is not overlay2 (as appears above) try switching to it, and then prune docker images again and check if that cleaned up that folder.
Another possible solution is mentioned in this thread, people are commenting that clearing logs solves this problem, so try the following:
Remove all log files:
find /var/lib/docker/containers/ -type f -name "*.log" -delete
Restart docker daemon (or entire machine):
sudo systemctl restart docker
or
docker-compose down && docker-compose up -d
or
shutdown -r now

Answer 2

in preparation for a possible removal of the folder
If you are going to delete all data from the Docker directory anyway it is safe to:
Stop Docker Daemon
Remove the /var/lib/docker directory entirely
Restart Docker Daemon
Docker will then recreate any needed data directories.
You can also add:
"log-driver": "json-file",
"log-opts": {"max-size": "20m", "max-file": "3"},
to your /etc/docker/daemon.json to restrict log size and growth in the future or set the log-driver to "journald" to eliminate log files entirely.

Answer 3

Thanks for your input and suggestions!
I believe that I am still using overlay2 as storage driver:
$ docker info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 19.03.8
Storage Driver: overlay2
<output truncated>
I also cleared the logs and restarted the daemon and actually also the entire machine. The problem however remained.
In the end I solved it by stoping the deamon, removing the entire docker folder and restarting the deamon, as suggested above.
df -h
sudo systemctl stop docker
sudo mv /var/lib/docker /var/lib/docker_old
sudo systemctl start docker
sudo rm -rf /var/lib/docker_old
df -h
I fear however that this will not be a permanent solutions and that the problem will come back, but this will hopefully last another year. :)

Answer 4

Try to prune everything including volumes (different than the original poster's command):
$ docker system prune --volumes
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all volumes not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N]
That freed up a bunch of space for me and solved my issue. I think the build cache was one of the culprits for me.

Answer 5

Two things will fill up /var/lib/docker/overlay2:
Docker images: Clean those with docker image prune -a. Note that any images not currently associated with a container will be deleted which requires pulling or building the image again if you needed it.
Container Specific Changes: any write to the container filesystem that isn't going to another mount (like a volume) will cause a copy-on-write that is stored in the container specific layer. You can see these changes with docker diff on a container. Even a metadata change like file ownership, permissions, or a timestamp, can trigger this copy-on-write of the entire file.
Things that are not included in this directory:
Volumes: Named volumes will be stored in /var/lib/docker/volumes by default. You can still prune these with docker volume prune but make sure you have backed up any important data first. A better cleanup is to remove unused anonymous volumes with a command like:
docker volume ls -qf dangling=true | egrep '^[a-z0-9]{64}$' | \
xargs --no-run-if-empty docker volume rm
Container Logs: Container logs will be written to /var/lib/docker/containers. If these are taking up space, it's best to have docker automatically rotate those. See this answer for details on rotating logs.

Answer 6

I had the same problem, /var/lib/docker/overlay2 was using 17 GB even after removing every docker image: docker image rm ...
When you stop a container, it is not automatically removed unless you started it with the --rm flag (prune-containers). Container size can be seen using this command: docker container ls -a -s
I managed to reclaim the space taken by stopped container using this command: docker container prune.
Answer of #Nick is perhaps even better as it cleans every unused docker files:
docker system prune --volumes