Docker run results in error: layer does not exist - docker

Question

I'm trying to setup redmine (with postgres) on my raspberry pi 3 using docker-compose. It already worked once, but then I tried to install plugins and somehow managed to bork my system.
Now it won't let me start my database container anymore. Even creating a new postgres:12.8 container, yields the error layer does not exist:
$ docker run --rm -it postgres:12.8 bash
docker: Error response from daemon: layer does not exist.
I've already restarted the system twice, which sadly didn't change anything.
How can I get this to work again?
Additional info:
$ docker info
Client:
Debug Mode: false
Server:
Containers: 5
Running: 4
Paused: 0
Stopped: 1
Images: 65
Server Version: 19.03.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.66-v7+
Operating System: Raspbian GNU/Linux 9 (stretch)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 926.1MiB
Name: raspberrypi
ID: <redacted>
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Update 1:
Removing the image (via docker rmi) doesn't stop the error.
$ docker run --rm -it postgres:12.8 bash
Unable to find image 'postgres:12.8' locally
12.8: Pulling from library/postgres
Digest: sha256:e10cd754296c1b3e93a121bcc64550d183df56f20bab47e08cacb123fb969b5e
Status: Downloaded newer image for postgres:12.8
docker: Error response from daemon: layer does not exist.
See 'docker run --help'.
Other images can be run as usual:
$ docker run --rm -it alpine bash
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
4ee0caa23b36: Pull complete
Digest: sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
Status: Downloaded newer image for alpine:latest
/ # whoami
root
/ #

Answer 1

Nuking /var/lib/docker seems to get the system working again - this removes all images and lots of other docker-related data.
This doesn't feel like a great solution, but it'll have to do for now.
$ sudo service docker stop
$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
$ sudo rm -rf /var/lib/docker
$ sudo service docker start
$ docker run --rm -it postgres:12.8 bash
Unable to find image 'postgres:12.8' locally
12.8: Pulling from library/postgres
607f77084e8a: Pull complete
ce89a6184ac9: Pull complete
2a4e6e7ac863: Pull complete
f3b6f439172b: Pull complete
2e803a3461b4: Pull complete
467430f26cab: Pull complete
1a81afc37621: Pull complete
4ceea61b7b5c: Pull complete
1f5bfa96a004: Pull complete
b40859b8e537: Pull complete
c7b3bf3cab26: Pull complete
653c63793cd5: Pull complete
aad7328542d8: Pull complete
Digest: sha256:e10cd754296c1b3e93a121bcc64550d183df56f20bab47e08cacb123fb969b5e
Status: Downloaded newer image for postgres:12.8
root#8ba418515688:/#
References:
https://github.com/coreos/bugs/issues/1808
https://github.com/coreos/bugs/issues/1313

Answer 2

I have the error below and a restart of docker daemon seems to fix it. Still wondering what causes it in the first place.
level=error msg="Handler for POST /v1.41/containers/create returned error: layer does not exist"