Run Docker on Raspberry Pi4 with overlay fs - docker

Question

I prefer to create a situation where on a Raspberry Pi4 Docker is running while the SD-card is read only. This with overlay fs.
In the dockercontainer a database is running, the data of the database is written to an USB-stick (volume mapping).
When overlayfs is activated (after reboot, enabled via “sudo raspi-config”), docker will not start-up any more.
The steps on https://docs.docker.com/storage/storagedriver/overlayfs-driver/
System information:
Linux raspberrypi 5.10.63-v8+ #1488 SMP PREEMPT Thu Nov 18 16:16:16 GMT 2021 aarch64 GNU/Linux
Docker information:
pi#raspberrypi:~ $ docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 1
Server Version: 20.10.11
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
………
Status docker after restart:
pi#raspberrypi:~ $ sudo systemctl status docker.*
Warning: The unit file, source configuration file or drop-ins of docker.service changed on disk. Run 'systemctl daemon-reload' to reload units.
● docker.socket - Docker Socket for the API
Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
Active: failed (Result: service-start-limit-hit) since Thu 2021-12-09 14:30:43 GMT; 1h 13min ago
Triggers: ● docker.service
Listen: /run/docker.sock (Stream)
CPU: 2ms
Dec 09 14:30:36 raspberrypi systemd[1]: Starting Docker Socket for the API.
Dec 09 14:30:36 raspberrypi systemd[1]: Listening on Docker Socket for the API.
Dec 09 14:30:43 raspberrypi systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2021-12-09 14:30:43 GMT; 1h 13min ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 992 (code=exited, status=1/FAILURE)
CPU: 162ms
Dec 09 14:30:43 raspberrypi systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Dec 09 14:30:43 raspberrypi systemd[1]: Stopped Docker Application Container Engine.
Dec 09 14:30:43 raspberrypi systemd[1]: docker.service: Start request repeated too quickly.
Dec 09 14:30:43 raspberrypi systemd[1]: docker.service: Failed with result 'exit-code'.
Dec 09 14:30:43 raspberrypi systemd[1]: Failed to start Docker Application Container Engine.
Running the command given in docker.service with additional overlay flag
pi#raspberrypi:~ $ sudo /usr/bin/dockerd --storage-driver=overlay -H fd:// --containerd=/run/containerd/containerd.sock
unable to configure the Docker daemon with file /etc/docker/daemon.json: the following directives are specified both as a flag and in the configuration file: storage-driver: (from flag: overlay, from file: overlay2)
pi#raspberrypi:~ $ sudo /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
INFO[2021-12-09T14:34:31.667296985Z] Starting up
failed to load listeners: no sockets found via socket activation: make sure the service was started by systemd
Which steps am I missing to be able to run Docker with overlay fs, such that the SD-card in the Raspberry is read only?
Without the overlay fs active it all works as expected.

Answer 1

I ran into this issue as well and found a way around it. In summary, you can't run the default Docker FS driver (overlay2) on overlayfs. Fortunately, Docker supports other storage drivers, including fuse-overlayfs. Switching to this driver resolves the issue but there's one final catch. When Docker starts, it attempts to rename /var/lib/docker/runtimes and since overlayfs doesn't support renames of directories already in lower layers, it fails. If you simply rm -rf this directory while Docker is stopped and before you enable RPi's overlayfs, everything should work.