docker start failed after adding daemon.json file - docker

Question

I'm trying to install Kubernetes on CentOS 7.7, therefore, I have to install docker first.
I followed Kubernetes Documentation to install docker-ce and modify daemon.json file.
$ yum install yum-utils device-mapper-persistent-data lvm2
$ yum-config-manager --add-repo \
https://download.docker.com/linux/centos/docker-ce.repo
$ yum update && yum install \
containerd.io-1.2.10 \
docker-ce-19.03.4 \
docker-ce-cli-19.03.4
$ mkdir /etc/docker
$ cat > /etc/docker/daemon.json <<EOF
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
$ mkdir -p /etc/systemd/system/docker.service.d
$ systemctl daemon-reload
$ systemctl start docker
When started docker service, it said:
Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
$ systemctl status -l docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Tue 2020-01-07 14:44:11 UTC; 7min ago
Docs: https://docs.docker.com
Process: 9879 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 9879 (code=exited, status=1/FAILURE)
Jan 07 14:44:09 love61y2222c.mylabserver.com systemd[1]: Failed to start Docker Application Container Engine.
Jan 07 14:44:09 love61y2222c.mylabserver.com systemd[1]: Unit docker.service entered failed state.
Jan 07 14:44:09 love61y2222c.mylabserver.com systemd[1]: docker.service failed.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: docker.service holdoff time over, scheduling restart.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: Stopped Docker Application Container Engine.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: start request repeated too quickly for docker.service
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: Failed to start Docker Application Container Engine.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: Unit docker.service entered failed state.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: docker.service failed.
$ journalctl -xe
.
.
-- Unit docker.service has begun starting up.
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.722780008Z" level=info msg="Starting up"
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728447514Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728479813Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728510943Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728526075Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.732325726Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.733844225Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.733880664Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.733898044Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.743421350Z" level=warning msg="Using pre-4.0.0 kernel for overlay2, mount failures may require
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: failed to start daemon: error initializing graphdriver: overlay2: the backing xfs filesystem is formatted without d_type
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: Unit docker.service entered failed state.
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: docker.service failed.
Could anyone tell me why docker service start failed after modifying daemon.json file? And how to specify cgroupdriver, default log-driver and default storage-driver in the right way?
Any suggestion will be greatly appreciated.
Thanks.

Answer 1

This error is pointing to an issue forcing docker to use overlay2 without the proper backing filesystem:
failed to start daemon: error initializing graphdriver: overlay2: the backing xfs filesystem is formatted without d_type
See docker's table for details on backing filesystem requirements for the different storage drivers: https://docs.docker.com/storage/storagedriver/#supported-backing-filesystems
The fix is to remove the storage driver settings, or fix the backing filesystem with the needed options to support overlay2:
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
}
}
For details on changing the xfs options, that appears to require rebuilding the filesystem. See this answer for more details on the needed steps.