ddev on ZFS: Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument - docker

Question

Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument
Happens with any new project after
This error happens when launching any of my ddev projects or when creating a new one with ddev config.
Update: This happens only when docker data (/var/lib/docker by default) is on a ZFS volume. The problem disappeared as soon as I moved it to EXT4.
It does not seem to be a problem with docker or docker compose, as I have launched the example/hello-world containers of both projects successfully.
I have tried removing ~/.ddev, downgrading ddev, and downgrading docker. I was not able to downgrade docker-compose.
Log:
$ ddev start
Starting test...
mkcert may not be properly installed, we suggest installing it for trusted https support, `brew install mkcert nss`, `choco install -y mkcert`, etc. and then `mkcert -install`
Running Command=ip address show dev docker0
Building db
Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument
Failed to start test: Failed to run docker-compose [-f /home/tester/htdocs/test/.ddev/.ddev-docker-compose-full.yaml up --build -d], err='exit status 1', stdout='Step 1/6 : ARG BASE_IMAGE
Step 2/6 : FROM $BASE_IMAGE
---> 33f3288968e4
Step 3/6 : ARG username
', stderr='Building db
Service 'db' failed to build : failed to copy files: copy file range failed: invalid argument'
config.yml
name: test
type: php
docroot: ""
php_version: "7.3"
webserver_type: nginx-fpm
router_http_port: "80"
router_https_port: "443"
xdebug_enabled: false
additional_hostnames: []
additional_fqdns: []
mariadb_version: "10.2"
mysql_version: ""
provider: default
use_dns_when_possible: true
composer_version: ""
System:
Docker 20.10.1
docker-compose 1.27.4
Manjaro Linux (Arch derivative)
ZFS 2.0.0-1 (on root)
docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-tp-docker)
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 27
Server Version: 20.10.1
Storage Driver: zfs
Zpool: rpool
Zpool Health: ONLINE
Parent Dataset: rpool/root
Space Used By Parent: 20958640128
Space Available: 282678809088
Parent Quota: no
Compression: off
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b.m
runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.2-2-MANJARO
Operating System: Manjaro Linux
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 30.9GiB
Name: testpc
ID: BKDP:GAWJ:BUCV:OVP6:QUKY:UQCZ:AESP:V3AN:EXHE:BWJU:4TT7:YPEO
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Update: This fixed itself after a complete system reinstall; unfortunately I have no idea how the reinstall might have helped.

Answer 1

(Don't have enough reputation to comment, but this isn't a full answer.)
I hit the same issue, also with Docker on Arch w/ZFS. I was able to "fix" it by passing DOCKER_BUILDKIT=0 docker build .... It appears to be an issue somewhere in buildkit.
I haven't dug in any further to see what specifically is causing the issue.