Random docker errors when starting containers - docker

Question

I am having trouble starting the docker containers on a particular machine: doing docker run gives random results, and that is the case whether I install atom, debian stretch of ubuntu 18.04. On the debian OSes, I am using a fresh install of Docker version 18.09.6, build 481bc77.
The most common issue is Error response from daemon: OCI runtime create failed
Here is what I see when I am trying to run the hello-world example (working ~1.5 times out of 7 times):
user#machine:~$ sudo docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
user#machine:~$ sudo docker run hello-world
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/02c7ab23649c89b19720d57a549eb703aa442805aa3b468e7610c19e6d8fa2eb/log.json: no such file or directory): runc did not terminate sucessfully: unknown.
ERRO[0001] error waiting for container: context canceled
user#machine:~$ sudo docker run hello-world
docker: Error response from daemon: ttrpc: client shutting down: read unix #->#/containerd-shim/moby/4de0da9c33103f4622907a3ab25535075325366e9a4d0f1c4849ec20ca3cb91f/shim.sock: read: connection reset by peer: unknown.
ERRO[0001] error waiting for container: context canceled
user#machine:~$ sudo docker run hello-world
docker: Error response from daemon: ttrpc: client shutting down: read unix #->#/containerd-shim/moby/151f1ba68a9b28260a00e9cff433c5009382880fb75a28ee79fa549ffdfb21a9/shim.sock: read: connection reset by peer: unknown.
ERRO[0001] error waiting for container: context canceled
user#machine:~$ sudo docker run hello-world
docker: Error response from daemon: OCI runtime create failed: unable to retrieve OCI runtime error (open /run/containerd/io.containerd.runtime.v1.linux/moby/32de5ca60771884d4a236e3e9d2704a48f18f03e93fc6dd195f4e39fb7b56501/log.json: no such file or directory): runc did not terminate sucessfully: unknown.
ERRO[0001] error waiting for container: context canceled
user#machine:~$ sudo docker run hello-world
docker: Error response from daemon: ttrpc: client shutting down: read unix #->#/containerd-shim/moby/dcbb905d8783c65302c1a3afe8fb7913c58e7d5765b5a79072d55fb36f7bc1ea/shim.sock: read: connection reset by peer: unknown.
ERRO[0001] error waiting for container: context canceled
user#machine:~$ sudo docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
(amd64)
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://hub.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/get-started/
docker: Error response from daemon: OCI runtime state failed: runc did not terminate sucessfully: SIGILL: illegal instruction
PC=0x55611122e30c m=3 sigcode=2
goroutine 20 [running]:
runtime.aeshashbody()
/.GOROOT/src/runtime/asm_amd64.s:939 +0x1c fp=0xc42002d6b8 sp=0xc42002d6b0 pc=0x55611122e30c
runtime.mapaccess1_faststr(0x556111a6ad00, 0xc42007f590, 0x5561116ceb02, 0x2, 0x556100000001)
/.GOROOT/src/runtime/hashmap_fast.go:233 +0x1d1 fp=0xc42002d728 sp=0xc42002d6b8 pc=0x5561111e3031
text/template/parse.lexIdentifier(0xc4200bab60, 0x556111ae6e70)
/.GOROOT/src/text/template/parse/lex.go:441 +0x138 fp=0xc42002d7b8 sp=0xc42002d728 pc=0x556111415128
text/template/parse.(*lexer).run(0xc4200bab60)
/.GOROOT/src/text/template/parse/lex.go:228 +0x39 fp=0xc42002d7d8 sp=0xc42002d7b8 pc=0x556111413f99
runtime.goexit()
/.GOROOT/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42002d7e0 sp=0xc42002d7d8 pc=0x55611122f3b1
created by text/template/parse.lex
/.GOROOT/src/text/template/parse/lex.go:221 +0x161
goroutine 1 [chan receive, locked to thread]:
text/template/parse.(*lexer).nextItem(...)
/.GOROOT/src/text/template/parse/lex.go:195
text/template/parse.(*Tree).next(...)
/.GOROOT/src/text/template/parse/parse.go:64
text/template/parse.(*Tree).nextNonSpace(0xc42009a200, 0x0, 0x0, 0x0, 0x0, 0x0)
/.GOROOT/src/text/template/parse/parse.go:102 +0x159
text/template/parse.(*Tree).parse(0xc42009a200)
/.GOROOT/src/text/template/parse/parse.go:284 +0x2fa
text/template/parse.(*Tree).Parse(0xc42009a200, 0x5561116cead5, 0xf0, 0x0, 0x0, 0x0, 0x0, 0xc42007f800, 0xc42007c6c0, 0x2, ...)
/.GOROOT/src/text/template/parse/parse.go:233 +0x228
text/template/parse.Parse(0x5561116b62fb, 0x5, 0x5561116cead5, 0xf0, 0x0, 0x0, 0x0, 0x0, 0xc42007c6c0, 0x2, ...)
/.GOROOT/src/text/template/parse/parse.go:55 +0x10a
text/template.(*Template).Parse(0xc42008c240, 0x5561116cead5, 0xf0, 0x5561112abfaa, 0x5561116c0486, 0x1d)
/.GOROOT/src/text/template/template.go:198 +0x11a
rax 0x5561116ceb02
rbx 0x55611122e2d0
rcx 0x2
rdx 0xc42002d6c8
rdi 0xc6b7000000000000
rsi 0x1
rbp 0xc42002d718
rsp 0xc42002d6b0
r8 0xc42002d728
r9 0x0
r10 0x3
r11 0x286
r12 0xc42006e468
r13 0xff
r14 0xff
r15 0xf
rip 0x55611122e30c
rflags 0x10202
cs 0x33
fs 0x0
gs 0x0
: unknown.
ERRO[0002] error waiting for container: context canceled
Does anyone know what the error could be?
I had some some weird networking errors when installing docker, but launching the same apt install again worked:
user#machine:~$ sudo apt-get install docker-ce docker-ce-cli containerd.io
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
aufs-tools cgroupfs-mount libltdl7 pigz
The following NEW packages will be installed:
aufs-tools cgroupfs-mount containerd.io docker-ce docker-ce-cli libltdl7 pigz
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 50.7 MB of archives.
After this operation, 243 MB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 pigz amd64 2.4-1 [57.4 kB]
Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 aufs-tools amd64 1:4.9+20170918-1ubuntu1 [104 kB]
E: Method https has died unexpectedly!
E: Sub-process https received signal 4.

Answer 1

If you are facing issues after the upgrade to containerd 1.4.0, downgrade to 1.3.4.
That is, for example, if you are on Arch Linux, probably you can do:
cd /var/cache/pacman/pkg/
sudo pacman -U containerd-1.3.4-2-x86_64.pkg.tar.zst
Specifically, this is the error message you might be facing:
docker: Error response from daemon: ttrpc: closed: unknown.
If you need 1.4.0 for some reason, there is an open issue tracking this issue on Github over here, best to track it's status from there: https://github.com/containerd/containerd/issues/4483