I'm trying to setup redmine (with postgres) on my raspberry pi 3 using docker-compose. It already worked once, but then I tried to install plugins and somehow managed to bork my system.
Now it won't let me start my database container anymore. Even creating a new postgres:12.8 container, yields the error layer does not exist:
$ docker run --rm -it postgres:12.8 bash
docker: Error response from daemon: layer does not exist.
I've already restarted the system twice, which sadly didn't change anything.
How can I get this to work again?
Additional info:
$ docker info
Client:
Debug Mode: false
Server:
Containers: 5
Running: 4
Paused: 0
Stopped: 1
Images: 65
Server Version: 19.03.13
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.19.66-v7+
Operating System: Raspbian GNU/Linux 9 (stretch)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 926.1MiB
Name: raspberrypi
ID: <redacted>
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Update 1:
Removing the image (via docker rmi) doesn't stop the error.
$ docker run --rm -it postgres:12.8 bash
Unable to find image 'postgres:12.8' locally
12.8: Pulling from library/postgres
Digest: sha256:e10cd754296c1b3e93a121bcc64550d183df56f20bab47e08cacb123fb969b5e
Status: Downloaded newer image for postgres:12.8
docker: Error response from daemon: layer does not exist.
See 'docker run --help'.
Other images can be run as usual:
$ docker run --rm -it alpine bash
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
4ee0caa23b36: Pull complete
Digest: sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
Status: Downloaded newer image for alpine:latest
/ # whoami
root
/ #
Nuking /var/lib/docker seems to get the system working again - this removes all images and lots of other docker-related data.
This doesn't feel like a great solution, but it'll have to do for now.
$ sudo service docker stop
$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
$ sudo rm -rf /var/lib/docker
$ sudo service docker start
$ docker run --rm -it postgres:12.8 bash
Unable to find image 'postgres:12.8' locally
12.8: Pulling from library/postgres
607f77084e8a: Pull complete
ce89a6184ac9: Pull complete
2a4e6e7ac863: Pull complete
f3b6f439172b: Pull complete
2e803a3461b4: Pull complete
467430f26cab: Pull complete
1a81afc37621: Pull complete
4ceea61b7b5c: Pull complete
1f5bfa96a004: Pull complete
b40859b8e537: Pull complete
c7b3bf3cab26: Pull complete
653c63793cd5: Pull complete
aad7328542d8: Pull complete
Digest: sha256:e10cd754296c1b3e93a121bcc64550d183df56f20bab47e08cacb123fb969b5e
Status: Downloaded newer image for postgres:12.8
root#8ba418515688:/#
References:
https://github.com/coreos/bugs/issues/1808
https://github.com/coreos/bugs/issues/1313
I have the error below and a restart of docker daemon seems to fix it. Still wondering what causes it in the first place.
level=error msg="Handler for POST /v1.41/containers/create returned error: layer does not exist"
I built a docker form source for my embedded platform(armv7) and install them successfully. But I got an error(docker: Error response from daemon: operation not supported) when trying load arm32v7/busybox image
I built docker(Version: 18.09.4) and its dependencies with buildroot and install it successfully
2.Then I run dockerd with command "dockerd -s overlay &"
3.And I load armv7/busybox image which was saved from my notebook.
(As my embeded board was limited by gateway and can't connect the Internet, I use my notebook save image for my embeded board)
4.finally I load the image successfully and run it failed with "docker: Error response from daemon: operation not supported"
5.run docker info on board output like this
# docker info
WARN[1970-01-02T02:46:52.315076016Z] Could not get operating system name: Error opening /usr/lib/os-release: open /usr/lib/os-release: no such file or directory
WARN[1970-01-02T02:46:52.328707599Z] failed to retrieve docker-init version: exec: "docker-init": executable file not found in $PATH
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 18.09.4
Storage Driver: overlay
Backing Filesystem: tmpfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version: v1.0.0-rc7
init version: N/A (expected: )
Kernel Version: 4.9.37
OSType: linux
Architecture: armv7l
CPUs: 2
Total Memory: 498.3MiB
Name: (none)
ID: VS4X:CF4M:UNNY:QKJW:2PKQ:5ZUL:7N7C:24AK:O2VN:ZRLM:BUZO:5BHO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
WARNING: the overlay storage-driver is deprecated, and will be removed in a future release.
1.on my notebook
leo#ubuntu:~$ sudo docker pull arm32v7/busybox
Using default tag: latest
latest: Pulling from arm32v7/busybox
2b35d97f9c81: Pulling fs layer
2b35d97f9c81: Pull complete
Digest: sha256:6b1ef683f696c503a0afb93d06684a9a70f8f793a90220eb0da569be116ce1d1
Status: Downloaded newer image for arm32v7/busybox:latest
leo#ubuntu:~$ sudo docker save arm32v7/busybox -o arm_busybox.tar
2.on my embeded board
# rz
rz waiting to receive.
Starting zmodem transfer. Press Ctrl+C to cancel.
Transferring arm_busybox.tar...
100% 1141 KB 1141 KB/sec 00:00:01 0 Errors
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
arm32v7/busybox latest 94491159407b Less than a second ago 944kB
# docker^C
# docker images rmi 94491159407b
"docker images" requires at most 1 argument.
See 'docker images --help'.
Usage: docker images [OPTIONS] [REPOSITORY[:TAG]]
List images
# docker rmi 94491159407b
Untagged: arm32v7/busybox:latest
Deleted: sha256:94491159407b8aca603c71cdcaf164be9e63ed17239b2b081ce3a6bf765f6e8a
Deleted: sha256:7dc29058f73fc3fe174d0cc6795bee0a4130ba807f971e6e0cf4a88d77a9fe53
#
# ls
arm_busybox.tar
# docker load -i arm_busybox.tar
7dc29058f73f: Loading layer [==================================================>] 1.16MB/1.16MB
Loaded image: arm32v7/busybox:latest
#
# docker run
"docker run" requires at least 1 argument.
See 'docker run --help'.
Usage: docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
Run a command in a new container
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
arm32v7/busybox latest 94491159407b Less than a second ago 944kB
# docker run arm32v7/busybox
docker: Error response from daemon: operation not supported.
See 'docker run --help'.
# docker run arm32v7/busybox ls
docker: Error response from daemon: operation not supported.
See 'docker run --help'.
# docker run arm32v7/busybox /bin/sh
docker: Error response from daemon: operation not supported.
See 'docker run --help'.
docker run arm32v7/busybox should return successfully. but get an error response.
why?
After CONFIG_TMPFS_XATTR and CONFIG_TMPFS_POSIX_ACL flags switch to [=y] (before compiling kernel),docker run don't return error, but block !!
I enable debug mode when starting dockerd. there is log of dockerd
# dockerd -D -s overlay
...
DEBU[1970-01-02T07:55:03.848247951Z] form data: {"AttachStderr":true,"AttachStdin":false,"AttachStdout":true,"Cmd":null,"Domainname":"","Entrypoint":null,"Env":[],"HostConfig":{"AutoRemove":false,"Binds":null,"BlkioDeviceReadBps":null,"BlkioDeviceReadIOps":null,"BlkioDeviceWriteBps":null,"BlkioDeviceWriteIOps":null,"BlkioWeight":0,"BlkioWeightDevice":[],"CapAdd":null,"CapDrop":null,"Cgroup":"","CgroupParent":"","ConsoleSize":[0,0],"ContainerIDFile":"","CpuCount":0,"CpuPercent":0,"CpuPeriod":0,"CpuQuota":0,"CpuRealtimePeriod":0,"CpuRealtimeRuntime":0,"CpuShares":0,"CpusetCpus":"","CpusetMems":"","DeviceCgroupRules":null,"Devices":[],"DiskQuota":0,"Dns":[],"DnsOptions":[],"DnsSearch":[],"ExtraHosts":null,"GroupAdd":null,"IOMaximumBandwidth":0,"IOMaximumIOps":0,"IpcMode":"","Isolation":"","KernelMemory":0,"Links":null,"LogConfig":{"Config":{},"Type":""},"MaskedPaths":null,"Memory":0,"MemoryReservation":0,"MemorySwap":0,"MemorySwappiness":-1,"NanoCpus":0,"NetworkMode":"default","OomKillDisable":false,"OomScoreAdj":0,"PidMode":"","PidsLimit":0,"PortBindings":{},"Privileged":false,"PublishAllPorts":false,"ReadonlyPaths":null,"ReadonlyRootfs":false,"RestartPolicy":{"MaximumRetryCount":0,"Name":"no"},"SecurityOpt":null,"ShmSize":0,"UTSMode":"","Ulimits":null,"UsernsMode":"","VolumeDriver":"","VolumesFrom":null},"Hostname":"","Image":"94491159407b","Labels":{},"NetworkingConfig":{"EndpointsConfig":{}},"OnBuild":null,"OpenStdin":false,"StdinOnce":false,"Tty":false,"User":"","Volumes":{},"WorkingDir":""}
DEBU[1970-01-02T07:55:03.907315451Z] container mounted via layerStore: &{/var/lib/docker/overlay/9efba7d0f8b20b508274373120276bfcead4c0afdc98ac6c813a41e64c61df9e/merged 0x2bdd510 0x2bdd510}
DEBU[1970-01-02T07:55:03.946627243Z] Calling POST /v1.39/containers/77d7e164a3a10e34bb25ff59852bdfac9c962639dedf543614d71107567f6417/attach?stderr=1&stdout=1&stream=1
DEBU[1970-01-02T07:55:03.947147743Z] attach: stdout: begin
DEBU[1970-01-02T07:55:03.953483993Z] Calling POST /v1.39/containers/77d7e164a3a10e34bb25ff59852bdfac9c962639dedf543614d71107567f6417/wait?condition=next-exit
DEBU[1970-01-02T07:55:03.954236410Z] attach: stderr: begin
DEBU[1970-01-02T07:55:03.964652451Z] Calling POST /v1.39/containers/77d7e164a3a10e34bb25ff59852bdfac9c962639dedf543614d71107567f6417/start
DEBU[1970-01-02T07:55:03.981005285Z] container mounted via layerStore: &{/var/lib/docker/overlay/9efba7d0f8b20b508274373120276bfcead4c0afdc98ac6c813a41e64c61df9e/merged 0x2bdd510 0x2bdd510}
DEBU[1970-01-02T07:55:03.982641076Z] Assigning addresses for endpoint boring_montalcini's interface on network bridge
DEBU[1970-01-02T07:55:03.982754368Z] RequestAddress(LocalDefault/172.17.0.0/16, <nil>, map[])
DEBU[1970-01-02T07:55:03.982917743Z] Request address PoolID:172.17.0.0/16 App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65533, Sequence: (0xc0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:2 Serial:false PrefAddress:<nil>
DEBU[1970-01-02T07:55:04.013580701Z] Assigning addresses for endpoint boring_montalcini's interface on network bridge
INFO[1970-01-02T07:55:04.015169910Z] /etc/resolv.conf does not exist
INFO[1970-01-02T07:55:04.015304618Z] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]
INFO[1970-01-02T07:55:04.015357285Z] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]
DEBU[1970-01-02T07:55:04.018542701Z] Programming external connectivity on endpoint boring_montalcini (dcf05afcaeb79ee2fc5fc4a6aceda37c6083f3a19fe608fabb4bb8d91f7e5f1b)
DEBU[1970-01-02T07:55:04.019416368Z] EnableService 77d7e164a3a10e34bb25ff59852bdfac9c962639dedf543614d71107567f6417 START
DEBU[1970-01-02T07:55:04.019500118Z] EnableService 77d7e164a3a10e34bb25ff59852bdfac9c962639dedf543614d71107567f6417 DONE
DEBU[1970-01-02T07:55:04.022324743Z] bundle dir created bundle=/var/run/docker/containerd/77d7e164a3a10e34bb25ff59852bdfac9c962639dedf543614d71107567f6417 module=libcontainerd namespace=moby root=/var/lib/docker/overlay/9efba7d0f8b20b508274373120276bfcead4c0afdc98ac6c813a41e64c61df9e/merged
DEBU[1970-01-02T07:55:04.030072118Z] event published ns=moby topic="/containers/create" type=containerd.events.ContainerCreate
INFO[1970-01-02T07:55:04.060624576Z] shim containerd-shim started address="/containerd-shim/moby/77d7e164a3a10e34bb25ff59852bdfac9c962639dedf543614d71107567f6417/shim.sock" debug=true pid=1383
DEBU[0000] registering ttrpc server
DEBU[0000] serving api on unix socket socket="[inherited from parent]"
mv root dir to a persistent path, not tmpfs, then:
stop dockerd
mv /var/lib/docker /root/docker
start dockerd
docker run arm32v7/busybox
I have ubuntu 14.04.5 installed as guest os in virtualbox 5.0.26 running on windows 10. I am not aware of any issues with the ubuntu installation, it seems to run fine and has a bridged internet connection so gets its own ip.
I have installed docker following the directions on docker docs for linux. The installation goes fine without any errors and the docker daemon starts ok.
Here is the docker info:
root#ubuntu-z9:~# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.12.0
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 0
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: overlay bridge host null
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: apparmor
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 10
Total Memory: 31.42 GiB
Name: ubuntu-z9
ID: 7MPO:OHFW:3OBJ:KUVX:3YCS:XP4U:RE6W:SFC3:O4KK:GJJU:M6WJ:HYLY
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Insecure Registries:
127.0.0.0/8
The machine can see the internet fine and access hub.docker.com from a browser.
However, when I run the simple hello-world test the daemon hangs
root#ubuntu-z9:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
with a timeout.
I can run docker-machine without any issues on the host windows 10 machine so I believe the issue lies in my setup of the ubuntu machine in virtualbox and docker.
Here is the logging output of the docker daemon on the ubuntu guest machine:
$ docker pull hello-world
DEBU[0093] Calling POST /v1.24/images/create?fromImage=hello-world&tag=latest
DEBU[0093] Trying to pull hello-world from https://registry-1.docker.io v2
DEBU[0094] Increasing token expiration to: 60 seconds
ERRO[0494] Error trying v2 registry: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"
ERRO[0494] Attempting next endpoint for pull after error: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"
DEBU[0494] Skipping v1 endpoint https://index.docker.io because v2 registry was detected
ERRO[0494] Handler for POST /v1.24/images/create returned error: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"
Any suggestions on a way forward to diagnose or fix the issue?
Many thanks.
It was a simple issue, undoubtedly documented somewhere but I missed it. I post an answer here in case someone else has the same.
The virtualbox os (ubuntu in my case) has to have a NAT network adapter and the NAT adapter has to have higher priority than a bridge adapter (if you have one). You don't need a bridged adapter to run docker (but if you want the virtualbox to have an ip on your local network then you do need to add a bridged adapter.)
VirtualBox configuration examples that work to run docker:
VBox Adapter 1: NAT (eth0), VBox Adapter 2: Host-only Adapter (eth1)
VBox Adapter 1: NAT (eth0), VBox Adapter 2: Bridged Adapter (eth1)
VirtualBox configuration examples that do not work to run docker:
VBox Adapter 1: Bridged Adapter (eth0)
VBox Adapter 1: Bridged Adapter (eth0), VBox Adapter 2: NAT (eth1)
Note in all four cases the virtualbox ubuntu os has access to the internet but docker can only pull images when NAT has priority over the bridged interface.
When I run docker info, it shows such information:
~ $ docker info
Containers: 0
Images: 8
Server Version: 1.9.1
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 9
Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.13-boot2docker
Operating System: Boot2Docker 1.9.1 (TCL 6.4.1); master : cef800b - Fri Nov 20 19:33:59 UTC 2015
CPUs: 1
Total Memory: 1.956 GiB
Name: default
ID: CXQT:PB43:QNMD:W5JY:66QM:QRI7:GJUB:X27R:RQ4U:53I2:QVJS:DYKV
Debug mode (server): true
File Descriptors: 17
Goroutines: 29
System Time: 2015-12-15T06:26:35.824197223Z
EventsListeners: 1
Init SHA1:
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Username: freewind
Registry: https://index.docker.io/v1/
Labels:
provider=virtualbox
You can notice at the bottom there is a Registry:
Registry: https://index.docker.io/v1/
I want to know to change this value, say, to change it to http://localhost:5000, so it will always pull images from http://localhost:5000?
PS: I already tried to add --engine-registry-mirror http://localhost:5000 when creating a docker machine:
docker-machine create -d virtualbox \
--engine-registry-mirror http://localhost:5000 default
And in the /mnt/sda1/var/lib/boot2docker/profile of the machine, it contains the content:
EXTRA_ARGS='
--label provider=virtualbox
--registry-mirror http://localhost:5000
'
In order to ask it to pull images from http://localhost:5000 first. But I still wonder how to change the global Registry value (shows in docker info)
how to change the global Registry value (shows in docker info)
This seems fixed in registry/config.go#L30-L31
// IndexServer is the v1 registry server used for user auth + account creation
IndexServer = DefaultV1Registry + "/v1/"
Note that the registry service will first look for v2 endpoint anyway.
func (s *Service) lookupEndpoints(repoName reference.Named) (endpoints []APIEndpoint, err error) {
endpoints, err = s.lookupV2Endpoints(repoName)
Issue 16974 asks "why not make mirror support a private v2 registry until now?"
Without going into detail, the main issue here is trust and provenance. Basically, the registry controls the naming, so once you point the daemon at another registry, you can no longer trust the content if layers collide. We avoid these issues by disallowing arbitrary mirroring.
This is a situation we are working on resolving but it will take some effort to appropriately address.
More specifically, this is followed in distribution PR 1136, with Proxying to other Registries:
A pull-through caching mode exists for the registry, but is restricted from within the docker client to only mirror the official Docker Hub.
This functionality can be expanded when image provenance has been specified and implemented in the distribution project.
For now, all you can do is to add --disable-legacy-registry to make sure to consider only V2 ones. That won't change the docker info output though.
I am trying to install a docker repo on an Ubuntu server, but it seems Docker has issues with DNS.
docker run -d -p 5000:5000 --restart=always --name registry registry:2
Unable to find image 'registry:2' locally
Pulling repository registry
Get https://index.docker.io/v1/repositories/library/registry/images: dial tcp: lookup index.docker.io: no such host
However, all other applications work fine. I can also do a wget on index.docker.io, so no issues there.
I am using an internal DNS server, which is a Synology NAS device.
resolv.conf of the server:
nameserver 192.168.10.2
search internal.mydomain.com
my /etc/default/docker options:
DOCKER_OPTS="--bip=192.168.11.0/24 --dns 192.168.10.2"
I am using 192.168.10.0/24 as my internal ip range. the .2 ip belongs to my NAS/DNS server.
Docker version:
Docker version 1.7.1, build 786b29d
Anyone a clue?
Update: changing dns to Google solved the download issue, but now it gives an error afterwards:
Error response from daemon: Cannot start container 33757f59f942583ff949f421fb5c266e6d1c2b0fdc1363565e77febf44feb60f: invalid argument
Some additional info about my setup:
jeroen#docker01:~$ docker info
Containers: 3
Images: 22
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 28
Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-32-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 2
Total Memory: 1.955 GiB
Name: docker01
ID: X6JB:IH7Z:OK5O:II5I:OJ6V:OERE:IPEM:PN6S:HDDM:G2J7:HRB2:4ZKO
WARNING: No swap limit support
I had the same issue, and I notice that you have "--bip=192.168.11.0/24"
Try changing this to an actual IP address, rather than a subnet. For example, try "--bip=192.168.11.1/24".
You will have to stop docker, remove the docker0 bridge (ip link delete docker0) and then restart using the new bip option.