Cannt run or build Docker images on CentOS 7 - docker

I'm learning using Docker now, I've installed docker on my server(CentOS 7). But when I follow the offical tutorial, I meet one problem which fails me to continue:
mkdir /var/lib/docker/overlay/7849ab40fd8072dcd724387dab14707bb4af0e94d9ab4f71795d0c478c3d49a9-init/merged/dev/shm: invalid argument
this appears when I build/run most images(few image that didnt fail is offical python:latest and hello-world)
What I'm trying to do is pulling offical image "docker/whalesay" and run it as follows:
docker run docker/whalesay
Unable to find image 'docker/whalesay:latest' locally
latest: Pulling from docker/whalesay
e190868d63f8: Pull complete
909cd34c6fd7: Pull complete
0b9bfabab7c1: Pull complete
a3ed95caeb02: Pull complete
00bf65475aba: Pull complete
c57b6bcc83e3: Pull complete
8978f6879e2f: Pull complete
8eed3712d2cf: Pull complete
Digest: sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b
Status: Downloaded newer image for docker/whalesay:latest
docker: Error response from daemon: mkdir /var/lib/docker/overlay/fb4b7f34f0963d158856dadccec49963e47716865c83066f7e6eaf0bae057a13-init/merged/dev/shm: invalid argument.
See 'docker run --help'.
Here is my docker info:
Containers: 5
Running: 0
Paused: 0
Stopped: 5
Images: 4
Server Version: 1.13.0
Storage Driver: overlay
Backing Filesystem: extfs
Supports d_type: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: 949e6fa
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.702 GiB
Name: iZ25d1y69iaZ
ID: VJAP:FBMM:CQ5I:KIV5:FO47:VJUJ:ECU2:5TOS:JZBE:EUSH:HUFF:NCAZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
I tried to search, not found the same issue. it looks like a file system problem, so here is my df -h and file -s /dev/vda1
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 40G 4.9G 33G 14% /
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 352K 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
tmpfs 380M 0 380M 0% /run/user/1000
.
/dev/vda1: Linux rev 1.0 ext4 filesystem data, UUID=80b9b662-0a1d-4e84-b07b-c1bf19e72d97 (needs journal recovery) (extents) (large files) (huge files)
I'm new to Docker, so this maybe same configuration problem or version issue, but i havnt find it out.
I appreciate any suggestions and answers!

I was in SLES12 and i fixed the issue by
service stop docker
vi /etc/docker/daemon.json (This is a new file)
Add following entry
{
"storage-driver": "devicemapper"
}
service start docker

Edit: GitHub issue with the discussion of this problem.
I remember there was an issue with 3.10 kernel and overlayfs/ext4/xfs filesystems, and some people noticed that it started working again with a more recent kernel (I think it was in 3.18 that the overlayfs module was added to the kernel).
So if upgrading the kernel is an option to you, you can check if overlayfs+ext4 works.
If a kernel upgrade is not an option, then I guess your only option is to use another storage driver (aufs should not be available, so device mapper)

Related

How to list only images located in a specific, private registry

I'm having problems getting a listing of images from a specific registry that I've set up on a local server, or, maybe, I'm having issues publishing them to that registry in the first place, as this is my first adventure into docker registries, I may just be confused with the terms used.
There's an old question, here, that kind of looks like what I want to achieve, but it appears that docker has gained built-in support for this, in the meanwhile, so the methods mentioned here are no longer relevant.
I have 2 servers (for the purpose of this question):
rancher-server: This server has a rancher:v2.6.0 container running and a registry:2 container.
k8s-server: This is just a freshly installed server, with the docker and kubernetes packages installed, that I want the rancher server to administer.
On k8s-server, I'm trying to spin up a docker image rancher/rancher-agent:v2.6.0 with a few arguments, that should let it relinquish control to the rancher server.
The trick here is, that this is all required to work without internet access (currently there IS internet access, but it's a PoC for a task that requires to be air-gapped). For the purposes of this question, I really just want to be able to spin up docker containers on k8s-server, using the registry on rancher-server.
Currently, this is the state of rancher-server:
# docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b9a15ea00d5e registry:2 "/entrypoint.sh /e..." About an hour ago Up About an hour 0.0.0.0:5000->5000/tcp local-registry
1b6bc6b88a8e 08c9693b4357 "entrypoint.sh 08c..." 26 hours ago Up 2 hours 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp goofy_minsky
# docker image ls --all (the list is big, this is just a sample):
REPOSITORY TAG IMAGE ID CREATED
rancher/rancher-agent v2.6.0 9c35a790aa16 2 weeks ago
rancher-server.example.com:5000/rancher/rancher-agent v2.6.0 9c35a790aa16 2 weeks ago
# docker info
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 225
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 66aedde759f33c190954815fb765eedc1d782dd9 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-1160.41.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 3.701 GiB
Name: rancher-server
ID: SA2T:G2IA:CGER:6BC5:HIV2:4T6T:LF3Q:2YVS:SYU7:SQ5V:ACUS:BMEX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
rancher-server.example.com:5000
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
On the k8s-server, I try to list the contents of that registry:
# docker image ls --all rancher-server.example.com:5000
REPOSITORY TAG IMAGE ID CREATED SIZE
# docker info
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 66aedde759f33c190954815fb765eedc1d782dd9 (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: fec3683b971d9c3ef73f284f176672c44b448662 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 3.10.0-1160.41.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 3
CPUs: 2
Total Memory: 3.701 GiB
Name: k8s-server
ID: QETJ:QSPQ:VS36:OOOA:ZPYL:CDHK:AJ5G:N4BD:ZQUH:UL6O:PHAB:5UOE
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
rancher-server.example.com:5000
127.0.0.0/8
Live Restore Enabled: false
Registries: docker.io (secure)
I had to jump through a few hoops to get there, in the first place, marking the registry as unsafe in /etc/docker/daemon.json on the k8s-server and disabling selinux on the rancher-server, for example.
I've tried to docker login rancher-server.example.com:5000 first, but that made no difference. It does look like, to me, that the k8s-server is configured correctly, but that the images on rancher-server haven't been tagged/pushed properly, but when I look back at the registry, I don't know how to do it differently, and, as far as I understand the registry, it looks fine to me?
I've changed the server names for anonymity and the output has been lightly edited for presentation.
EDIT:
I think I found a clue to what's happening here, it turns out that I can actually run the images from this registry remotely, just fine, it just so happens that I have no way to discover the names of the images, however, if I do a docker run -d --privileged --restart=unless-stopped --net=host -v /etc/kubernetes:/etc/kubernetes -v /var/run:/var/run rancher-server.example.com:5000/rancher/rancher-agent:v2.6.0 --server https://rancher-server.example.com:5000 --token <token> --ca-checksum <ca-checksum> --etcd --controlplane it actually pulls and runs the container, so it looks like the registry itself is fine, but maybe the index isn't?

ESP8266: "No space left on device" error when building nodemcu firmware

I am trying build the nodemcu firmware with a docker on a windows 10 system.
When I try to build the nodemcu firmware, I have this error:
(...)
PRUNE libmain.a libc.a
/opt:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/nodemcu-firmware/tools/toolchains/esp8266-linux-x86_64-20190731.0/bin
xtensa-lx106-elf-ar: /opt/nodemcu-firmware/sdk/esp_iot_sdk_v3.0-e4434aa/lib/libc.a: No space left on device
Makefile:334: recipe for target '/opt/nodemcu-firmware/sdk/.pruned-3.0-e4434aa' failed
make: *** [/opt/nodemcu-firmware/sdk/.pruned-3.0-e4434aa] Error 1
make: Leaving directory '/opt/nodemcu-firmware'
I tried docker system prune but this error persists.
I tried execute docker info but doesn't help me:
Client:
Debug Mode: false
Server:
Containers: 1
Running: 0
Paused: 0
Stopped: 1
Images: 12
Server Version: 19.03.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.14.131-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.419GiB
Name: docker-desktop
ID: RCEX:VYON:IG6T:AXCF:CSLX:3NVK:V453:DTSP:HW4Y:EKBY:PCVW:2UOJ
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 28
Goroutines: 44
System Time: 2020-12-21T02:04:22.387341144Z
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
When I execute docker system df, I have this result:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 4 1 673.8MB 209.8MB (31%)
Containers 1 0 0B 0B
Local Volumes 10 0 0B 0B
Build Cache 0 0 0B 0B
I am new in docker and I don't know what do to from here. Can anyone help me?
Docker tends to need cleanup after you work with it sometimes. There may be previous resources that were created in the past that are no longer in use, this is especially true if you are making use of volumes.
Sometimes, images are not fully created and end up becoming untagged dangling images which for the most part are undesirable. So you have to remove these undesirables that consume space.
docker image prune
docker image prune removes all dangling images.
2. docker volume prune
docker volume prune removes all 'volumes' which are not attached to any containers. These should always be ran before pruning containers otherwise all containers will get deleted this is because pruning containers removes all containers which are not running at the time the command is ran. For the most part in my experience volumes is often what consumes the most amount of resources in terms of space.
docker container prune
For the most part this is optional and you can choose to do this or not
Now even after deleting dangling images there might still be other images that do not get deleted but you no longer need, like for example if you had older versions of the redis image and later download the most recent version of it, you may need to remove the older one if you no longer use it. You will have to delete this manually. Use the following commands:
docker image ls
docker rmi <image id>
The first command gives you a list of all the images you have downloaded, and the second command deletes it. sometimes you may have to use the -f flag on the delete command to force the deletion of an image:
docker rmi -f <image id>

error creating overlay mount to /var/lib/docker/overlay2/

I am getting the below error message:
{"changed": false, "msg": "Error creating container: 500 Server Error:
Internal Server Error ("error creating overlay mount to
/var/lib/docker/overlay2/e17accf17aa46c9d12dbc4da10e399121b476698039cdcf54a5bb42443ffc260-init/merged:
invalid argument")"}
docker info shows below:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: false
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Authorization: rhel-push-plugin
Swarm: inactive
Runtimes: oci runc
Default Runtime: oci
Init Binary: /usr/libexec/docker/docker-init-current
containerd version: 1556cce7e5c5349fdffeb123c092a7681769866a (expected: aa8187dbd3b7ad67d8e5e3a15115d3eef43a7ed1)
runc version: 1556cce7e5c5349fdffeb123c092a7681769866a-dirty (expected: 9df8b306d01f59d3a8029be411de015b7304dd8f)
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Security Options:
seccomp
WARNING: You're not using the default seccomp profile
Profile: /etc/docker/seccomp.json
selinux
Kernel Version: 4.18.7-200.fc28.x86_64
Operating System: Fedora 28 (Server Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 24
Total Memory: 94.34 GiB
Name: xxxxxx.com
ID: U63T:T7ZF:HWDJ:4NKZ:YUW2:KO6W:7RI5:Y6MY:VKSQ:DJSL:FGD5:QMZS
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true
Registries: docker.io (secure), registry.fedoraproject.org (secure), quay.io (secure), registry.access.redhat.com (secure), registry.centos.org (secure), docker.io (secure)
Here Backing filesystem shows unknown here. How to change it to xfs or somerelevant to work?
I was able to resolve this by doing a factory reset in Docker.
Docker for Mac -> Troubleshooting (it is the "bug" icon next to the Settings icon) -> Reset Factory Defaults
$ vim /etc/docker/daemon.json
{
"max-concurrent-uploads": 1
}
$ service docker restart
Posting it in case it helps someone.
I had the same issue with the docker. It was not able to push images to GCR. The above was able to solve my issue.
First, I think you can check the /var/lib/docker/overlay2 directory using xfs_info whether or not it can be supported the overlay2.
For instance,
$ xfs_info /var/lib/docker/overlay2 | grep ftype
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
If the output is ftype=1, then it can be supported the overlay2.
I hope it help you. :^)
I have just meet the same issue. And I fixed it by change the disk volume type used by docker into Ext4 or change the path into a Ext4 volume by this command in docker.service
ExecStart=/usr/bin/dockerd --graph /new-path/docker
It might happen because daemon.json doesn't exist. For create it you can do following steps: open "Docker Desktop" > "Preferences" > "Docker Engine" > edit the texteditor under "Configure the Docker daemon by typing a json Docker daemon configuration file" > "Apply & Restart".

docker is full, all inodes are used

got huge problem, all my inodes seems to be used.
I've cleaned all volumes unused
Cleaned all container and images
with command -> docker prune
but still it seems that it stay full :
Filesystem Inodes IUsed IFree IUse% Mounted on
none 3200000 3198742 1258 100% /
tmpfs 873942 16 873926 1% /dev
tmpfs 873942 13 873929 1% /sys/fs/cgroup
/dev/sda1 3200000 3198742 1258 100% /images
shm 873942 1 873941 1% /dev/shm
tmpfs 873942 1 873941 1% /sys/firmware
docker info
Containers: 5
Running: 3
Paused: 0
Stopped: 2
Images: 23
Server Version: 17.06.1-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 53
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 6.668GiB
Name: serveur-1
ID: CW7J:FJAH:S4GR:4CGD:ZRWI:EDBY:AYBX:H2SD:TWZO:STZU:GSCX:TRIC
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
The only thing i think can do this, is a build i'm doing on this machine.
This build runs a npm install with many files.
Can these files stays on server ?
is there any chance i have to delete these temporary files ?
Is there any dangling volumes left in the system? If you have dangling volumes, it may fill up your disk space.
List all dangling volumes
docker volume ls -q -f dangling=true
Remove all dangling volumes
docker volume rm `docker volume ls -q -f dangling=true`
Found the error,
this seems to be Docker 17.06.1-ce error.
This version seems not correctly deleting images, and keeping files in /var/lib/docker/aufs/mnt/
So just upgrade to new docker version and this will be fine.
now df show me
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 51558236 3821696 45595452 8% /
udev 10240 0 10240 0% /dev
tmpfs 1398308 57696 1340612 5% /run
tmpfs 3495768 0 3495768 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 3495768 0 3495768 0% /sys/fs/cgroup
This is better :)
I had the same problem. Had Jenkins running inside Docker with a volume attached to it. After a few weeks Jenkins told me "npm WARN tar ENOSPC: no space left on device". After some googling I found out that all inodes are taken with sudo df -ih. With sudo find . -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n I could locate the folder using up all the inodes and it was a certain build with npm. Deleted that folder and now I'm good to go again.
This can also be an effect of a lot of stopped containers, for example if there's a cron job running that use a container, and the docker run commandline used does not include --rm - in that case, every cron invocation will leave a stopped container on the filesystem.
In this case, the output of docker info will show a high number under Server -> Containers -> Stopped.
To cure this:
docker container prune
Add --rm to your docker run command line in the cron job.
In my case it was dangling build cache because removing dangling images does not solve the issue.
This cache can be removed by following command: docker system prune --all --force, but be careful maybe you still need some volumes or images.
In my case, pods with ingress-nginx and modsecurity active are creating a lot of dirs & files on container volumes in mod security structure (/var/lib/docker/overlay2/...) after more than 80 days of execution.
Restarting the pods remove the problem.
This case can be generalize to other cases with container internal storage no controlled

docker-registry disk full and no ideas to diet it

I have a server with a docker registry, and have pushed a lot of times a build the same :latest tag now my DD is full and I can't get how to diet it.
disk is full
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 48G 45G 397M 100% /
udev 10M 0 10M 0% /dev
tmpfs 794M 81M 713M 11% /run
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
/dev/dm-1 9.8G 56M 9.2G 1% /var/lib/docker/devicemapper/mnt/2e895760700ac3e1575e496a4ac6adde4de6129226febba8c0c3126af1655ad9
shm 64M 0 64M 0% /var/lib/docker/containers/5aa47e34d1b8be22deeae473729b4e587e6e4bfe7fb3e262eda891bad4b05042/shm
there is no dangling volume nor images
# docker volume ls -qf dangling=true
#
# docker images -f "dangling=true" -q
#
docker images
[root#kvm22:/etc/cron.daily] # docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
jwilder/nginx-proxy 0.5.0 72b65b5a6f38 4 weeks ago 248.4 MB
registry 2 c9bd19d022f6 11 weeks ago 33.27 MB
registry 2.5 c9bd19d022f6 11 weeks ago 33.27 MB
disk usage
# du -h -d 7 /var/lib/docker/volumes/
12K /var/lib/docker/volumes/24000fbe2e81da06924be8f7ce81e07101824036bca5f87d4d811f2a6f7bfa7b/_data
16K /var/lib/docker/volumes/24000fbe2e81da06924be8f7ce81e07101824036bca5f87d4d811f2a6f7bfa7b
42G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/blobs/sha256
42G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/blobs
5.9M /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/repositories/labor-prod
5.9M /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2/repositories
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry/v2
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker/registry
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data/docker
43G /var/lib/docker/volumes/registry_docker-registry-volume/_data
43G /var/lib/docker/volumes/registry_docker-registry-volume
43G /var/lib/docker/volumes/
Output of docker version:
# docker --version
Docker version 1.12.4, build 1564f02
Output of docker info:
# docker info
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 5
Server Version: 1.12.4
Storage Driver: devicemapper
Pool Name: docker-8:1-1184923-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: ext4
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 1.058 GB
Data Space Total: 107.4 GB
Data Space Available: 3.036 GB
Metadata Space Used: 2.142 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.145 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.90 (2014-09-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options:
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.873 GiB
Name: kvm22
ID: G6OC:EKKY:ER4W:3JVZ:25BI:FF2Y:YXVA:RZRR:WPAP:SASB:AJJA:DM6J
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Insecure Registries:
127.0.0.0/8
I had the same problem. I can't believe there's no ready solution for this. Anyway I hacked a tool together and it seems to work.
You can find it here: https://github.com/Richie765/docker-tools
Basically it uses a bash script to find out which manifests are untagged. Then deletes them through the registry API. Afterwards you can run a garbage collection to actually delete the data.
I'm sure the script isn't perfect. Any improvements are welcome!
Found another tool here.
https://github.com/burnettk/delete-docker-registry-image
Included a clean_old_version.py script.
I give it a try too.
In case anyone still has this problem:
This is the 'reset' way how I solved it:
You can stop and delete the registry
docker stop registry && docker rm -v registry
and restart it afterwards:
docker run -d -p 5000:5000 --restart=always --name registry registry:2
Then you would have to rebuild your images localy and push them to the registry again.

Resources