Docker eating up disk space /var/lib/docker/devicemapper/devicemapper - docker

We are using Docker and have a production issue. We are running out of space in our root volume since docker device mapper is eating up space.
Is there a way to reduce the disk space to a proper size without having to remove Docker and reinstall from scratch?
The docker version we are using is
Client:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
We have 20 GB root volume and 12GB is used by docker. We are left with only few GB of space. The below docker file is 12GB
/var/lib/docker/devicemapper/devicemapper/data
We have also tried removing unused old docker images by using the command
docker images -q |xargs docker rmi
but still it doesn't work.
Appreciate if someone could let me know if there is any way to reduce the disk space to a proper size WITHOUT having to remove docker and reinstall from scratch? Also we don't want to copy files to a new location and do a softlink.
The current docker command outputs are
[root#ip-10-202-69-58 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
92558da1291a dockyardaws.kdc.capitalone.com/statim/plexus:27.6 "/root/go/src/plexus/" 19 hours ago Up 19 hours quality
2a0497fdbaa7 dockyardaws.kdc.capitalone.com/statim/plexus:21.9 "/root/go/src/plexus/" 21 hours ago Up 21 hours rmCallMetrics
1f5f93604e8f dockyardaws.kdc.capitalone.com/statim/loanservicing:8.3 "/root/go/src/loanser" 2 days ago Up 2 days ls_vznagtingestor
88b4d764c03c dockyardaws.kdc.capitalone.com/statim/plexus:21.3 "/root/go/src/plexus/" 3 days ago Up 3 days dlrMgmt
bd0ecfe2f9ae dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days userRMAttributesIngestor
8bd3d626f7d7 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhPointsPerHour
383e257d2f31 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhPrimeSameDayContracts
6727c76221e4 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days rbcARMAgentsIngestor
527327182f0a dockyardaws.kdc.capitalone.com/statim/plexus:19.6 "/root/go/src/plexus/" 3 days ago Up 3 days dhDealerOrg
1ece1b0f61a7 dockyardaws.kdc.capitalone.com/statim/plexus:25.6 "/root/go/src/plexus/" 3 days ago Up 3 days shawMasterFileLvl1
8aefc887e1f5 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhExhaustEveryEffort
94b89a019b8c dockyardaws.kdc.capitalone.com/statim/plexus:22.1 "/root/go/src/plexus/" 3 days ago Up 3 days asmSummaryWeeklyMetrics
e64e9fe9c7e2 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhForecastedMetrics
f39c5fef33d3 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhMTDContracts
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]# docker ps -qa
92558da1291a
2a0497fdbaa7
1f5f93604e8f
88b4d764c03c
bd0ecfe2f9ae
8bd3d626f7d7
383e257d2f31
6727c76221e4
527327182f0a
1ece1b0f61a7
8aefc887e1f5
94b89a019b8c
e64e9fe9c7e2
f39c5fef33d3
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
dockyardaws.kdc.capitalone.com/statim/plexus 27.6 75fd00165ca4 19 hours ago 976.7 MB
dockyardaws.kdc.capitalone.com/statim/loanservicing 8.3 84bf8f83af5b 2 days ago 973.4 MB
dockyardaws.kdc.capitalone.com/statim/plexus 25.6 672f56435428 13 days ago 976.9 MB
dockyardaws.kdc.capitalone.com/statim/plexus 22.1 3f0d00cf334a 3 weeks ago 976.4 MB
dockyardaws.kdc.capitalone.com/statim/plexus 21.9 783a4e812972 3 weeks ago 976.4 MB
dockyardaws.kdc.capitalone.com/statim/plexus 21.3 28a70e5a5621 3 weeks ago 976.3 MB
dockyardaws.kdc.capitalone.com/statim/plexus 19.6 b85f1fd3e822 4 weeks ago 975.4 MB
dockyardaws.kdc.capitalone.com/statim/plexus 12.1 ce572ad85088 9 weeks ago 971.8 MB
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]# docker images -q
75fd00165ca4
84bf8f83af5b
672f56435428
3f0d00cf334a
783a4e812972
28a70e5a5621
b85f1fd3e822
ce572ad85088
[root#ip-10-202-69-58 ~]#

Try removing docker containers which are not running:
docker rm -v $(docker ps -a -q -f status=exited)
And then remove the unused docker images:
docker rmi $(docker images -q)

Related

Docker no space left on device on Mac M1

I want to run container and receive error:
docker run -ti --rm grafana/promtail:2.5.0 -config.file=/etc/promtail/config.yml
docker: Error response from daemon: mkdir /var/lib/docker/overlay2/0cad6a6645e2445a9985d5c9e9c6909fa74ee1a30425b407ddfac13684bd9d31-init: no space left on device.
At first, I thought I have a lot of volumes and images cached. So I clean docker with:
docker prune
docker builder prune
But in a while, the same error occur. When I check my Docker Desktop configuration, I can see I am using all available disk size for images:
Disk image size:
59.6 GB (59.5 GB used)
I have 13 images on my system and together its less than 5GB:
REPOSITORY TAG IMAGE ID CREATED SIZE
logstashloki latest 157966144f3b 3 days ago 761MB
minio/minio <none> 717586e37f7f 4 days ago 232MB
grafana/grafana <none> 31a8875955e5 9 days ago 277MB
docker.elastic.co/beats/filebeat 8.3.2 e7b210caf528 3 weeks ago 295MB
k8s.gcr.io/kube-apiserver v1.24.0 b62a103951f4 2 months ago 126MB
k8s.gcr.io/kube-scheduler v1.24.0 b81513b3bfb4 2 months ago 50MB
k8s.gcr.io/kube-controller-manager v1.24.0 59fad34d4fe0 2 months ago 116MB
k8s.gcr.io/kube-proxy v1.24.0 66e1443684b0 2 months ago 106MB
k8s.gcr.io/etcd 3.5.3-0 a9a710bb96df 3 months ago 178MB
grafana/promtail 2.5.0 aa21fd577ae2 3 months ago 177MB
grafana/loki 2.5.0 369cbd28ef9b 3 months ago 60MB
k8s.gcr.io/pause 3.7 e5a475a03805 4 months ago 514kB
k8s.gcr.io/coredns/coredns v1.8.6 edaa71f2aee8 9 months ago 46.8MB
From output of docker system df there is no suspicious size of container, images or volumes:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 13 13 2.35GB 69.57MB (2%)
Containers 21 21 35.15kB 0B (0%)
Local Volumes 2 0 2.186MB 2.186MB (100%)
Build Cache 20 0 0B 0B
I am new to MacOS and cannot determine what take all my space and how to clean all that space and where are all that data stored on system?

What does "in use" mean for an image?

What does "in use" mean and how can I get that info from the CLI?
Reference: docker ps -a --format "table {{.ID}}\t{{.Names}}\t{{.State}}\t{{.Image}}"
CONTAINER ID
NAMES
STATE
IMAGE
07bce6924796
laughing_wozniak
exited
vsc-volume-bootstrap
6d37d8744a77
angry_brahmagupta
exited
vsc-quickstarts-d91f349952ba5208420f1403c31b2955-uid
0bce117a827c
dapr_placement
running
daprio/dapr:1.5.0
1232bf715593
dapr_zipkin
running
openzipkin/zipkin
c128e546a0b6
dapr_redis
running
redis
dc44e1006831
miked
exited
my-first-fsharp-web
ce3cf77a0eb9
minikube
running
gcr.io/k8s-minikube/kicbase:v0.0.28
Reference: docker image ls
REPOSITORY
TAG
IMAGE ID
CREATED
SIZE
vsc-microsoftvscodeinsiders-572e00dcd0f79c5ee8d7d39c18e7c701-features
latest
9f05ea6535d4
12 hours ago
6.36GB
vsc-volume-bootstrap
latest
81646861762b
12 hours ago
180MB
vsc-quickstarts-d91f349952ba5208420f1403c31b2955-uid
latest
453bd2943e10
40 hours ago
9.7GB
vsc-quickstarts-d91f349952ba5208420f1403c31b2955
latest
10de525681a7
40 hours ago
9.7GB
openzipkin/zipkin
latest
6a9714eacfd9
2 days ago
153MB
miked.azurecr.io/my-first-fsharp-web
96e7948ee30c
5 days ago
211MB
my-first-fsharp-web
latest
e36fabe64a1c
5 days ago
211MB
miked.azurecr.io/my-first-fsharp-web
1
e36fabe64a1c
5 days ago
211MB
miked.azurecr.io/my-first-fsharp-web
latest
e36fabe64a1c
5 days ago
211MB
counter-image
latest
22dfe0305c55
7 days ago
208MB
redis
latest
40c68ed3a4d2
8 days ago
113MB
daprio/dapr
1.5.0
bff1855a0302
2 weeks ago
214MB
vsc-azure-container-apps-demo-41dcd784881293406771e08c255554b9
latest
1af591496e8a
4 weeks ago
337MB
gcr.io/k8s-minikube/kicbase
v0.0.28
e2a6c047bedd
8 weeks ago
1.08GB
It indicates if the image is used by a container (running or already stopped).
You cannot get this via the CLI using docker images, but listing the containers docker ps -a you can see the associated image.

Diffrence between real and showed by docker system df disc usage

I have docker in version 18.03.1-ce that support command docker system df. His output:
Images space usage:
REPOSITORY TAG IMAGE ID CREATED ago SIZE SHARED SIZE UNIQUE SiZE CONTAINERS
registry.gitlab.com/precisesale/app latest b7833546c2cf About an hour ago ago 252.1MB 123.8MB 128.4MB 1
healthdiary/app latest 565c6d3906e6 2 days ago ago 312.2MB 123.8MB 188.4MB 1
mongo latest f93ff881751f 5 days ago ago 367.6MB 0B 367.6MB 2
nginx latest b175e7467d66 6 weeks ago ago 108.9MB 0B 108.9MB 1
jwilder/docker-gen latest 8959ee34c769 2 months ago ago 19.91MB 4.148MB 15.77MB 1
jrcs/letsencrypt-nginx-proxy-companion latest 17939ceb7a52 2 months ago ago 86.86MB 4.148MB 82.71MB 1
Containers space usage:
CONTAINER ID IMAGE COMMAND LOCAL VOLUMES SIZE CREATED ago STATUS NAMES
c20dc3438552 healthdiary/app "./entrypoint.sh nod…" 0 0B 8 minutes ago ago Up 8 minutes healthdiary_app_1
bf8c4307dcbb mongo:latest "docker-entrypoint.s…" 1 0B 8 minutes ago ago Up 8 minutes healthdiary_mongo_1
47fced8d18fe registry.gitlab.com/precisesale/app "./entrypoint.sh nod…" 0 0B 9 minutes ago ago Up 9 minutes precisesale_app_1
597d97d5c1fa mongo:latest "docker-entrypoint.s…" 1 0B 9 minutes ago ago Up 9 minutes precisesale_db_1
b5bb14faa910 jwilder/docker-gen "/usr/local/bin/dock…" 0 0B 7 hours ago ago Up 19 minutes nginx-gen
8eee2bee084a nginx "nginx -g 'daemon of…" 0 2B 7 hours ago ago Up 19 minutes nginx-web
6b8b0cd5d938 jrcs/letsencrypt-nginx-proxy-companion "/bin/bash /app/entr…" 0 1.66kB 7 hours ago ago Up 19 minutes nginx-letsencrypt
Local Volumes space usage:
VOLUME NAME LINKS SIZE
0a40fac6ca98e776dad972c8193362a51a485b3305979e58996545d97310a3c7 1 0B
929b0b88849ad4d390efd4666e6a0e5f82e0e6dd34f7a09f609de90b190e6148 1 0B
Build cache usage: 0B
Even if I do not take into account savings from shared space from two first containers summary size is 1147.5 MB
But if I measure size of docker overlay2 on disc by du I get
du -hs /var/lib/docker/overlay2/
2.7G /var/lib/docker/overlay2/
Where is reason of difference in size of containers measured by docker system df and du?
I was wondering the same thing some time ago.
It’s not a bug, it’s a feature :-)
du -sh /var/lib/docker/overlay2
is not showing objective value because merge folders have been mounted using overlay driver and du output is not actual disk allocation size.
You can see the actual disk allocation size by examining only diff folders like:
du -shc /var/lib/docker/overlay2/*/diff
You can test this in your environment like this:
run
df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2
Now start 20 centos containers and observe what has change:
for i in {1..20}; do docker run -itd centos bash; done
df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2
You can see that the actual disk allocation (df command) is just cca 200MB more than before, but “du” on whole folder outputs 4.2G allocation.
“du” on “diff” folders shows 212M what is correct.
This is how Docker works and what makes it great!

deleting old images in Docker - OSX

I've been toying with a docker image for Tensorflow.
To summarize, I first installed the standard image, then realized I needed nodejs, so added it and did a docker commit. Then realized I needed expressJS, added it an did a commit
I am running docker v1.12.5 (so the new gc/prune commands are not there)
At this stage, docker images -a shows:
REPOSITORY TAG IMAGE ID CREATED SIZE
tensor-node-express latest f2f59eb61aae 15 hours ago 2.104 GB
gcr.io/tensorflow/tensorflow latest-devel 308238445d5c 2 days ago 1.995 GB
gcr.io/tensorflow/tensorflow <none> 74435614a991 9 days ago 1.52 GB
I only want to keep tensor-node-express and delete the older images.
$ docker rmi 308238445d5c
Error response from daemon: conflict: unable to delete 308238445d5c (cannot be forced) - image has dependent child images
$docker rmi gcr.io/tensorflow/tensorflow:latest-devel
Error response from daemon: conflict: unable to remove repository reference "gcr.io/tensorflow/tensorflow:latest-devel" (must force) - container 03de9d864e31 is using its referenced image 308238445d5c
I assumed that this means docker commits store differential images, but when I go to ~/.docker/machine/machines/default, I see:
40894464 Mar 13 13:57 boot2docker.iso
5043847168 Mar 16 08:34 disk.vmdk
I suppose the 5G file is a composite of my images, which seems to show each docker commit is the full image!
Any thoughts on how I can only use the latest docker image (tensor-node-express) and free my HD of the invasion of docker?
Supplementary info - here is the output of docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e6dcd2915991 tensor-node-express "/bin/bash" 15 hours ago Exited (130) 15 hours ago flamboyant_bose
fb44b19a21c2 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 18 hours ago Exited (130) 15 hours ago compassionate_bose
075001a687e3 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 18 hours ago Exited (0) 18 hours ago nervous_sinoussi
a80ce2d2e688 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 19 hours ago Exited (130) 18 hours ago happy_euclid
f493bd3c8712 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 19 hours ago Exited (1) 19 hours ago friendly_cori
03de9d864e31 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 2 days ago Exited (255) 23 minutes ago 6006/tcp, 8888/tcp tender_hopper
2dd1e83d62d3 gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 2 days ago Exited (0) 15 hours ago modest_einstein
3067ed171b1c gcr.io/tensorflow/tensorflow:latest-devel "/bin/bash" 2 days ago Exited (0) 2 days ago dazzling_bhabha
62c699afd3fd 74435614a991 "/bin/bash" 2 days ago Exited (127) 2 days ago inspiring_austin
9523ffe2945c 74435614a991 "/bin/bash" 2 days ago Exited (0) 2 days ago kickass_leakey
e06958ea517c 74435614a991 "/bin/bash" 2 days ago Exited (0) 2 days ago objective_euler
ccf922954667 74435614a991 "/bin/bash" 2 days ago Exited (255) 2 days ago dreamy_bartik
fad0d92a07a3 74435614a991 "/bin/bash" 2 days ago Exited (130) 2 days ago elastic_dubinsky
f2a98d4e11ea 74435614a991 "/bin/bash" 2 days ago Exited (0) 2 days ago heuristic_kilby
f07e46367b17 74435614a991 "/bin/bash" 2 days ago Exited (130) 2 days ago trusting_darwin
5bbf9cf992b8 74435614a991 "/bin/bash" 2 days ago Exited (0) 2 days ago flamboyant_knuth
I tried
docker ps --filter "status=exited" | grep "days ago" | awk '{print $1}' | xargs docker rm (credit)
I ran the above manually as well for some of the containers it missed
That pruned the ps list to:
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e6dcd2915991 tensor-node-express "/bin/bash" 15 hours ago Exited (130) 15 hours ago flamboyant_bose
But even then I can't delete old images - same error.
Further update, I tried to list dependencies in images (credit)
using this script:
for i in $(docker images -q)
do
docker history $i | grep -q 74435614a991 && echo $i
done | sort -u
And it told me:
308238445d5c
74435614a991
f2f59eb61aae
This means my new images are child images of the old image. But the size is not a differential looking at the disk size.
Thoughts?
docker-machine uses a Linux VM
When you looked at the docker-machine .vdmk and .iso files, what you are looking at is files for a Linux VM running on your Mac. This is needed because Docker requires Linux kernel features to run, it cannot run directly on the Mac's microkernel.
So your Mac is running a Linux virtual machine, and inside that virutal machine is running the Docker daemon and all of your containers.
Therefore the file size of the .vmdk and .iso tell you nothing about any one image.
docker images have parent/child relationships
As you may already know, docker images have parents and/or children. For instance when you build an image with a Dockerfile like this:
FROM ubuntu:latest
RUN apt-get update && apt-get install nginx
You will end up with a new image that you have perhaps tagged my-nginx. But it requires the ubuntu:latest image as its parent; you cannot delete ubuntu:latest with this image still around, as it requires its parent.
docker commit creates those relationships
When you use docker commit, you are basically doing a dynamic snapshot build. It is similar to the above, except there's no Dockerfile involved.
The above example has a FROM line which indicates the image to use as a base. When using commit, there is a base implied - whatever image was used to launch the running container that you are committing.
The above example has a RUN command which will create new contents in the built image, above and beyond the base image. In a real Dockerfile there are usually multiple commands that do various things which build on the base image. When you use commit, you don't have that. Instead, anything that has been written to the container on top of the base image is your new content. It exists in a read-write filesystem layer in the container. That is the thing you are committing; it is written as a new read-only layer and you get that back as a new (immutable, read-only) docker image. With a parent.
Based on your comments, and the question itself, you appear to have believed that using docker commit would create a new full image that had no dependencies on other images. That is not true. You can craft images like that if you build them yourself from scratch, just not this way.
You can untag the image
If what you want is for the image to not show up in your list, that's easy. Just untag it.
docker rmi gcr.io/tensorflow/tensorflow:latest-devel
However, this is more or less cosmetic. The image will still be there, as another image requires it. All this does is remove the tag, so it doesn't appear in the docker images list anymore without the -a flag.
The reason trying this did not work for you is you tried to rmi the image using its ID, not using its tag.

Docker pull not getting image tagnames

I'm running docker with boot2docker on OS X 10.10.
I'm following the main tutorial and doing a docker pull ubuntu
It gets the image okay, however when I then do docker images it only lists ubuntu:latest
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ubuntu latest 5506de2b643b 3 weeks ago 199.3 MB
instead of the full list of images as the tutorial says:
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
training/webapp latest fc77f57ad303 3 weeks ago 280.5 MB
ubuntu 13.10 5e019ab7bf6d 4 weeks ago 180 MB
ubuntu saucy 5e019ab7bf6d 4 weeks ago 180 MB
ubuntu 12.04 74fe38d11401 4 weeks ago 209.6 MB
ubuntu precise 74fe38d11401 4 weeks ago 209.6 MB
ubuntu 12.10 a7cf8ae4e998 4 weeks ago 171.3 MB
ubuntu quantal a7cf8ae4e998 4 weeks ago 171.3 MB
ubuntu 14.04 99ec81b80c55 4 weeks ago 266 MB
ubuntu latest 99ec81b80c55 4 weeks ago 266 MB
ubuntu trusty 99ec81b80c55 4 weeks ago 266 MB
ubuntu 13.04 316b678ddf48 4 weeks ago 169.4 MB
ubuntu raring 316b678ddf48 4 weeks ago 169.4 MB
ubuntu 10.04 3db9c44f4520 4 weeks ago 183 MB
ubuntu lucid 3db9c44f4520 4 weeks ago 183 MB
If I do docker images -a I see this:
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ubuntu latest 5506de2b643b 3 weeks ago 199.3 MB
<none> <none> 22093c35d77b 3 weeks ago 199.3 MB
<none> <none> 3680052c0f5c 3 weeks ago 192.7 MB
<none> <none> e791be0477f2 3 weeks ago 192.7 MB
<none> <none> ccb62158e970 3 weeks ago 192.7 MB
<none> <none> d497ad3926c8 3 weeks ago 192.5 MB
<none> <none> 511136ea3c5a 17 months ago 0 B
Anybody knows why this is happening?
When you do a docker pull <image> you will only get the latest tag for that image. This is expected behaviour.
To pull a specific tag, use docker pull <image>:<tag>.
The list there in the documentation should only be expected if you've followed the full guide and used all those images. You usually only need one tag for an image.

Resources