I want to run container and receive error:
docker run -ti --rm grafana/promtail:2.5.0 -config.file=/etc/promtail/config.yml
docker: Error response from daemon: mkdir /var/lib/docker/overlay2/0cad6a6645e2445a9985d5c9e9c6909fa74ee1a30425b407ddfac13684bd9d31-init: no space left on device.
At first, I thought I have a lot of volumes and images cached. So I clean docker with:
docker prune
docker builder prune
But in a while, the same error occur. When I check my Docker Desktop configuration, I can see I am using all available disk size for images:
Disk image size:
59.6 GB (59.5 GB used)
I have 13 images on my system and together its less than 5GB:
REPOSITORY TAG IMAGE ID CREATED SIZE
logstashloki latest 157966144f3b 3 days ago 761MB
minio/minio <none> 717586e37f7f 4 days ago 232MB
grafana/grafana <none> 31a8875955e5 9 days ago 277MB
docker.elastic.co/beats/filebeat 8.3.2 e7b210caf528 3 weeks ago 295MB
k8s.gcr.io/kube-apiserver v1.24.0 b62a103951f4 2 months ago 126MB
k8s.gcr.io/kube-scheduler v1.24.0 b81513b3bfb4 2 months ago 50MB
k8s.gcr.io/kube-controller-manager v1.24.0 59fad34d4fe0 2 months ago 116MB
k8s.gcr.io/kube-proxy v1.24.0 66e1443684b0 2 months ago 106MB
k8s.gcr.io/etcd 3.5.3-0 a9a710bb96df 3 months ago 178MB
grafana/promtail 2.5.0 aa21fd577ae2 3 months ago 177MB
grafana/loki 2.5.0 369cbd28ef9b 3 months ago 60MB
k8s.gcr.io/pause 3.7 e5a475a03805 4 months ago 514kB
k8s.gcr.io/coredns/coredns v1.8.6 edaa71f2aee8 9 months ago 46.8MB
From output of docker system df there is no suspicious size of container, images or volumes:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 13 13 2.35GB 69.57MB (2%)
Containers 21 21 35.15kB 0B (0%)
Local Volumes 2 0 2.186MB 2.186MB (100%)
Build Cache 20 0 0B 0B
I am new to MacOS and cannot determine what take all my space and how to clean all that space and where are all that data stored on system?
Related
What does "in use" mean and how can I get that info from the CLI?
Reference: docker ps -a --format "table {{.ID}}\t{{.Names}}\t{{.State}}\t{{.Image}}"
CONTAINER ID
NAMES
STATE
IMAGE
07bce6924796
laughing_wozniak
exited
vsc-volume-bootstrap
6d37d8744a77
angry_brahmagupta
exited
vsc-quickstarts-d91f349952ba5208420f1403c31b2955-uid
0bce117a827c
dapr_placement
running
daprio/dapr:1.5.0
1232bf715593
dapr_zipkin
running
openzipkin/zipkin
c128e546a0b6
dapr_redis
running
redis
dc44e1006831
miked
exited
my-first-fsharp-web
ce3cf77a0eb9
minikube
running
gcr.io/k8s-minikube/kicbase:v0.0.28
Reference: docker image ls
REPOSITORY
TAG
IMAGE ID
CREATED
SIZE
vsc-microsoftvscodeinsiders-572e00dcd0f79c5ee8d7d39c18e7c701-features
latest
9f05ea6535d4
12 hours ago
6.36GB
vsc-volume-bootstrap
latest
81646861762b
12 hours ago
180MB
vsc-quickstarts-d91f349952ba5208420f1403c31b2955-uid
latest
453bd2943e10
40 hours ago
9.7GB
vsc-quickstarts-d91f349952ba5208420f1403c31b2955
latest
10de525681a7
40 hours ago
9.7GB
openzipkin/zipkin
latest
6a9714eacfd9
2 days ago
153MB
miked.azurecr.io/my-first-fsharp-web
96e7948ee30c
5 days ago
211MB
my-first-fsharp-web
latest
e36fabe64a1c
5 days ago
211MB
miked.azurecr.io/my-first-fsharp-web
1
e36fabe64a1c
5 days ago
211MB
miked.azurecr.io/my-first-fsharp-web
latest
e36fabe64a1c
5 days ago
211MB
counter-image
latest
22dfe0305c55
7 days ago
208MB
redis
latest
40c68ed3a4d2
8 days ago
113MB
daprio/dapr
1.5.0
bff1855a0302
2 weeks ago
214MB
vsc-azure-container-apps-demo-41dcd784881293406771e08c255554b9
latest
1af591496e8a
4 weeks ago
337MB
gcr.io/k8s-minikube/kicbase
v0.0.28
e2a6c047bedd
8 weeks ago
1.08GB
It indicates if the image is used by a container (running or already stopped).
You cannot get this via the CLI using docker images, but listing the containers docker ps -a you can see the associated image.
I have docker in version 18.03.1-ce that support command docker system df. His output:
Images space usage:
REPOSITORY TAG IMAGE ID CREATED ago SIZE SHARED SIZE UNIQUE SiZE CONTAINERS
registry.gitlab.com/precisesale/app latest b7833546c2cf About an hour ago ago 252.1MB 123.8MB 128.4MB 1
healthdiary/app latest 565c6d3906e6 2 days ago ago 312.2MB 123.8MB 188.4MB 1
mongo latest f93ff881751f 5 days ago ago 367.6MB 0B 367.6MB 2
nginx latest b175e7467d66 6 weeks ago ago 108.9MB 0B 108.9MB 1
jwilder/docker-gen latest 8959ee34c769 2 months ago ago 19.91MB 4.148MB 15.77MB 1
jrcs/letsencrypt-nginx-proxy-companion latest 17939ceb7a52 2 months ago ago 86.86MB 4.148MB 82.71MB 1
Containers space usage:
CONTAINER ID IMAGE COMMAND LOCAL VOLUMES SIZE CREATED ago STATUS NAMES
c20dc3438552 healthdiary/app "./entrypoint.sh nod…" 0 0B 8 minutes ago ago Up 8 minutes healthdiary_app_1
bf8c4307dcbb mongo:latest "docker-entrypoint.s…" 1 0B 8 minutes ago ago Up 8 minutes healthdiary_mongo_1
47fced8d18fe registry.gitlab.com/precisesale/app "./entrypoint.sh nod…" 0 0B 9 minutes ago ago Up 9 minutes precisesale_app_1
597d97d5c1fa mongo:latest "docker-entrypoint.s…" 1 0B 9 minutes ago ago Up 9 minutes precisesale_db_1
b5bb14faa910 jwilder/docker-gen "/usr/local/bin/dock…" 0 0B 7 hours ago ago Up 19 minutes nginx-gen
8eee2bee084a nginx "nginx -g 'daemon of…" 0 2B 7 hours ago ago Up 19 minutes nginx-web
6b8b0cd5d938 jrcs/letsencrypt-nginx-proxy-companion "/bin/bash /app/entr…" 0 1.66kB 7 hours ago ago Up 19 minutes nginx-letsencrypt
Local Volumes space usage:
VOLUME NAME LINKS SIZE
0a40fac6ca98e776dad972c8193362a51a485b3305979e58996545d97310a3c7 1 0B
929b0b88849ad4d390efd4666e6a0e5f82e0e6dd34f7a09f609de90b190e6148 1 0B
Build cache usage: 0B
Even if I do not take into account savings from shared space from two first containers summary size is 1147.5 MB
But if I measure size of docker overlay2 on disc by du I get
du -hs /var/lib/docker/overlay2/
2.7G /var/lib/docker/overlay2/
Where is reason of difference in size of containers measured by docker system df and du?
I was wondering the same thing some time ago.
It’s not a bug, it’s a feature :-)
du -sh /var/lib/docker/overlay2
is not showing objective value because merge folders have been mounted using overlay driver and du output is not actual disk allocation size.
You can see the actual disk allocation size by examining only diff folders like:
du -shc /var/lib/docker/overlay2/*/diff
You can test this in your environment like this:
run
df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2
Now start 20 centos containers and observe what has change:
for i in {1..20}; do docker run -itd centos bash; done
df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2
You can see that the actual disk allocation (df command) is just cca 200MB more than before, but “du” on whole folder outputs 4.2G allocation.
“du” on “diff” folders shows 212M what is correct.
This is how Docker works and what makes it great!
I need to mount host directory /data to 6 containers h1,h2,h3,h4,h5,h6
/data is an external hard disk mounted on the host. The 6 containers can be opened and closed easily.
The 6 containers will go into their own sub-directories of /data to analyze data independently and produce new data locally. All sub-directories have nothing to do with each other.
A relevant question is here, but no preferred answer is given.
How to do that? Below are the containers and images I have now.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d9bd9334a1e7 ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h6
23679fe7252b ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h5
e2864e38e746 ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h4
c8996a304638 ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h3
9acd2a223d86 ubuntu "/usr/bin/bash" 19 hours ago Up 18 hours h2
5690b8c7b6da ubuntu "/usr/bin/bash" 2 days ago Up 12 hours h1
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/hello-world latest f2a91732366c 2 months ago 1.85 kB
docker.io/ubuntu 27 422dc563ca32 2 months ago 252 MB
docker.io/ubuntu latest 422dc563ca32 2 months ago 252 MB
IF those containers are already running, you cannot easily add /data to them.
Except maybe with docker cp.
But the best practice remains either:
make images with /data already in it (Dockerfile ADD)
or use the existing image and launch your container, but with the -v (volume) option. See Use volumes.
We are using Docker and have a production issue. We are running out of space in our root volume since docker device mapper is eating up space.
Is there a way to reduce the disk space to a proper size without having to remove Docker and reinstall from scratch?
The docker version we are using is
Client:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
Server:
Version: 1.8.2
API version: 1.20
Go version: go1.4.2
Git commit: 0a8c2e3
Built: Thu Sep 10 19:08:45 UTC 2015
OS/Arch: linux/amd64
We have 20 GB root volume and 12GB is used by docker. We are left with only few GB of space. The below docker file is 12GB
/var/lib/docker/devicemapper/devicemapper/data
We have also tried removing unused old docker images by using the command
docker images -q |xargs docker rmi
but still it doesn't work.
Appreciate if someone could let me know if there is any way to reduce the disk space to a proper size WITHOUT having to remove docker and reinstall from scratch? Also we don't want to copy files to a new location and do a softlink.
The current docker command outputs are
[root#ip-10-202-69-58 ~]# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
92558da1291a dockyardaws.kdc.capitalone.com/statim/plexus:27.6 "/root/go/src/plexus/" 19 hours ago Up 19 hours quality
2a0497fdbaa7 dockyardaws.kdc.capitalone.com/statim/plexus:21.9 "/root/go/src/plexus/" 21 hours ago Up 21 hours rmCallMetrics
1f5f93604e8f dockyardaws.kdc.capitalone.com/statim/loanservicing:8.3 "/root/go/src/loanser" 2 days ago Up 2 days ls_vznagtingestor
88b4d764c03c dockyardaws.kdc.capitalone.com/statim/plexus:21.3 "/root/go/src/plexus/" 3 days ago Up 3 days dlrMgmt
bd0ecfe2f9ae dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days userRMAttributesIngestor
8bd3d626f7d7 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhPointsPerHour
383e257d2f31 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhPrimeSameDayContracts
6727c76221e4 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days rbcARMAgentsIngestor
527327182f0a dockyardaws.kdc.capitalone.com/statim/plexus:19.6 "/root/go/src/plexus/" 3 days ago Up 3 days dhDealerOrg
1ece1b0f61a7 dockyardaws.kdc.capitalone.com/statim/plexus:25.6 "/root/go/src/plexus/" 3 days ago Up 3 days shawMasterFileLvl1
8aefc887e1f5 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhExhaustEveryEffort
94b89a019b8c dockyardaws.kdc.capitalone.com/statim/plexus:22.1 "/root/go/src/plexus/" 3 days ago Up 3 days asmSummaryWeeklyMetrics
e64e9fe9c7e2 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhForecastedMetrics
f39c5fef33d3 dockyardaws.kdc.capitalone.com/statim/plexus:12.1 "/root/go/src/plexus/" 3 days ago Up 3 days dhMTDContracts
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]# docker ps -qa
92558da1291a
2a0497fdbaa7
1f5f93604e8f
88b4d764c03c
bd0ecfe2f9ae
8bd3d626f7d7
383e257d2f31
6727c76221e4
527327182f0a
1ece1b0f61a7
8aefc887e1f5
94b89a019b8c
e64e9fe9c7e2
f39c5fef33d3
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
dockyardaws.kdc.capitalone.com/statim/plexus 27.6 75fd00165ca4 19 hours ago 976.7 MB
dockyardaws.kdc.capitalone.com/statim/loanservicing 8.3 84bf8f83af5b 2 days ago 973.4 MB
dockyardaws.kdc.capitalone.com/statim/plexus 25.6 672f56435428 13 days ago 976.9 MB
dockyardaws.kdc.capitalone.com/statim/plexus 22.1 3f0d00cf334a 3 weeks ago 976.4 MB
dockyardaws.kdc.capitalone.com/statim/plexus 21.9 783a4e812972 3 weeks ago 976.4 MB
dockyardaws.kdc.capitalone.com/statim/plexus 21.3 28a70e5a5621 3 weeks ago 976.3 MB
dockyardaws.kdc.capitalone.com/statim/plexus 19.6 b85f1fd3e822 4 weeks ago 975.4 MB
dockyardaws.kdc.capitalone.com/statim/plexus 12.1 ce572ad85088 9 weeks ago 971.8 MB
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]#
[root#ip-10-202-69-58 ~]# docker images -q
75fd00165ca4
84bf8f83af5b
672f56435428
3f0d00cf334a
783a4e812972
28a70e5a5621
b85f1fd3e822
ce572ad85088
[root#ip-10-202-69-58 ~]#
Try removing docker containers which are not running:
docker rm -v $(docker ps -a -q -f status=exited)
And then remove the unused docker images:
docker rmi $(docker images -q)
Let the commands speak for themselves:
on a host called: coreworker
core#coreworker-1 ~ $ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059 1 bbd5d4d98156 31 minutes ago 139.3 MB
ec2-54-169-239-164.ap-southeast-1.compute.amazonaws.com:5000/hub-action-repository latest 66ecb895d185 14 hours ago 856.4 MB
ec2-54-169-239-164.ap-southeast-1.compute.amazonaws.com:5000/hub-ext-node-base 0.12.7 f2f1afc202e4 8 days ago 136.6 MB
...
core#coreworker-1 ~ $
on a host called devhost
core#devhost ~ $ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
<none> <none> 67e45ce93dee 48 minutes ago 725 MB
node 0.12.7 a4b45afffe4a 5 days ago 642.2 MB
jwnintex/mesos-worker latest 42f1b41b0089 5 weeks ago 504 MB
jwnintex/nginx-port-router latest 11edcdf1a5fc 9 weeks ago 126.4 MB
jwnintex/consul latest e66fb6787628 10 weeks ago 69.4 MB
jwnintex/mesos-master latest 187d84106a3e 3 months ago 561.8 MB
jwnintex/marathon latest b1d8dd91146a 3 months ago 699.3 MB
jwnintex/zookeeper latest 9b72d56707c9 4 months ago 304.3 MB
jwnintex/registrator latest b1c29d1a74a9 6 months ago 11.79 MB
but when I do:
core#devhost ~ $ docker images -a hub-docker-repo:5000
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
now run the image on devhost residing in hub-docker-repo
veryify it's up:
core#devhost ~ $ ping hub-docker-repo
PING hub-docker-repo.service.consul (172.17.8.150) 56(84) bytes of data.
64 bytes from coreworker-1.node.dc1.consul (172.17.8.150): icmp_seq=1 ttl=63 time=8.27 ms
64 bytes from coreworker-1.node.dc1.consul (172.17.8.150): icmp_seq=2 ttl=63 time=5.19 ms
now try it
core#devhost ~ $ docker run -d hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059:1
Unable to find image 'hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059:1' locally
Pulling repository hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059
FATA[0000] Error: image 485d5874-c786-4b90-93ac-8db5342a6059:1 not found
Now try it on coreworker which is actually hosting the registry (as a docker image)
core#coreworker-1 ~ $ docker run -d hub-docker-repo:5000/485d5874-c786-4b90-93ac-8db5342a6059:1
98d642c1bafd30569d853e92167c7b4fe720bd67f65ec0d0719ec5a36bb6616f
Why am I not able to run that remote image? Or better, even discover it?
It turns out the image had never been pushed and so couldn't be seen by the other host.