Is there a way to check how many layers a docker container is composed without going over each docker file that it “inherits” to count how many RUN are there?
yes.
docker history -q <imagenameorid> | wc -l
This will give you the count of layer in the given image
Related
My FS has been filled by my docker container (docker pull dependencytrack/apiserver). It seems it has duplicated itself in many subfolders.
find /var/lib/docker/overlay2/ -name alpine-1.10.2.jar | wc -l
1550
(answer should be closer to 1, I guess).
They all reside in /var/lib/docker/overlay2/[diff|merged]/<random_number1>/tmp/jetty-0_0_0_0-8080-dependency-track-apiserver_jar-dtrack-api-any-<random_number2>/webapp/WEB-INF/lib/
where random_number1 is fixed and random_number2 has many differents values.
Is this problem related to my image, or is it docker ? I don't really know how to proceed from here.
Summary
Given that:
The storage driver docker users is ZFS;
Only docker creates legacy datasets;
Bash:
$ docker ps -a | wc -l
16
$ docker volume ls | wc -l
12
$ zfs list | grep legacy | wc -l
157
16 containers (both running and stopped). 12 volumes. 157 datasets. This seems like an awful lot of legacy datasets. I'm wondering if a lot of them are so orphaned that not even docker knows about them anymore, so they don't get cleaned up.
Rationale
There is a huge list of legacy volumes in my Debian zfs pool. They started appearing when I started using Docker on this machine:
$ sudo zfs list | grep legacy | wc -l
486
They are all in the form of:
pool/var/<64-char-hash> 202K 6,18T 818M legacy
This location is used solely by docker.
$ docker info | grep -e Storage -e Dataset
Storage Driver: zfs
Parent Dataset: pool/var
I started cleaning up.
$ docker system prune -a
(...)
$ sudo zfs list | grep legacy | wc -l
154
That's better. However, I'm only running about 15 containers, and after running docker system prune -a, the history or every container shows that only the last image layer is still available. The rest are <missing> (because they are cleaned up).
$ docker images | wc -l
15
If all containers use only the last image layer after pruning the rest, shouldn't docker only use 15 image layers and 15 running containers, totalling 30 volumes?
$ sudo zfs list | grep legacy | wc -l
154
Can I find out if they are in use by a container/image? Is there a command that traverses all pool/var/<hash> datasets in ZFS and figures out to what docker container/image they belong? Either a lot of them can be removed, or I don't understand how to figure out (beyond just trusting docker system prune) they cannot.
The excessive use of zfs volumes by docker messes up my zfs list command, both visually and performance-wise. Listing zfs volumes now takes ~10 seconds in stead of <1.
Proof that docker sees no more dangling counts
$ docker ps -qa --no-trunc --filter "status=exited"
(no output)
$ docker images --filter "dangling=true" -q --no-trunc
(no output)
$ docker volume ls -qf dangling=true
(no output)
zfs list example:
NAME USED AVAIL REFER MOUNTPOINT
pool 11,8T 5,81T 128K /pool
pool/var 154G 5,81T 147G /mnt/var
pool/var/0028ab70abecb2e052d1b7ffc4fdccb74546350d33857894e22dcde2ed592c1c 1,43M 5,81T 1,42M legacy
pool/var/0028ab70abecb2e052d1b7ffc4fdccb74546350d33857894e22dcde2ed592c1c#211422332 10,7K - 1,42M -
# and 150 more of the last two with different hashes
I had the same question but couldn't find a satisfactory answer. Adding what I eventually found, since this question is one of the top search results.
Background
The ZFS storage driver for Docker stores each layer of each image as a separate legacy dataset.
Even just a handful of images can result in a huge number of layers, each layer corresponding to a legacy ZFS dataset.
Quote from the Docker ZFS driver docs:
The base layer of an image is a ZFS filesystem. Each child layer is a ZFS clone based on a ZFS snapshot of the layer below it. A container is a ZFS clone based on a ZFS Snapshot of the top layer of the image it’s created from.
Investigate
You can check the datasets used by one image by running:
$ docker image inspect [IMAGE_NAME]
Example output:
...
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:f2cb0ecef392f2a630fa1205b874ab2e2aedf96de04d0b8838e4e728e28142da",
...
...
...
"sha256:2e8cc9f5313f9555a4decca744655ed461e21fbe48a0f078ed5f7c4e5292ad2e",
]
},
...
This explains why you can see 150+ datasets created when only running a dozen containers.
Solution
Prune and delete unused images.
$ docker image prune -a
To avoid a slow zfs list, specify the dataset of interest.
Suppose you store docker in tank/docker and other files in tank/data. List only the data datasets by the recursive option:
# recursively list tank/data/*
$ zfs list tank/data -r
I use docker-in-docker containers that also generates a lot of unused snapshot.
based on #Redsandro comment I've used the following command
sudo zfs list -t snapshot -r pool1| wc -l
sudo zpool list
(sudo zfs get mounted |grep "mounted no" | awk '/docker\// { print $1 }' | xargs -l sudo zfs destroy -R ) 2> /dev/null
as just delete all snapshot ruined the consistency of docker.
But as docker mounts all images that uses under /var/lib/docker/zfs/graph
(same for the docker-in-docker images) so ignoring those that mounted only should delete dangling images/volumes/containers that was not properly freed up. You need to run this till the number of snapshot decreasing.
Prune introductions on docker.com.
I assume your docker version is lower than V17.06. Since you’ve executed docker system prune -a, the old layers’ building information and volumes are missing. And -a/--all flag means all images without at least one container would be deleted. Without -a/--all flag, just dangling images would be deleted.
In addition, I think you have misunderstanding about <missing> mark and dangling images. <missing> doesn't mean that the layers marked as missing are really missing. It just means that these layers may be built on other machines. Dangling images are non-referenced images. Even the name and tag are marked <none>, the image still could be referenced by other images, which could check with docker history image_id.
In your case, these layers are marked as missing, since you have deleted the old versions of images which include building information. You said above--only latest version images are available--thus, only the latest layer are not marked missing.
Note this: docker system prune is a lazy way to manage all objects(image/container/volume/network/cache) of Docker.
Say I have a build machine which builds many docker images myimage:myversion. I consume about 1GB disk space per 100 images created and I certainly don't need all of them. I'd like to keep, say, the most recent 10 images (and delete everything older) but I want to make sure I have all of the cached layers from the 10 builds/image. If I have all of the layers cached, then I'm more likely to have a fast build on my next run.
The problem is all of the images (very old and brand new) share a lot of layers so I can't blindly delete the old ones as there is a ton of overlap with the new ones.
I don't want to use docker image prune (https://docs.docker.com/config/pruning/#prune-images) as that depends on which containers I have (regardless of state) and am deleting the containers so prune will end up deleting way too much stuff.
Is there a simple command I can run periodically to achieve the state I described above?
Simple, no, but some shell wizardry is possible. I think this shell script will do what you want:
#!/bin/sh
docker images --filter 'reference=myimage:*' \
--format '{{ .CreatedAt }}/{{ .ID }}/{{ .Repository }}:{{ .Tag }}' \
| sort -r \
| tail +11 \
| cut -d / -f 2 \
| xargs docker rmi
(You might try running this one step at a time to see what comes out.)
In smaller pieces:
List all of the myimage:* images in a format that starts with their date. (If you're using a private registry you must include the registry name as a separate part and you must explicitly include the tag; for instance to list all of your GCR images you need -f 'reference=gcr.io/*/*:*'.)
Sort them, by the date, newest first.
Skip the first 10 lines and start printing at the 11th.
Take only the second slash-separated field (which from the --format option is the hex image ID).
Convert that to command-line arguments to docker rmi.
The extended docker images documentation lists all of the valid --format options.
We have a system where user may install some docker containers. We dont have a limit on what he can install. After some time, we need to clean up - delete all the images that are not in used in the swarm.
What would be the solution for that using docker remote API?
Our idea is to have background image-garbage-collector thread that:
lists all the images
try to delete some
if it fails, just ignore
Would this make sense? Would this affect swarm somehow?
Cleaner way to list and (try to) remove all images
The command docker rmi $(docker images -q) would do the same as the answer by #tpbowden but in a cleaner way. The -q|--quiet only list the images ID.
It may delete frequently used images (not running at the cleaning date)
If you do this, when the user will try to swarm run deleted-image it will:
Either pull the image (< insert network consumption warning here />)
Either just block as the pull action is not automatic in swarm if I remember it right (< insert frequent support request warning here about misunderstood Swarm behavior />).
"dangling=true" filter:
A useful option is the --filter "dangling=true". Executing swarm images -q --filter "dangling=true" will display not-currently-running images.
Though challenge
You issue reminds me the memory management in a computer. Your real issue is:
How to remove image that won't be used in the future?
Which is really hard and really depends on your policy. If your policy is old images are to be deleted the command that could help is: docker images --format='{{.CreatedSince}}:{{ .ID}}'. But then the hack starts... You may need to grep "months" and then cut -d ':' -f 2.
The whole command would result as:
docker rmi $(docker images --format='{{.CreatedSince}}:{{ .ID}}' G months | cut -d ':' -f 2)
Note that this command will need to be run on every Swarm agent as well as the Swarm manager, not only the Swarm manager.
Swarm and registry
Be aware than a swarm pull image:tag will not pull the image on Swarm agents! Each Swarm agent must pull the image itself. Thus deleting still used images will result in network load.
I hope this answer helps. At this time there is no mean to query "image not used since a month" AFAIK.
All you need is 'prune'
$ docker image prune --filter until=72h --force --all
docker images | tail -n+2 | awk '{print $3}' | xargs docker rmi
This will list all images, strip the top line with column headings, grab the 3rd column (image ID hash) and then attempt to remove them all. Docker will prevent you from removing any images that are currently used by running containers.
If you want to do this in a slightly less 'hacky' way, you could use Docker's API to get images which aren't being used and delete them that way.
I have two Docker images, let's call them base and derived. As the names suggest, the one is FROMed from the other.
Now, among other containers, I start multiple instances of derived.
How can I get a list of all of them? In other words: How can I identify containers whose image is derived from a specific base image?
Any hints?
You can get the dockerfile from image https://github.com/CenturyLinkLabs/dockerfile-from-image that will get this information for you. Some scripting based on this anddocker psshould do what you want.
Does this work for you?
sudo docker ps | grep base | awk '{print $1}'