What is a dangling image and what is an unused image? - docker

In the docker documentation of docker image prune it is possible to use the -a flag to
Remove all unused images, not just dangling ones
and later
Remove all dangling images. If -a is specified, will also remove all images not referenced by any container.
Can someone explain to me what dangling images are and what's the difference between dangling and unused images?

An unused image means that it has not been assigned or used in a container. For example, when running docker ps -a - it will list all of your exited and currently running containers. Any images shown being used inside any of containers are a "used image".
On the other hand, a dangling image just means that you've created the new build of the image, but it wasn't given a new name. So the old images you have becomes the "dangling image". Those old image are the ones that are untagged and displays "<none>" on its name when you run docker images.
When running docker system prune -a, it will remove both unused and dangling images. Therefore any images being used in a container, whether they have been exited or currently running, will NOT be affected.

Safest and Easiest way to cleanup Dangling Images
docker images --quiet --filter=dangling=true | xargs --no-run-if-empty docker rmi
Docker images consist of multiple layers. Dangling images, are layers that have no relationship to any tagged images. They no longer serve a purpose and consume disk space.
Note: I recommend not to use prune in production, because docker system prune -a will remove all the images which are not referenced by the container, by which we can't roll back to the previous release.
To list dangling images by adding the filter flag, -f with a value of dangling=true to the docker images.
List Dangling images
docker images -f dangling=true
Remove Dangling Images
docker rmi $(docker images -f dangling=true -q)
OR
docker images --quiet --filter=dangling=true | xargs --no-run-if-empty docker rmi
When we run any cron jobs to delete tha dangling stuff use the above to make sure the job runs successfully. Like in Jenkins if we run a free style job with beloow commad job will never fail even if no dangling stuff exists in the machine.
This is the safest and easiest way to cleanup dangling images and get back our disk space back for use.

Images in docker are referenced by a sha256 digest, often referred to as the image id. That digest is all you need for the image to exist on the docker host. Typically, you will have tags that point to these digests, e.g. the tag 'busybox:latest' currently points to image id c30178c523... on my system.
Multiple tags can point to the same image, and any tag can be changed to point to a different id, e.g. when you pull a new copy of busybox:latest or build a new version of your application image.
Dangling images are images which do not have a tag, and do not have a child image (e.g. an old image that used a different version of FROM busybox:latest), pointing to them. They may have had a tag pointing to them before and that tag later changed. Or they may have never had a tag (e.g. the output of a docker build without including the tag option).
These are typically safe to remove as long as no containers are still running that reference the old image id. The main reason to keep them around is for build caching purposes.
In addition, you may have downloaded images that you are not currently used by containers (including stopped containers). These are entirely different from dangling images and may be safe to remove as long as you don't plan to use them in the future or don't mind downloading another copy when you do need them.

Dangling images are layers that have no relationship to any tagged images. They no longer serve a purpose and consume disk space.
An unused image is an image that has not been assigned or used in a container.
To list dangling images:
docker images -f dangling=true

Dangling images are untagged images. The following command gives a list of dangling images.
docker images --filter "dangling=true"
docker image prune deletes all dangling images.
Unused images are images that have tags but currently not being used as a container. You may or may not need it in future.
docker image prune -a delete all dangling as well as unused images.
You generally don't want to remove all unused images until some time. Hence it is better to remove with a filter.
docker image prune -a -f --filter "until=6h"

In the images screenshot, "none" name are dangling images.
A dangling image just means that you've created the new build of the image, but it wasn't given a new name. So the old images you have becomes the "dangling image". Those old image are the ones that are untagged and displays "" on its name when you run docker images.
docker system prune -a, it will remove both unused and dangling images. Therefore, any images being used in a container, whether they have been exited or currently running, will NOT be affected.

I saw useful commands (aliases) for removing dangling images, courtesy of andyneff here: https://forums.docker.com/t/how-to-delete-cache/5753:
alias docker_clean_images='docker rmi $(docker images -a --filter=dangling=true -q)'
alias docker_clean_ps='docker rm $(docker ps --filter=status=exited --filter=status=created -q)'
The first one
cleans all dangling images. This is useful for removing intermediate
images left over from multiple builds. The second one is for removing
stopped containers. These are aliases I use for routine maintenance
If you want to remove ALL of your cache, you first have to make sure
all containers are stopped and removed, since you cannot remove an
image in use by a container. So something similar
docker kill $(docker ps -q) docker_clean_ps docker rmi $(docker images
-a -q)
This would kill and remove all images in your cache.

Related

Docker: "You don't have enough free space in /var/cache/apt/archives/"

I have a dockerfile which when I want to build results in the error
E: You don't have enough free space in /var/cache/apt/archives/
Note that the image sets up a somewhat complex project with several dependencies that require quite a lot of space. For example, the list includes Qt. This is only a thing during the construction of the image, and in the end, I expect it to have a size of maybe 300 MB.
Now I found this: https://unix.stackexchange.com/questions/578536/how-to-fix-e-you-dont-have-enough-free-space-in-var-cache-apt-archives
Given that, what I tried so far is:
Freeing the space used by docker images so far by calling docker system prune
Removing unneeded installation files by calling sudo apt autoremove and sudo apt autoclean
There was also the suggestion to remove data in var/log, which has currently a size of 3 GB. However, I am not the system administrator and thus wary to do such a thing.
Is there any other way to increase that space?
And, preferably, is there a more sustainable solution, allowing me to build several images without having to search for spots where I can clean up the system?
Try this suggestion. You might have a lot of unused images that need to be deleted.
https://github.com/onyx-platform/onyx-starter/issues/5#issuecomment-276562225
Converting a #Dre suggestion into the code, you might want to use Docker prune command for containers, images & volumes
docker container prune
docker image prune
docker volume prune
You can use these commands in sequence:
docker container prune; docker image prune; docker volume prune
Free Space without removing your latest images
Use the following command to see the different types of reclaimable storage (the -v verbose option provides more detail):
docker system df
docker system df -v
Clear the build cache (the -a option will remove unused build cache):
docker builder prune -a
Remove dangling images ( tagged images, old and previous image builds):
docker rmi -f $(docker images -f "dangling=true" -q)
Increase Disk image size using Docker UI
Docker > Preferences > Resources > Advanced > adjust Disk image size > Apply & Restart
TLDR;
run
docker system prune -a --volumes
I tried to increase the disk space and prune the images, containers and volumes manually but was facing the issue again and again. When I tried to check the memory consumption on my machine, I found a lot of memory consumed by ~/Library/Containers/com.docker.docker location. Did a system prune which cleaned up a lot of space and docker builds started working again.

Eliminate docker <none> images when building using docker-compose

I am working on a microservice with several docker containers. Using docker-compose creates the images I want but also several other copies of images which fill up my disk space. What are the use of this other child images and can I stop them from being created since they are using up my memory. Please note they are not really "dangling" they just appear on build.
The images are the intermediate layers resulted from docker build. They are the parent layers for your final image and cannot be removed as your latest image actually refers to them.
Only those images which are not referenced by any other layers can be removed. These images are called dangling. You can use the following command to remove the dangling images:
docker rmi $(docker images -f "dangling=true" -q)

What are Dangling Images in Docker/How they created/How to remove them

How do dangling images get created? Whenever I tried to create them it is not happening.
I had used the same concept as mentioned in docker docs. If my DockerFile contains FROM alphine:3.4 then I build the image as docker build -t image1 .
After some time I update it as FROM alphine:3.5 and again building it as docker build -t image1 .
But it is showing me three images no dangling one??
What are Dangling Images in Docker?
Dangling Images are temporary images that are created at the time of building the images in docker server. Since images are build on layers of commits.
How they created?
There are created automatically. You can see them using docker images -f dangling=true
How to remove them?
You can do system purge to remove danggling images like this docker images purge.
If you really want to create a dangling image, simply interrupt the build process before it's finished.
Honestly don't understand why you want to though; they are a by-product of a failed build, that's all.

docker pull always show "Already exists" for layers during pull even after deleting all images

here is my input and output:
shshenhx#shshenhx:~/Desktop/Docker$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
python latest 336d482502ab 4 days ago 692 MB
shshenhx#shshenhx:~/Desktop/Docker$ docker rmi 336
Untagged: python:latest
Untagged: python#sha256:bf0718e2882cabc144c9c07d758b09c65efc104a6ddc72a9a976f8b26f67c2ee
Deleted: sha256:336d482502ab564b0b2964b2ed037529ba40c7b4ade2117ca11d74edbf11f99e
Deleted: sha256:1e2f72b0bf844de7bfe57a32115b660770355843ef8078fb55629e720147e6a9
Deleted: sha256:b5818ba96f33835318f3e9d7b4094e1007be247f04ab931ea9ef83b45b451f90
Deleted: sha256:0c2d7cafdab1084ebbd0112b3bedf76458ae6809686fb0ad9bae8022f11c3a84
shshenhx#shshenhx:~/Desktop/Docker$ docker pull python
Using default tag: latest
latest: Pulling from library/python
4176fe04cefe: Already exists
851356ecf618: Already exists
6115379c7b49: Already exists
aaf7d781d601: Already exists
40cf661a3cc4: Already exists
975fe2fd635f: Pull complete
bf4db784e7fd: Pull complete
0491f7e9426b: Pull complete
Digest: sha256:bf0718e2882cabc144c9c07d758b09c65efc104a6ddc72a9a976f8b26f67c2ee
Status: Downloaded newer image for python:latest
My question is, I have already rm python image, why it still shows already exist for some of layers? How can I totally delete all the python layers.
Thanks.
What helped for me was to run docker-system prune after removing all containers and images. So the whole process I got it to work was:
Remove all containers docker rm -vf $(docker ps -a -q)
Remove all images docker rmi -f $(docker images -a -q)
Prune system with docker system prune
From reference of docker images command
Docker images have intermediate layers that increase reusability, decrease
disk usage, and speed up docker build by allowing each step to be cached.
These intermediate layers are not shown by default.
Maybe those Already exists are intermediate layers. By default, they are hided when you run docker images, please try docker images --all.
Docker has a solution for this on docker dashboard, just press the button below and presto it works! (This is in the troubleshooting section. or the bug icon)
There is a docker command that deletes images that do not have an associated running container:
docker image prune
allows you to clean up dangling images (a dangling image is not tagged and not referenced by any container)
docker image prune -a
will remove all images that are not used by any existing containers
docker image prune -a --filter "until=48h"
you can add a filter (--filter) to limit the images to be pruned, as in this example, only images that are older than 48 hours
You can add the -f or --force flag to prune the images and not prompt for confirmation
You could use this in a scheduled bash script to keep your drive tidy
docker system prune -a helped me.It will remove both unused and dangling images.
You can use docker system prune but it will clear everything in docker.
Its not an image problem its actually cache problem so deleting your images wont work because the problem relies in layer caching.
There is a --no-cache flag in docker build but I don't know if it works with docker pull too.
For someone interested in: (1) using the command line; (2) removing cache only for images that have been removed with docker rmi (i.e., not visible with docker images --all), but for which caches are still present, and for which the Already exists shows for layers when pulling image; you can try this command:
docker builder prune
(It may take up to 15 minutes to run this command, so be patient.)
This command seems to have solved the problem for me. I did not want to use the general, "system-wide" prune / purge / clean docker commands and functionalities, because I had some images and containers running (not related to the image that I have already removed), that I wished to have left as they were.
Source of the solution: https://forums.docker.com/t/how-to-delete-cache/5753/7
I think that the situation of "invisible" cached layers for images removed with docker rmi may happen due to the fact of using docker compose (as suggested also by John Doe in his comment to one of the answers in the current question: docker pull always show "Already exists" for layers during pull even after deleting all images).
So after a lot of searching, I have a solution although I don't think it's necessarily a great one.
The "sledge hammer solution" would be to do:
rm -rf /var/lib/docker
This basically factory resets docker, so you would lose everything. This was fine for my case, but be very careful and think this through before using this one.
Please try this command docker rmi 336d482502ab and then try to pull it again.

Cached Docker image?

I created my own image and pushed it to my repo on docker hub. I deleted all the images on my local box with docker rmi -f ...... Now docker images shows an empty list.
But when I do docker run xxxx/yyyy:zzzz it doesn't pull from my remote repo and starts a container right away.
Is there any cache or something else? If so, what is the way to clean it all?
Thank you
I know this is old now but thought I'd share still.
Docker will contain all those old images in a Cache unless you specifically build them with --no-cache, to clear the cache down you can simply run docker system prune -a -f and it should clear everything down including the cache.
Note: this will clear everything down including containers.
You forced removal of the image with -f. Since you used -f I'm assuming that the normal rmi failed because containers based on that image already existed. What this does is just untag the image. The data still exists as a diff for the container.
If you do a docker ps -a you should see containers based on that image. If you start more containers based on that same previous ID, the image diff still exists so you don't need to download anything. But once you remove all those containers, the diff will disappear and the image will be gone.

Resources