I have a bunch of docker images I want to get rid of and it would be convenient if I could remove them by specifying the repository name as it appears when I run docker images. For example if docker images returns:
REPOSITORY TAG IMAGE ID CREATED SIZE
ui_test 191127_manual 41a7ca9824d6 24 hours ago 1.42GB
ui git-24fa8d1a cdd254eff918 24 hours ago 1.44GB
ui git-31a4b052 9b4740060a62 25 hours ago 1.45GB
ui_test 191122_manual ba9cb04ce2d8 6 days ago 1.39GB
ui git-68110e426 f26ef80abc25 6 days ago 1.38GB
what command would I use to remove all of the ui_test images?
You can pass image IDs you want to delete to docker rmi:
docker rmi $(docker images -q 'ui_test')
From the docs:
The docker images command takes an optional [REPOSITORY[:TAG]] argument that restricts the list to images that match the argument. If you specify REPOSITORYbut no TAG, the docker images command lists all images in the given repository.
Related
Docker documentation mentions image reference in many places. However, running docker images command gives the list of images with the following properties: REPOSITORY, TAG, IMAGE ID, CREATED, SIZE - no reference. Is 'reference' a synonym for ID or digest, or something else?
The docker image reference is the combination of the REPOSITORY and TAG in this format REPOSITORY:TAG where they are both separated by :. So if you have an image with a REPOSITORY of IMAGE1 and a tag of latest the image reference would be IMAGE1:latest. The knowledge of an image reference would help you to filter by docker image list by reference by running:
docker images --filter=reference='myDocker*:*dev'
The above command will return all docker images that the repository name starts with myDocker and the tag name ends with dev.
To add on to Kelvin's answer, Reference is the Repository which you will use with the tag. Have a look the below example.
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
busybox latest e02e811dd08f 5 weeks ago 1.09 MB
busybox uclibc e02e811dd08f 5 weeks ago 1.09 MB
busybox musl 733eb3059dce 5 weeks ago 1.21 MB
busybox glibc 21c16b6787c6 5 weeks ago 4.19 MB
As you can see above, my reference would be respectively
busybox:latest
busybox:uclibc
busybox:musl
busybox:glibc
If you only use the reference as busybox, by default it will use the latest tag.
You can filter the images on the reference filter as well.
docker images --filter=reference='busy*:*libc'
REPOSITORY TAG IMAGE ID CREATED SIZE
busybox uclibc e02e811dd08f 5 weeks ago 1.09 MB
busybox glibc 21c16b6787c6 5 weeks ago 4.19 MB
You use the image reference in Dockerfile as well when you build the image by using the FROM directive.
FROM busybox #Reference is used as you can see and automatically, `latest` tag will be pulled
...
Ref:- https://docs.docker.com/engine/reference/commandline/images/
Ref:- https://docs.docker.com/engine/reference/builder/#from
We have added a dockerised build agent to our development Kubernetes cluster which we use to build our applications as part of our Azure Devops pipelines. We created our own image based on the deprecated Microsoft/vsts-agent-docker on Github.
The build agent uses Docker outside of Docker (DooD) to create images on our development cluster.
This agent was working well for a few days but then an error would occasionally occur on the docker commands in our build pipeline:
Error response from daemon: No such image: fooproject:ci-3284.2
/usr/local/bin/docker failed with return code: 1
We realised that the build agent was creating tons of images that weren't being removed. There were tons of images that were blocking up the build agent and there were missing images, which would explain the "no such image" error message.
By adding a step to our build pipelines with the following command we were able to get our build agent working again:
docker system prune -f -a
But of course this then removes all our images, and they must be built from scratch every time, which causes our builds to take an unnecessarily long time.
I'm sure this must be a solved problem but I haven't been able to locate any documentation on the normal strategy for dealing with a dockerised build agent becoming clogged over time. Being new to docker and kubernetes I may simply not know what I am looking for. What is the best practice for creating a dockerised build agent that stays clean and functional, while maintaining a cache?
EDIT: Some ideas:
Create a build step that cleans up all but the latest image for the given pipeline (this might still clog the build server though).
Have a cron job run that removes all the images every x days (this would result in slow builds the first time after the job is run, and could still clog the build server if it sees heavy usage.
Clear all images nightly and run all builds outside of work hours. This way builds would run quickly during the day. However heavy usage could still clog the build server.
EDIT 2:
I found someone with a docker issue on Github that seems to be trying to do exactly the same thing as me. He came up with a solution which he described as follows:
I was exactly trying to figure out how to remove "old" images out of my automated build environment without removing my build dependencies. This means I can't just remove by age, because the nodejs image might not change for weeks, while my app builds can be worthless in literally minutes.
docker image rm $(docker image ls --filter reference=docker --quiet)
That little gem is exactly what I needed. I dropped my repository name in the reference variable (not the most self-explanatory.) Since I tag both the build number and latest the docker image rm command fails on the images I want to keep. I really don't like using daemon errors as a protection mechanism, but its effective.
Trying to follow these directions, I have applied the latest tag to everything that is built during the process, and then run
docker image ls --filter reference=fooproject
If I try to remove these I get the following error:
Error response from daemon: conflict: unable to delete b870ec9c12cc (must be forced) - image is referenced in multiple repositories
Which prevents the latest one from being removed. However this is not exactly a clean way of doing this. There must be a better way?
Probably you've already found a solution, but it might be useful for the rest of the community to have an answer here.
docker prune has a limited purpose. It was created to address the issue with cleaning up all local Docker images. (As it was mentioned by thaJeztah here)
To remove images in the more precise way it's better to divide this task into two parts:
1. select/filter images to delete
2. delete the list of selected images
E.g:
docker image rm $(docker image ls --filter reference=docker --quiet)
docker image rm $(sudo docker image ls | grep 1.14 | awk '{print $3}')
docker image ls --filter reference=docker --quiet | xargs docker image rm
It is possible to combine filters clauses to get exactly what you what:
(I'm using Kubernetes master node as an example environment)
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.14.2 5c24210246bb 3 months ago 82.1MB
k8s.gcr.io/kube-apiserver v1.14.2 5eeff402b659 3 months ago 210MB
k8s.gcr.io/kube-controller-manager v1.14.2 8be94bdae139 3 months ago 158MB
k8s.gcr.io/kube-scheduler v1.14.2 ee18f350636d 3 months ago 81.6MB # before
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
k8s.gcr.io/coredns 1.3.1 eb516548c180 7 months ago 40.3MB # since
k8s.gcr.io/etcd 3.3.10 2c4adeb21b4f 8 months ago 258MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 20 months ago 742kB
$ docker images --filter "since=eb516548c180" --filter "before=ee18f350636d"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
$ docker images --filter "since=eb516548c180" --filter "reference=quay.io/coreos/flannel"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
$ docker images --filter "since=eb516548c180" --filter "reference=quay*/*/*"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
$ docker images --filter "since=eb516548c180" --filter "reference=*/*/flan*"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
As mentioned in the documentation, images / image ls filter is much better than docker prune filter, which supports until clause only:
The currently supported filters are:
• dangling (boolean - true or false)
• label (label=<key> or label=<key>=<value>)
• before (<image-name>[:<tag>], <image id> or <image#digest>) - filter images created before given id or references
• since (<image-name>[:<tag>], <image id> or <image#digest>) - filter images created since given id or references
If you need more than one filter, then pass multiple flags
(e.g., --filter "foo=bar" --filter "bif=baz")
You can use other linux cli commands to filter docker images output:
grep "something" # to include only specified images
grep -v "something" # to exclude images you want to save
sort [-k colN] [-r] [-g]] | head/tail -nX # to select X oldest or newest images
Combining them and putting the result to CI/CD pipeline allows you to leave only required images in the local cache without collecting a lot of garbage on your build server.
I've copied here a good example of using that approach provided by strajansebastian in the comment:
#example of deleting all builds except last 2 for each kind of image
#(the image kind is based on the Repository value.)
#If you want to preserve just last build modify to tail -n+2.
# delete dead containers
docker container prune -f
# keep last 2 builds for each image from the repository
for diru in `docker images --format "{{.Repository}}" | sort | uniq`; do
for dimr in `docker images --format "{{.ID}};{{.Repository}}:{{.Tag}};'{{.CreatedAt}}'" --filter reference="$diru" | sed -r "s/\s+/~/g" | tail -n+3`; do
img_tag=`echo $dimr | cut -d";" -f2`;
docker rmi $img_tag;
done;
done
# clean dangling images if any
docker image prune -f
I created two new images anubh_custom_build_image/ubuntu_bionic:version1 & ubuntu_bionic_mldev:version1 from the base image ubuntu:bionic. The purpose of creating the custom-built ubuntu-docker images was to use Linux system on windows platform. I have faced many issues in past one such is installing a new version of tensorflow library ! pip install -q tf-nightly, I can't find a substitute of ! to run this command on windows cmd-prompt/PowerShell. Moreover, I want to invest more time on my codebase rather than fixing the issues on different OS. So, I pull the latest Ubuntu image from docker, installed a bunch of libraries for my usage and committed using docker commit command:
docker commit 503130713dff ubuntu_bionic_MLdev:version1
I can see the images using :
PS C:\Users\anubh> docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu_bionic_mldev version1 e7d1b154b69f 21 hours ago 9.33GB
anubh_custom_build_image/ubuntu_bionic version1 3c98f8954731 22 hours ago 9.33GB
tensorflow/tensorflow latest 2c8d1fd8bde4 2 days ago 1.25GB
ubuntu bionic 735f80812f90 2 weeks ago 83.5MB
ubuntu latest 735f80812f90 2 weeks ago 83.5MB
floydhub/dl-docker cpu 0b9fc622f1b7 2 years ago 2.87GB
When I tried to spin-up the containers using these images, The following command ran without any error.
PS C:\Users\anubh> docker run anubh_custom_build_image/ubuntu_bionic:version1
PS C:\Users\anubh> docker run ubuntu_bionic_mldev:version1
EDIT:
The issue is that run command is executing but the containers aren't spinning up for the above two images. I apologize for attaching the wrong error message in the first post, I edited it now. The below two containers were spined-up using docker run -it -p 8888:8888 tensorflow/tensorflow & docker run ubuntu:bionic commands.
PS C:\Users\anubh> docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
94d59b217b70 tensorflow/tensorflow "/run_jupyter.sh --a…" 21 hours ago Up 21 hours 6006/tcp, 8888/tcp boring_clarke
503130713dff ubuntu:bionic "bash" 38 hours ago Up 38 hours awesome_bardeen
Could anyone suggest what I am missing for running these images anubh_custom_build_image/ubuntu_bionic:version1 & ubuntu_bionic_mldev:version1 from the base image ubuntu:bionic as
containers?
Also, I can't find the location of any of these images on my disk.
Could anyone also suggest where to look for inside Windows OS?
NOTE: I will write a dockerfile in future to build a custom image, but for now, I want to use the commit command to create new image & use them.
Your docker run command doesn't work because you don't have the :version1 tag at the end of it. (Your question claims you do, but the actual errors you quote don't have it.)
However: if this was anything more than a simple typo, the community would probably be unable to help you, because you have no documentation about what went into your image. (And if not "the community", then "you, six months later, when you discover your image has a critical security vulnerability".) Learn how the Dockerfile system works and use docker build to build your custom images. Best-practice workflows tend to not use docker commit at all.
The output of sudo docker images shows images that were created ages ago, I don't see any use for them, sudo docker system prune does not remove them either.
How can I easily delete old images? Do I have to write a script to conditionally delete based on date created within the output of docker images?
I don't think you can filter images by created date, but you can use another image as a reference of time.
The flag (-f or--filter) with before shows only images created before the image with given id or reference.
For example, having these images:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
image1 latest eeae25ada2aa 4 minutes ago 188.3 MB
image2 latest dea752e4e117 9 minutes ago 188.3 MB
image3 latest 511136ea3c5a 25 minutes ago 188.3 MB
Filtering with before (image1) would give:
$ docker images --filter "before=eeae25ada2aa"
REPOSITORY TAG IMAGE ID CREATED SIZE
image2 latest dea752e4e117 9 minutes ago 188.3 MB
image3 latest 511136ea3c5a 25 minutes ago 188.3 MB
To Remove images before image1(eeae25ada2aa)
$ docker rmi $(docker images --filter "since=511136ea3c5a" -q)
dea752e4e117
511136ea3c5a
Assume that you can create a 500MB docker image with following command
docker build -t demo:1.0 .
It seems to me that the docker image (demo:1.0) is overwritten when I re-run the same command. However, I noticed that free disk space decreases as I re-run the command. Why does this happen? Is docker creating another set of the image somewhere in the file system? I'd like to know how I can find old-generations of the docker images and delete them.
It's standard docker behavior, if you run
docker images
you may see a dangling image like this
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
demo 1.0 a1b38444260e 6 days ago 500 MB
<none> <none> 46250b7172c5 6 days ago 500 MB
You can remove the dangling images with the command
docker rmi $(docker images -qa -f "dangling=true")
Docker doesn’t delete such images automatically, despite, it's a logical feature to have. For now this command can be used to do a manual deletion.
While the images are stored in /var/lib/docker/{driver-name}, you wouldn't remove images manually from that path.