How to use docker images filter - docker

I can write
docker images --filter "dangling=true"
What other filters can I use?
I can use something like this?
docker images --filter "running=false"

Docker v1.13.0 supports the following conditions:
-f, --filter value Filter output based on conditions provided (default [])
- dangling=(true|false)
- label=<key> or label=<key>=<value>
- before=(<image-name>[:tag]|<image-id>|<image#digest>)
- since=(<image-name>[:tag]|<image-id>|<image#digest>)
- reference=(pattern of an image reference)
Or use grep to filter images by some value:
$ docker images | grep somevalue
References
docker images filtering
docker docs

You can also use the REPOSITORY argument to docker images to filter the images.
For example, suppose we have the images:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
local-foo latest 17864104b328 2 months ago 100 MB
example.com/bar latest b94c37de2801 9 months ago 285 MB
example.com/baz latest a004e3ac682c 2 years ago 221 MB
We can explicitly filter for all images with a given name:
$ docker images example.com/bar
REPOSITORY TAG IMAGE ID CREATED SIZE
example.com/bar latest b94c37de2801 9 months ago 285 MB
Docker also supports globbing:
$ docker images "example.com/*"
REPOSITORY TAG IMAGE ID CREATED SIZE
example.com/bar latest b94c37de2801 9 months ago 285 MB
example.com/baz latest a004e3ac682c 2 years ago 221 MB
Official docs here.

In Docker v1.7:
The currently supported filters are:
dangling (boolean - true or false)
label (label=<key> or label=<key>=<value>)

For me,
docker images -q | while read IMAGE_ID; do
docker inspect --format='{{.Created}}' --type=image ${IMAGE_ID}
done
did the trick. The date command is able to produce output in the same format via
date -Ins --date='10 weeks ago'
which allows me to compare timestamps. I still use the filter for dangling images for convenience, though.

I'm wanted to find a match for both local images and images that were tagged with a remote repo (my-repo.example.com in example below).
For example,
docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
my-good-app latest 9a8742ad82d3 24 hours ago 126MB
my-repo.example.com/mine/my-good-app latest 9a8742ad82d3 24 hours ago 126MB
my-repo.example.com/mine/demo-docker latest c10baf5231a1 2 weeks ago 200MB
I got tired of trying to figure out how filtering worked, so I just fell back to what I knew.
docker images | grep my-good-app | awk '{print $3}' | uniq
This would match any image names that had the pattern my-good-app. I could not get other answers to include both (images without a repo and images with a reponame like my-repo.example.com in my example).
Then to delete the images matched above, I ran:
docker rmi -f $(docker images | grep my-good-app | awk '{print $3}' | uniq)

sudo docker images --filter "running=false"
For cleaning up old stopped containers you can use:
docker container prune
To remove untagged images you can use:
docker image prune

Docker CLI lacks in the proper filtering if you're looking for a solution to remove images based on both the creation date and repository / tag:
docker image prune does accept the timestamps via --since and --until flags (e.g.: --until=24h), but does not allow filtering by repo/tag.
docker images and docker image ls does accept the repo/tag filters (e.g.: --filter=reference='registry.gitlab.com/group/*/*) but does not accept timestamps - only other images ids in --since and --before (e.g. --since=586026f10754)
I wrote a (seven-lines) one-liner for what I believe is a common need: removing images based on both their repo and creation date. Here you go:
docker image rm $(docker images \
--filter=reference='registry.gitlab.com/group/*/*/*:*' \
--format "{{.ID}}-{{.CreatedAt}}'" | \
cut -d " " -f 1 | \
sed 's/-/ /'| \
awk -v date="$(date --date='3 days ago' +%Y-%m-%d)" '$NF < date' \
cut -d " " -f 1)
Customizations:
Name filtering: 2nd line, value of reference= - see docs
Date filtering: 6th line, value of date= - see docs
Requirements: bash, awk, cut, sed and obviously docker.

In Powershell use this example:
docker images --format "{{.Repository}}:{{.Tag}}" | findstr "some_name"
To delete images you can combine this with the delete command like so:
docker rmi $(docker images --format "{{.Repository}}:{{.Tag}}"|findstr "some_name")

To add to the original answer on how to use images filter, just to add a use case for a similar scenario.
My CI pipeline re-builds docker and tags them with last commit number every time, sends them to docker repository.
However, this results in residual & un-used/un-wanted images on the CI build machine.
As a post step, I need to clean them up all, even the ones build just now, but at the same time, want to leave my base downloaded images ( such as OpenJDK, PostGre ) un-deleted to avoid download every time
Add a/any label in Docker file ( unique and is not expected to be contained in my base images)
LABEL built=XYZ
Using images filter and just to get the image identifiersfor the images I created
docker images --quiet --filter label=built=XYZ
Delete them as a post build action
docker rmi -f $(docker images --quiet --filter label=built=XYZ)

There's another example, works with version 17.09++:
sudo docker rmi $(sudo docker images -f=reference="registry.gitlab.com/example-app" -f "dangling=true" -q)
Explanation:
reference - we are referencing images by repository name;
dangling=true - we are removing untagged images;
-q - means quiet, showing only numeric IDs of images, instead of a whole line.
This command removes all images that have a repository name "registry.gitlab.com/example-app" and untagged (have <none> in a tag column)
Reference link: https://docs.docker.com/engine/reference/commandline/images/#filtering

Docker builtin filtering feature simply doesn't cut it for certain use cases. Mainly, the glob pattern not matching the forward slash makes it impossible to match images not sharing the same count of forward slashes.
I find it's easier to rely on an external tool such as awk. For example, here I'm pulling/updating all the images from a certain repository and its subdirectories tagged with any SNAPSHOT version :
docker images --filter=since=65f20cac3aa5 | awk '$1~repo && $2~tag { print $1 ":" $2}' repo=my.repo.com/subdirectory tag=SNAPSHOT | xargs -r -L 1 docker pull
Notice that I combined it with the builtin filter "since".

FYI, without filter, but for delete all images when you use as testing or learning,
docker image rm -f $(docker image ls)
Greetings.

Related

How to prune old docker images, but only for a selected container?

We know that docker image prune has a --filter argument that can be used to select (and remove) images older than a given number of hours (e.g.--filter "until=7*24h").
We also know that docker images has a similar --filter argument that supports a before key (e.g. docker images --filter "before=ubuntu:22.04"), but that can only filter images created before a given image or reference (not a date).
But pruning as described above would apply to all containers, which is rather too broad. What if we wanted to prune the "old" images more selectively, restricting the pruning to the images of just a single container (e.g. to spare older base containers, etc.)?
I've come up with something looking ugly, but apparently rather effective.
The example below removes (forcefully) all images older than 2 weeks (the shortest period for which this implementation works - can be tweaked to any period though) of the an mirekphd/ml-cache container (caution: as a special case it can remove all images of this container):
$ MAX_WEEK_NUM=2 && REPO=mirekphd && CONTAINER=ml-cache && docker images --format "{{.ID}} {{.CreatedSince}}" --filter=reference="$REPO/$CONTAINER" | grep "[$MAX_WEEK_NUM-9999] weeks\|[1-999] months\|[1-99] years" | awk '{print $1}' | xargs docker rmi -f

How can I delete a docker images by their tag <none> and repository name?

I would like to ask if it's possible to delete docker images for its tag or the created label, I have something like this:
Repository TAG IMAGE_ID CREATED SIZE
registry.someRegistry. latest fa8767676 5 hours ago 119MB
registry.someRegistry. <none> 878787874 29 hours ago 119MB
registry.someRegistry. <none> jkj7jjjk4 2 days ago 119MB
registry.someRegistry. <none> d99090iii 3 days ago 119MB
I need to remove all the images but not the most recently one in this case is the one tagged with latest or the one with the most recent Created date. I can use the next command to delete all the images until 5 hours of creation, but it's not appropriate, because I could delete more images not just the last one, would be better to delete images by the tag=latest.
docker image prune -a --force --filter "until=5h"
I need also an additional filter for the registry I think, in order to remove just the images for the registry I need.
Any good way to achieve that?
The simplest would be the rmi command, i.e.:
docker rmi 878787874
This is a "remove image" command and uses the IMAGE_ID you list.
None of the replies were helpful, and there does not seem to be another thread either. It is way too late but I had the same problem and found this solution:
docker rmi $(docker images | grep "<none>" | grep "registry.someRegistry." | awk {'print $3'})
This is not 100% proof because grep looks for a string in the name and the tag, so theoretically if your registry name appears in some tag and vice versa you could accidentally delete it.
you can remove dangling images with filter
docker images --filter dangling=true
It is possible to find all dangling images from one specific repository with
docker images repository --filter dangling=true
If you want to filter by more than one repository based on the registry, it is also possible to do
docker images --filter=reference='registry/someRegistry/*:*' --filter dangling=true
Once you have the list of the ones you want to remove you can do it with:
docker rmi $(docker images --filter=reference='registry/someRegistry/*:*' --filter dangling=true -q)
Edit to add that the wildcard * does not apply to /, so if the registry has a more complex folder structure, the expression should be amended to something like
'registry/someRegistry/*/*:*'

How to remove multiple <none> docker images blocked by stopped containers quickly [duplicate]

This question already has answers here:
How to remove old and unused Docker images
(29 answers)
Closed 2 years ago.
I was trying to remove all dangling images including the images which have <none> as repo name or tag while listing with docker images command.
Now, if you try to remove the images , you may have seen that the docker rmi <image_id> commands might fail due to presence of invalid docker containers which were Exited abruptly and persisted as below.
It is tedious to remove all the containers and dependent images one by. Hence, is there a good way of removing such images quickly or in automated way ?
Easier way to remove such images and dependent containers to use following commands sequentially...
docker ps -a | grep "Exited " | awk '{print $1}' | xargs docker rm
docker images -f dangling=true | grep "none" | awk '{print $3}'| xargs docker rmi
you can also use docker image prune to remove all dangling images first.
Above commands will remove all dangling and <none> images from your system. You can also use docker image prune -a to remove ALL UNUSED images, but use it with CAUTIONs.

How do I see extra tags on a local Docker image?

When I build a Docker image locally with a tag, then assign a secondary tag, I am unable to list all of the tags for that image.
I'm losing tags either on the build, save or load of docker images and I'm trying to track the state of my image at every step. Please note that this is a local Docker file (as in on my laptop), and can't be sent to a remote repository. When I run docker images I can see the image created with one of the tags, but not all of them.
$ docker build -t my/repo:first_tag
$ docker tag my/repo:first_tag my/repo:second_tag
When I run docker images I only see the first tag shown as the tag.
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
my/repo first_tag abcd1234 6 seconds ago 100MB
I would like to have some way of showing that image with all of the tags, something like:
my/repo:
first_tag
second_tag
Does a command like this exist and, if so, what is it?
EDIT: It turns out I was adding tags wrong and the code show above wasn't representative of it. What I was doing was actually:
$ docker build -t my/repo:first_tag
$ docker tag my/repo:first_tag my/repo
Note that the second tag doesn't have an actual tag on it. When I actually run the code in the question I get the following output:
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
my/repo second_tag abcd1234 6 seconds ago 100MB
my/repo first_tag abcd1234 6 seconds ago 100MB
If you want to list tags for a particular image, you can run :
docker images my/repo --format="{{ .Tag }}"
--format is used here to only keep interesting columns. By default, docker images my/repo will return all info about my/repo (repository, tag, image id, created and size), like docker images (without filtering repository) does.
Edit : group images by tags
Since --format option placeholders are limited (https://docs.docker.com/engine/reference/commandline/images/#format-the-output), you cannot access some interesting properties as .RepoTags to retrieve all tags associated with an image digest. However, you could do this by docker inspecting desired images and using some shell and jq magic (jq is a tool to manipulate JSON) :
docker images my/repo --format="{{ .ID }}" | \
sort | uniq | \
xargs docker inspect | \
jq -r '.[] | [ .Id, (.RepoTags | join(",")) ] | #csv'
Explanations :
first, retrieve all images IDs (--format="{{ .ID }}") related to my/repo
remove duplicates (sort | uniq) if any (because of multiple tags per image ID)
for each image (xargs), inspect that image (docker inspect) :
use jq to retrieve a string like <id of the image>,<tags related to the image>. Tags related to this image are separated by a comma (join(","))
Example result :
"sha256:abcd1234...","my/repo:v1"
"sha256:abcd2345...","my/repo:v2"
"sha256:abcd3456...","my/repo:v3,my/repo:tag"
Get RepoTags by example:
docker image inspect my/backend:latest-qa | jq -r '.[] | (.RepoTags)'
ouput:
[
"my/backend:0.12.3-qa",
"my/backend:latest-qa"
]

Correctly keeping docker VSTS / Azure Devops build agent clean yet cached

We have added a dockerised build agent to our development Kubernetes cluster which we use to build our applications as part of our Azure Devops pipelines. We created our own image based on the deprecated Microsoft/vsts-agent-docker on Github.
The build agent uses Docker outside of Docker (DooD) to create images on our development cluster.
This agent was working well for a few days but then an error would occasionally occur on the docker commands in our build pipeline:
Error response from daemon: No such image: fooproject:ci-3284.2
/usr/local/bin/docker failed with return code: 1
We realised that the build agent was creating tons of images that weren't being removed. There were tons of images that were blocking up the build agent and there were missing images, which would explain the "no such image" error message.
By adding a step to our build pipelines with the following command we were able to get our build agent working again:
docker system prune -f -a
But of course this then removes all our images, and they must be built from scratch every time, which causes our builds to take an unnecessarily long time.
I'm sure this must be a solved problem but I haven't been able to locate any documentation on the normal strategy for dealing with a dockerised build agent becoming clogged over time. Being new to docker and kubernetes I may simply not know what I am looking for. What is the best practice for creating a dockerised build agent that stays clean and functional, while maintaining a cache?
EDIT: Some ideas:
Create a build step that cleans up all but the latest image for the given pipeline (this might still clog the build server though).
Have a cron job run that removes all the images every x days (this would result in slow builds the first time after the job is run, and could still clog the build server if it sees heavy usage.
Clear all images nightly and run all builds outside of work hours. This way builds would run quickly during the day. However heavy usage could still clog the build server.
EDIT 2:
I found someone with a docker issue on Github that seems to be trying to do exactly the same thing as me. He came up with a solution which he described as follows:
I was exactly trying to figure out how to remove "old" images out of my automated build environment without removing my build dependencies. This means I can't just remove by age, because the nodejs image might not change for weeks, while my app builds can be worthless in literally minutes.
docker image rm $(docker image ls --filter reference=docker --quiet)
That little gem is exactly what I needed. I dropped my repository name in the reference variable (not the most self-explanatory.) Since I tag both the build number and latest the docker image rm command fails on the images I want to keep. I really don't like using daemon errors as a protection mechanism, but its effective.
Trying to follow these directions, I have applied the latest tag to everything that is built during the process, and then run
docker image ls --filter reference=fooproject
If I try to remove these I get the following error:
Error response from daemon: conflict: unable to delete b870ec9c12cc (must be forced) - image is referenced in multiple repositories
Which prevents the latest one from being removed. However this is not exactly a clean way of doing this. There must be a better way?
Probably you've already found a solution, but it might be useful for the rest of the community to have an answer here.
docker prune has a limited purpose. It was created to address the issue with cleaning up all local Docker images. (As it was mentioned by thaJeztah here)
To remove images in the more precise way it's better to divide this task into two parts:
1. select/filter images to delete
2. delete the list of selected images
E.g:
docker image rm $(docker image ls --filter reference=docker --quiet)
docker image rm $(sudo docker image ls | grep 1.14 | awk '{print $3}')
docker image ls --filter reference=docker --quiet | xargs docker image rm
It is possible to combine filters clauses to get exactly what you what:
(I'm using Kubernetes master node as an example environment)
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.14.2 5c24210246bb 3 months ago 82.1MB
k8s.gcr.io/kube-apiserver v1.14.2 5eeff402b659 3 months ago 210MB
k8s.gcr.io/kube-controller-manager v1.14.2 8be94bdae139 3 months ago 158MB
k8s.gcr.io/kube-scheduler v1.14.2 ee18f350636d 3 months ago 81.6MB # before
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
k8s.gcr.io/coredns 1.3.1 eb516548c180 7 months ago 40.3MB # since
k8s.gcr.io/etcd 3.3.10 2c4adeb21b4f 8 months ago 258MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 20 months ago 742kB
$ docker images --filter "since=eb516548c180" --filter "before=ee18f350636d"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
$ docker images --filter "since=eb516548c180" --filter "reference=quay.io/coreos/flannel"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
$ docker images --filter "since=eb516548c180" --filter "reference=quay*/*/*"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
$ docker images --filter "since=eb516548c180" --filter "reference=*/*/flan*"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
As mentioned in the documentation, images / image ls filter is much better than docker prune filter, which supports until clause only:
The currently supported filters are:
• dangling (boolean - true or false)
• label (label=<key> or label=<key>=<value>)
• before (<image-name>[:<tag>], <image id> or <image#digest>) - filter images created before given id or references
• since (<image-name>[:<tag>], <image id> or <image#digest>) - filter images created since given id or references
If you need more than one filter, then pass multiple flags
(e.g., --filter "foo=bar" --filter "bif=baz")
You can use other linux cli commands to filter docker images output:
grep "something" # to include only specified images
grep -v "something" # to exclude images you want to save
sort [-k colN] [-r] [-g]] | head/tail -nX # to select X oldest or newest images
Combining them and putting the result to CI/CD pipeline allows you to leave only required images in the local cache without collecting a lot of garbage on your build server.
I've copied here a good example of using that approach provided by strajansebastian in the comment:
#example of deleting all builds except last 2 for each kind of image
#(the image kind is based on the Repository value.)
#If you want to preserve just last build modify to tail -n+2.
# delete dead containers
docker container prune -f
# keep last 2 builds for each image from the repository
for diru in `docker images --format "{{.Repository}}" | sort | uniq`; do
for dimr in `docker images --format "{{.ID}};{{.Repository}}:{{.Tag}};'{{.CreatedAt}}'" --filter reference="$diru" | sed -r "s/\s+/~/g" | tail -n+3`; do
img_tag=`echo $dimr | cut -d";" -f2`;
docker rmi $img_tag;
done;
done
# clean dangling images if any
docker image prune -f

Resources