REPOSITORY <none> TAG <none> - docker

I am new here trying to learn docker, I started this tutorial
https://docs.docker.com/engine/examples/nodejs_web_app/
Building your image
$ docker build -t mlotfi/centos-node-hello .
mlotfi is my username in https://hub.docker.com/
when I did docker images, I got:
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
<none> <none> sha256:189cb 27 seconds ago 485.1 MB
centos centos6 sha256:d0a31 12 days ago 228.9 MB
instead of:
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
mlotfi/centos-node-hello latest sha256:189cb 27 seconds ago 485.1 MB
centos centos6 sha256:d0a31 12 days ago 228.9 MB
UPDATE:
at the end of the build I see :
Complete!
---> e053d8f57e5c
Removing intermediate container 060f921fd08c
Step 4 : COPY package.json /src/package.json
SECURITY WARNING: You are building a Docker image from Windows against a non-Windows Docker host. All files and
directories added to build context will have '-rwxr-xr-x' permissions. It is recommended to double check and r
eset permissions for sensitive files and directories.
lstat package.json: no such file or directory
But I already have package.json in the src directory.

That can happen if your docker build sequence does not got all the way, meaning it stops on error at some point in the Dockerfile.
The result of that interrupted process is the last intermediate image built by the Dockerfile line that succeeded, just before the Dockerfile line that fail to execute properly.
Other reasons are listed in "What are Docker <none>:<none> images? "
Each docker image is composed of layers, with these layers having a parent-child hierarchical relationship with each other.
All docker file system layers are by default stored at /var/lib/docker/graph. Docker calls it the graph database
<none>:<none> images stand for. They stand for intermediate images and can be seen using docker images -a.
Another style of <none>:<none> images are the dangling images which can cause disk space problems.
A dangling image and needs to be pruned. When our hello_world image was rebuilt using the Dockerfile, its reference to old Fedora became untagged and dangling.
you can see an example of dangling image in "What are <none> repository and tags? Why do they appear when I use docker build?".
The next command can be used to clean up these dangling images.
docker rmi $(docker images -f "dangling=true" -q)
In your case, if this was the first time your used docker build, this should not be a dangling image, but an intermediate one as I explained first in this answer.

Related

Is there a way not to create dependant child images when building a docker container [duplicate]

When I run the docker-compose build command to rebuild an image in Docker because I had changed something in Dockerfile, sometimes I get "none" image tags. How can we avoid this fact? I want to rebuild the image but the none image should not appear.
REPOSITORY TAG IMAGE ID CREATED SIZE
magento2 latest b4dce4dcbd4f 16 hours ago 516MB
<none> <none> b4ffce2bf91e 16 hours ago 519MB
<none> <none> a1aedb60c82a 17 hours ago 516MB
<none> <none> ec9a14ae856c 20 hours ago 519MB
<none> <none> ef8eba6874cc 23 hours ago 516MB
<none> <none> 0e53a8b8c303 23 hours ago 516MB
php 7.1-apache 93e6fb4b13e1 3 weeks ago 369MB
mysql 5.6.39 079344ce5ebd 7 months ago 256MB
Below are some parts from What are Docker <none>:<none> images?
The Good <none>:<none>
These are intermediate images and can be seen using docker images -a. They don't result into a disk space problem but it is definitely a screen "real estate" problem. Since all these <none>:<none> images can be quite confusing as what they signify.
The Bad <none>:<none>
These images are the dangling ones, which can cause disk space problems. These <none>:<none> images are being listed as part of docker images and need to be pruned.
(a dangling file system layer in Docker is something that is unused and is not being referenced by any images. Hence we need a mechanism for Docker to clear these dangling images)
So,
if your case has to do with dangling images, it's ok to remove them with:
docker rmi $(docker images -f "dangling=true" -q)
There is also the option of docker image prune but the client and daemon API must both be at least v1.25 to use this command.
if your case has to do with intermediate images, it's ok to keep them, other images are pointing references to them.
Related documentation:
docker rmi
docker image rm
docker image prune
In my experience most of the <none> images are held by temporary containers. Due to Docker architecture those containers are preserved even after they stop. You can verify how many stopped containers you have using
docker ps -a
So to remove the <none> images you first need to remove the unneeded containers:
docker container prune
docker image prune
The above two commands can be abbreviated to
docker system prune
There is little to add based on what #tgogos said except that it needs more upvoting.
You can check image sizes of dangling and non-dangling images here:
docker system df -v
Don't be bugged by intermediate images. This way you oversee that the build process has been made more efficient by keeping intermediate images for each line of a Dockerfile, i.e. such a line can be skipped during the build process if no change occurred.
This will remove all dangling docker images in Windows:
for /f %x in ('docker images -f "dangling=true" -q') do docker rmi %x
You can remove dangling images using
docker rmi $(docker images -f "dangling=true" | grep "<none>.*<none>" | awk '{ print $3; }')
To remove <none> layers use:
docker rmi $(docker images --filter "dangling=true" -q --no-trunc)

Correctly keeping docker VSTS / Azure Devops build agent clean yet cached

We have added a dockerised build agent to our development Kubernetes cluster which we use to build our applications as part of our Azure Devops pipelines. We created our own image based on the deprecated Microsoft/vsts-agent-docker on Github.
The build agent uses Docker outside of Docker (DooD) to create images on our development cluster.
This agent was working well for a few days but then an error would occasionally occur on the docker commands in our build pipeline:
Error response from daemon: No such image: fooproject:ci-3284.2
/usr/local/bin/docker failed with return code: 1
We realised that the build agent was creating tons of images that weren't being removed. There were tons of images that were blocking up the build agent and there were missing images, which would explain the "no such image" error message.
By adding a step to our build pipelines with the following command we were able to get our build agent working again:
docker system prune -f -a
But of course this then removes all our images, and they must be built from scratch every time, which causes our builds to take an unnecessarily long time.
I'm sure this must be a solved problem but I haven't been able to locate any documentation on the normal strategy for dealing with a dockerised build agent becoming clogged over time. Being new to docker and kubernetes I may simply not know what I am looking for. What is the best practice for creating a dockerised build agent that stays clean and functional, while maintaining a cache?
EDIT: Some ideas:
Create a build step that cleans up all but the latest image for the given pipeline (this might still clog the build server though).
Have a cron job run that removes all the images every x days (this would result in slow builds the first time after the job is run, and could still clog the build server if it sees heavy usage.
Clear all images nightly and run all builds outside of work hours. This way builds would run quickly during the day. However heavy usage could still clog the build server.
EDIT 2:
I found someone with a docker issue on Github that seems to be trying to do exactly the same thing as me. He came up with a solution which he described as follows:
I was exactly trying to figure out how to remove "old" images out of my automated build environment without removing my build dependencies. This means I can't just remove by age, because the nodejs image might not change for weeks, while my app builds can be worthless in literally minutes.
docker image rm $(docker image ls --filter reference=docker --quiet)
That little gem is exactly what I needed. I dropped my repository name in the reference variable (not the most self-explanatory.) Since I tag both the build number and latest the docker image rm command fails on the images I want to keep. I really don't like using daemon errors as a protection mechanism, but its effective.
Trying to follow these directions, I have applied the latest tag to everything that is built during the process, and then run
docker image ls --filter reference=fooproject
If I try to remove these I get the following error:
Error response from daemon: conflict: unable to delete b870ec9c12cc (must be forced) - image is referenced in multiple repositories
Which prevents the latest one from being removed. However this is not exactly a clean way of doing this. There must be a better way?
Probably you've already found a solution, but it might be useful for the rest of the community to have an answer here.
docker prune has a limited purpose. It was created to address the issue with cleaning up all local Docker images. (As it was mentioned by thaJeztah here)
To remove images in the more precise way it's better to divide this task into two parts:
1. select/filter images to delete
2. delete the list of selected images
E.g:
docker image rm $(docker image ls --filter reference=docker --quiet)
docker image rm $(sudo docker image ls | grep 1.14 | awk '{print $3}')
docker image ls --filter reference=docker --quiet | xargs docker image rm
It is possible to combine filters clauses to get exactly what you what:
(I'm using Kubernetes master node as an example environment)
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.14.2 5c24210246bb 3 months ago 82.1MB
k8s.gcr.io/kube-apiserver v1.14.2 5eeff402b659 3 months ago 210MB
k8s.gcr.io/kube-controller-manager v1.14.2 8be94bdae139 3 months ago 158MB
k8s.gcr.io/kube-scheduler v1.14.2 ee18f350636d 3 months ago 81.6MB # before
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
k8s.gcr.io/coredns 1.3.1 eb516548c180 7 months ago 40.3MB # since
k8s.gcr.io/etcd 3.3.10 2c4adeb21b4f 8 months ago 258MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 20 months ago 742kB
$ docker images --filter "since=eb516548c180" --filter "before=ee18f350636d"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
$ docker images --filter "since=eb516548c180" --filter "reference=quay.io/coreos/flannel"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
$ docker images --filter "since=eb516548c180" --filter "reference=quay*/*/*"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
$ docker images --filter "since=eb516548c180" --filter "reference=*/*/flan*"
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/coreos/flannel v0.11.0-amd64 ff281650a721 6 months ago 52.6MB
As mentioned in the documentation, images / image ls filter is much better than docker prune filter, which supports until clause only:
The currently supported filters are:
• dangling (boolean - true or false)
• label (label=<key> or label=<key>=<value>)
• before (<image-name>[:<tag>], <image id> or <image#digest>) - filter images created before given id or references
• since (<image-name>[:<tag>], <image id> or <image#digest>) - filter images created since given id or references
If you need more than one filter, then pass multiple flags
(e.g., --filter "foo=bar" --filter "bif=baz")
You can use other linux cli commands to filter docker images output:
grep "something" # to include only specified images
grep -v "something" # to exclude images you want to save
sort [-k colN] [-r] [-g]] | head/tail -nX # to select X oldest or newest images
Combining them and putting the result to CI/CD pipeline allows you to leave only required images in the local cache without collecting a lot of garbage on your build server.
I've copied here a good example of using that approach provided by strajansebastian in the comment:
#example of deleting all builds except last 2 for each kind of image
#(the image kind is based on the Repository value.)
#If you want to preserve just last build modify to tail -n+2.
# delete dead containers
docker container prune -f
# keep last 2 builds for each image from the repository
for diru in `docker images --format "{{.Repository}}" | sort | uniq`; do
for dimr in `docker images --format "{{.ID}};{{.Repository}}:{{.Tag}};'{{.CreatedAt}}'" --filter reference="$diru" | sed -r "s/\s+/~/g" | tail -n+3`; do
img_tag=`echo $dimr | cut -d";" -f2`;
docker rmi $img_tag;
done;
done
# clean dangling images if any
docker image prune -f

docker pull always show "Already exists" for layers during pull even after deleting all images

here is my input and output:
shshenhx#shshenhx:~/Desktop/Docker$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
python latest 336d482502ab 4 days ago 692 MB
shshenhx#shshenhx:~/Desktop/Docker$ docker rmi 336
Untagged: python:latest
Untagged: python#sha256:bf0718e2882cabc144c9c07d758b09c65efc104a6ddc72a9a976f8b26f67c2ee
Deleted: sha256:336d482502ab564b0b2964b2ed037529ba40c7b4ade2117ca11d74edbf11f99e
Deleted: sha256:1e2f72b0bf844de7bfe57a32115b660770355843ef8078fb55629e720147e6a9
Deleted: sha256:b5818ba96f33835318f3e9d7b4094e1007be247f04ab931ea9ef83b45b451f90
Deleted: sha256:0c2d7cafdab1084ebbd0112b3bedf76458ae6809686fb0ad9bae8022f11c3a84
shshenhx#shshenhx:~/Desktop/Docker$ docker pull python
Using default tag: latest
latest: Pulling from library/python
4176fe04cefe: Already exists
851356ecf618: Already exists
6115379c7b49: Already exists
aaf7d781d601: Already exists
40cf661a3cc4: Already exists
975fe2fd635f: Pull complete
bf4db784e7fd: Pull complete
0491f7e9426b: Pull complete
Digest: sha256:bf0718e2882cabc144c9c07d758b09c65efc104a6ddc72a9a976f8b26f67c2ee
Status: Downloaded newer image for python:latest
My question is, I have already rm python image, why it still shows already exist for some of layers? How can I totally delete all the python layers.
Thanks.
What helped for me was to run docker-system prune after removing all containers and images. So the whole process I got it to work was:
Remove all containers docker rm -vf $(docker ps -a -q)
Remove all images docker rmi -f $(docker images -a -q)
Prune system with docker system prune
From reference of docker images command
Docker images have intermediate layers that increase reusability, decrease
disk usage, and speed up docker build by allowing each step to be cached.
These intermediate layers are not shown by default.
Maybe those Already exists are intermediate layers. By default, they are hided when you run docker images, please try docker images --all.
Docker has a solution for this on docker dashboard, just press the button below and presto it works! (This is in the troubleshooting section. or the bug icon)
There is a docker command that deletes images that do not have an associated running container:
docker image prune
allows you to clean up dangling images (a dangling image is not tagged and not referenced by any container)
docker image prune -a
will remove all images that are not used by any existing containers
docker image prune -a --filter "until=48h"
you can add a filter (--filter) to limit the images to be pruned, as in this example, only images that are older than 48 hours
You can add the -f or --force flag to prune the images and not prompt for confirmation
You could use this in a scheduled bash script to keep your drive tidy
docker system prune -a helped me.It will remove both unused and dangling images.
You can use docker system prune but it will clear everything in docker.
Its not an image problem its actually cache problem so deleting your images wont work because the problem relies in layer caching.
There is a --no-cache flag in docker build but I don't know if it works with docker pull too.
For someone interested in: (1) using the command line; (2) removing cache only for images that have been removed with docker rmi (i.e., not visible with docker images --all), but for which caches are still present, and for which the Already exists shows for layers when pulling image; you can try this command:
docker builder prune
(It may take up to 15 minutes to run this command, so be patient.)
This command seems to have solved the problem for me. I did not want to use the general, "system-wide" prune / purge / clean docker commands and functionalities, because I had some images and containers running (not related to the image that I have already removed), that I wished to have left as they were.
Source of the solution: https://forums.docker.com/t/how-to-delete-cache/5753/7
I think that the situation of "invisible" cached layers for images removed with docker rmi may happen due to the fact of using docker compose (as suggested also by John Doe in his comment to one of the answers in the current question: docker pull always show "Already exists" for layers during pull even after deleting all images).
So after a lot of searching, I have a solution although I don't think it's necessarily a great one.
The "sledge hammer solution" would be to do:
rm -rf /var/lib/docker
This basically factory resets docker, so you would lose everything. This was fine for my case, but be very careful and think this through before using this one.
Please try this command docker rmi 336d482502ab and then try to pull it again.

Many <none> images created after build a docker image

FROM scratch
MAINTAINER Aario <AarioAi#gmail.com>
ENV SHARED_GROUP docker
I build a docker image with above dockerfile.
After runing docker build -t "aario/centos" .
It creates this image aario/centos and the <none> image:
Is it ok? And how to solve it?
When I run docker rmi 2f??????? to remove the aario/centeros image. The <none> image will be removed on the same time.
Docker image is composed of layers:
docker history aario/centos
Each row you see using command above is a separate layer. Also, with time, a number of "orphaned" layers will fill up your disk space. You can safely remove them with:
docker image prune
Those are intermediate image layers. I.e, all of the steps of your Dockerfile.
You can check what these images are made of with inspect command:
docker image inspect 2f???????
They do not use additional disk space, since they are part of your named image. They can save space when you have multiple Dockerfiles with the same steps (in the same order) because they work as cache.
A longer and more complete explanation can be found here.
Eventually, if you delete your image or change build steps, you could have dangling none images. Those can be deleted, as #Mike mentioned, with:
docker image prune

docker build creates another set of docker image even the build command is the same?

Assume that you can create a 500MB docker image with following command
docker build -t demo:1.0 .
It seems to me that the docker image (demo:1.0) is overwritten when I re-run the same command. However, I noticed that free disk space decreases as I re-run the command. Why does this happen? Is docker creating another set of the image somewhere in the file system? I'd like to know how I can find old-generations of the docker images and delete them.
It's standard docker behavior, if you run
docker images
you may see a dangling image like this
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
demo 1.0 a1b38444260e 6 days ago 500 MB
<none> <none> 46250b7172c5 6 days ago 500 MB
You can remove the dangling images with the command
docker rmi $(docker images -qa -f "dangling=true")
Docker doesn’t delete such images automatically, despite, it's a logical feature to have. For now this command can be used to do a manual deletion.
While the images are stored in /var/lib/docker/{driver-name}, you wouldn't remove images manually from that path.

Resources