If there are RUN commands in docker file then it does create some intermediate images. My question is whether such intermediate images occupy any memory of your hard drive? If yes, docker build --rm should be enough?
Yes intermediate layers occupy disk space and it is a good thing usually. This facilitates re-use of layers and speedy builds. What you should be concentrating on instead is to reduce the number of layers by optimising the dockerfile. Your final docker image is actually a combination of all the layers. So you cannot remove the layers unless you remove the final image and no other image is using the layers.
docker build --rm does not save any extra disk space. To understand why, you should know how docker build works - Each instruction (e.g., RUN) in a dockerfile starts a new container, after the instruction completes, the container exits, and is committed to an image.
docker build --rm removes these intermediate containers. --rm option is true by default and so docker build --rm has not extra affect compared to docker build. For some reason if you want to keep the intermediate containers then you can turn it off with --rm=False.
If there are any layers which are not being used by any other images, you can remove them. These are called dangling layers. You can remove them with following command -
docker rmi $(docker images -f "dangling=true" -q)
RUN does not create intermediate images, but intermediate layers/containers.
If you run docker build --rm these containers will be deleted after they're generated, so yes it will save space in your disk but will need more time each time you rebuild the same image, since it will create those layers everytime as they are not cached.
Edit (thanks #Shashank V): The --rm option is set by default, so having it or not makes no difference.
Related
I execute the docker(Docker version 19.03.1, build 74b1e89) clean command in the server:
docker system prune
to my surprise, this command delete the container that stopped. That's the problem, some container stopped by some reason but I still want to use it in the furture. Now it is deleted, is it possible to recover the mistaking deleted container that stopped?
No, it is not possible. You have to repull the image docker pull image, or rebuild the image.
Docker images and containers are not the same. See this answer.
docker system prune tells you:
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cache
Are you sure you want to continue? [y/N]
So it should be no surprise that it removed your container and possibly also the image it was based on (if no other container was running based on that same image).
I believe it is not possible to recover your image or container, however you can rebuild them. Depending on how the image was obtained you have to run:
docker pull <image> # for an image on dockerhub or other registry
docker build <arguments> # for an image based on a dockerfile
After that you will have your image and you can run a container again with:
docker run <image>
I have a dockerfile which when I want to build results in the error
E: You don't have enough free space in /var/cache/apt/archives/
Note that the image sets up a somewhat complex project with several dependencies that require quite a lot of space. For example, the list includes Qt. This is only a thing during the construction of the image, and in the end, I expect it to have a size of maybe 300 MB.
Now I found this: https://unix.stackexchange.com/questions/578536/how-to-fix-e-you-dont-have-enough-free-space-in-var-cache-apt-archives
Given that, what I tried so far is:
Freeing the space used by docker images so far by calling docker system prune
Removing unneeded installation files by calling sudo apt autoremove and sudo apt autoclean
There was also the suggestion to remove data in var/log, which has currently a size of 3 GB. However, I am not the system administrator and thus wary to do such a thing.
Is there any other way to increase that space?
And, preferably, is there a more sustainable solution, allowing me to build several images without having to search for spots where I can clean up the system?
Try this suggestion. You might have a lot of unused images that need to be deleted.
https://github.com/onyx-platform/onyx-starter/issues/5#issuecomment-276562225
Converting a #Dre suggestion into the code, you might want to use Docker prune command for containers, images & volumes
docker container prune
docker image prune
docker volume prune
You can use these commands in sequence:
docker container prune; docker image prune; docker volume prune
Free Space without removing your latest images
Use the following command to see the different types of reclaimable storage (the -v verbose option provides more detail):
docker system df
docker system df -v
Clear the build cache (the -a option will remove unused build cache):
docker builder prune -a
Remove dangling images ( tagged images, old and previous image builds):
docker rmi -f $(docker images -f "dangling=true" -q)
Increase Disk image size using Docker UI
Docker > Preferences > Resources > Advanced > adjust Disk image size > Apply & Restart
TLDR;
run
docker system prune -a --volumes
I tried to increase the disk space and prune the images, containers and volumes manually but was facing the issue again and again. When I tried to check the memory consumption on my machine, I found a lot of memory consumed by ~/Library/Containers/com.docker.docker location. Did a system prune which cleaned up a lot of space and docker builds started working again.
Is there a way to do docker prune image to get rid of images if either they or their intermediary layers have not been used in the last X hours?
The purpose is: Say a build has created intermdiary and final image(F1). A subsequent build can use the intermediary image. While this subsequent build is running, if I run docker image prune -f -a in another window, then Docker will delete the image F1 (assuming it is unused) and it's intermediary images. Thus the build in progress even if it had used intermediary image will have to rebuild the intermediary image. If there is a filter setting on the prune command based on last used then it will be helpful.
Based on "last used" it might not be very easy, there is still issue going on: https://github.com/moby/moby/issues/4237
Tool named 'docuum' is created for it if you are ready to use it. Also less maintained docker-gc is available.
You can prune based on "last created" as hours for example:
docker image prune -a --force --filter "until=5h"
Source: https://docs.docker.com/engine/reference/commandline/image_prune/
I am working on a microservice with several docker containers. Using docker-compose creates the images I want but also several other copies of images which fill up my disk space. What are the use of this other child images and can I stop them from being created since they are using up my memory. Please note they are not really "dangling" they just appear on build.
The images are the intermediate layers resulted from docker build. They are the parent layers for your final image and cannot be removed as your latest image actually refers to them.
Only those images which are not referenced by any other layers can be removed. These images are called dangling. You can use the following command to remove the dangling images:
docker rmi $(docker images -f "dangling=true" -q)
FROM scratch
MAINTAINER Aario <AarioAi#gmail.com>
ENV SHARED_GROUP docker
I build a docker image with above dockerfile.
After runing docker build -t "aario/centos" .
It creates this image aario/centos and the <none> image:
Is it ok? And how to solve it?
When I run docker rmi 2f??????? to remove the aario/centeros image. The <none> image will be removed on the same time.
Docker image is composed of layers:
docker history aario/centos
Each row you see using command above is a separate layer. Also, with time, a number of "orphaned" layers will fill up your disk space. You can safely remove them with:
docker image prune
Those are intermediate image layers. I.e, all of the steps of your Dockerfile.
You can check what these images are made of with inspect command:
docker image inspect 2f???????
They do not use additional disk space, since they are part of your named image. They can save space when you have multiple Dockerfiles with the same steps (in the same order) because they work as cache.
A longer and more complete explanation can be found here.
Eventually, if you delete your image or change build steps, you could have dangling none images. Those can be deleted, as #Mike mentioned, with:
docker image prune