I have a CI-pipeline that builds a docker image for my app for every run of the pipeline (and the pipeline is triggered by a code-push to the git repository.)
The docker image consists of several intermediate layers which progressively become very large in size. Most of the intermediate images are identical for each run, hence the caching mechanism of docker is significantly utilized.
However, the problem is that the final couple layers are different for each run, as they result from a COPY statement in dockerfile, where the built application artifacts are copied into the image. Since the artifacts are modified for every run, the already cached bottommost images will ALWAYS be invalidated. These images have a size of 800mb each.
What docker command can I use to identify (and delete) these image that gets replaced by newer images, i.e. when they get invalidated?
I would like to have my CI-pipeline to remove them at the end of the run so they don't end up dangling on the CI-server and waste a lot of disk space.
If I understand correctly: With every code push, CI pipeline creates new image, where new version of application is deployed. As a result, previously created image becomes outdated, so you want to remove it. To do so, you have to:
Get rid of all outdated containers, which where created from outdated image
display all containers with command docker ps -a
if still running, stop outdated containers with command docker stop [containerID]
remove them with command docker rm [containerID]
Remove outdated images with command: docker rmi [imageID]
To sum up why this process is needed: you cannot remove any image, until it is used by any existing container (even stopped containers still require their images). For this reason, you should first stop and remove old containers, and then remove old images.
Detection part, and automation of deletion process should be based on image versions and container names, which CI pipeline generates while creating new images.
Edit 1
To list all images, which have no relationship to any tagged images, you can use command: docker images -f dangling=true. You can delete them with the command: docker images purge.
Just one thing to remember here: If you build an image without tagging it, the image will appear on the list of "dangling" images. You can avoid this situation by providing a tag when you build it.
Edit 2
The command for image purging has changed. Right now the proper command is:
docker image prune
Here is a link with a documentation
Related
I want all running containers on my server to always use the latest version of an official base image e.g. node:16.3 in order to get security updates. To achieve that I have implemented an image update mechanism for all container images in my registry using a CI workflow which has some limitations described below.
I have read the answers to this question but they either involve building or inspecting images on the target server which I would like to avoid.
I am wondering whether there might be an easier way to achieve the container image updates or to alleviate some of the caveats I have encountered.
Current Image Update Mechanism
I build my container images using the FROM directive with the minor version I want to use:
FROM node:16.13
COPY . .
This image is pushed to a registry as my-app:1.0.
To check for changes in the node:16.3 image compared to when I built the my-app:1.0 image I periodically compare the SHA256 digests of the layers of the node:16.3 with those of the first n=(number of layers of node:16.3) layers of my-app:1.0 as suggested in this answer. I retrieve the SHA256 digests with docker manifest inpect <image>:<tag> -v.
If they differ I rebuild my-app:1.0 and push it to my registry thus ensuring that my-app:1.0 always uses the latest node:16.3 base image.
I keep the running containers on my server up to date by periodically running docker pull my-app:1.0 on the server using a cron job.
Limitations
When I check for updates I need to download the manifests for all my container images and their base images. For images hosted on Docker Hub this unfortunately counts against the download rate limit.
Since I always update the same image my-app:1.0 it is hard to track which version is currently running on the server. This information is especially important when the update process breaks a service. I keep track of the updates by logging the output of the docker pull command from the cron job.
To be able to revert the container image on the server I have to keep previous versions of the my-app:1.0 images as well. I do that by pushing incremental patch version tags along with the my-app:1.0 tag to my registry e.g. my-app:1.0.1, my-app:1.0.2, ...
Because of the way the layers of the base image and the app image are compared it is not possible to detect a change in the base image where only the uppermost layers have been removed. However I do not expect this to happen very frequently.
Thank you for your help!
There are a couple of things I'd do to simplify this.
docker pull already does essentially the sequence you describe, of downloading the image's manifest and then downloading layers you don't already have. If you docker build a new image with an identical base image, an identical Dockerfile, and identical COPY source files, then it won't actually produce a new image, just put a new name on the existing image ID. So it's possible to unconditionally docker build --pull images on a schedule, and it won't really use additional space. (It could cause more redeploys if neither the base image nor the application changes.)
[...] this unfortunately counts against the download rate limit.
There's not a lot you can do about that beyond running your own mirror of Docker Hub or ensuring your CI system has a Docker Hub login.
Since I always update the same image my-app:1.0 it is hard to track which version is currently running on the server. [...] To be able to revert the container image on the server [...]
I'd recommend always using a unique image tag per build. A sequential build ID as you have now works, date stamps or source-control commit IDs are usually easy to come up with as well. When you go to deploy, always use the full image tag, not the abbreviated one.
docker pull registry.example.com/my-app:1.0.5
docker stop my-app
docker rm my-app
docker run -d ... registry.example.com/my-app:1.0.5
docker rmi registry.example.com/my-app:1.0.4
Now you're absolutely sure which build your server is running, and it's easy to revert should you need to.
(If you're using Kubernetes as your deployment environment, this is especially important. Changing the text value of a Deployment object's image: field triggers Kubernetes's rolling-update mechanism. That approach is much easier than trying to ensure that every node has the same version of a shared tag.)
On my PC, in one window suppose if a large docker build is in progress (about 30 layers). While the build is in progress (say it has reached on the 20th layer), in another window suppose I run docker image prune -a -f (goal is to remove dangling and unused images), then will the layers of the build that are in progress be considered as dangling and removed causing the build to fail? Is this deterministic behaviour?
The docker image prune -a -f command will delete all images, including tagged images, so not just the dangling ones (this is the -a option). There is likely a race condition in that process depending on where the build is at the time the prune command scans for in use images. If the build is currently inside of a RUN step and using the image, then the prune should skip all of the layers used by that image.
If you are between RUN steps when the prune command looks for images to delete, it may attempt to delete the image being used as the parent in your build. That will likely error out the prune if during the delete that image is then in use again. However, it could also cause an error in the build if that prune deletes the parent image the build expected to extend.
Typically I'd recommend against doing a prune where you expect the images cleaned up will include those being actively used in builds. You will cause builds to take a longer time as deleted images are downloaded against, and you may cause more space used on the registry and other nodes pulling images if the layers are recreated rather than reusing the cached layers from previous builds.
For an active build server, using buildkit with it's built in garbage collection is a much better option. The garbage collection looks at when a layer was used rather than originally created, and is based on size of the layers so you can allocate a specific amount of disk for your image builds. And since buildkit runs on containerd instead of the docker engine directly, the cache itself is not mixed with the docker images and you can prune images without impacting the build cache. There's an example of these garbage collection options in my DockerCon presentation.
The docker docs said:
The Docker daemon runs the instructions in the Dockerfile one-by-one, committing the result of each instruction to a new image if necessary, before finally outputting the ID of your new image. The Docker daemon will automatically clean up the context you sent.
So against the quote above, my question is below:
When is it necessary to commit the result of one instruction to a new image?
If the new images is generated, why I can not see any new image except the final image when finishing the build process?
Thanks.
As a Dockerfile author, you never need to manually commit anything. Almost every Dockerfile instruction results in a new layer; the only exceptions I can think of are FROM (which starts a new stage in a multi-stage build) and maybe ARG (documented to "not persist into the built image").
docker images doesn't display images that don't have tags but do have additional images built on top of them. docker images -a will display everything. It's usually an implementation detail, though, and the long list of <none> images tends to be more a source of confusion.
Also note that intermediate image IDs are printed out by docker build as it executes each step, and you should also be able to find them via docker history. Running docker run on an intermediate image is a pretty useful debugging technique if you can tell the input to some step isn't right but you're not sure why.
Docker novice here. Is Docker analogous to GitHub in that you can commit changes to an image without having to re-build the image from scratch? If yes, what commands are used to do that?
Right now every time I make a change to my code I delete the current Docker image using docker system prune -a and re-build the image using docker build -t appname.
There's no need to delete the existing image first, you can rebuild and create a tag to the same image name that already exists. Images themselves are resolved to an immutable image ID that does not change. To change the contents of an image, you must build a new image that has a new image ID. And then to use it, you need to start new containers referencing that new image.
A rebuild from scratch will reuse the cache, so only commands in your Dockerfile that changed, or are after a change, will result in a rebuild. The layers in the beginning of your Dockerfile that are same as previous builds will be reused between images. Those layers need to be built previous on this host (or there's a --cache-from option if you are building in ephemeral cloud environments). Order matters with the build cache, as does the exact hash of the files and their metadata that you copy into your image.
The docker image prune command is useful after you rebuild an image with the same image name. In that scenario, docker will delete old image ID's that no longer have a reference (image name) pointing to it, and do not currently have a container using it. Note that this also removes those old images from the build cache, so you may want to keep some old images around to speed up builds should a change get reverted from a commit.
I am new to docker container. I met a problem about how to catch changes of my code. Some lines in my local style.css file have been changed. Then I built docker image again, but actually nothing changed when I browsed my app.
Here are some methods I found online and tried, but didn't work.
remove image and build again
--no-cache=true
add a comment in Dockerfile to make it different
docker system prune
--pull
(I also used git pull to get code on my cloud instance, these files were checked to be the latest.)
I know little about docker mechanism, could anyone tell me what the problem is?
Extra info I found:
After stopping container and removing image, I restart my instance, then build image, run container again. Only in this way, can I catch those changes. Does anyone know the problem?
Many thanks!
There appears to be a disconnect on the difference between a container and an image. The container is the running instance of your application. It is based on an image that you have already built. That image has a tag for easier referencing, but the real reference to the image is a sha256 hash and building a new image will change the hash that the tag points to without impacting any of your running containers.
Therefore, the workflow to update your running application in docker is to:
Build a new image
Stop the running container
Start a new container pointing to that image
Cleanup any old images or stopped containers
If you are using docker-compose, it automates the middle two steps with a docker-compose up command, and that even deletes the old container. Most users keep a few copies of older images to allow easy rollback.