I use docker to build a web app (a Rails app specifically).
Each build is tagged with the git SHA value and the :latest tag points to the latest SHA value (e.g. 4bfcf8d) in this case.
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
feeder_web 4bfcf8d c2f766746901 About a minute ago 1.61GB
feeder_web latest c2f766746901 About a minute ago 1.61GB
feeder_web c14c3e6 4cb983fbf407 13 minutes ago 1.61GB
feeder_web cc1ecd9 3923b2c0c77f 18 minutes ago 1.61GB
Each version only differs by some minor copy in the app's frontend, but other than that they are largely the same.
Each one is listed at 1.61GB. Does it really require an additional 1.61GB for each build if I just change a few lines in the web app? And if so, should I actively be clearing old builds?
Each version only differs by some minor copy in the app's frontend, but other than that they are largely the same. Each one is listed at 1.61GB. Does it really require an additional 1.61GB for each build if I just change a few lines in the web app?
Whether or not you can benefit from layer caching depends largely on how you write your Dockerfile.
For example if you write
FROM debian
COPY ./code /code
RUN apt-get update && all that jazz ... ...
...
and you change one iota of that ./code, the whole layer is tossed and every layer after it. Docker has to rerun (and re-store) your layer, creating another few hundred meg layers every time you build. But if you run
FROM debian
RUN apt-get song and dance my-system deps && clean up after myself
MKDIR /code
COPY ./code/requirements /code/requirements
RUN pip or gem thingy
COPY /code/
...
Now you don't have to install requirements every time. So the bulk of your environment (system and language libraries) doesn't need to change. You should only need the space for whatever you copy in ./code and thereafter in this case - still usually in the 0.1 gig or so magnitude.
The community tends to tout minimization of layers in an image and as far as steps with the same lifespan and dependencies (apt-get install / cleanup ) it makes sense. But this is actually contrary to efficiency if you can make good use of caching. For example, if you need to change gem file, probably don't need to change all of system libraries, so no need to rebuild that image unless you want to update the lower layers. This also drastically increases build time if you don't have to install libffi-dev or whatever every time.
Likely the biggest thing you can do to keep final image sizes down is use multi stage builds. Python and Ruby containers often end up pulling in complex build time dependencies that are then kept in the final image. Removing them from the end image is also a potential security bonus, at least security overhead in terms of CVE exposure. So look up multi stage builds if you haven't yet and spend an hour seeing if it's fairly easy to get some of the build time dependencies out of your final image. Full disclosure: I cannot be sure at the moment whether these build stages are automatically cleaned up.
And if so, should I actively be clearing old builds?
Since disk space is a fundamentally limited resource, the only question is how actively, and to what extent do you want to mitigate by increasing hard disk space.
And don't forget to clean up old containers ,too. I try to make docker run --rm a habit whenever possible, but still find myself pruning them after they inevitably build up.
If one build requires 1.6G then just changing a few lines is not going to change the size.
If you are not planning to use them anymore, I would suggest clearing out the old builds. Every so often I do a docker system prune -a which removes unused data (images, containers, volumes, etc.).
Each build is a full container, not just a layer on top of an older container. In other words, if you update your image 4 times and rebuild, you have 4 full copies of you app running. Each one is going to take up approximately the same amount of space because they are very similar to one another other than the incremental changes you’ve made between each build.
In this context, I don’t think there’s any reason not to clear out the old containers. They are clearly consuming a lot of space. Presumably, you’ve updated them for a reason, so unless you are continuing to use/test the old version of your app and need to keep it for that reason, it probably makes sense to at least periodically clear them out. This is assuming you’re doing all this in a development environment (ie, your personal/work machine) and having old versions lying around dormant doesn’t pose a security risk because they aren’t actively connecting to external services. If these are running on a live server somewhere, dump the old ones as soon as they are no longer being used so that you aren’t unnecessarily exposing any extra attack surface on your production servers.
Related
Having needed several times in the last few days to upload a 1Gb image after some micro change, I can't help but wonder why there isnt a deploy path built into docker and related tech (e.g. k8s) to push just the application files (Dockerfile, docker-compose.yml and app related code) and have it build out the infrastructure from within the (live) docker host?
In other words, why do I have to upload an entire linux machine whenever I change my app code?
Isn't the whole point of Docker that the configs describe a purely deterministic infrastructure output? I can't even see why one would need to upload the whole container image unless they make changes to it manually, outside of Dockerfile, and then wish to upload that modified image. But that seems like bad practice at the very least...
Am I missing something or this just a peculiarity of the system?
Good question.
Short answer:
Because storage is cheaper than processing power, building images "Live" might be complex, time-consuming and it might be unpredictable.
On your Kubernetes cluster, for example, you just want to pull "cached" layers of your image that you know that it works, and you just run it... In seconds instead of compiling binaries and downloading things (as you would specify in your Dockerfile).
About building images:
You don't have to build these images locally, you can use your CI/CD runners and run the docker build and docker push from the pipelines that run when you push your code to a git repository.
And also, if the image is too big you should look into ways of reducing its size by using multi-stage building, using lighter/minimal base images, using few layers (for example multiple RUN apt install can be grouped to one apt install command listing multiple packages), and also by using .dockerignore to not ship unnecessary files to your image. And last read more about caching in docker builds as it may reduce the size of the layers you might be pushing when making changes.
Long answer:
Think of the Dockerfile as the source code, and the Image as the final binary. I know it's a classic example.
But just consider how long it would take to build/compile the binary every time you want to use it (either by running it, or importing it as a library in a different piece of software). Then consider how indeterministic it would download the dependencies of that software, or compile them on different machines every time you run them.
You can take for example Node.js's Dockerfile:
https://github.com/nodejs/docker-node/blob/main/16/alpine3.16/Dockerfile
Which is based on Alpine: https://github.com/alpinelinux/docker-alpine
You don't want your application to perform all operations specified in these files (and their scripts) on runtime before actually starting your applications as it might be unpredictable, time-consuming, and more complex than it should be (for example you'd require firewall exceptions for an Egress traffic to the internet from the cluster to download some dependencies which you don't know if they would be available).
You would instead just ship an image based on the base image you tested and built your code to run on. That image would be built and sent to the registry then k8s will run it as a black box, which might be predictable and deterministic.
Then about your point of how annoying it is to push huge docker images every time:
You might cut that size down by following some best practices and well designing your Dockerfile, for example:
Reduce your layers, for example, pass multiple arguments whenever it's possible to commands, instead of re-running them multiple times.
Use multi-stage building, so you will only push the final image, not the stages you needed to build to compile and configure your application.
Avoid injecting data into your images, you can pass it later on-runtime to the containers.
Order your layers, so you would not have to re-build untouched layers when making changes.
Don't include unnecessary files, and use .dockerignore.
And last but not least:
You don't have to push images from your machine, you can do it with CI/CD runners (for example build-push Github action), or you can use your cloud provider's "Cloud Build" products (like Cloud Build for GCP and AWS CodeBuild)
why we don't use CMD apt update instead of RUN apt update on Dockerfile
we use RUN apt update for update an image this is for one time but why we don't use CMD apt update for update every container we create ? ? ? ?
As it sounds like you already know, RUN is intended "xecute any commands in a new layer on top of the current image and commit the results", and CMD is intended to "xecute any commands in a new layer on top of the current image and commit the results". So RUN is a build-time instruction, while CMD is a run-time instruction.
There are a few reasons this won't be a good idea:
Containers should be fast
Containers are usually expected to consume as few resources as possible, and startup and shutdown quickly and easily. If we update a container's packages EVERY time we want to run a container, it might take the container many minutes or even hours on a bad network before it can even start running whatever process it is intended for.
Unexpected behavior
Part of the process when developing a new container image is ensuring that the packages that are necessary for the container to work, play well together. But if we are upgrading all the packages each time the container is run on whatever system it is run on, it is possible (if not inevitable) that there will eventually be a package that will be published that introduces a breaking change to the container, and this is obviously not ideal.
Now this could be avoided by removing the default repositories and replacing them with your own where you can vet each package upgrade, test them together, and publish them, but this is probably a much greater effort than what would make sense unless the repos would be serving multiple container images.
Image Versioning
Many container images (ex Golang) will version their images based on the version of Golang they support; however, when the underlying packages on the container are changing how would you start to version the image?
Now this isn't necessarily a deal breaker, but it could cause confusion among the containers user-base and ultimately undercut their trust in the container.
Unexpected network traffic
Even if well documented, most developers would not expect this type of functionality and would lead to development issues when your container requires access to the internet. For example, in a K8s environment networking can be extremely strict and the developer would need to manually open up a route to the internet (or a set of custom repos).
Additionally, even if the networking is not an issue, if you expected a lot of these containers to be started, you might be clogging the network with the upgrade packages and cause network performance issues.
Wouldn't work for base images
While it sounds like you are probably not developing an image intended to serve as a base image for anything else... but obviously the CMD likely would be overriden for the base image.
I few months ago I decided to setup the CI of my project building docker images with the no-cache flag: I thought it would be best not to take the risk of letting docker use an old cache layer.
I realized only now that the sha of the layers of my image are always different (even if the newly built image should generate a layer identical to the previous built) and whenever I pull the newly built image all layers are always downloaded from zero.
I'm thinking now that the issue is the --no-cache flag, I know it sounds obvious, but honestly I thought that the --no-cache was only slower to execute, but also thought that it was implemented in a functional way (same command + same content = same layer).
Can someone confirm that the --no-cache flag is the problem?
The thing with containers is that practically speaking, you will never build the same layer with the same sha, ever. You can only have the same sha if you use the same layer you previously built.
Think about it this way: every time you build a layer, there will be at least a log file, a timestamp, something that is different - and then we have not yet mentioned external dependencies pulled in.
The --no-cache flag will simply stop the Docker engine from using the cached layers and it will download & build everything again. So that flag is indeed the (indirect) reason why your hashes are different, but that is the intended behavior. Building from the cache means that your builds will be faster, but reuse previously built layers, hence having the same sha (this may result in reusing previous stale results and whatnot, that is why we have the flag).
Have a look at this article for further info:
https://thenewstack.io/understanding-the-docker-cache-for-faster-builds/
If you wish to guarantee some layers have the same sha but still not want to use the cache, you may want to look into multiphase Docker builds:
https://docs.docker.com/develop/develop-images/multistage-build/
This way you can have a base image which is fixed and build everything else on top of that.
One of the main benefits of Docker is reproducibility. One can specify exactly which programs and libraries get installed how and where, for example.
However, I'm trying to think this through and can't wrap my head around it. How I understand reproducibility is that if you request a certain tag, you will receive the same image with the same contents every time. However there are two issues with that:
Even if I try to specify a version as thoroughly as possible, for example python:3.8.3, I seem to have no guarantee that it points to a static non-changing image? A new version could be pushed to it at any time.
python:3.8.3 is a synonym for python:3.8.3-buster which refers to the Debian Buster OS image this is based on. So even if Python doesn't change, the underlying OS might have changes in some packages, is that correct? I looked at the official Dockerfile and it does not specify a specific version or build of Debian Buster.
If you depend on external docker images, your Docker image indeed has no guarantee of reproducability. The solution is to import the Python:3.8.3 image into your own Docker Registry, ideally a docker registry that can prevent overriding of tags (immutability), e.g. Harbor.
However, reproducibility if your Docker image is harder then only the base image you import. E.g. if you install some pip packages, and one of the pip packages does not pin a version of a package they depend on, you still have no guarantee that rebuilding your Docker image leads to the same image. Hosting those python packages in your own pip artifactory is again the solution here.
Addressing your individual concerns.
Even if I try to specify a version as thoroughly as possible, for example python:3.8.3, I seem to have no guarantee that it points to a static non-changing image? A new version could be pushed to it at any time.
I posted this in my comment on your question, but addressing it here as well. Large packages use semantic versioning. In order for trust to work, it has to be established. This method of versioning introduces trust and consistency to an otherwise (sometimes arbitrary) system.
The trust is that when they uploaded 3.8.3, it will remain as constant as possible for the future. If they added another patch, they will upload 3.8.4, if they added a feature, they will upload 3.9.0, and if they broke a feature, they would create 4.0.0. This ensures you, the user, that 3.8.3 will be the same, every time, everywhere.
Frameworks and operating systems often backport patches. PHP is known for this. If they find a security hole in v7 that was in v5, they will update all versions of v5 that had it. While all the v5 versions were updated from their original published versions, functionality remained constant. This is important, this is the trust.
So, unless you were "utilizing" that security hole to do what you needed to do, or relying on a bug, you should feel confident that 3.8.3 from DockerHub should always be used.
NodeJS is a great example. They keep all their old deprecated versions available in Docker Hub for archival sake.
I have been utilizing named tags (NOT latest) from Docker Hub in all my projects for work and home, and I've never into an issue after deployment where a project crashed because something changed "under my feet". In fact, just last week, I rebuilt and updated some code on an older version of NodeJS (from 4 years ago) which required a repull, and because it was a named version (not latest), it worked exactly as expected.
python:3.8.3 is a synonym for python:3.8.3-buster which refers to the Debian Buster OS image this is based on. So even if Python doesn't change, the underlying OS might have changes in some packages, is that correct? I looked at the official Dockerfile and it does not specify a specific version or build of Debian Buster.
Once a child image (python) is built off a parent image (buster), it is immutable. The exception is if the child image (python) was rebuilt at a later date and CHOOSES to use a different version of the parent image (buster). But this is considered bad-form, sneaky and undermines the PURPOSE of containers. I don't know any major package that does this.
This is like doing a git push --force on your repository after you changed around some commits. It's seriously bad practice.
The system is designed and built on trust, and in order for it to be used, adopted and grow, the trust must remain. Always check the older tags of any container you want to use, and be sure they allow their old deprecated tags to live on.
Thus, when you download python:3.8.3 today, or 2 years from now, it should function exactly the same.
For example, if you docker pull python:2.7.8, and then docker inspect python:2.7.8 you'll find that it is the same container that was created 5 years ago.
"Created": "2014-11-26T22:30:48.061850283Z",
Assuming I have 2 options to add docker layers.
Option1:
RUN python -m nltk.downloader punkt averaged_perceptron_tagger brown
Option2:
RUN python -m nltk.downloader punkt
RUN python -m nltk.downloader brown
RUN python -m nltk.downloader averaged_perceptron_tagger
I understand that the 2nd option adds 3 layers whereas 1st option adds only 1 layer.
Does number of layers have implication on size, setup time or performance of current and future docker images?
Note: Current means the current image. Future means any image that may use some of layers from an existing image thus speeding up the setup.
It does impact setup time and might impact size. It should not performance, meaning by performance the actual performance of the running app.
It does impact setup time because, the better you define your layers, the better they can be reusable for other images. At the end of the day, a layer is just a cache, if it can be shared for other images, the build time will be improved.
Regarding size, it really depends on how you build the image. For example, if you have build dependencies that are not needed on runtime, the image will be bigger because it will have such dependencies. Speaking of python, usually you will want to install build-essential to build your app, however, once the packages are installed, you no longer need build-essential. If you don't remove it, the image will be bigger.
To remove it, you have two options:
Either you use a long RUN statement in which you install build-essential, install the packages you need, and then remove build-essential, all in the same RUN.
Just use multi-staging, and have different stages for building and running.
At a practical level, and especially at the level of single layers, it makes no difference. There used to be a documented limit of 127 layers in an image; most practical images have fewer than 20. In principle going through the Docker filesystem layers could be slower if there are more of them, but Linux kernel filesystem caching applies, and for most performance-sensitive things, avoiding going to disk at all is usually best.
As always with performance considerations, if it really matters to you, measure it in the context of your specific application.
I'd say there are really three things to keep in mind about Docker image layers:
Adding a layer never makes an image smaller. If you install something in an earlier RUN step, and remove it in a later RUN step, your image winds up with all of the installed content from the earlier layer, and an additional placeholder layer that says "this stuff is deleted now". This particularly happens around build tools. #eez0 discuses this case a little more in their answer.
Layers are the unit of Docker image caching. If you repeat a docker build step and you're running an identical command on an exact layer that already existed, Docker will skip actually running it and reuse the resulting layer from the previous build. This has a couple of impacts on Dockerfile style (you always want to RUN apt-get update && apt-get install in the same command, in case you change the list of packages that get installed) but doesn't really impact performance.
You can docker run the result of an individual step. This is a useful debugging technique: the output of docker build includes an image ID for each step, and you can docker run your intermediate results. If some step is failing you can get a debugging shell on the image up to the start of that step and see what's actually in the filesystem.
In your example, the two questions worth asking are how much overhead an individual downloader step has, and how likely this set of plugins is to change. If you're likely to change things often, separate RUN commands will let you cache layers better on later builds; if the downloader tool itself has high overhead (maybe it assembles all downloaded plugins into a single zip file) then running it just once could conceivably be faster.
Usual practice is to try to pack things together into a single RUN command, but it is kind of a micro-optimization that usually won't make much of a difference in practice. In the case of package installations I'm used to seeing just one apt-get install or pip install line in a Dockerfile and style-wise I might expect that here too. If you're in the process of developing things, one command per RUN line is easier to understand and debug.