I have a base which is being used by 100 applications. All 100 applications have the common base image in their Dockerfile. Now, I am upgrading the base image for OS upgrade or some other upgrade and bumping up the version and I also tag with the latest version.
Here, the problem is, whenever I change the base image, all 100 application needs to change the base image in their dockerfile and rebuild the app for using the latest base image.
Is there any better way to handle this?
Note :- I am running my containers in Kubernetes and Dockerfile is there in GIT for each application.
You can use a Dockerfile ARG directive to modify the FROM line (see Understand how ARG and FROM interact in the Dockerfile documentation). One possible approach here would be to have your CI system inject the base image tag.
ARG base=latest
FROM me/base-image:${base}
...
This has the risk that individual developers would build test images based on an older base image; if the differences between images are just OS patches then you might consider this a small and acceptable risk, so long as only official images get pushed to production.
Beyond that, there aren't a lot of alternatives beyond modifying the individual Dockerfiles. You could script it
# Individually check out everything first
BASE=$(pwd)
TAG=20191031
for d in *; do
cd "$BASE/$d"
sed -i.bak "s#FROM me/base-image.*#FROM:me/base-image:$TAG/" Dockerfile
git checkout -b "base-image-$TAG"
git commit -am "Update Dockerfile to base-image:$TAG"
git push
hub pull-request --no-edit
done
There are automated dependency-update tools out there too and these may be able to manage the scripting aspects of it for you.
You don't need to change the Dockerfile for each app if it uses base-image:latest. You'll have to rebuild app images though after base image update. After that you'll need to update the apps to use the new image.
For example using advises from this answer
whenever I change the base image, all 100 application needs to change the base image in their dockerfile and rebuild the app for using the latest base image.
That's a feature, not a bug; all 100 applications need to run their tests (and potentially fix any regressions) before going ahead with the new image...
There are tools out there to scan all the repos and automatically submit pull requests to the 100 applications (or you can write a custom one, if you don't have just plain "FROM" lines in Dockerfiles).
If you need to deploy that last version of base image, yes, you need to build, tag, push, pull and deploy each container again. If your base image is not properly tagged, you'll need to change your dockerfile on all 100 files.
But you have some options, like using sed to replace all occurrences in your dockerfiles, and execute all build commands from a sh file pointing to every app directory.
With docker-compose you may update your running 100 apps with one command:
docker stack deploy --compose-file docker-compose.yml
but still needs to rebuild the containers.
edit:
with docker compose you can build too your 100 containers with one command, you need to define all of them in a compose file, check the docks for the compose file.
Related
When processing a Dockerfile, how do I instruct docker build to build the image specified in FROM locally using another Dockerfile if it is not already available?
Here's the context. I have a large Dockerfile that starts from base Ubuntu image, installs Apache, then PHP, then some custom configuration on top of that. Whether this is a good idea is another point, let's assume the build steps cannot be changed. The problem is, every time I change anything in the config, everything has to be rebuilt from scratch, and this takes a while.
I would like to have a hierarchy of Dockerfiles instead:
my-apache : based on stock Ubuntu
my-apache-php: based on my-apache
final: based on my-apache-php
The first two images would be relatively static and can be uploaded to dockerhub, but I would like to retain an option to build them locally as part of the same build process. Only one container will exist, based on the final image. Thus, putting all three as "services" in docker-compose.yml is not a good idea.
The only solution I can think of is to have a manual build script that for each image checks whether it is available on Dockerhub or locally, and if not, invokes docker build.
Are there better solutions?
I have found this article on automatically detecting dependencies between docker files and building them in proper order:
https://philpep.org/blog/a-makefile-for-your-dockerfiles
Actual makefile from Philippe's git repo provides even more functionality:
https://github.com/philpep/dockerfiles/blob/master/Makefile
What is the best practices for getting code into a docker container?
Here are some possible approaches:
ADD call in docker container
git clone or git wget from repo in docker container
Mount an external device
Usually, we use mount for dev/local environment with docker to have the code changes instantly applied.
You can use RUN git clone but you will need to install git and have access to the repository.
The easiest and most used/recommended way is putting the Dockerfile inside your repo and use the ADD call. But use the COPY directive instead because it’s more explicit
You should always COPY your application code in the Dockerfile. You generally shouldn't mount something over that, and you almost definitely shouldn't run git in your Dockerfile.
Running Git has a number of obvious issues. You can only build whatever specific commit the Dockerfile mentions; in real development it's not unusual to want to build a branch or a past commit. You need to install both git itself and relevant credentials in the image, with corresponding space and security issues. Docker layer caching means that docker build by default won't repeat a git clone step if it thinks it's already done that, which means you need to go out of your way to get a newer build.
It's very common in SO questions to mount code into a container. This very much defeats the point of Docker, IMHO. If you go to run your application in production you want to just distribute the image and not also separately distribute its code, but if you've been developing in an environment where you always hide the code built into the image, you've never actually run the image itself. There are also persistent performance and consistency problems in various environments. I don't really recommend docker run -v or Docker Compose volumes: as a way of getting code into a container.
I've seen enough SO questions that have a Dockerfile that's essentially just FROM python, for example, and then use Docker Compose volumes: and command: options to get a live-development environment on top of that. It's much easier to just install Python, and have the tools you need right in front of you without needing to go through a complex indirection sequence to run them in Docker.
I'm just starting with docker, and this question probably clearly shows that I'm not really understanding the base concepts yet, but I can't figure it out.
So, I want to use this image as my "base" image: https://registry.hub.docker.com/u/phusion/baseimage/
Now, this base image has a number of tags (versions). The most recent one is 0.9.11.
So, let's say I'll spin up a number of images based on this "base" image and push those to production.
Then Phusion guys will push some updates to that image and I would want to upgrade not just the actual base image but also all of the images I already use on prod (based on the "base" image).
So how would I do that?
=================
Extra question:
The other case I assume should be perfectly possible:
The base image has some common lib, openssl, for example. Now there's a new bug discovered and I need to upgrade to newer openssl version.
Is this possible to upgrade the openssl on the base image, commit it to my local registry, and pull that change on all images that are based on that "base" image?
When you build a Dockerfile, instructions are read from top to bottom using cache as much as possible.
The first time it encounters a command that changed or a new command, the cache is busted.
The FROM directive is usually at the top of the Dockerfile, so if you change the tag of your base image, the whole Dockerfile will be re-built from scratch.
And that's how you "update" a base image, you rebuild all your containers from their Dockerfiles, you don't "push" the changes.
I'm playing with docker by creating a Dockerfile with some nodejs instructions. Right now, every time I make changes to the dockerfile I recreate the image by running sudo docker build -t nodejstest . in my project folder however, this creates a new image each time and swallows my ssd pretty soon.
Is there a way I can update an existing image when I change the dockerfile or I'm forced to create a new one each time I make changes to the file?
Sorry if it's a dumb question
Docker build support caching as long as there is no ADD instruction. If you are actively developing and changing files, only what is after the ADD will be rebuilt.
Since 0.6.2 (scheduled today), you can do docker build --rm . and it will remove the temporary containers. It will keep the images though.
In order to remove the orphan images, you can check them out with docker images, and perform a docker rmi <id> on one of them. As of now, there is an auto-prune and all untagged images (orphans, previous builds) will be removed.
According to this best practices guide if you keep the first lines of your dockerfile the same it'll also cache them and reuse the same images for future builds
During development, it makes less sense to re-build a whole container for every commit. Later, you can automate building a Docker container with your latest code as part of your QA/deployment process.
Basically, you can choose to make a minimal container that pulls in code (using git when starting the container, or using -v /home/myuser/mynode:/home/myuser/mynode with ENTRYPOINT to run node).
See my answer to this question:
Docker rails app and git
If you are making a service with a Dockerfile is it preferred for you to build an image with the Dockerfile and push it to the registry -- rather than distribute the Dockerfile (and repo) for people to build their images?
What use cases favour Dockerfile+repo distribution, and what use case favour Registry distribution?
I'd imagine the same question could be applied to source code versus binary package installs.
Pushing to a central shared registry allows you to freeze and certify a particular configuration and then make it available to others in your organisation.
At DevTable we were initially using a Dockerfile that was run when we deployed our servers in order to generate our Docker images. As our docker image become more complex and had more dependencies, it was taking longer and longer to generate the image from the Dockerfile. What we really needed was a way to generate the image once and then pull the finished product to our servers.
Normally, one would accomplish this by pushing their image to index.docker.io, however we have proprietary code that we couldn't publish to the world. You may also end up in such a situation if you're planning to build a hosted product around Docker.
To address this need in to community, we built Quay, which aims to be the Github of Docker images. Check it out and let us know if it solves a need for you.
Private repositories on your own server are also an option.
To run the server, clone the https://github.com/dotcloud/docker-registry to your own server.
To use your own server, prefix the tag with the address of the registry's host. For example:
# Tag to create a repository with the full registry location.
# The location (e.g. localhost.localdomain:5000) becomes
# a permanent part of the repository name
$ sudo docker tag 0u812deadbeef your_server.example.com:5000/repo_name
# Push the new repository to its home location on your server
$ sudo docker push your_server.example.com:5000/repo_name
(see http://docs.docker.io.s3-website-us-west-2.amazonaws.com/use/workingwithrepository/#private-registry)
I think it depends a little bit on your application, but I would prefer the Dockerfile:
A Dockerfile...
... in the root of a project makes it super easy to build and run it, it is just one command.
... can be changed by a developer if needed.
... is documentation about how to build your project
... is very small compared with an image which could be useful for people with a slow internet connection
... is in the same location as the code, so when people checkout the code, they will find it.
An Image in a registry...
... is already build and ready!
... must be maintained. If you commit new code or update your application you must also update the image.
... must be crafted carefully: Can the configuration be changed? How you handle the logs? How big is it? Do you package an NGINX within the image or is this part of the outer world? As #Mark O'Connor said, you will freeze a certain configuration, but that's maybe not the configuration someone-else want to use.
This is why I would prefer the Dockerfile. It is the same with a Vagrantfile - it would prefer the Vagrantfile instead of the VM image. And it is the same with a ANT or Maven script - it would prefer the build script instead of the packaged artifact (at least if I want to contribute code to the project).