I need different images for dev,stage, and prod environments, how should I store images in dokckerhub?
should I use tags
my_app:prod
my_app:dev
my_app:stage
or maybe include env name in image like this
my_app_stage
my_app_stage
my_app_stage
Tags are primarily meant for versioning, as the default tag latest implies. If you use it for other meaning without versioning info, like tagging environment as my_app:dev and my_app:prod, there's no strict rule to prohibit that, but it could cause problem for deployment of the containers.
Imagine you have a container defined in docker-compose.yml that specifies my_app:prod as image. It's fine when you're developing locally, but when you deploy to production with Docker Compose or an orchestration service like Kubernetes, depending on policy, the controller can choose to reuse images from its local cache instead of pulling from registry every time. Now you just completed a new version of the image, and pushed it to Docker Hub feeling assured. Too bad it's still under the same name and tag, so the controller considers it's the same and uses the cached image, causing your old version to be deployed.
It could be worse than that. Not all nodes or clusters are configured the same, some will pull the latest version from the registry while some don't. Your swarm or deployment now contains a mixed set of old and new container versions, producing erratic behavior at best.
Now you know better and push your new version as my_app/prod:v2.0 and update the config. All controllers see the new version and pull down to use for replacing and scaling containers. Everything is consistent.
A simple version number as tag may sound a bit too simple, as practically you could have many properties that you find useful to add to an image, to help with documentation or query maybe. Or you need a specific name and tag so you can push to a certain cloud provider. Luckily you don't have to sacrifice versioning to do that, as Docker allows you to apply as many tags as you like:
docker build -t my_app:latest -t my_app:v2.0 -t my_app:prod -t cloud_user/app_image_id:v2.0 .
Related
I want all running containers on my server to always use the latest version of an official base image e.g. node:16.3 in order to get security updates. To achieve that I have implemented an image update mechanism for all container images in my registry using a CI workflow which has some limitations described below.
I have read the answers to this question but they either involve building or inspecting images on the target server which I would like to avoid.
I am wondering whether there might be an easier way to achieve the container image updates or to alleviate some of the caveats I have encountered.
Current Image Update Mechanism
I build my container images using the FROM directive with the minor version I want to use:
FROM node:16.13
COPY . .
This image is pushed to a registry as my-app:1.0.
To check for changes in the node:16.3 image compared to when I built the my-app:1.0 image I periodically compare the SHA256 digests of the layers of the node:16.3 with those of the first n=(number of layers of node:16.3) layers of my-app:1.0 as suggested in this answer. I retrieve the SHA256 digests with docker manifest inpect <image>:<tag> -v.
If they differ I rebuild my-app:1.0 and push it to my registry thus ensuring that my-app:1.0 always uses the latest node:16.3 base image.
I keep the running containers on my server up to date by periodically running docker pull my-app:1.0 on the server using a cron job.
Limitations
When I check for updates I need to download the manifests for all my container images and their base images. For images hosted on Docker Hub this unfortunately counts against the download rate limit.
Since I always update the same image my-app:1.0 it is hard to track which version is currently running on the server. This information is especially important when the update process breaks a service. I keep track of the updates by logging the output of the docker pull command from the cron job.
To be able to revert the container image on the server I have to keep previous versions of the my-app:1.0 images as well. I do that by pushing incremental patch version tags along with the my-app:1.0 tag to my registry e.g. my-app:1.0.1, my-app:1.0.2, ...
Because of the way the layers of the base image and the app image are compared it is not possible to detect a change in the base image where only the uppermost layers have been removed. However I do not expect this to happen very frequently.
Thank you for your help!
There are a couple of things I'd do to simplify this.
docker pull already does essentially the sequence you describe, of downloading the image's manifest and then downloading layers you don't already have. If you docker build a new image with an identical base image, an identical Dockerfile, and identical COPY source files, then it won't actually produce a new image, just put a new name on the existing image ID. So it's possible to unconditionally docker build --pull images on a schedule, and it won't really use additional space. (It could cause more redeploys if neither the base image nor the application changes.)
[...] this unfortunately counts against the download rate limit.
There's not a lot you can do about that beyond running your own mirror of Docker Hub or ensuring your CI system has a Docker Hub login.
Since I always update the same image my-app:1.0 it is hard to track which version is currently running on the server. [...] To be able to revert the container image on the server [...]
I'd recommend always using a unique image tag per build. A sequential build ID as you have now works, date stamps or source-control commit IDs are usually easy to come up with as well. When you go to deploy, always use the full image tag, not the abbreviated one.
docker pull registry.example.com/my-app:1.0.5
docker stop my-app
docker rm my-app
docker run -d ... registry.example.com/my-app:1.0.5
docker rmi registry.example.com/my-app:1.0.4
Now you're absolutely sure which build your server is running, and it's easy to revert should you need to.
(If you're using Kubernetes as your deployment environment, this is especially important. Changing the text value of a Deployment object's image: field triggers Kubernetes's rolling-update mechanism. That approach is much easier than trying to ensure that every node has the same version of a shared tag.)
I've setup a staging and a production server. The flow is as follows. I develop and push my new image to the staging registry. There the code is tested and if everything fits I want to push the image to the production server. I also want to use versioning. I thought about incrementing the image tag.( Not just ass latest) I deploy my application with docker stack and a compose file.
Now I want to ask about a best practice.
Example:
current image: xyz:0.1
new image: xyz:0.2
In my compose file I referance the image xyz:auto_lastest_version_here
I want to be able to see the version string and not just the latest Tag.
Is there already a mechanism or a way to reduce an update to docker pull - > pull the latest version available
stack deploy ... to update the specific container.
EDIT: I guess I can write a script where I extract the latest Tag from the images and refer the this by using an env var in my compose file. I just thought there might be an easier way or standard way provided by docker.
Image repository names and tags are just strings attached to a blob which is the actual image data. Docker and the docker registry don't really have a concept of the most recent version of a blob -- the ":latest" tag doesn't even have internal significance - it's just a string which is used by default when building, and there's nothing preventing you from tagging an older image as :latest.
Fortunately, you can tag an image with multiple tags, and that provides a reasonable solution. For me, I tag with a string identifying the version I want on my production server, like ":live" or ":production", in addition to tagging with the actual version number. Thus, lets say you have images myimage:1.0.0, myimage:1.0.1, and myimage:1.1.0, you could run:
docker tag "myimage:1.1.0" "myimage:production"
...to add a production tag to it. The stack file you deploy on the production server would always then refer to myimage:production.
The real advantage to that is that if users start complaining about problems when you switch to 1.1.0, you can simply tag myimage:1.0.1 as myimage:production and redeploy, and it would switch back to the older version.
Not sure if this is the "best" practice, but it's the one I use.
We have a base image with the tag latest. This base image is being used for bunch of applications. There might be some update on the base image like ( OS upgrade, ...).
Do we need to rebuild and redeploy all applications when there is a change in the base image? Or, since the tag is latest and the new base image also will be with the tag latest, it will be updating in the docker layer and will be taken care without a restart?
Kubernetes has an imagePullPolicy: setting to control this. The default is that a node will only pull an image if it doesn’t already have it, except that if the image is using the :latest tag, it will always pull the image.
If you have a base image and then some derived image FROM my/base:latest, the derived image will include a specific version of the base image as its lowermost layers. If you update the base image and don’t rebuild the derived images, they will still use the same version of the base image. So, if you update the base image, you need to rebuild all of the deployed images.
If you have a running pod of some form and it’s running a :latest tag and the actual image that tag points at changes, Kubernetes has no way of noticing that, so you need to manually delete pods to force it to recreate them. That’s bad. Best practice is to use some explicit non-latest version tag (a date stamp works fine) so that you can update the image in the deployment and Kubernetes will redeploy for you.
There are two levels to this question.
Docker
If you use something like FROM baseimage:latest, this exact image is pulled down on your first build. Docker caches layers on consecutive builds, so not only will it build from the same baseimage:latest, but it will also skip execution of the Dockerfile elements untill first changed/not-cached one. To make the build notice changes to your baseimage you need to run docker pull baseimage:latest prior to the build, so that next run uses new content under latest tag.
The same goes for versioned tags when they aggregate minor/patch versions like when you use baseimage:v1.2 but the software is updated from baseimage:v1.2.3 to v1.2.4, and by the same process content of v1.2.4 is published as v1.2. So be aware of how versioning for particular image is handled.
Kubernetes
When you use :latest to deploy to Kubernetes you usually have imagePullPolicy: Always set. Which as for Docker build above, means that the image is always pulled before run. This is far from ideal, and far from immutable. Depending on the moment of container restart you might end up with two pods running at the same time, both the same :latest image yet the :latest for both of them will mean different actual image underneath it.
Also, you can't really change image in Deployment from :latest to :latest cause that's no change obviously, meaning you're out of luck for triggering rolling update, unless you pass version in label or something.
The good practice is to version your images somehow and push updates to cluster with that version. That is how it's designed and intended to use in general. Some versioning schemas I used were :
semantic (ie. v1.2.7) : nice if your CI/CD tool supports it well, I used it in Concourse CI
git_sha : works in many cases but is problematic for rebuilds that are not triggered by code changes
branch-buildnum or branch-sha-buildnum : we use it quite a lot
that is not to say I completely do not use latest. In fact most of my builds are built as branch-num, but when they are released to production that are also tagged and pushed to registry as branch-latest (ie. for prod as master-latest), which is very helpful when you want to deploy fresh cluster with current production versions (default tag values in our helm charts are pointing to latest and are set to particular tag when released via CI)
I'm using docker cloud at the moment. I'm trying to figure out a development to production workflow using docker with docker compose to pull application code for multiple applications of the same type, but simply changing the repository each pulls from. I understand the concept of mounting a volume, but all the examples show the source code in the same repo with the dockerfile and docker compose file. example. I want the app code from this example to come from a remote, dynamic repo. Would I set an environment variable in the docker image? If so how?
Any example or link to a workflow example is appreciated.
If done right, the code "baked" into Docker images should be immutable and the only thing that should change at runtime is configurable parameters like environment variables (e.g. to set the port the app will listen on).
Ideally, you should bake your code into the image. Otherwise you're losing a lot of the benefit of using Docker in the first place.
The problem is..
.. your use case does not match with the best practice. You want an image without any code embedded in it, but rather fetched at each update. If you browse the docker hub you'll find many image named as service:version. That's one of the benefit of Docker, offering different versions of the same service. If you want to always get the most up-to-date code your workflow may have some down sides.
One solution could be
Webhooks, especially if your code is versionned on GH. Or any tools of continuous integration.
I want to base my container off centos:centos6. But for some reason, centos:centos6 is updated somehow on the registry. This results in possible unfavorable different images when being built on different machines at different time. Such change caused segmentation fault in our application recently.
Is there a way I can specify the exact version for the from declaration so the build shall be the same even when the container being built on different machines at different time?
No, you can't tell a Dockerfile to be pinned to a particular image / layer ID, you have to use a tag (and if you don't use a tag, the tag latest is assumed and used as the default.
If you're worried that a remote registry image will change, you should take a copy of the Dockerfile and build your own version of the image yourself. You can either host it on the Docker Hub under your account or run your own private registry.
That way, you're in complete control over what's in there and when it gets updated (i.e., if you need to pin a particular package to an old version).