I have a local docker image that was pushed to private Azure Container Registry. Then in Azure Kubernetes Service I have a cluster where I am using this image - from ACR.
Now I wanted to update the image (realised that I needed to install zip and unzip). I started a local container, made changes, committed them and pushed the new image to ACR. Unfortunately, that`s not enough. My pods are still using the previous version of the image, without zip.
Bit more details and what I tried:
Inside the helm chart I am using "latest" tag;
Compared the digest sha of my local "latest" image and what I have in ACR - they are the same;
Started the "latest" container locally (docker run -it --rm -p 8080:80 My-REPO.azurecr.io/MY-IMAGE:latest) - it has zip installed
Deleted existing pods in kubernetes; newly created ones are still missing zip
Deleted the release and recreated it - still nothing.
I am pushing to ACR using docker push MY-REPO.azurecr.io/MY-IMAGE:latest
So my question is - what am I missing? How to properly update this setup?
You should be looking for a setup like this:
Your Docker images have some unique tag, not latest; a date stamp will generally work fine.
Your Helm chart should take the tag as a parameter in the values.yaml file.
You should use a Kubernetes Deployment (not a bare Pod); in its pod spec part specify the image as something like image: MY-REPO.azurecr.io/MY-IMAGE:{{ .Values.tag }}.
When you have a new build, you can helm update --set tag=20190214; this will push an updated Deployment spec to Kubernetes; and that that will cause it to create new Pods with the new image and then destroy the old Pods with the old image.
The essential problem you're running into is that some textual difference in the YAML file is important to make Kubernetes take some action. If it already has MY-IMAGE:latest, and you try to kubectl apply or equivalent the same pod or deployment spec with exactly the same image string, it will decide that nothing has changed and it doesn't need to do anything. Similarly, when you delete and recreate the pod, the node decides it already has a MY-IMAGE:latest image and doesn't need to go off and pull anything; it just reuses the same (outdated) image it already has.
Some best practices related to the workflow you describe:
Don't use a ...:latest image tag (or any other fixed string); instead, use some unique value like a timestamp, source control commit ID, or release version, where every time you do a deployment you'll have a different tag.
Don't use bare pods; use a higher-level controller instead, most often a Deployment.
Don't use docker commit ever. (If your image crashed in production, how would you explain "oh, I changed some stuff by hand, overwrote the image production is using, and forcibly restarted everything, but I have no record of what I actually did"?) Set up a Dockerfile, check it into source control, and use docker build to make images. (Better still, set up a CI system to build them for you whenever you check in.)
Related
I want all running containers on my server to always use the latest version of an official base image e.g. node:16.3 in order to get security updates. To achieve that I have implemented an image update mechanism for all container images in my registry using a CI workflow which has some limitations described below.
I have read the answers to this question but they either involve building or inspecting images on the target server which I would like to avoid.
I am wondering whether there might be an easier way to achieve the container image updates or to alleviate some of the caveats I have encountered.
Current Image Update Mechanism
I build my container images using the FROM directive with the minor version I want to use:
FROM node:16.13
COPY . .
This image is pushed to a registry as my-app:1.0.
To check for changes in the node:16.3 image compared to when I built the my-app:1.0 image I periodically compare the SHA256 digests of the layers of the node:16.3 with those of the first n=(number of layers of node:16.3) layers of my-app:1.0 as suggested in this answer. I retrieve the SHA256 digests with docker manifest inpect <image>:<tag> -v.
If they differ I rebuild my-app:1.0 and push it to my registry thus ensuring that my-app:1.0 always uses the latest node:16.3 base image.
I keep the running containers on my server up to date by periodically running docker pull my-app:1.0 on the server using a cron job.
Limitations
When I check for updates I need to download the manifests for all my container images and their base images. For images hosted on Docker Hub this unfortunately counts against the download rate limit.
Since I always update the same image my-app:1.0 it is hard to track which version is currently running on the server. This information is especially important when the update process breaks a service. I keep track of the updates by logging the output of the docker pull command from the cron job.
To be able to revert the container image on the server I have to keep previous versions of the my-app:1.0 images as well. I do that by pushing incremental patch version tags along with the my-app:1.0 tag to my registry e.g. my-app:1.0.1, my-app:1.0.2, ...
Because of the way the layers of the base image and the app image are compared it is not possible to detect a change in the base image where only the uppermost layers have been removed. However I do not expect this to happen very frequently.
Thank you for your help!
There are a couple of things I'd do to simplify this.
docker pull already does essentially the sequence you describe, of downloading the image's manifest and then downloading layers you don't already have. If you docker build a new image with an identical base image, an identical Dockerfile, and identical COPY source files, then it won't actually produce a new image, just put a new name on the existing image ID. So it's possible to unconditionally docker build --pull images on a schedule, and it won't really use additional space. (It could cause more redeploys if neither the base image nor the application changes.)
[...] this unfortunately counts against the download rate limit.
There's not a lot you can do about that beyond running your own mirror of Docker Hub or ensuring your CI system has a Docker Hub login.
Since I always update the same image my-app:1.0 it is hard to track which version is currently running on the server. [...] To be able to revert the container image on the server [...]
I'd recommend always using a unique image tag per build. A sequential build ID as you have now works, date stamps or source-control commit IDs are usually easy to come up with as well. When you go to deploy, always use the full image tag, not the abbreviated one.
docker pull registry.example.com/my-app:1.0.5
docker stop my-app
docker rm my-app
docker run -d ... registry.example.com/my-app:1.0.5
docker rmi registry.example.com/my-app:1.0.4
Now you're absolutely sure which build your server is running, and it's easy to revert should you need to.
(If you're using Kubernetes as your deployment environment, this is especially important. Changing the text value of a Deployment object's image: field triggers Kubernetes's rolling-update mechanism. That approach is much easier than trying to ensure that every node has the same version of a shared tag.)
i have simple docker-copose.yml which builds 4 containers. The containers run's on EC2.
docker-compose change ~ twice a day on master branch, and each change we need to deploy the new containers on production
this is what i'm doing:
docker-compose down --rmi all
git pull origin master
docker-compose build -d
i'm removing images to avoid conflicts so that once i'm starting the service i have fresh images
This process takes me around ~ 1 minutes,
what is the best practice to spin up docker-compose, any suggestion to improve this ?
You can do the set of commands you show natively in Docker, without using git or another source-control tool as part of the deployment process.
Whenever you have a change to your source tree, build a new Docker image and push it to a Docker repository. This can be Docker Hub, or if you're on AWS already, Amazon ECR. Each build should have a unique image tag, such as a source control commit ID or a time stamp. You can set up a continuous-integration tool to do all of this for you automatically.
Once you have this, your docker-compose.yml file needs to be updated with the version number to deploy. If you only have a single image you're deploying, you can straightforwardly use Compose variable substitution to fill it in
image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/myimage:${TAG:-latest}
If you have multiple images you can set multiple environment variables or produce an updated docker-compose.yml file with the values filled in, but you will need to know all of the image versions together at deployment time.
Now when you go to deploy it you only need to run
TAG=20200317.0412 docker-compose up -d
to set the environment variable and trigger Compose. Compose will see that the image you're trying to run for that container is different from what's already running, pull the updated image, and replace the container for you. You don't need to manually remove the old containers or stop the entire stack.
If git is part of your workflow now, it's probably because you're mounting application code into your container. You will also need to delete any volumes: that overwrite the content in the image. Also make sure you make this change in your CI system (so you're testing the actual image you're deploying to production) and in development (similarly).
This particular task becomes slightly easier with a cluster-management system like Kubernetes (or Amazon EKS), though it brings many other complexities elsewhere. In Kubernetes you need to send an updated Deployment spec to the Kubernetes API server, but you can do this without direct ssh access to the target system and only needing to know the specific version of the one image you're updating, and with multiple replicas you can get a zero-downtime upgrade. Both using a Docker repository and using a unique image tag per build are basically required in this setup: images are the only way code gets into the cluster, and changing the image tag string is what triggers code to be redeployed.
If I have one ubuntu container and I ssh to it and make one file after the container is destroyed or I reboot the container the new file was destroyed because the kubernetes load the ubuntu image that does not contain my changes.
My question is what should I do to save any changes?
I know it can be done because some cloud provider do that.
For example:
ssh ubuntu#POD_IP
mkdir new_file
ls
new_file
reboot
after reboot I have
ssh ubuntu#POD_IP
ls
ls shows nothing
But I want to it save my current state.
And I want to do it automatically.
If I use docker commit I can not control my images because it makes hundreds of images. because I should make images by every changes.
If I want to use storage I should mount /. but kubernetes does not allow me to mount /. and it gives me this error
Error: Error response from daemon: invalid volume specification: '/var/lib/kubelet/pods/26c39eeb-85d7-11e9-933c-7c8bca006fec/volumes/kubernetes.io~rbd/pvc-d66d9039-853d-11e9-8aa3-7c8bca006fec:/': invalid mount config for type "bind": invalid specification: destination can't be '/'
You can try to use docker commit but you will need to ensure that your Kubernetes cluster is picking up the latest image that you committed -
docker commit [OPTIONS] CONTAINER [REPOSITORY[:TAG]]
This is going to create a new image out of your container which you can feed to Kubernetes.
Ref - https://docs.docker.com/engine/reference/commandline/commit/
Update 1 -
In case you want to do it automatically, you might need to store the changed state or the files at a centralized file system like NFS etc & then mount it to all running containers whenever required with the relevant permissions.
K8s ref - https://kubernetes.io/docs/concepts/storage/persistent-volumes/
Docker and Kubernetes don't work this way. Never run docker commit. Usually you have very little need for an ssh daemon in a container/pod and you need to do special work to make both the sshd and the main process both run (and extra work to make the sshd actually be secure); your containers will be simpler and safer if you just remove these.
The usual process involves a technique known as immutable infrastructure. You never change code in an existing container; instead, you change a recipe to build a container, and tell the cluster manager that you want an update, and it will tear down and rebuild everything from scratch. To make changes in an application running in a Kubernetes pod, you typically:
Make and test your code change, locally, with no Docker or Kubernetes involved at all.
docker build a new image incorporating your code change. It should have a unique tag, often a date stamp or a source control commit ID.
(optional but recommended) docker run that image locally and run integration tests.
docker push the image to a registry.
Change the image tag in your Kubernetes deployment spec and kubectl apply (or helm upgrade) it.
Often you'll have an automated continuous integration system do steps 2-4, and a continuous deployment system do the last step; you just need to commit and push your tested change.
Note that when you docker run the image locally in step 3, you are running the exact same image your production Kubernetes system will run. Resist the temptation to mount your local source tree into it and try to do development there! If a test fails at this point, reduce it to the simplest failing case, write a unit test for it, and fix it in your local tree. Rebuilding an image shouldn't be especially expensive.
Your question hints at the unmodified ubuntu image. Beyond some very early "hello world" type experimentation, there's pretty much no reason to use this anywhere other than the FROM line of a Dockerfile. If you haven't yet, you should work through the official Docker tutorial on building and running custom images, which will be applicable to any clustering system. (Skip all of the later tutorials that cover Docker Swarm, if you've already settled on Kubernetes as an orchestrator.)
We have a base image with the tag latest. This base image is being used for bunch of applications. There might be some update on the base image like ( OS upgrade, ...).
Do we need to rebuild and redeploy all applications when there is a change in the base image? Or, since the tag is latest and the new base image also will be with the tag latest, it will be updating in the docker layer and will be taken care without a restart?
Kubernetes has an imagePullPolicy: setting to control this. The default is that a node will only pull an image if it doesn’t already have it, except that if the image is using the :latest tag, it will always pull the image.
If you have a base image and then some derived image FROM my/base:latest, the derived image will include a specific version of the base image as its lowermost layers. If you update the base image and don’t rebuild the derived images, they will still use the same version of the base image. So, if you update the base image, you need to rebuild all of the deployed images.
If you have a running pod of some form and it’s running a :latest tag and the actual image that tag points at changes, Kubernetes has no way of noticing that, so you need to manually delete pods to force it to recreate them. That’s bad. Best practice is to use some explicit non-latest version tag (a date stamp works fine) so that you can update the image in the deployment and Kubernetes will redeploy for you.
There are two levels to this question.
Docker
If you use something like FROM baseimage:latest, this exact image is pulled down on your first build. Docker caches layers on consecutive builds, so not only will it build from the same baseimage:latest, but it will also skip execution of the Dockerfile elements untill first changed/not-cached one. To make the build notice changes to your baseimage you need to run docker pull baseimage:latest prior to the build, so that next run uses new content under latest tag.
The same goes for versioned tags when they aggregate minor/patch versions like when you use baseimage:v1.2 but the software is updated from baseimage:v1.2.3 to v1.2.4, and by the same process content of v1.2.4 is published as v1.2. So be aware of how versioning for particular image is handled.
Kubernetes
When you use :latest to deploy to Kubernetes you usually have imagePullPolicy: Always set. Which as for Docker build above, means that the image is always pulled before run. This is far from ideal, and far from immutable. Depending on the moment of container restart you might end up with two pods running at the same time, both the same :latest image yet the :latest for both of them will mean different actual image underneath it.
Also, you can't really change image in Deployment from :latest to :latest cause that's no change obviously, meaning you're out of luck for triggering rolling update, unless you pass version in label or something.
The good practice is to version your images somehow and push updates to cluster with that version. That is how it's designed and intended to use in general. Some versioning schemas I used were :
semantic (ie. v1.2.7) : nice if your CI/CD tool supports it well, I used it in Concourse CI
git_sha : works in many cases but is problematic for rebuilds that are not triggered by code changes
branch-buildnum or branch-sha-buildnum : we use it quite a lot
that is not to say I completely do not use latest. In fact most of my builds are built as branch-num, but when they are released to production that are also tagged and pushed to registry as branch-latest (ie. for prod as master-latest), which is very helpful when you want to deploy fresh cluster with current production versions (default tag values in our helm charts are pointing to latest and are set to particular tag when released via CI)
I want to set up a Gitlab CD to Kubernetes and I read this article
However, I am wondering, how is it that my K8 cluster would be updated with my latest Docker images?
For example, in my .gitlab-ci.yaml file I will have a build, test, and release stage that ultimately updates my cloud Docker images. By setting up the deploy stage as instructed in the article:
deploy:
stage: deploy
image: redspreadapps/gitlabci
script:
- null-script
would Spread then know to "magically" update my K8 cluster (perhaps by repulling all images, perform rolling-updates) as long as I set up my directory structure of K8 resources as is specified by Spread?
I don't have a direct answer, but from looking at the spread project it seems pretty dead. Last commit in Aug last year with a bunch of issues and not supporting any of the newer kubernetes constructs (e.g. deployments).
The typical way to update images in kubernetes nowadays is to run a command like kubectl set image <deployment-name> <image>. This will in turn perform a rolling update on the deployment and shutting down a POD at a time updating it with the new image. See this doc.
Since spread is from before that, I assume they must use rolling update replication controller with a command like kubectl rolling-update NAME -f FILE and picking up the new image from the configuration file in their project folder (assuming it changed). See this doc.