Using Kubernetes for deployment:
Considering I have a Dockerfile, I build, then push to registry.
If I run a container on a host, the image gets pulled and the container is ran.
Now, if i update the Dockerfile, build and push again, without changing its tag then the image is changed in the registry, but the host has the image pulled, and it doesn't seem to go look for updates.
How do i force a pull to get the latest image when running a container?
I can manually pull the image, but I'd like to know if there is a 'formal way' of doing this (in the pod or rc templates?)
Thanks for insight.
Set an imagePullPolicy of Always on the container
Related
I have a Docker build environment where I build containers locally and test them. When I'm done, I push them to our Dev GitLab container registry to be deployed to Kubernetes.
I've run into a situation where either Docker isn't pushing up the newest layers or GitLab is seeing layers from a previous version and just mounting that layer so when the container is deployed in Kubernetes, the container, despite the new tag, is running the old container image.
I've tried completely wiping my Docker image repository, rebuilding, and repushing and that didn't fix it. I tried using the red trash icon in GitLab to delete the old version of the tag I'm trying to use.
I added some echo's in the console output for the container so I know the new bits aren't being run but I can't figure out if the problem is Docker or GitLab or how to fix it. Anyone have any ideas?
TIA!
Disregard -- my workers had a docker image on them and because my imagePullPolicy in my YAML was not set, it was defaulting to using the cached image in Docker. I just had to clear the image on my worker node (docker rmi -f <image name>) and then I updated my deployment YAML to use an imagePullPolicy: Always.
I am running two VMs. One VM is used for running nextflow, on the other VM there is a Jenkins build server. Jenkins is responsible for building new Docker images and pushing new Docker images to our private google container registry.
My nextflow.config file looks something like this:
process {
withLabel: awesome_image {
container = "eu.gcr.io/best-project-1234/coolest_os:latest"
}
}
After building a new image using the Jenkins server I was running a new nextflow script and I noticed that nextflow was still using the old image. After some research (https://stackoverflow.com/a/58539792/1820480), I realized that this has to do with the fact that I am using the latest tag, and since there is already an image called latest on the nextflow VM, nextflow uses that one and does not bother checking the registry.
Question: How can I ensure that before every run of nextflow, it checks the registry for newer images? Or, is there a script/program that I can run on the VM that checks the registry (instead of nextflow)?
Thank you.
Nextflow just runs your command(s) in a container using docker run. If you specify an image that you haven't pulled yet, docker run will first do a docker pull to download/localize the image. To check the registry again for newer images, you'll just need to make sure you call docker pull (for each image) before running Nextflow. If you want to instead check the registry for newer images each time a process is spawned, please see below.
After some research, it looks like the latest Docker cli (v20.10.0) now has a flag to modify the pull behavior when running containers:
--pull string Pull image before running ("always"|"missing"|"never") (default "missing")
This is nice because it means it should now be possible pass this through in your nextflow.config:
docker {
enabled = true
runOptions = '--pull=always'
}
But this will have the overhead of doing a docker pull for each process spawned and, depending on when new images are pushed to your registry, may mean some processes get different containers during your workflow execution. This may not be a concern though if you only need the 'latest' containers and do not care for reproducibility.
I load an image (.tar) into local docker repository and then Kubernetes pulls that Image into a container. After the application is up and running I want to remove that image from local docker repository, is that right way or Kubernetes still needs that image inside docker local repository?
I have tested when deleting that image, my Application still works, but if I want to scale up or scale down, I got a problem because the image is missing.
I presume, if the Pod goes down, Kubernetes will seek for Image again in docker repository?
Is there any way to cache that Image inside Kubernetes, so no need to pull again from docker repository?
The reason why I want to delete these images right after Application run on Kubernetes is security, I don't want to leave my images inside docker repository so a user can't extract that image and export it somewhere...
I presume, if the Pod goes down, Kubernetes will seek for Image again in docker repository?
Yes.
Is there any way to cache that Image inside Kubernetes, so no need to pull again from docker repository?
When you pull an image it's cached locally by docker or your runtime manager.
The reason why I want to delete these images right after Application run on Kubernetes is security, I don't want to leave my images inside docker repository so a user can't extract that image and export it somewhere...
If you are concerned about security you shouldn't put sensitive information in the image (passwords, credentials, keys). That's why there are Kubernetes Secrets or tools like Hashicorp Vault.
If you are concerned about keeping intellectual property private you should consider using a private container image repository or private Docker registry
I have 2 machines(separate hosts) running docker and I am using the same image on both the machines. How do I keep both the images in sync. For eg. suppose I make changes to the image in one of the hosts and want the changes to reflect in the other host as well. I can commit the image and copy the image over to the other host. Is there any other efficient way of doing this??
Some ways I can think of:
1. with a Docker registry
the workflow here is:
HOST A: docker commit, docker push
HOST B: docker pull
2. by saving the image to a .tar file
the workflow here is:
HOST A: docker save
HOST B: docker load
3. with a Dockerfile and by building the image again
the workflow here is:
provide a Dockerfile together with your code / files required
everytime your code has changed and you want to make a release, use docker build to create a new image.
from the hosts that you want to take the update, you will have to get the updated source code (maybe by using a version control software like Git), and then docker build the image
4. CI/CD pipeline
you can see a video here: docker.com/use-cases/cicd
Keep in mind that containers are considered to be ephemeral. This means that updating an image inside another host will then require:
to stop and remove any old container (running with the outdated image)
to run a new one (with the updated image)
I quote from: Best practices for writing Dockerfiles
General guidelines and recommendations
Containers should be ephemeral
The container produced by the image your Dockerfile defines should be as ephemeral as possible. By “ephemeral,” we mean that it can be stopped and destroyed and a new one built and put in place with an absolute minimum of set-up and configuration.
You can perform docker push to upload you image to docker registry and perform a docker pull to get the latest image from another host.
For more information please look at this
Does Kubernetes download Docker image automatically when i create a pod or should I use Docker pull manually to Download the image locally?
You do not need to run docker pull manually. The pod definition contains the image name to pull and Kubernetes will pull the image for you. You have several options in terms of defining how Kubernetes will decide to pull the image, using the imagePullPolicy: definition in your pod spec. Much of this is documented here, but basically you can pull if the image is not present, pull always, never update (once the image is local). Hopefully that doc can get you started.