Consider this partial k8s deployment manifest for a basic rest application called myapp
spec:
replicas: 1
...
containers:
name: myapp
image: us.gcr.io/my-org/myapp.3
...
resources:
limits:
cpu: 1000m
memory: 1.5Gi
We store incremental builds in Google Container Registry (GCR) eg (myapp.1, myapp.2, myapp.3). In the current setup, our CI system (Jenkins) does the following:
Docker builds new image myapp.4 and uploads to GCR
Runs kubectl set image myapp.4 to update the deployment.
This works well for most deploys, but what about changes to the deployment manifest itself? For example, if we changed resource > cpu to 1500m, we'd now have to manually run kubectl apply. This step needs to be automated, which brings me to my point: instead of using kubectl set image couldn't the build system itself just run kubectl apply each time? When is it appropriate to use kubectl set image vs. kubectl apply in a CI/CD pipeline?
As long as the new image were provided, wouldn't kubectly apply handle both image updates AND other config changes? If so, what are the pros/cons against just kubcetl set image?
PS - our deploys are simple and mainly rely on single replica and 100% uptime is not necessarily required.
With a kubectl set image, you only patch the image deployed in your deployment. To patch the other values (CPU, memory, replicas,...) you can use other commands like path or set repiclas
The problem is that you loose the consistency with your original YAML definition. If your cluster crash and you want to recreate one, you won't have the exact same deployment because your YAML file will be outdated (the "patches" won't be known)
With kubectl apply you overwrite the existing control plane config and you set the exact content of your YAML. It's more consistent and it's a common practice when you are in GitOps mode.
Which one to use? All depends on what you need and what you want to achieve. I prefer the kubectl apply mode for its consistency and its replayability, but it's up to you!
Use whatever suits your case the best. Just remember that the CI will be the "source of truth" when it comes to where resources and such are applied. If you change it somewhere else and run the CI job, it will update it again.
Most CI engines have the ability to only trigger a certain job in case a file have been changed.
That way, you could patch the image in case the manifest have not changed and run a full apply in case the manifest have changed.
I usually use the image patch method personally, as I prefer to have my Kube files separated from my sourcecode, but as said, whatever fits your case the best!
Each time you make changes to your application code or Kubernetes configuration, you have two options to update your cluster: kubectl apply or kubectl set image.
You can use kubectl set to make changes to an object's image, resources (compute resource such as CPU and memory), or selector fields.
You can use kubectl apply to update a resource by applying a new or updated configuration
Related
I have a cluster to manage which has Pods running with Image-CachePolicy set to "IfNotPresent".
containers:
- name: test-app
image: myimages/test-app
imagePullPolicy: "IfNotPresent"
I am already familiar with the difference between IfNotPresent and Always.
In our case, if the Pod gets respawned, K8s will due to "IfNotPresent" load the image from some cache instead of fetching the image from docker-hub.
What we want to achieve is to find this place where kubectl is caching the image and delete the cache in order to make K8s re-fetching the image on the next time, when the Pod gets restarted.
We don't want to apply a new Policy with imagePullPolicy set to Always because this would already fetch the latest version of the Pod, which we don't want yet. Instead, it shall only fetch new newest image on the next restart of the pod.
Is there any way to do this? Where to find this caching-location?
If you imagined Kubernetes was running Docker, it'd do the equivalent of:
Look at docker images.
If the image isn't already there, docker pull it.
docker run the resulting image.
(imagePullPolicy: Always skips the first step and always pulls the image; Never skips the pull, and will fail if the image isn't already there.)
Each node will have its own copy of the images; where exactly depends on the specific container runtime in use. Each node in turn knows how to garbage collect images not currently in use, though old images will generally stick around until the local disk fills up (or until a tool like the cluster autoscaler deletes the node).
A general assumption in Kubernetes is that a given registry.example.com/name/image:tag string uniquely identifies an image. Your CI/CD system should cooperate with this, and assign a distinct tag to each image build (maybe based on the commit ID or a date stamp). If you do this, you shouldn't need to adjust imagePullPolicy: from its default or try to manually remove images from nodes.
Kubernetes itself doesn't cache the images, it is the underlying OCI runtime that does the caching.
If the k8s cluster is using docker, an image can be removed from a node with:
docker image rm <name_or_checksum>
Welcome to stackOverflow, related to where images are cached, as explained by #David Maze and #Matt above it depends on your OCI.
I would like to understand a little more about this statement
> We don't want to apply a new Policy with imagePullPolicy set to Always because this would already fetch the latest version of the Pod,
which we don't want yet. Instead, it shall only fetch new newest image
on the next restart of the pod.
When you mention "we don't want yet" does that mean your current image on that tag is not the right one yet. Is there a possibility to modify your CI/CD so that image tags (tag used in Kubernetes manifest) to be generated or created only based on a promotion flow (meaning only that at that time when you want that particular tag ready for Kubernetes clusters)? This is the normal way of handling images for Kubernetes uses.
The reason for my suggestion is, there could be several reasons why your pod can also get re-scheduled to other nodes (nodes without any cache and k8s can instruct OCI to refetch, as that node may not have this cache) at any point in time. Assuming pods lifetime might give some unexpected outcomes.
I would suggest stick to Kubernetes image policy and modify CI/CD process or promotion flows to handle which image to be ready for which Kubernetes environment.
If you still feel there is a need to modify and play with image caches, explain more on the actual use case, there could be other alternatives.
Dears,
I am very new to Kubernetes and I'm currently working on the update process of my services (traefik, prometheus, ...). I want to avoid the compulsive real-time updates that may lead to bugs or crash. I am used to keep the control about what need to be updated and what does not.
So far, I understood that Kubernetes provides the field spec.updateStrategy.type with 2 possible values:
RollingUpdate: permanent auto-update
OnDelete: auto-update after the manual deletion of a pod
I am surprised to not find the same steps than with apt Debian tool: when I use apt update; apt upgrade, I get a list of what will be updated and I choose what I want to be updated.
When I came to Kubernetes, I imagined updates would allow to keep this two-steps spirit, something like :
Execute a command to compare the current docker images that are deployed on the cluster to the repos. This command would print the new existing version of each images.
Execute another command to choose what will be updated.
There is no stable, unstable, testing channels like Linux repositories with docker, then I have no way to make difference between the testing update and the trustworthy updates. I am affraid that RollingUpdate would deploy each new image without distinction.
Which lead to my main question: is it completely safe to blindly trust RollingUpdate ?
It is safe to trust the RollingUpdate StatefulSet update strategy. However, it is important to note that this is not an "auto-update".
A StatefulSet (and also a Deployment) has a template Pod spec that has an image:. That tends to name a fairly specific version of the image to run. In general Kubernetes will check to see if an image with that name and tag is already on the node, and if so, tries to run it without updating it (see Updating images, which discusses the imagePullPolicy: field).
So if your StatefulSet says image: prom/prometheus:2.26.0, for example, your Pods will use exactly that version of Prometheus, and no other. If you just say image: prom/prometheus with no tag (or with ...:latest) then each node will pull whatever version happens to be current at the time the Pod starts up, and you can easily wind up with out-of-sync versions.
The place the update strategy gets used, then, is when you change image: prom/prometheus:2.27.0. The image in a Pod spec can't be modified, so the existing pods need to be deleted and recreated. If you have updateStrategy: { type: RollingUpdate } (the default) then the StatefulSet controller will do this for you, specifically deleting and recreating the pods in order. If you have it set to OnDelete then you need to manually delete the pods.
Since you as the administrator control the image: then you can pick exactly what version is getting used; there is no such thing as a "stable update channel" that will automatically update. Correspondingly, you can deploy the updated StatefulSet YAML configuration in a pre-production environment to test out before you promote it to production.
I'm attempting to deploy a parallelized data processing task that uses many containers of the same docker image which would each run with different incrementing environment variables. The image is setup to read env vars in order to determine which segment of a larger list to process.
Background: I was originally using a bash script that passed down an incrementing env var to docker run commands but now I'd like a better way to manage/monitor all the containers. I've only had experience using Kubernetes for application services but it seems like it may be a better way to orchestrate my multi-container task as well.
Wondering if this sort of dynamic environment variable passing is possible within the Kubernetes YAML configs as I'd prefer declarative config vs a shell script. I'm also not sure of the best approach to do this in Kubernetes whether it's multiple separate pods, multi-container pod, or to use replicas in some way.
I'm open to suggestions, I know other tools like Terraform may be helpful for this sort of programatic infrastructure also.
What about using Parallel Processing Using Work Queue for passing different environment variables to your k8s job pods with .spec.parallelism. Although having a separate service for work queue may be little too much depending on what you are trying to do.
The other idea can be using helm templating power to create a k8s manifest file. I create a sample helm chart to give an idea of templating parallel processing. See git repo - helm-parallel-jobs. Once you have the git repo cloned, you can install the helm chart for parallel processing like this. The template for job is same as used by k8s documentation. As seen in the output below, three different environment variables - apple,banana,cherry are provided, which creates 3 different pods with environment variables passed to them.
[root#jr]# helm install --set envs='{apple,banana,cherry}' --name jobs ./helm-parallel-jobs/example/parallel-jobs
NAME: jobs
LAST DEPLOYED: Sun Aug 26 16:29:23 2018
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/Job
NAME DESIRED SUCCESSFUL AGE
process-item-apple 1 0 0s
process-item-banana 1 0 0s
process-item-cherry 1 0 0s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
process-item-apple-dr6st 0/1 ContainerCreating 0 0s
process-item-banana-d2wwq 0/1 ContainerCreating 0 0s
process-item-cherry-wvlxz 0/1 ContainerCreating 0 0s
My understanding is you'd like to do something like https://kubernetes.io/docs/tasks/job/parallel-processing-expansion/ where jobs are created from a template, one for each data item in a list. But you don't want it to be shell-scripted.
I imagine helm could be used to replace the Job and it has a range function so a chart could be set up to create jobs for each entry in a section of a values.yaml. So it could occupy a space similar to what you suggested for terraform. Ansible could also be an option.
However, the direction of travel of this question seems to be towards batch scheduling. I am wondering if your jobs will evolve to end up having dependencies between them etc. If so Helm and Kubernetes: Is there a barrier equivalent for jobs? and https://www.quora.com/Is-Kubernetes-suited-for-long-running-batch-jobs help here. Currently Kubernetes has facilities for running batch jobs and the tooling to enable a batch scheduling system to run or be built on it but it doesn't itself contain an out of the box batch scheduling system. So people are currently using a range of different approaches to suit their needs.
I am quite familiar with Docker, but I have zero experience on Kubernetes.
I have a Docker stack (multi-container) software that I can deploy in a Docker swarm cluster. I was wondering if Kubernetes has something similar? I don't need replicas, auto scaling and so on... I just need a group of containers working together with its dependencies and networks defined in single text file.
I have searched and found a tool called kompose that translates the Docker stack file to Kubernetes syntax... However, it looks like the output is a list of *.yaml files, instead of a single file.
So, I came to the conclusion that kubernetes does not have this exact functionality.. Am I missing something?
You can copy the content of the generated files into one file and separate them with ---.
For instance, if you've got 3 Kubernetes files: service.yml, deployment.yml and configmap.yml, your file should look something like:
# content of service.yml
....
---
# content of deployment.yml
....
---
# content of configmap.yml
....
You would use the same kubectl commands to CRUD using this spec file.
A single 'Docker Stack' yml definition is equivalent to a collection of Kubernetes Deployments and Services. Each service in a Docker Stack definition is also available to one another via a default overlay network automatically created by docker at deploy time. To simulate this in Kubernetes you would need to define multiple deployments/services with-in the same file so that they could be created and deleted as a single 'stack'.
I want to set up a Gitlab CD to Kubernetes and I read this article
However, I am wondering, how is it that my K8 cluster would be updated with my latest Docker images?
For example, in my .gitlab-ci.yaml file I will have a build, test, and release stage that ultimately updates my cloud Docker images. By setting up the deploy stage as instructed in the article:
deploy:
stage: deploy
image: redspreadapps/gitlabci
script:
- null-script
would Spread then know to "magically" update my K8 cluster (perhaps by repulling all images, perform rolling-updates) as long as I set up my directory structure of K8 resources as is specified by Spread?
I don't have a direct answer, but from looking at the spread project it seems pretty dead. Last commit in Aug last year with a bunch of issues and not supporting any of the newer kubernetes constructs (e.g. deployments).
The typical way to update images in kubernetes nowadays is to run a command like kubectl set image <deployment-name> <image>. This will in turn perform a rolling update on the deployment and shutting down a POD at a time updating it with the new image. See this doc.
Since spread is from before that, I assume they must use rolling update replication controller with a command like kubectl rolling-update NAME -f FILE and picking up the new image from the configuration file in their project folder (assuming it changed). See this doc.