I'm trying to use argo events to trigger a workflow where I push changes to a database, then I have to restart certain pods so that changes are taken into consideration. I know how to use argo to create kubernetes objects, but I don't know how I can use this to restart a pod from within a kubernetes object. Alternatively I can also launch a pod from within argo and its container would restart a docker container, is this possible? If so how?
You can a do zero downtime rolling update via argo rollouts.
A RollingUpdate slowly replaces the old version with the new version. As the new version comes up, the old version is scaled down in order to maintain the overall count of the application. This is the default strategy of the deployment object
Argo Rollouts also supports Canary and BlueGreen.
Related
My team is currently working on migrating a Discord chat bot to Kubernetes. We plan on using a StatefulSet for the main bot service, as each Shard (pod) should only have a single connection to the Gateway. Whenever a shard connects to said Gateway, it tells it its ID (in our case the pod's ordinal index) and how many shards we are running in total (the amount of replicas in the StatefulSet).
Having to tell the gateway the total number of shards means that in order to scale our StatefulSet up or down we'd have to stop all pods in that StatefulSet before starting new ones with the updated value.
How can I achieve that? Preferrably through configuration so I don't have to run a special command each time.
Try kubectl rollout restart sts <sts name> command. It'll restart the pods one by one in a RollingUpdate way.
Scale down the sts
kubectl scale --replicas=0 sts <sts name>
Scale up the sts
kubectl scale --replicas=<number of replicas> sts <sts name>
One way of doing this is,
Firstly get the YAML configuration of StatefulSets by running the below command and save it in a file:
kubectl get statefulset NAME -o yaml > sts.yaml
And then delete the StatefulSets by running the below command:
kubectl delete -f sts.yaml
And Finally, again create the StatefulSets by using the same configuration file which you got in the first step.
kubectl apply -f sts.yaml
I hope this answers your query to only delete the StatefulSets and to create the new StatefulSets as well.
Before any kubectl scale, since you need more control on your nodes, you might consider a kubectl drain first
When you are ready to put the node back into service, use kubectl uncordon, which will make the node schedulable again.
By draining the node where your pods are maanged, you would stop all pods, with the opportunity to scale the statefulset with the new value.
See also "How to Delete Pods from a Kubernetes Node" by Keilan Jackson
Start at least with kubectl cordon <nodename> to mark the node as unschedulable.
If your pods are controlled by a StatefulSet, first make sure that the pod that will be deleted can be safely deleted.
How you do this depends on the pod and your application’s tolerance for one of the stateful pods to become temporarily unavailable.
For example you might want to demote a MySQL or Redis writer to just a read-only slave, update and release application code to no longer reference the pod in question temporarily, or scale up the ReplicaSet first to handle the extra traffic that may be caused by one pod being unavailable.
Once this is done, delete the pod and wait for its replacement to appear on another node.
I am toying around with Kubernetes and have managed to deploy a statefull application (jenkins instance) to a single node.
It uses a PVC to make sure that I can persist my jenkins data (jobs, plugins etc).
Now I would like to experiment with failover.
My cluster has 2 digital ocean droplets.
Currently my jenkins pod is running on just one node.
When that goes down, Jenkins becomes unavailable.
I am now looking on how to accomplish failover in a sense that, when the jenkins pod goes down on my node, it will spin up on the other node. (so short downtime during this proces is ok).
Of course it has to use the same PVC, so that my data remains intact.
I believe, when reading, that a StatefulSet kan be used for this?
Any pointers are much appreciated!
Best regards
Digital Ocean's Kubernetes service only supports ReadWriteOnce access modes for PVCs (see here). This means the volume can only be attached to one node at a time.
I came across this blogpost which, while focused on Jenkins on Azure, has the same situation of only supporting ReadWriteOnce. The author states:
the drawback for me though lies in the fact that the access mode for Azure Disk persistent volumes is ReadWriteOnce. This means that an Azure disk can be attached to only one cluster node at a time. In the event of a node failure or update, it could take anywhere between 1-5 minutes for the Azure disk to get detached and attached to the next available node.
Note, Pod failure and node failures are different things. Since DO only supports ReadWriteOnce, there's no benefit to trying anything more sophisticated than what you have right now in terms of tolerance to node failure. Since it's ReadWriteOnce the volume will need to be unmounted from the failing node and re-mounted to the new node, and then a new Pod will get scheduled on the new node. Kubernetes will do this for you, and there's not much you can do to optimize it.
For Pod failure, you could use a Deployment since you want to read and write the same data, you don't want different PVs attached to the different replicas. There may be very limited benefit to this, you will have multiple replicas of the Pod all running on the same node, so it depends on how the Jenkins process scales and if it can support that type of scale horizontal out model while all writing to the same volume (as opposed to simply vertically scaling memory or CPU requests).
If you really want to achieve higher availability in the face of node and/or Pod failures, and the Jenkins workload you're deploying has a hard requirement on local volumes for persistent state, you will need to consider an alternative volume plugin like NFS, or moving to a different cloud provider like GKE.
Yes, you would use a Deployment or StatefulSet depending on the use case. For Jenkins, a StatefulSet would be appropriate. If the running pod becomes unavailable, the StatefulSet controller will see that and spawn a new one.
What you are describing is the default behaviour of Kubernetes for Pods that are managed by a controller, such as a Deployment.
You should deploy any application as a Deployment (or another controller) even if it consists just of a single Pod. You never really deploy Pods directly to Kubernetes. So, in this case, there's nothing special you need to do to get this behaviour.
When one of your nodes dies, the Pod dies too. This is detected by the Deployment controller, which creates a new Pod. This is in turn detected by the scheduler, which assigns the new Pod to a node. Since one of the nodes is down, it will assign the Pod to the other node that is still running. Once the Pod is assigned to this node, the kubelet of this node will run the container(s) of this Pod on this node.
Ok, let me try to anwser my own question here.
I think Amit Kumar Gupta came the closest to what I believe is going on here.
Since I am using a Deployment and my PVC in ReadWriteOnce, I am basically stuck with one pod, running jenkins, on one node.
weibelds answer made me realise that I was asking questions to about a concept that Kubernetes performs by default.
If my pod goes down (in my case i am shutting down a node on purpose by doing a hard power down to simulate a failure), the cluster (controller?) will detect this and spawn a new pod on another node.
All is fine so far, but then I noticed that my new pod as stuck in ContainerCreating state.
Running a describe on my new pod (the one in ContainerCreating state) showed this
Warning FailedAttachVolume 16m attachdetach-controller Multi-Attach error for volume "pvc-cb772fdb-492b-4ef5-a63e-4e483b8798fd" Volume is already used by pod(s) jenkins-deployment-6ddd796846-dgpnm
Warning FailedMount 70s (x7 over 14m) kubelet, cc-pool-bg6u Unable to mount volumes for pod "jenkins-deployment-6ddd796846-wjbkl_default(93747d74-b208-421c-afa4-8d467e717649)": timeout expired waiting for volumes to attach or mount for pod "default"/"jenkins-deployment-6ddd796846-wjbkl". list of unmounted volumes=[jenkins-home]. list of unattached volumes=[jenkins-home default-token-wd6p7]
Then it started to hit me, this makes sense.
It's a pitty, but it makes sense.
Since I did a hard power down on the node, the PV went down with it.
So now the controller tries to start a new pod, on a new node but it cant transfer the PV, since the one on the previous pod became unreachable.
As I read more on this, I read that DigitalOcean only supports ReadWriteOnce , which now leaves me wondering, how the hell can I achieve a simple failover for a stateful application on a Kubernetes Cluster on Digital Ocean that consists of just a couple of simple droplets?
I am setting up an application in kubernetes and want to restart the pod if the new image is pushed into docker hub.
I am not able to restart the pod if the new image is pushed into docker hub registry?
I have included
"imagePullPolicy": "Always"
"terminationGracePeriodSeconds": 30
in deployment.yaml file
How to pull the new image to pod without stopping the existing pod?
Even in plain Docker, you can never update a container to a newer image without deleting and recreating it. This is also impossible in Kubernetes: the list of containers in a pod spec “cannot currently be added or removed...cannot be updated”, which means you can never change the image on an existing Pod.
In Kubernetes, the best practice is to always use an explicit version tag (never latest or anything similar that’s expected to change) and to manage Pods with Deployments. When you have a new build, change your Deployment spec to have the new tag. Kubernetes will notice that the new Deployment is different from the old one, and will, in order:
Start a new Pod with the new image;
Wait for its health checks to pass; and
Delete the old Pod.
This results in a zero-downtime upgrade.
A pod launched is immutable, if you make a image change in a deployment it will change only when the new containers are launched and the old ones deleted.
Use deployments to launch your pods. Launch the following command to change the image:
kubectl set image deployment/DEPLOY_NAME container-name=image_path:version
The deploy will take care of killing the old pods and launching new ones.
I'm running more than one replicas of pods with kubernetes deployment
and I'd like to update the replicas to use updated configmap in a rolling way. same like rolling-update works.
So that kubernetes will terminate pod and start sending traffic to the newly updated pods one at a time until all pods will be updated.
Can I use rolling-update with deployment?
Applying a change to the Deployment object will trigger a rolling-update. From the docs:
A Deployment’s rollout is triggered if and only if the Deployment’s pod template (that is, .spec.template) is changed, for example if the labels or container images of the template are updated. Other updates, such as scaling the Deployment, do not trigger a rollout.
So if you want to trigger a rolling update to update your configmap I would suggest you update a metadata label. Perhaps a CONFIG_VER key.
To automatically perform a rolling update of deployment on configmap update, you can also use a tool that my team has built and opensourced: Reloader which we are also using in production clusters of our customers.
Reloader watches changes in ConfigMap and Secret and updates the associated Deployments, Deamonsets and Statefulsets, based on the configured update strategy.
I'm very new to kubernetes and all I want to do at this point is restart my cluster and have it run an updated dockerfile. I'm running kubernetes in google-cloud-platform by the way.
kubectl from version 1.15 should contain kubectl rollout restart
(according to this comment https://github.com/kubernetes/kubernetes/issues/33664#issuecomment-497242094)
You can use rolling update mechanism to update the service without outage which will update one pod at a time until the desired state match, and still your services are up and running. Of course we must have to update our containers inside the pod to protect our data and to get latest features out. Kubernetes makes it easy to roll out updates to your applications by modifying the deployments and managing them. It's major update time and we'll use easy way to tweak them.
Suppose you have front end, auth and back-end deployments and there is change into the auth or newer version, so you want to update auth deployment configuration file in which you could change its respective auth container image to newer version after building the new docker image and simply changing the image version in your .yaml file and apply as below
$ kubectl apply -f deployments/auth.yaml
Check that it succeed with the deployment describe command, you could see the rolling update strategy and figure out that right number of pods are always available. That uses the new replica set to ensure that we are running the latest version of auth container.
$ kubectl describe deployments auth
Once, the rolling update is complete, we can view the running pods for the auth service.
$ kubectl get pods
Check the time frame for which it is running. The new version of the auth pod has replaced the previous one. Once again check with the id of the new auth pod and verify. Updating deployment this way keeps us with a clean declarative approach to roll out changes to our application weather you have single or thousands of pods running.
You can use kubectl patch to trigger a redeploy for example adding a new label.
$ kubectl patch deployment your_deployment -p "{\"spec\": {\"template\": {\"metadata\": { \"labels\": { \"redeploy\": \"$(date +%s)\"}}}}}"
And now you should see a new ReplicaSet trying to deploy new pods for you!
https://www.kevinsimper.dk/posts/trigger-a-redeploy-in-kubernetes
(I also made it into a shortcut that can apply it to all deploys that match some sort of filter)