my Kubernetes setup:
v1.16.2 on bare metal
1 master node: used for Jenkins Master + Docker registry
5 slave nodes: used for Jenkins JNPL slaves
I use kubernetes-plugin to run slave docker agents. All slave k8 nodes labeled as "jenkins=slave". When I use nodeSelector ("jenkins=slave") for podTemplate, kubernetes always schedule new pod on same node regardless the amount of started Jenkins jobs.
Please give me advice, how I can configure kubernetes or kubernetes-plugin to schedule each next build by round-robin (across all labeled nodes in kubernetes cluster)
Thank you.
This is generally handled by the inter-pod anti affinity configuration https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity. You would set this in the pod template for your builder deployment. That said, it's more common to use the Kubernetes plugin for Jenkins which runs each build as a temporary pod, rather than having long-lived JNLP builders.
Related
I want to run my Jenkins behind k8s. We can achieve that with any standard helm chart or our own manifest files. In this case, Jenkins (master only) will run inside a container (Pod).
Now I also want to have a pipeline job that uses docker agent as described here
I am getting confused, about
how and where this docker container will be run (on the same node where Jenkins is running? and suppose the node capacity is over then it needs to run docker agent on a different node)
how does Jenkins will authenticate to run containers on k8s nodes?
I saw the Kubernetes plugin/docker plugin. But those plugins create containers beforehand (or at least we need to set up a template, which decides how containers will start, which image will be used and many more) and connects Jenkins with help of JNLP / ssh. I lose the flexibility to have an image as an agent in that case.
going further, I also like to build custom images on the fly with help of Dockerfile shipped along with code. An example is available in the same link.
I believe this documentation is answering all of your questions: https://devopscube.com/jenkins-build-agents-kubernetes/
With this method, you are not losing your flexibility because your Jenkins master going to create a K8s pod on the fly. Yes, additionally you need JNLP authentication but you can think of that as a sidecar container.
About your first question: If you use exactly that way, your Jenkins jobs going to run under Jenkins master with the same Docker that your Jenkins Master is using.
I would like to setup a Kubernetes cluster as follows:
Kubernetes will be installed on top of VMs depicted in pink.
I am going to use statefulsets or replicasets to deploy Jenkins master and Jenkins executors. I would like that the workspace folder on the Jenkins master to be always in sync on all replicas in eventuality of losing any worker VMs or server.
Can be achieved using internal mechanisms of replicasets or statefulsets or is any other way of keeping the workspace in sync?
Thank you,
Albert
You can't just assume that statefulset will do the job for you. You can configure a NFS server and point the PV to it and bind your PVC to this PV and your STS can point to your PVC. So, basically
STS -> PVC -> PV -> NFS Server
So, even if one worker node goes down, it won't impact the others.
I configured Jenkins on a K8s cluster and setup Jenkins build pipeline. Once build execute It creates the jenkins-slave pod and after the build, the pod will terminate.
The use case is basically if all my workers in Jenkins goes full, I want to auto-scale (increase the number of slaves) if it comes back down, I would like to reduce the slaves count.
Is it possible? and How I can do it from k8s.
I set up the kubernetes on EC2 instances over ubuntu platform, everything working fine. But now I have deployed a Jenkins pod, after that, I am able to access the Jenkins in the browser. Now my question is how can I connect Jenkins with my machine.
My Jenkins pod IP:- 10.43.0.8
My Kubernetes Master private IP:- 192.168.105.229
I am able to ping with each other. But how I can access my master machine using Jenkins, so I can create the pods through Jenkins.
There are a few ways to do this. A couple that I can think of:
Use the Jenkins Kubernetes Plugin. If you install this on your Jenkins master (which is also running on a pod) and also if you configure it to talk to the same Kubernetes cluster (meaning the kube-apiserver) then you can create/remove pods using the plugin. More on how to configure authentication and RBAC in Kubernetes
Manually configure Jenkins slaves running as 'user'. Then on the 'user's some directory create a ~/.kube/config that has the configuration to talk to your Kubernetes cluster. From there you can just issue kubectl commands to create/remove pods. You would still need to configure authentication/RBAC.
If you want to access a pod running in Kubernetes you need to expose it as a service and possibly an ingress resource
https://kubernetes.io/docs/concepts/services-networking/service/
I have set up a kubernetes cluster and is working fine.
As of now, Kubernetes cluster deploying container in the master as well. I dont want this to be happened . Can anybody let me know, how to prevent to deploy container in the master ?
If you want no Pods to be scheduled on the master you will need to taint the master node:
kubectl taint nodes nameofmaster dedicated=master:NoSchedule
Read up on taints and tolerations to understand the consequences and how to schedule specific Pods on the now tainted master node. To undo use kubectl taint nodes nameofmaster dedicated-