I’d like to set Jenkins in fault tolerant mode, so if one of masters is down, second master should pick up failed job and re-trigger it.
I don’t have to care about slaves because slaves are on-demand Docker containers.
Any ideas about how it could be done?
Any ideas about how it (Jenkins HA) could be done?
CloudBees provides a commercial Jenkins HA solution.
Related
I am running jenkins multi branch job, suddenly it not allow me to change the configuration changes, its keep on loading without any timeout issue.
Can you please some one help me on this ?
You could have a look at the Jenkins master machine CPU and memory. Look what is consuming them. I have seen this happening when the CPU is nearly 100 %. In this case, restarting the Jenkins process or Jenkins master machine could help.
Try to remember/ask colleagues if there are any recent changes to Jenkins master machine. We had similar issues after installing plugins.
Avoid executing jobs on Jenkins master, use slave agents.
You may need to clean up old builds if you are not doing this already.
in my case, after disabling / enabling all plugins one by one, it was the "AWS SQS Build Trigger Plugin", causing the "save / apply" buttons to move, and not be functional
I have a linux slave and windows master configured in Jenkins. However if i am scheduling periodic job on the linux slave and i shutdown my master (Windows), will the scheduled job execute ?
Answer in case of Jenkins is definitive no, but here is what you have as an alternative https://www.rundeck.com is the modern it operations management platform in other words it is scheduller with UI wich can a lot of power and if just need to run simple cases one per week it will do that job.
Ofc, you can always use cron with some shell scripts we it is an *nix system
No! You cannot.
All the jobs will be configured on Master and Master tells slaves to do what and when. So, its quite obvious that you cannot have what you are asking.
We are looking at Open source Jenkins masters failover scenario, and currently backing up jenkins jobs and configurations using SCM sync plugin. any ideas on how to restore Jenkins for high availablity
when master goes down.
Docker images work great for this. In essence the master is just an image which you configure with all your jobs. Logging of course should not be stored on the docker image but piped to AWS S3 or some datastore.
Each job you run launches a new docker slave to handle that task. Offers HA with lots of room for horizontal scaling.
If docker/containers is not your thing, configuration management is the way to go (chef, puppet, ansible). Take your pick and use these tools to build out your consistent Jenkins master and restore from latest backup.
I would like to take suggestion and inputs from wider audience regarding an issue with Jenkins master
I would like to have Jenkins setup in the following way to have the other backup so that it runs even if the master is down!
my plan for Jenkins master
The black arrows indicate that it is the primary configuration for Jenkins1 master and if it is down, we can switch apache to communicate with Jenkins2 instantly as a hot backup
It is possible and can be achieved and is helpful while performing OS upgrades or any other maintenance activities on master machine
I am planning to have it in such a way that to have a set of Jenkins master, which can be used in a manner, that the load is delegated to each master from Apache and entire Jenkins is not offline when we restart a master as other masters are also online
Please let me know a better way to achieve this if you have any
CloudBees have already addressed this issue with an Enterprise plugin for HA
The High Availability plugin, offered within CloudBees Jenkins
Enterprise, eliminates downtime due to master failures. Multiple
Jenkins masters act as backups waiting for a primary master failure.
Once a failure is detected, a backup master automatically boots up and
acts as a failover.The best part? Project work continues.
This is a paid solution but if you have a mission critical configuration then it might be worth the cost
I am trying to scale Jenkins for a large organization. Is there a way to have multiple Jenkins masters share a slave pool? For example, if I had 200 Jenkins Masters and I want them to share the same set of 50 Linux slaves.
That is, assuming each slave only has 1 executor, if Master A submits a job to the slave pool and it is running on Slave 1, if Master B submits a job to the slave pool, it would try to run on one of the other free slaves, since Slave 1 is already occupied.
I know multiple masters could share a single slave if I configured the slave to have a new workspace and executor for each master. However, I want to be able to set the slave up once, instead of having a slave.jar running on the slave for each master.
Cloudbees Op Center appears to provide this functionality, but looking for a way to do this with the Open Source version. If not, how difficult do you think it would be to extend Jenkins to have this functionality? I have Java development experience and have done a little work with Jenkins plugin development.
Thanks,
As you've noted it's not hard to share slaves between masters, just setup multiple workspaces and each master will install it's own slave jar. The trick is to share resources properly.
One such resource manager is Apache Mesos. A Jenkins Mesos plugin exists enabling the creation of slaves on a managed cluster.
This approach is very new and Ebay have blogged on how they've evolved their Jenkins setup to use Mesos:
Ebay CI solutin part 1
Ebay CI solution part 2
Hope this helps.
There is a Gearman Plugin developed by Open Stack to handle sharing slaves by multiple masters.
If it were me, I'd set up all the masters with a cloud plugin for slaves. For example, you could install the kubernetes plugin or the nomad plugin and connect all masters up to the same kubernetes or nomad cluster. Nomad or Kubernetes would take care of resource management, and the masters would only be submitting jobs to a shared pool of resources. This concept can easily be applied to other cloud providers, like AWS, but IMHO if you just want to set up an on-prem pool of resources for your jenkins masters from scratch, nomad is the easiest option.