We have multiple Jenkin masters and have enabled Jenkins Prometheus plugin and connected these masters as data sources to Grafana. I am currently interested in finding jobs which are waiting for executors for more than a certain time and create an alert based on this. I looked at Jenkins metrics but did not find any suitable metric to have monitoring for this use case. How can I achieve this?
You can access the Jenkins build queue via Groovy and check for entries waiting for too long.
It is possible to run Groovy scripts via the Jenkins REST API, which then will be your interface for polling for such "blocked" jobs.
Related
I would like to monitor multiple Jenkins servers health checks in a single Jenkins server as dashboard.Can some one let me know what all the options do we have for this ??
Thanks!!
There is no option for managing multiple Jenkins servers health checks in a single Jenkins Server specially on Open source Jenkins. But we can find some similar thing on CloudBees Jenkins where we can integrate Multiple masters together.
CloudBees Jenkins Analytics also make the monitoring of multiple masters easier by adding a number of new graphs.
Please refer the link : https://www.cloudbees.com/blog/jenkins-operations-and-continuous-delivery-scale-cloudbees-jenkins-enterprise
I am working on a Robotic process automation where i need to automate 10 different process flows.Robot needs to run 24/7.My solution is hosted in AWS cloud and i have got 10 cloud machines to run the scripts.
I have a master Jenkins job which will retrieve the list of automated jobs to execute from a database and i have 10 different jobs configured in Jenkins server.Number of jobs that i need to run at the same time varies from time to time.It may be N different scripts or N instances of the same script with different data combinations.
Challenge i am facing is in post build action i am not able to control the list of scripts/jobs that i need to run based on the output from Jenkins master job.Is there any way to run only the job i need based on the output from a build command?
I was able to achieve it using Jenkins Flexible Publish plugin.
I have a jenkins instance running around 200 jobs. What I need is a plugin to show the build statistics for all the jobs.
Total Builds for each project
Failures
Success
Average time per build.
Searched a lot, but couldn't find a proper report plugin. Please help
These are few which you can look depending on how much customization/features you want to do/display:
https://wiki.jenkins-ci.org/display/JENKINS/Global+Build+Stats+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/build-metrics-plugin
https://wiki.jenkins-ci.org/display/JENKINS/Project+Statistics+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/eXtreme+Feedback+Panel+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/InfluxDB+Plugin
https://wiki.jenkins-ci.org/display/JENKINS/CouchDB+Statistics+Plugin
And there is Dashboard-View too.
For collecting nodes machine metrics (CPU Time/ Used Memory / Build Time per node etc.), I found the monitoring plugin to be the best.
https://wiki.jenkins.io/display/JENKINS/Monitoring
When it comes to aggregates build times group by job, I couldn't find anything good within Jenkins UI, but if you have a datadog license, you can just enable the datadog Jenkins plugin, configure the datadog api key and hostname in Jenkins Config, and you are good to go.
https://www.datadoghq.com/blog/monitor-jenkins-datadog/
I have several integration tests within my Jenkins jobs. They run on several application servers, and I want to make sure that only one integration test job is run at the same time on one application server.
I would need something like a tag or variable within my jobs which create a group of jobs and then configure the logic that within that group, only one job may run at the same time.
Could I use the Exclusion plugin for that? Does anyone have experience with it?
Use the Throttle Concurrent Builds Plugin. It replaces the Locks and Latches plugin, and provides the capability to restrict the number of jobs running for specific labels.
For example: you create a project category 'Integration Test Server A' and tie jobs to it with a maximum concurrent count of 1, and a second 'Integration Test Server B' label and tie other jobs to it, both categories will only run a single concurrent build (assuming you've set a max job count of 1), and the other jobs in that category will queue until the 'lock' has cleared.
Using this method, you don't have to restrict the number of executors available on any specific Jenkins instance, and can easily add further slaves in the future without having to reconfigure all your jobs.
If I understand you right, you have a pool of application servers and it doesn't matter on what server your tests run. They only need to be the only test on that server.
I haven't seen a plugin that can do that. However, you can get easily around it. You need to configure a slave for each application server. (1 slave = 1 app server) You need to assign the same label to all slaves and every slave can only have one executor. Then you assign the jobs that run the integration tests, to run on that label. Jenkins will assign the jobs then to the next available slave (or node) that has that label.
Bare in mind that you can have more than one slave running on the same piece of hardware and even a master and a slave can coexist on the same server.
Did you check below parameter in the Jenkins -> Manage Jenkins -> Configure system
# of executors
The above parameter helps you restrict the number of jobs to be executed at a time.
A Jenkins executor is one of the basic building blocks which allow a build to run on a node/agent (e.g. build server). Think of an executor as a single “process ID”, or as the basic unit of resource that Jenkins executes on your machine to run a build. Please see Jenkins Terminology for more details regarding executors, nodes/agents, as well as other foundational pieces of Jenkins.
You can find information on how to set the number of Jenkins executors for a given agent on the Remoting Best Practices page, section Number of executors.
Source - https://support.cloudbees.com/hc/en-us/articles/216456477-What-is-a-Jenkins-Executor-and-how-can-I-best-utilize-my-executors
I want a way to automatically discover Jenkins master servers and automatically monitor the health of the jobs on those Jenkins master servers so that I can look at a single console(using nagios host) to detect issues when a job is failing anywhere in integration.
Could someone help me out to finding Jenkins master servers using nagios?
There's a nagios plugin for retrieving job health information from Jenkins, but it looks like it requires manual configuration for each job, see Nagios Jenkins plugin.
I'm not familiar enough with nagios to know how any built-in auto-discovery works, but it looks like there are several example scripts (check_find_new_hosts and device discovery) for generating the necessary configuration from a network scan. You'll have to do some work to integrate the results of the scan into your nagios instance. (IIRC, you need to restart nagios after writing new configuration?)
To get the list of Jenkins servers, you can build on one of the existing network scan scripts for nagios. The script should scan an IP range and identify devices that respond to http://IP:8080/api/xml. The resulting XML document (JSON results are also supported) should contain a root tag named <hudson> (in my instance, maybe this will change to "jenkins" in a future release). If the server responds to this request, then you'll want your script to generate the nagios configuration for monitoring it.
In addition, the XML response will contain a list of jobs, like:
<job>
<name>My Job</name>
<url>http://jenkins:8080/job/My%20job/</url>
<color>blue</color>
</job>
By iterating through this list, you get the job names, job urls (for more details or polling for status), and the current statuses (blue means success). This list of jobs can provide input to the Nagios Jenkins plugin configuration.
The Jenkins Remote API is documented on your Jenkins instance, just go to http://jenkins:8080/api.