I manage a Jenkins server with a few hundred projects in the whole ecosystem. Many of the projects rely on upstream servers, that, unfortunately, are not always responsive. When I have a lag on these servers, my build queue can get to 10 or more. Is there a plugin or setting to send a warning email when the build queue exceeds a particular length?
I have been unable to find a plugin that does this, but you can query Jenkins for the information as detailed here: Jenkins command to get number of builds in queue.
If you have a Jenkins slave available you could set up a job that runs every 15 minutes and just hit each of the other Jenkins servers with the API call to get build queue counts (this is easy if you have just one master and many slaves.)
If you wanted to stay completely outside of Jenkins (not add another job to the mix) you could write a script to poll the Jenkins API for the information. You could then run that script under, say, a 15 minute (or some other relevant time step) timer using cron (or windows scheduled task). Admittedly then you have to dedicate some resources to running this job.
It looks like you could use python to get the build queue and check the length of the returned list. get_queue_info()
I haven't mucked about with the Jenkins API much myself so I'm not sure offhand exactly what the script would need, but it should be simple enough once you dig into it.
Related
I am not sure whats the best way to implement multiple parallel calls in Jenkins remotely.. Any inputs will be greatly appreciated.
How to get the build number for the multiple parallel calls (2-10 multiple calls) to the Jenkins Server for a parameteried job if it gets triggered remotely. One requirement is there will be no change in the build parameter. The development team is using tool/python program to invoke 50 POST calls in that case how do we track the build number.
Scenario -- I have a freestyle parameterized job with Enable concurrent build if necessary box checked for this job Sequential call for with same build parameters request when initiated remotely, we see build number using https://jenkinsurl/queue/item with filtering out the build number and then https://jenkins url/build/Consoletext -- This works
Scenario 2 -- Same request with no change with parameters when triggered multiple times more than 2 calls we can see the build number /Consoletext for the first call and later ones were unable to track with the build number.
Sorry, I am a beginner and trying to implement multiple parallel calls. My Jenkins job is configured to run a python script on the Jenkins server that will return success along with work id and other responses that the dev Team needs for further processing.
When the team triggers API remotely 50 times, only for the first call we see the build number/full response from the ConsoleText for the rest of the calls we don't see any build number. I don't see any failures also in Jenkins. FYI This is a free-style parameterized job concurrent build option enabled.
I would like to monitor the estimated time of all of my builds to catch the cases where this value is shown as 'N/A'.
In these cases the build gets stuck (probably due to network issues in my environment) and it won't start new builds for that job until killed manually.
What I am missing is how to get that data for each job, either from api or other source.
I would appreciated any suggestion.
Thanks.
For each job, you can click "Trend" on the job run history table, and it will show you the currently executing progress along with a graph of "usual" execution times.
Using the API, you can go to http://jenkins/job/<your_job_name>/<build_number>/api/xml (or /json) and the information is under <duration> and <estimatedDuration> fields.
Finally, there is a Jenkins Timeout Plugin that you can use to automatically take care of "stuck" builds
I have a job A running in Jenkins, which kicks off a process A on a VM, waits for it to finish, picks up the report generated by it and sends it as an attachment to the build notification. The problem is this process A takes too long to finish and job A keeps waiting on it. Is there any way I can start this process A, stop job A and when process A is done, trigger a new job B which would pick up the report generated by process A and sends it out with build sucess/failure status.
Any help is appreciated.
Thanks
Jenkins provides an API for kicking off jobs via simple HTTP requests. You kick off job B using curl or something like that, as the final step in process A on the VM.
The docs are on the Jenkins site. You can use your own Jenkins find the specific URLs for kicking off particular jobs; there's a link in the bottom right hand corner of the Jenkins page.
Perhaps an even better match for your use case would be a job of type "Monitor an external job". I have not used it myself, but from the documentation it sounds like a useful tool. The docs are at: https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs
I set my Jenkins job to build automaticlally many times a day by the scheduler.
If the build is failed, it will send mail to my team.
However I don't want to spamming the mail box. How can I set a condition to stop the build scheduler if it was failed more than 10 times ?
Rather than scheduling the job continuously, try the continuous integration paradigm, like this:
Unconditionally schedule the job only rarely. Perhaps once per day, just to ensure than any external factors (missing resources, changed interfaces, etc.) haven't come into play.
Trigger the job when any known source or dependency changes (e.g. source code, jar in your artifact repository, DB schema change, etc.)
Use a suitable plugin to retry failures.
I recommend the Naginator plugin for this. It can nag a limited number of times, and it auto-throttles: it nags frequently to begin with, then less frequently after a protacted period of failure.
Even if you don't change how the job is trigger, Naginator is probably a good solution for you. Use it to send your emails, instead of using an unconditional on-failure step.
We need to be able to run a Jenkins job that consumes two slaves. (Or, two jobs, if we can guarantee that they run at the same time, and it's possible for at least one to know what the other is.) The situation is that we have a heavy weight application that we need to run tests against. The tests run on one machine, the application runs on another. It's not practical to have them on the same host.
Right now, we have a Jenkins job that uses a script to kick a dedicated application server up, install the correct version, the correct data, and then run the tests against it. That means that I can't use the dedicated application server to run other tasks, when there aren't the heavy weight testing going on. It also pretty much limits us to one loop. Being able to assign the app server dynamically would allow more of them.
There's clearly no way to do this in the core jenkins, but I'm hoping there's some plugin or hackery to make this possible. The current test build is a maven 2 job, but that's configurable, if we have to wrap it in something else. It's kicked off by the successful completion of another job, which could be changed to start two, or whatever else is required.
I just learned from that the simultaneous allocation of multiple slaves can be done nicely in a pipeline job, by nesting node clauses:
node('label1') {
node('label2') {
// your code here
[...]
}
}
See this question where Mateusz suggested that solution for a similar problem.
Let me see if I understood the problem.
You want to have dynamically choose a slave and start the App Server on it.
When the App server is running on a Slave you do not want it to run any other job.
But when the App server is not running, you want to use that Slave as any other Slave for other jobs.
One way out will be to label the Slaves. And use "Restrict where this project can be run" to make the App Server and the Test Suite run on the machines with Slave label.
Then in the slave nodes, put " # of executors" to 1. This will make sure that at anytime only one Job will run.
Next step will be to create a job to start the App Server and then kick off the Test job once the App Server start job is successful..
If your test job needs to know the server details of the machine where your App server is running then it becomes interesting.