I have a jenkins job that runs a bash script.
In the bash script I perform effectively two actions, something like
java ApplicationA &
PID_A=$!
java ApplicationB
kill $PID_A
but if the job is manually aborted, the ApplicationA remains alive (as can be seen with a ps -ef on the node machine). I cannot use trapping and so on, because that won't work if jenkins sends a 9 signal (trapping doesn't work for 9).
It would be ideal if this job could be configured to simply kill all processes that it spawns, how can I do that?
Actually, by default, Jenkins has a feature called ProcessTreeKiller which is responsible to make sure there are no processes left running after the job execution.
The link above explain how to disable that feature. Are you sure you don't have that disabled by mistake somehow?
Edit:
Following the comments by the author, based on the information about disabling ProcessTreeKiller, to achieve the inverse one must set the environment variable BUILD_ID to the build id of Jenkins job. This way, when ProcessTreeKiller looks through the running processes to kill, it will find this as well
export BUILD_ID=$BUILD_ID
You can also use the Build Result Trigger plugin, configure a second job to clean up your applications, and configure it to monitor the first job for ABORTED state as a trigger.
Related
What I'd like to do:
Before/after any build allocates a slave: Reset slave to a pristine state (eg restore/delete files, kill rogue processes, etc. In other words, run some script). I know how to do this in one job, I don't know how to have Jenkins trigger it for every job of every type without modifying them all.
Unsatisfactory Ways I can think of accomplishing this:
Enforce a pre/post step in every AbstractProject (old style jobs). This is only scalable if using the Job DSL Plugin to generate all jobs.
Enforce a pre/post step in every PipelineJob. This is only scalable if using a Jenkins shared library (eg: myBuild() that encapsulates these steps).
Make use of the Global Post Script plugin and run a groovy script after every build. This works, but the script runs after the slave has been released, so another build may have grabbed it already (it's too late). Figuring out what nodes were allocated during the build is also fairly complicated
Switch all slaves to one-shot type (ie Docker) that require no clean-up. This doesn't work for my use cases, but may work for someone else.
Periodically run a job that uses a System Groovy Script to edit all other jobs to add a "clean-up" "pre" build step if not present. (Will not work for Pipeline jobs)
(I currently do #1 and #2)
Ideal Solution
Theoretical options:
A plugin to hook into some event (ie WorkspaceListener.beforeUse() ) and execute something then (unfortunately, WorkspaceListener does not apply to Pipeline Jobs). This should trigger right before a slave is used. It gets a little complicated when a slave has multiple executors (mine don't).
A plugin to enforce execution of some steps, similar to "Execute builders from job X", but in every job. (also doesn't work for Pipeline jobs)
Assuming that I'm using Swarm for all slaves, modify the swarm client to handle this logic (perform a task when slave becomes idle). A poor man's way would be to make the swarm client run in "one-shot" mode, in a bash loop.
Question:
What am I overlooking? Is there a better way?
I am writing a Groovy script to perform an automatic reboot of Windows servers. In the script, I am first taking the nodes offline, then checking to see if there are any builds, if there aren't, then perform a restart.
I wanted to use the safeRestart() method but it doesn't support the import statement I am using when looping through the nodes. I have seen an execute() method which basically executes a shell line of code in groovy.
How would I execute a restart of the Windows computers using execute()?
Not sure if this will answer your question directly, but will point in the right direction ...
You can leverage this S/O question: Run a remote command on all Jenkins slaves via Masters's script console or this Gist: run_command_on_all_slaves.groovy
btw: Jenkins API does seem to support running a script directly on the server (Computer).
Your actual command should be shutdown /r`
I don't believe you can do this unless the Node is on-line. Disconnecting the node stops the Jenkins slave process, then there's nothing running on the node, so not sure what control you'd have. Instead you want to block the queue and let the existing jobs finish:
Jenkins.instance.getNode('Node-Name').toComputer().setAcceptingTasks(false)
and check:
Jenkins.instance.getNode('Node-Name').toComputer().countBusy() == 0
Then run your (work on server) restart command
When the server is available again, launch the node and open the queue.
Jenkins.instance.getNode('Node-Name').getComputer().launch()
Jenkins.instance.getNode('Node-Name').getComputer().setAcceptingTasks(true)
Hope that helps.
(I'm new to Jenkins and curl, so please forgive any imprecision below.)
I am running a Jenkins job on one network that sends a curl command to a second network that is used to start a Jenkins job on that second network.
Sometimes I have to log onto that second network and restart the job using the Rebuild button provided by the Rebuild plugin.
I need to know how to determine whether the job on the second network was started by the original curl command or restarted via the Rebuild plugin, without the user having to do anything but restart the job with the same parameters.
I could use an extra boolean-parameter in the job on the second network that can be set to true by the curl command and to false when using the Rebuild button, but that requires the user to manually change the value of that parameter. I don't want the user to have to do that.
I think this only possible by using different users, or lets say an own user for the remote invocation. Anything more informative would have to be done manually for instance by an own paramter for the job. Normally jobs are started by developers, schedulers, hooks or by other jobs on the same instance. If triggered remotelly the job is triggered by a user as well, without authentication its the anonymus user who triggers the job.
Do you know the Jenkins CLI, could replace your curl commands?
https://jenkins.io/doc/book/managing/cli/
I am trying to run some automated acceptance tests on a windows VM but am running into some problems.
Here is what I want, a job which runs on a freshly reverted VM all the time. This job will get an MSI installer from an upstream job, install it, and then run some automated tests on it, in this case using robotframework (but that doesn't really matter in this case)
I have setup the slave in the vSphere plugin to only have one executor and to disconnect after one execution. On disconnect is shutsdown and reverts. My hope was this meant that it would run one Jenkins job and then revert, the next job would get a fresh snapshot, and so would the next and so on.
The problem is if a job is in queue waiting for the VM slave, as soon as the first job finishes the next one starts, before the VM has shutdown and reverted. The signal to shutdown and revert has however been sent, so the next job is almost immedieatly failed as the VM shuts down.
Everything works fine as long as jobs needing the VM aren't queued while another is running, but if they are I run into this problem.
Can anyone suggest a way to fix this?
Am I better off using vSphere build steps rather than setting up a build slave in this fashion, if so how exactly do I go about getting the same workflow to work using buildsteps and (i assume) pipelined builds.
Thanks
You can set a 'Quiet period' - it's in Advanced Project Options when you create a build. You should set it at the parent job, and this is the time to wait before executing the dependent job
If you'll increase the wait time, the server will go down before the second job starts...
Turns out the version of the vSphere plugin I was using was outdated, this bug problem is fixed in the newer version
Is there an elegant way to temporarily prevent Jenkins from executing any further builds in a defined time frame (say e.g. daily between 6am and 7am)?
Rather than stopping Jenkins, you can put it into "Quiet Down" mode, which prevents any new builds from taking place.
You can enable this via the URLs /quietDown and /cancelQuietDown, or via the CLI commands [cancel-]quiet-down.
Depening on what you want to achieve exactly you can probably use the Exclusive Execution Plugin. This plugin allows you to schedule a job which will block execution of all other jobs by putting Jenkins in shutdown mode (which is canceled when the job is done). You can make this job start at 6am and make ik run a simple ant script which sleeps for an hour.
However, if you are trying to use that window to e.g. run a backup you could actually run your backup from within that job, which will make 100% sure your backup won't start until all
running jobs are completed and it will make Jenkins available again as soon as the backup is done.
Alternatively you could consider using cron or the windows scheduler (depending on your OS) to stop Jenkins completely at 6am and restart it at 7am.