I have a build flow that runs multiple build flows
Main Flow
parallel (
{ build("flow1") },
{ build("flow2") },
{ build("flow3") },
{ build("flow4") }
)
Each sub flow runs a set of jobs
Sub-flow
build(job1)
build(job2)
Each of the jobs are configured to run on a Jenkins slave with a certain label but I am unable to figure out how to get each job to run on a slave that doesn't have a flow already on it.
ie. both flow 1 and flow 2 queue on the same slave when I have 2 slaves available with the defined label.
Anyone able to help me with configuring blocking/throttling for build flows.
I think the problem lies with the fact they don't take up an executor on the slave.
Thanks
Typically jobs build on the last node they built on. So, I'm assuming both the flyweight jobs last ran on the same slave?
Here are some ways to make one of the flyweights move to another slave.
Take one of the slaves offline and kick off flow1. Then switch places and take the other offline and start up the remaining. Kick off flow 2. This should put the last place where each flyweight built on different slaves.
Add a different label to each slave and target each one separately. For example, Flow1 targets Node1 and Flow2 targets Node2. Each flyweight should only go to that label being targeted.
Install the Job Restrictions Plugin and create some kind of complex logic to determine whether the flyweight should build on the node.
Related
I have a Jenkins server on K8s (using Rancher).
Job Y is configured to run after job X (image is attached).
Job X runs successfully, but at its end, job Y is not triggered.
Both jobs are on the same Jenkins master.
This problem has never happened to me before, on our Jenkins on prem servers.
I have checked everything I could think of, including creating test jobs that basically do nothing, just in order to test if the triggering is working for other jobs on this master, it does not.
Checking the logs - job x is in the log (image is also attached), job Y is missing from the log completely.
I will mention that on another master on the same cluster, triggering is working. Any ideas?
check the trigger condition for job Y.
Below is the one way you can trigger jobs in Jenkins
Chain jobs together
Chain Jenkins job together using the “Build other projects” option in the “Post Build Actions” of a job. The downside to this is that it introduces a dependency between jobs. For example, I want to be able to run the database backup job without doing a deployment every time.
Parameterized Trigger Plugin
When you have the Parameterized Trigger Plugin installed:
Create a wrapper job for your sequential jobs
For each sequential job
Select Build->Add build step->Trigger/call builds on other projects
Enter the sequential job name
Check the ‘Block until the triggered projects finish their builds’ checkbox (this only appears when you have the Parameterized Trigger Plugin installed)
Now when you run the wrapper job, all the triggered jobs will be run in order and sequentially. You of course also have the option of using the parameters functionality of the plugin too.
Throttle Concurrent Builds Plugin
The docs for the Throttle Concurrent Builds Plugin seem to be a little lacking, but it works well. With it installed, a “Throttle Concurrent Builds” section shows up in the Jenkins config section (Jenkins -> Manage Jenkins -> Configure System). There, you create a “Multi-Project Throttle Category”
Next, for each job that needs to be run sequentially, you
Check the ‘Throttle Concurrent Builds’ check box
Select the ‘Throttle this project as part of one or more categories’ radio button
Check the checkbox of the “Multi-Project Throttle Category” you created above
Save
Finally, you create a wrapper job that triggers all your to-be-run-sequentially jobs and when you run it, you should see your jobs running sequentially and in order.
Although a fairly complex option, this approach also allows you to run some jobs sequentially and others in parallel. For example, your wrapper job can run jobs A and B sequentially and jobs C and D sequentially, but trigger A and C in parallel. You can also combine this plugin with the Parameterized Trigger Plugin for maximum flexibility.
Join Plugin
Although I haven’t used the Join Plugin myself, it seems worth mentioning. Its docs state that
This plugin allows a job to be run after all the immediate downstream jobs have completed. In this way, the execution can branch out and perform many steps in parallel, and then run a final aggregation step just once after all the parallel work is finished.
That’s it. Several options for running Jenkins jobs in parallel. Choose the one that best suits your needs…
I have Jenkins set up with 3 nodes (build-01, build-02 and build-03), labeled with the "build" label , and a Freestyle-type project configuration, in which "Restrict where this project can be run" is set to "build".
When I start a project, it runs OK on build-01, however, when I start another project (with different parameters), it is queued by Jenkins to the same node, even if two other nodes are idle.
How could I configure Jenkins to assign a project to a next idle node?
Thanks!
Do you have labels on those nodes? Typically you'd see labels being utilized for this situation and Jenkins would schedule based on available nodes.
Project A executes continuously at around every minute
How can I also execute Project B every 2 hours?
At the time of Project B's execution Project A must be disabled / not executed.
That would depend on how you want things to run. If your Jenkins instance only have 1 slave with 1 job concurrency configured, you don't have to worry about build concurrency if there is two projects trying to build. One of them is gonna be picked and built, and then the another.
If i'm not wrong, Jenkins has 3 builders per slave for default, so probably the 2 projects would run at the same time. But you can always go on Configurations and tweak that.
pipeline {
options { lock resource: 'build-lock' }
stages {...}
}
use this in both of your pipelines. At any given point in time, either A or B will execute.
For more info Lockable resources
On Jenkins is there a way to setup a job to have two distinct builds that can be triggers for different reasons.
As an example
(1) First build handles releases and is only started manually.
(2) Second build runs automatically every time a commit occurs.
I know that I can split this into two different jobs each with its only build and triggers however I prefer not to.
Do the builds have anything in common? If they share the same build process, but the release requires extra pre/post-build steps, then Release Plugin is exactly what you need.
It allows to define additional pre/post-build steps that are executed only when manually triggering a release build.
However this goes against the CI concept. In CI, your eventually-released-build would have been promoted from a regular automatic CI build.
I'd like to be able to run several builds of the same Jenkins job simultaneously.
Example:
Build [*jenkins_job_1*]: calls an ant script with parameter 'A'
Build [*jenkins_job_1*]: calls an ant script with parameter 'B'
repeat as necessary
each instance of the job runs simultaneously, rather than through a queue.
The reason I'd like to do this is to avoid having to create several jobs that are nearly identical, all of which would need to be maintained.
Is there a way to do this, or maybe another solution (ie — dynamically create a job from a base job and remove it after it's finished)?
Jenkins has a check box: "Execute concurrent builds if necessary"
If you check this, then it'll start multiple builds for a job.
This works with the "This build is parameterized" checkbox.
You would still trigger the builds, passing your A or B as parameters. You can use another job to trigger them or you could do it manually via a script.
You can select Build a Multi-configuration project (Matrix build) when you create the job. Then, under the job's configuration, you can define the Configuration Matrix which lets you specify one or more parameters (axes) for different builds. Regarding running simultaneously, you should be able to run as many simultaneous builds as you have executors (with the appropriate label).
Unfortunately, the Jenkins wiki lacks documentation about this setup. There are a couple previous SO questions, here and here, that might provide a little guidance. There was a "recent" blog post about setting up a multi-configuration job to perform builds on various platforms.
A newer (and better) solution is the Jenkins Job DSL Plugin.
We've been using it with great success. Our job configurations are now disposable... we can set up a huge stack of complicated jobs from some groovy files and a couple template jobs. It's great.
I'm liking it a lot more than the matrix builds, which were complicated and harder to understand.
Nothing stopping you doing this using the Jenkins pipeline DSL.
We have the same pipeline running in parallel in order to model combined loads for an application that exposes web services, provides a database to several external applications, receives data via several work queues and has a GUI front end. The business gives us non-functional requirements (NFRs) which our application must meet that guarantees its responsiveness even at busy times.
The different instances of the pipeline are run with different parameters. The first instance might be WS_Load, the second GUI_Load and the third Daily_Update_Load, modelling a large data queue that needs processing within a certain time-frame. More can be added depending on which combination of loads we're wanting to test.
Other answers have talked about the checkboxes for concurrent builds, but I wanted to mention another issue: resource contention.
If your pipeline uses temporary files or stashes files between pipeline stages, the instances can end up pulling the rug from under each others' feet. For example you can end up overwriting a file in one concurrent instance while another instance expects to find the pre-overwritten version of the same stash. We use the following code to ensure stashes and temporary filenames are unique per concurrent instance:
def concurrentStash(stashName, String includes) {
/* make a stash unique to this pipeline and build
that can be unstashed using concurrentUnstash() */
echo "Safe stashing $includes in ${concurrentSafeName(stashName)}..."
stash name: concurrentSafeName(stashName), includes: includes
}
def concurrentSafeName(name) {
/* make a name or name component unique to this pipeline and build
* guards against contention caused by two or more builds from the same
* Jenkinsfile trying to:
* - read/write/delete the same file
* - stash/unstash under the same name
*/
"${name}-${BUILD_NUMBER}-${JOB_NAME}"
}
def concurrentUnstash(stashName) {
echo "Safe unstashing ${concurrentSafeName(stashName)}..."
unstash name: concurrentSafeName(stashName)
}
We can then use concurrentStash stashName and concurrentUnstash stashName and the concurrent instances will have no conflict.
If, say, the two pipelines both need to store stats, we can do something like this for filenames:
def statsDir = concurrentSafeName('stats')
and then the instances will each use a unique filename to store their output.
You can create a build and configure it with parameters. Click the This build is parameterized checkbox and add your desired param(s) in the Configuration of the build. You can then fire off simultaneous builds using different parameters.
Side note: The "Bulk Builder" in Jenkins might push it into a queue, but there's also a This bulk build is parameterized checkbox.
I was having a pretty large build queue and I performed below steps to run jobs in
parallel in jenkins to reduce number of jobs waiting in queue
For each job you need to navigate to configure and select the checkbox stating
"Execute concurrent builds if necessary"
Navigate to Manage -> Configure System -> look for "# of executors" and set the no
of parallel executors you want (in my case it was set to 0 and I updated it to 2)