I have many Jenkins Jobs that I need to run on every Build,
At present time I have 4 slave servers.
I would like the jobs to run in parallel as much as possible, hence I defined the jobs as follow:
Execute concurrent builds if necessary - Disabled
Restrict where this project can be run - Enabled with the following values SalveLinux1HT||SalveLinux2HT||SalveLinux3HT||SalveLinux4HT
To my understanding if Job A and B are triggered at the same time, one should use 1HT and the other should use 2HT and they can run in parallel
however Jenkins build job A on all 4 slaves and only after it's finished he will build job B on all 4 slaves
This is the opposite of my goal
Any ideas?
Thanks in advance
You can use
Build Flow Plugin
You can find both installation and configuration instructions of this plugin at the above mentioned link.
If you want to run any jobs in parallel you can use following scripts:
parallel (
// job A and B will be scheduled in parallel.
{ build("jobA") },
{ build("jobB") }
)
// jobC will be triggered after jobs A and B are completed
build("jobC")
Related
I have setup a slave.
For a job that executes a shell script, I configured to run either on the slave and on the master.
If I launch 2 instances of the same job, I observe that the job is run only by the master, and the 2nd instance waits to the 1st one to finish, and it will be run also by the master.
I expect master and slave to work simultaenously.
Why is the slave always idle?
Is there a way to priorize one slave?
UPDATE: In my use case, a job uses the database for destructive tests, so is not good for reliability to have more than one instance of the same job in a node. Each node has a copy of the database.
First, go to the job configuration page and check "Execute concurrent builds if necessary". This will allow multiple instances of your job to execute at the same time.
Next, go to the configuration pages of your build nodes (via the link "Build Executor Status" on the main page) and set "# of executors" to 1 for each one (both master and slave) . This will prevent one build node from running multiple jobs at the same time.
The result should be that if you launch 2 instances of the same job, one will execute on the master and one will execute on the slave.
The solution with a jenkins pipeline script:
node("master") {
parallel (
"masterbuild" : {
node ("master") {
mybuild()
}
},
"slavebuild" : {
node ("slave") {
mybuild()
}
}
)
}
def mybuild() {
sh 'echo build on `hostname`'
}
This is an improvement on Wim answer:
Go to the job configuration page and check "Execute concurrent builds if necessary". This will allow multiple instances of your job to execute at the same time.
Next, use the Throttle Concurrent Builds Plug-in.
In this way, only one execution per node is allowed, and the load is balanced between different nodes.
In this way, a node doesn't loose the ability to run simultaneously several unrelated jobs.
I have one pipeline build called, for example UAT. This build is scheduled every 3 minutes. And another build called DEV. DEV is scheduled every minute. The task is: to run UAT only if the last DEV execution was SUCCESS. If not - skip the execution. And run it after other 3 minutes with the same condition.
How can I achieve that ?
Don't schedule your UAT job as a separate job but instead trigger the launch once your first DEV pipeline finishes with success.
As you are using pipelines you actually have 2 solutions :
1)
Don't call another job but just call a Groovy function to integrate the DEV part, such as :
node() {
stage "UAT"
// Your existing UAT pipeline content here
stage "DEV"
git 'http://urlToYourGit/projectContainingYourDevScript'
pipeline = load 'functions.groovy'
pipeline.dev()
}
2) Just call a second Jenkins job with this kind of line :
node() {
stage "UAT"
// Your existing UAT pipeline content here
build job: "dev-job"
}
With these 2 solutions you can configure your first job to run every minute and it will trigger the second part/job only if the first one finishes with success (otherwise Jenkins will just fail the build as it would normally do).
I'm new to Jenkins and I'm trying to setup a project which will use few build executors.
The flow shall be as follows:
two build executors with webservice label return their IP addresses and wait for the third build executor to finish its job
third build executor with tester label collects those IP addresses and performs some long running job (e.x. sends HTTP requests to the webservices deployed on those two agents)
How to achieve such behavior in Jenkins?
I've found that when an build executor finishes its job it is immediately released and I don't know how to make it wait for other build executors to finish their jobs.
Edit:
I forgot to mention that I want the build executors with the webservice label to be reserved (not available for other jobs) till the build executor with the tester label will finish its long-running job.
Also all these build executors should be on separate slaves each. That means each slave has only one build executor.
I've finally managed to do this using Pipeline and below script:
node('webservice') {
def firstHostname = getHostname()
node('webservice') {
def secondHostname = getHostname()
node('tester') {
println 'Running tests against ' + firstHostname + ' and ' + secondHostname
// ...
}
}
}
def getHostname() {
sh 'hostname > output'
readFile('output').trim()
}
It acquires two build executors with webservice label. I'm getting their hostnames (I'm using them instead of the IP addresses) and pass them to the build executor with a tester label. Finally the tester runs some long-running tests.
Those two webservice build executors are blocked till the tester finishes its job, and no other project may use them during that time.
As Alex O mentioned, you can configure the master and slave relationship between the projects /executors inside the Jenkins projects /executors. There is option for that, "Build Triggers" -> Build after other projects are built
or use plugin to achieve it
https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin
or
https://wiki.jenkins-ci.org/display/JENKINS/Join+Plugin
What you actually want is probably that your job uses three slaves at the same time.
Re-thinking the setup in that way, it won't be necessary to consider the collection of IPs and the subsequent usage of the slaves as three different steps that must be aligned in some way.
Unfortunately, Jenkins does not support using multiple slaves for one build out-of-the box, but it will be possible to achieve what you want e.g. using the Multijob plugin and the Join plugin that Aaron mentioned already.
See also this question for information on how to use two slaves at the same time.
i have 2 pipelines in jenkins and i need to run a final job if last 2 jobs in 2 pipelines are successfull.
job 1 ( which will build periodically at 7PM ) will call 2 jobs job_pipeline1_1 and job_pipeline2_1.
job1
job_pipeline1_1 -- job_pipeline1_2
job_pipeline2_1 -- job_pipeline2_2
job_final (should be called only after job_pipeline1_2, job_pipeline2_2 are successfull)
job_pipeline1_1 and job_pipeline1_2 are independent of job_pipeline2_1 and job_pipeline2_2 and will run on differnt servers.
job_final should be called only if job_pipeline1_2 and job_pipeline2_2 are successfull in that particular build.
job_final should be in the pipeline.
check this image "http://i.stack.imgur.com/58Upc.png"
Can any one help me in this regard?
Thanks in advance.
You can use Jenkins plugin "Build Flow Plugin" to run your jobs in parallel.
In that case your final job will be executed after completion of parallel jobs.
I have 4 jobs which needs to be executed in the following sequence
JOB A
|------> JOB B
|------> JOB C
|------> JOB D
In the above
A should trigger B & C parallely and C inturn triggers D.
A should hold the job as running till all 3 of them completed.
I tried the following plugins and couldn't achieve what I am looking for
Join Plugin
Multijob Plugin
Multi-Configuration Project
Paramterized Trigger Plugin
Is there any plugin which I haven't tried would help me in resolving this. Or is this can be achieved in a different way. Please advise.
Use DSL Script with Build Flow plugin.
try this Example for your execution:
build("job A")
parallel
(
{build("job B")}
{build("job C")}
)
build("job D")
Try the Locks and Latches plugin.
This may not be optimal way, but it should work. Use the Parameterized Trigger Plugin. To Job A, add a build step (NOT a Post Build Action) to start both Jobs B and C in the same build step AND block until they finish. In Job C, add a build step (NOT a Post Build Action) that starts Job D AND blocks until it is finished. That should keep Job A running for the full duration.
This isn't really optimal though: Job A is held open waiting for B and C to finish. Then C is held open until D is finished.
Is there some reason that Job A needs to remain running for the duration? Another possibility is to have Job A terminate after B and C are started, but have a Promotion on Job A that will execute your final actions after jobs B, C and D are successful.
I am trying to build a same system. I am building a certification pipeline where I need to run packager/build/deploy jobs and and corresponding test jobs. When all of them are successful, I want to aggregate the test results and trigger the release job that can do an automated maven release.
I selected Build pipeline plugin for visualization of the system. Initially tried with Parameterized trigger Plugin with blocking builds. I could not setup archiving the artifacts/fingerprinting and downstream build relationship this way since archiving the artifacts works only in postbuild. Then I put the Parameterized trigger in Post build activity. This way I was able to setup downstream builds, fingerprinting, aggregate test results but the build failures were not bubbling to upstream job chain and upstream jobs were non blocking
I was finally able to achieve this using these plugins-
Build Pipeline
MultiJob Plugin
FingerPrint Plugin
Copy Artifacts Plugin
Join Plugin
I'm using Jenkins 1.514
System looks like this
Trigger Job --> build (and deploy) Job (1..n) ---> Test Job (1..n)
Trigger Job -
Create as MultiJob and create a fingerprint file in shell exec
echo date +%s > fingerprint.txt
Trick is that file needs to be archived during the build, to do that execute this script-
ARCHIVEDIR=$JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_ID/archive
mkdir $ARCHIVEDIR
cp fingerprint.txt $ARCHIVEDIR
Create MultiJob Phase consisting of build/deploy job.
Build/deploy job is itself a multijob
follow the same steps for creating build/deploy job as above relative
to fingerprinting.
Copy the fingerprint.txt artifact from upstream job
Setup MultiJob phase in deploy job that triggers the test job
create a new fingerprint file and force archive it similar to above step
Collect Junit results in the final test job.
In the trigger Job, use Join Plugin to execute the Release Job by choosing 'Run Post Build Actions at join' and execute the release project only on stable build of Trigger Job.
This way all the steps are showing up in Build Pipeline view and Trigger job is blocking for all downstream builds to finish and sets its status as the worst downstream build to give a decision point for release job.
Multijob Plugin
If you'd like to stop the mess with downstream / upstream jobs chains definitions. Or when you want to add a full hierarchy of Jenkins jobs that will be executed in sequence or in parallel. Add context to your buildflow implementing parameter inheritance from the MultiJob to all its Phases and Jobs. Phases are sequential while jobs inside each Phase are parallel.
https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin