Is there a way to run Jenkins build step in parallel.
I know I can run multiple jobs in parallel, but I need to collate the unit test result and code coverage and use it generate a report.
So jobs in parallel might not be achieve the purpose, so I need to run build steps in parallel.
Any ideas?
Sounds like a job for the Jenkins Multijob plugin. You can create 2 phases:
Phase 1 : Builds jobs which run in parallel
Phase 2 : Collate results and generate report. You can specify this job to run only after the first phase is complete, and only if its successful etc.
Related
I am faced with the task of automating the build and test process for 4 software products in Jenkins. As many steps as possible should run in parallel.
The build of all four products can run in parallel. However, the test can only be parallelized by products 3 and 4. The test process of 1 and 2 must be sequential.
How would you basically design the automation in Jenkins?
If I create 4 separate pipelines, it needs to be ensured that the test process of 1 and 2 does not run in parallel.
Is it easier to create a single pipeline and prevent parallelization of 1 and 2 using job dependencies?
A quick solution off the top of my head would be something like this:
Run all the build jobs in one stage and have all of them inside a parallel block.
Then you create a new stage that runs after all the build stages have completed. This stage will also have a parallel block. Tests 3 and 4 will run in parallel while 1 and 2 run sequentially.
I'm pretty new to Jenkins and trying to run or be able to run multiple jenkins jobs in parallel with different goals (inside my goals will have a mvn command, also cucumber #tags). Basically I want to run multiple cucumber tags on multiple jenkins jobs at the same time. From the research I've done so far looks like I have a few options - multijob or pipeline plugins..please advice. Thanks!
If you don't worry about any resource consumption, you can use separated jobs as below and also you can hook them each other or trigger them with one job via Jenkins features:
Example job configurations :
1. job :
mvn clean test -Dcucumber.options="--tags #Smoke"
2. job :
mvn clean test -Dcucumber.options="--tags #Regression"
I started using Jenkins in my project and I am trying to parallelize my test suite (Rspec test cases) written in 4 files in Jenkins
spec/features/
|-- test1.rb
|-- test2.rb
|-- test3.rb
|-- test4.rb
We can run all test cases with below command, it will run all tests written in test1.rb ..test4.rb sequentially which will take around 1 hour.
script spec/features/
If you want to excute test cases from each test file we can run like
script spec/features/test1.rb
Now I want to parallelize these test cases which can reduce the run from 1hr to 15 mins, All these test cases can run in one machine in parallel
I followed below approach in Jenkins
1) Set a new job "Main_Test_job"
2)
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test1.rb
Block until the triggered projects finish their builds ( Not selected this)
Add trigger --->
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test2.rb
Block until the triggered projects finish their builds ( Not selected this)
Add trigger --->
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test3.rb
Block until the triggered projects finish their builds ( Not selected this)
Add trigger --->
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test4.rb
Block until the triggered projects finish their builds ( Not selected this)
3)
Created job "Child_test_job" as which was included in main_test_job like below
Select Build step "Execute Shell" with below command
script spec/$TEST_UNIT
When I start "Main_Test_job", it will automatically start 4 Child_Test_Jobs in same machine, which will reduce my total run time to 15 mins.
But in this case "Main_test_job" has no way to monitor statuses of 4
child_test_jobs, It always succeeds immediately after starting 4
child jobs
"Block until the triggered projects finish their builds" this option
monitors child jobs but if we select this option for all child jobs,
they are running sequentially instead of parallel.
I can't use join plugin as I am not running different jobs instead triggering same job multiple times.
My Ideas:
have separate jobs for each test.rb and use join trigger to monitor
statuses of all jobs
have some shell script as the post-build task of "Main_Test_job"
which will aggregate/monitor statuses/results of each jobs.
I think this must be a common scenario in many organizations and there must be a easy way in Jenkins to achieve this.
Please let me know your approaches/ideas. May be I am missing some thing here.
If your jobs can run in parallel on same machine then Multijob plugin might be of interest to you. It starts the jobs in parallel but waits till all of them finish.
You can also use Build Flow Plugin
You can run any type of jobs by using this plugin.
I have 4 jobs which needs to be executed in the following sequence
JOB A
|------> JOB B
|------> JOB C
|------> JOB D
In the above
A should trigger B & C parallely and C inturn triggers D.
A should hold the job as running till all 3 of them completed.
I tried the following plugins and couldn't achieve what I am looking for
Join Plugin
Multijob Plugin
Multi-Configuration Project
Paramterized Trigger Plugin
Is there any plugin which I haven't tried would help me in resolving this. Or is this can be achieved in a different way. Please advise.
Use DSL Script with Build Flow plugin.
try this Example for your execution:
build("job A")
parallel
(
{build("job B")}
{build("job C")}
)
build("job D")
Try the Locks and Latches plugin.
This may not be optimal way, but it should work. Use the Parameterized Trigger Plugin. To Job A, add a build step (NOT a Post Build Action) to start both Jobs B and C in the same build step AND block until they finish. In Job C, add a build step (NOT a Post Build Action) that starts Job D AND blocks until it is finished. That should keep Job A running for the full duration.
This isn't really optimal though: Job A is held open waiting for B and C to finish. Then C is held open until D is finished.
Is there some reason that Job A needs to remain running for the duration? Another possibility is to have Job A terminate after B and C are started, but have a Promotion on Job A that will execute your final actions after jobs B, C and D are successful.
I am trying to build a same system. I am building a certification pipeline where I need to run packager/build/deploy jobs and and corresponding test jobs. When all of them are successful, I want to aggregate the test results and trigger the release job that can do an automated maven release.
I selected Build pipeline plugin for visualization of the system. Initially tried with Parameterized trigger Plugin with blocking builds. I could not setup archiving the artifacts/fingerprinting and downstream build relationship this way since archiving the artifacts works only in postbuild. Then I put the Parameterized trigger in Post build activity. This way I was able to setup downstream builds, fingerprinting, aggregate test results but the build failures were not bubbling to upstream job chain and upstream jobs were non blocking
I was finally able to achieve this using these plugins-
Build Pipeline
MultiJob Plugin
FingerPrint Plugin
Copy Artifacts Plugin
Join Plugin
I'm using Jenkins 1.514
System looks like this
Trigger Job --> build (and deploy) Job (1..n) ---> Test Job (1..n)
Trigger Job -
Create as MultiJob and create a fingerprint file in shell exec
echo date +%s > fingerprint.txt
Trick is that file needs to be archived during the build, to do that execute this script-
ARCHIVEDIR=$JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_ID/archive
mkdir $ARCHIVEDIR
cp fingerprint.txt $ARCHIVEDIR
Create MultiJob Phase consisting of build/deploy job.
Build/deploy job is itself a multijob
follow the same steps for creating build/deploy job as above relative
to fingerprinting.
Copy the fingerprint.txt artifact from upstream job
Setup MultiJob phase in deploy job that triggers the test job
create a new fingerprint file and force archive it similar to above step
Collect Junit results in the final test job.
In the trigger Job, use Join Plugin to execute the Release Job by choosing 'Run Post Build Actions at join' and execute the release project only on stable build of Trigger Job.
This way all the steps are showing up in Build Pipeline view and Trigger job is blocking for all downstream builds to finish and sets its status as the worst downstream build to give a decision point for release job.
Multijob Plugin
If you'd like to stop the mess with downstream / upstream jobs chains definitions. Or when you want to add a full hierarchy of Jenkins jobs that will be executed in sequence or in parallel. Add context to your buildflow implementing parameter inheritance from the MultiJob to all its Phases and Jobs. Phases are sequential while jobs inside each Phase are parallel.
https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin
I have three jobs that I would like to serialize in Jenkins.
They should run as a block after a job that triggers them:
Job1 -> [A,B,C]
Job2 -> [A,B,C]
Right now when Job1 is triggered twice, I get the following behavior:
Order that jobs are run now:
-Job1
-Job2
-A
-B
-(job A or C)
-Order is not guaranteed after this
What I would like to see is:
Order that jobs are run:
-Job1
-Job2
-A (from Job1)
-B (from Job1)
-C (from Job1)
------------
-A (from Job2)
-B (from Job2)
-C (from Job2)
Have a look at the Jenkins Locks and Latches plugin.
So, this is the solution I found:
Using the Jenkins Parameterized Trigger plug-in, an extra Build step is available:
Trigger/call builds on other projects
On Job1 add three build triggers:
Projects to Build: JobA
[x] Block until the triggered projects finish
their builds
Projects to Build: JobB
[x] Block until the triggered projects finish
their builds
Projects to Build: JobC
[x] Block until the triggered projects finish
their builds
Do the same on Job2.
Add a lock that is shared between the two Jobs 1 and 2.
This way when Job 1 and Job 2 are triggered they will wait for each other. And each Job won't be marked as passed until all the Jobs A->B>C run and are successful.
More options are available to control the return value of each build step , whether it should depend on the result of the triggered job or continue regardless of the result.
Fail this build step if the triggered build is worse or equal to
Mark this build as failure if the triggered build is worse or equal to
Mark this build as unstable if the triggered build is worse or equal to
I cannot think of a way to do exactly what you want in Jenkins. Maybe Jenkins is not the tool for you?
Another way to approach this would be: Your question actually presents a solution to your problem (a bunch of jobs and their execution order) and you are asking how to implement it.
Would you be willing to explain what is your goal, what is it you want to achieve? Maybe some Jenkins expert can then tell you how you can do it using Jenkins.