I am faced with the task of automating the build and test process for 4 software products in Jenkins. As many steps as possible should run in parallel.
The build of all four products can run in parallel. However, the test can only be parallelized by products 3 and 4. The test process of 1 and 2 must be sequential.
How would you basically design the automation in Jenkins?
If I create 4 separate pipelines, it needs to be ensured that the test process of 1 and 2 does not run in parallel.
Is it easier to create a single pipeline and prevent parallelization of 1 and 2 using job dependencies?
A quick solution off the top of my head would be something like this:
Run all the build jobs in one stage and have all of them inside a parallel block.
Then you create a new stage that runs after all the build stages have completed. This stage will also have a parallel block. Tests 3 and 4 will run in parallel while 1 and 2 run sequentially.
Related
I'm pretty new to Jenkins and trying to run or be able to run multiple jenkins jobs in parallel with different goals (inside my goals will have a mvn command, also cucumber #tags). Basically I want to run multiple cucumber tags on multiple jenkins jobs at the same time. From the research I've done so far looks like I have a few options - multijob or pipeline plugins..please advice. Thanks!
If you don't worry about any resource consumption, you can use separated jobs as below and also you can hook them each other or trigger them with one job via Jenkins features:
Example job configurations :
1. job :
mvn clean test -Dcucumber.options="--tags #Smoke"
2. job :
mvn clean test -Dcucumber.options="--tags #Regression"
Suppose that I have one jenkins node with one executor and I have to run i.e. tests and build stages in parallel.
node("jenkinsNodeWithOneExecutor") {
parallel {
test: {echo "run test" ... }
build: {echo "run build" ...}
}
}
How this will work? In the logs I see that that it runs parallel. Is it possible? I thought that every parallel task needs own executor...
PS. When trying to run job twice: Waiting for next available executor on ...
When you run a job in parallel the job will branch off and create two different jobs which will (ideally) execute simultaneously.
In your example you'd actually have three jobs: the original job, the "test" job, and the "build" job. You would need three executors available to process the script in parallel. If you're running the initial job on the same node you're trying to run parallel jobs on and you only have one executor this job would always hang indefinitely waiting for an executor that will never be available. If you're running the original job on another node it'd process your jobs sequentially since you only have one executor.
Are you sure you want to run your tests in parallel with your build? Typically tests have a dependency on whatever you've built. This would run the tests at the same time as the build and your build artifacts may not be available yet.
Is there a way to run Jenkins build step in parallel.
I know I can run multiple jobs in parallel, but I need to collate the unit test result and code coverage and use it generate a report.
So jobs in parallel might not be achieve the purpose, so I need to run build steps in parallel.
Any ideas?
Sounds like a job for the Jenkins Multijob plugin. You can create 2 phases:
Phase 1 : Builds jobs which run in parallel
Phase 2 : Collate results and generate report. You can specify this job to run only after the first phase is complete, and only if its successful etc.
I have many Jenkins Jobs that I need to run on every Build,
At present time I have 4 slave servers.
I would like the jobs to run in parallel as much as possible, hence I defined the jobs as follow:
Execute concurrent builds if necessary - Disabled
Restrict where this project can be run - Enabled with the following values SalveLinux1HT||SalveLinux2HT||SalveLinux3HT||SalveLinux4HT
To my understanding if Job A and B are triggered at the same time, one should use 1HT and the other should use 2HT and they can run in parallel
however Jenkins build job A on all 4 slaves and only after it's finished he will build job B on all 4 slaves
This is the opposite of my goal
Any ideas?
Thanks in advance
You can use
Build Flow Plugin
You can find both installation and configuration instructions of this plugin at the above mentioned link.
If you want to run any jobs in parallel you can use following scripts:
parallel (
// job A and B will be scheduled in parallel.
{ build("jobA") },
{ build("jobB") }
)
// jobC will be triggered after jobs A and B are completed
build("jobC")
I started using Jenkins in my project and I am trying to parallelize my test suite (Rspec test cases) written in 4 files in Jenkins
spec/features/
|-- test1.rb
|-- test2.rb
|-- test3.rb
|-- test4.rb
We can run all test cases with below command, it will run all tests written in test1.rb ..test4.rb sequentially which will take around 1 hour.
script spec/features/
If you want to excute test cases from each test file we can run like
script spec/features/test1.rb
Now I want to parallelize these test cases which can reduce the run from 1hr to 15 mins, All these test cases can run in one machine in parallel
I followed below approach in Jenkins
1) Set a new job "Main_Test_job"
2)
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test1.rb
Block until the triggered projects finish their builds ( Not selected this)
Add trigger --->
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test2.rb
Block until the triggered projects finish their builds ( Not selected this)
Add trigger --->
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test3.rb
Block until the triggered projects finish their builds ( Not selected this)
Add trigger --->
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test4.rb
Block until the triggered projects finish their builds ( Not selected this)
3)
Created job "Child_test_job" as which was included in main_test_job like below
Select Build step "Execute Shell" with below command
script spec/$TEST_UNIT
When I start "Main_Test_job", it will automatically start 4 Child_Test_Jobs in same machine, which will reduce my total run time to 15 mins.
But in this case "Main_test_job" has no way to monitor statuses of 4
child_test_jobs, It always succeeds immediately after starting 4
child jobs
"Block until the triggered projects finish their builds" this option
monitors child jobs but if we select this option for all child jobs,
they are running sequentially instead of parallel.
I can't use join plugin as I am not running different jobs instead triggering same job multiple times.
My Ideas:
have separate jobs for each test.rb and use join trigger to monitor
statuses of all jobs
have some shell script as the post-build task of "Main_Test_job"
which will aggregate/monitor statuses/results of each jobs.
I think this must be a common scenario in many organizations and there must be a easy way in Jenkins to achieve this.
Please let me know your approaches/ideas. May be I am missing some thing here.
If your jobs can run in parallel on same machine then Multijob plugin might be of interest to you. It starts the jobs in parallel but waits till all of them finish.
You can also use Build Flow Plugin
You can run any type of jobs by using this plugin.