I have 5 spring batches inside a jar. I have 1 spring dataflow app and 1 task. To launch a particular job, I am using below command :
java -jar jobName=job1 --spring.cloud.task.executionid=10
Everytime I launch a task, I have to provide jobName parameter. My question is how do I avoid this? Can I create 5 tasks (with single jar) corresponding to 5 jobs, so that I can start task1 and job1 is launched?
Basically, how to manage multiple jobs(contained in single jar) in spring dataflow?
Once you combine all the batches into one single jar, then you need a way to control which batch to launch based on some command line arg or a job parameter.
But, to answer your question, yes you can create multiple task definitions using the same task application which has all the batch applications.
For instance, if you have a task application registered as mytask1 which is the jar file.
Then, you can create multiple task definitions for the same task app:
task-batch1=mytask1
task-batch2=mytask1
..
task-batch5=mytask1
This way, you can launch the batches individually or using composed tasks.
But, you still need a way to launch a specific batch job inside the jar.
Related
I have two runners file in my cucumber framework, 1. The main runner.java which executes all of my feature files and 2. FailedRunner which only needs to be triggered when the first runner reports any failures.
Now the current execution method I am using is
gradle clean build -Denv=QA -Dcucumber.filter.tags=#QA
Now, I would Also like to pass my runner file name so that, I can create a upstream/downstream job to run the failedRunner class for retries for MainRunner file.
Thanks
I have a script to run to which i give multiple parameters in a loop, it takes each parameter completes the cycle and then the next one.
I need to run this on Jenkins, is there any option that i can run multiple builds on a single job ? I mean each parameter should be a single build and all the builds should run in queue not parallel.
We have two options :
1.) In my case i use a shell script and csv file as an input to it. So i wrote a simple groovy script for getting the inputs from csv file. Then i used parameterized plugin to run the script
2.) In case if you want to see the workflow on GUI you can go for pipeline
I want to know how can we parameterize build in jenkins. I need to create a jenkins job which will contain 5 sub jobs in it. i need to create a drop down , selelct any of the module and build it. But the script used is different for every sub build? can any1 guide on the same is it possible.
string parameters in Jenkins result in environment variables of the same name.
So, you could write a wrapper script in bash which would look for the environment variables that could be set as a result of the parameterized build (i.e. your 5 sub-jobs) in a series of if-elif statements, and within each one, you would invoke the necessary build script from there.
The build script that you would have Jenkins run would be the wrapper script.
I started using Jenkins in my project and I am trying to parallelize my test suite (Rspec test cases) written in 4 files in Jenkins
spec/features/
|-- test1.rb
|-- test2.rb
|-- test3.rb
|-- test4.rb
We can run all test cases with below command, it will run all tests written in test1.rb ..test4.rb sequentially which will take around 1 hour.
script spec/features/
If you want to excute test cases from each test file we can run like
script spec/features/test1.rb
Now I want to parallelize these test cases which can reduce the run from 1hr to 15 mins, All these test cases can run in one machine in parallel
I followed below approach in Jenkins
1) Set a new job "Main_Test_job"
2)
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test1.rb
Block until the triggered projects finish their builds ( Not selected this)
Add trigger --->
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test2.rb
Block until the triggered projects finish their builds ( Not selected this)
Add trigger --->
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test3.rb
Block until the triggered projects finish their builds ( Not selected this)
Add trigger --->
Selected "Trigger/Call builds on other projects"
projects to build " Child_test_job"
Build on same node
Predefined Parameters TEST_UNIT=test4.rb
Block until the triggered projects finish their builds ( Not selected this)
3)
Created job "Child_test_job" as which was included in main_test_job like below
Select Build step "Execute Shell" with below command
script spec/$TEST_UNIT
When I start "Main_Test_job", it will automatically start 4 Child_Test_Jobs in same machine, which will reduce my total run time to 15 mins.
But in this case "Main_test_job" has no way to monitor statuses of 4
child_test_jobs, It always succeeds immediately after starting 4
child jobs
"Block until the triggered projects finish their builds" this option
monitors child jobs but if we select this option for all child jobs,
they are running sequentially instead of parallel.
I can't use join plugin as I am not running different jobs instead triggering same job multiple times.
My Ideas:
have separate jobs for each test.rb and use join trigger to monitor
statuses of all jobs
have some shell script as the post-build task of "Main_Test_job"
which will aggregate/monitor statuses/results of each jobs.
I think this must be a common scenario in many organizations and there must be a easy way in Jenkins to achieve this.
Please let me know your approaches/ideas. May be I am missing some thing here.
If your jobs can run in parallel on same machine then Multijob plugin might be of interest to you. It starts the jobs in parallel but waits till all of them finish.
You can also use Build Flow Plugin
You can run any type of jobs by using this plugin.
I am using Jenkins to run a bunch of my scripts on regular basis. And the Execute shell section of my job looks like:
runner.py my_script A.config run
The problem is I have a bunch of configs A.config, B.config .... S.config. I am currently creating separate jobs for each config. And have a bunch of jobs. Is there any plugin you would recommend so that I can just have "runner.py my_script" and pass in A.config or B.config... using a simpler option than having a bunch of job like I have right now?
You can a achieve this with the multi configuration job in Jenkins. Select it as the radio button entry when naming the job
Here is more information Jenkins and multi-configuration (matrix) jobs
It is a standard Jenkins feature so doesn't need a plugin and you can add your own labels in.