I need to run migration script in batches
I have tried with writing script and fetch all data and process with the use of Promise.all
Related
I have 5 spring batches inside a jar. I have 1 spring dataflow app and 1 task. To launch a particular job, I am using below command :
java -jar jobName=job1 --spring.cloud.task.executionid=10
Everytime I launch a task, I have to provide jobName parameter. My question is how do I avoid this? Can I create 5 tasks (with single jar) corresponding to 5 jobs, so that I can start task1 and job1 is launched?
Basically, how to manage multiple jobs(contained in single jar) in spring dataflow?
Once you combine all the batches into one single jar, then you need a way to control which batch to launch based on some command line arg or a job parameter.
But, to answer your question, yes you can create multiple task definitions using the same task application which has all the batch applications.
For instance, if you have a task application registered as mytask1 which is the jar file.
Then, you can create multiple task definitions for the same task app:
task-batch1=mytask1
task-batch2=mytask1
..
task-batch5=mytask1
This way, you can launch the batches individually or using composed tasks.
But, you still need a way to launch a specific batch job inside the jar.
I use jenkins for my countinuous integration testing + possibility of starting manual checks.
My utilization is a job that :
Poll mercurial repo every 10 mn
Once a commit is done, start a generation (clone + make)
Launch a python suite test script
Gather result
Sometimes I want to be able to gracefully stop a job without using the "stop" button that simply aborts the test.
I managed to do it using a trick that check the presence of a file in the log directory used by the Python test suite
But I'm looking for a way to do it inside jenkins job itself.
Is there a way to have a customizable button for that purpose ?
I tried "batch task" plugin that would have been perfect BUT it waits for the python script to complete before execution ... So useless in my case (but the code works)
Thanks in advance for your help
I have built a Jenkins job to run automated ZAP-Proxy scans.
I used curl -X POST -u YOUR_USER:YOUR_USER_PASSWORD http://YOUR_JENKINS_URL/job/YOUR_JOB to build the job from the terminal. Is there a way to display the console output in the terminal while the job is building?
It is possible, but it's a little more complicated for two reasons:
When you trigger a new build via curl, then the build will not start immediately. The build will enter the build queue, and it will only start executing once Jenkins found a suitable executor. Before that time, there is no build URL at all.
Technically, the (continuous) console output is delivered in multiple fragments that must be retrieved individually via HTTP.
So, once you triggered the build, you need to find its URL after if left the build queue. It can be done nicely in Groovy -- as a simpler heuristic, you could just wait for certain time and then use the lastBuild reference.
For fetching the console log fragments, you'll use the <buildUrl>/logText/progressiveText end point. Create a loop fetching that URL and checking the X_More_Data and X_Text_Size HTTP headers for information on whether (and what) console output is available. You can do that in bash with curl in a loop; I found this Groovy example on the Web.
In the end, the most elegant solution is probably to
trigger a new build by submitting a Groovy script that will trigger the build and then waits for the build to leave the build queue, returning the build URL.
Then use another script that will poll/fetch/display that build's console output.
If you use the CLI interface for submitting the script, then you can do all those steps in a single script. If you use the REST API, then second part ("continuous output") probably won't work due to output buffering on REST API side.
I have a script to run to which i give multiple parameters in a loop, it takes each parameter completes the cycle and then the next one.
I need to run this on Jenkins, is there any option that i can run multiple builds on a single job ? I mean each parameter should be a single build and all the builds should run in queue not parallel.
We have two options :
1.) In my case i use a shell script and csv file as an input to it. So i wrote a simple groovy script for getting the inputs from csv file. Then i used parameterized plugin to run the script
2.) In case if you want to see the workflow on GUI you can go for pipeline
I have a gulp task that runs a couple other tasks. One of the tasks is to run karma. A couple of the tests are failing, which causes the gulp to fail. Is there a way to force a gulp task to completion via the command line, or a way to manipulate the return code?
I am writing a Jenkins job for another project that I don't own, so I am not responsible for the failing tests. That said, I don't have commit privs to the SVN repo that their code is stored on. Yes, it would be easiest to just change the gulpfile, but I'd rather not write a script in the Jenkins job to append a new task. Although, that might be the only option.
I do not know of a 'gulp-native' way to do this. A very similar scenario to what you laid out is discussed in this gulpjs/gulp issue - Continue gulp on error? #75.
The ultimate resolution was to use gulp-plumber