Is there a way to trigger an action (such as sending an email to an administrator) if a task spawned by gnu parallel times out?
Use --joblog. Exitval=-1 means timed out.
seq 100000 | parallel --joblog jl.log echo >> foo &
# Parse jl.log and do something with that
tail -n+1 -f jl.log | parallel --header : echo {Exitval}
Related
I would like to keep track of GNU parallel in a simple log file and would like it to emit the name of each as it starts / ends (either or both are equally fine). It seems verbose is too verbose for this.
If you make a profile that does the logging:
echo 'echo {} >> my.log;' > ~/.parallel/log
Then you can do this:
parallel -J log seq {} ::: 1 2 3
But since the profile uses {} you need to mention {} explicitly.
THIS DOES NOT WORK:
parallel -J log seq ::: 1 2 3
If you are not looking for --joblog then please explain how your needs differ.
--joblog is covered in 7.7 (p. 59) in GNU Parallel 2018 (paper copy: http://www.lulu.com/shop/ole-tange/gnu-parallel-2018/paperback/product-23558902.html or download it at: https://doi.org/10.5281/zenodo.1146014).
I want to use GNU parallel for the following problem:
I have a few files each with several lines of text. I would like to understand how I can run a script (code.sh) on each line of text of each file and for each file in parallel. I should be able to write out the output of the operation on each input file to an output file with a different extension.
Seems this is a case of multiple parallel commands running parallel over all files and then running parallel for all lines inside each file.
This is what I used:
ls mydata_* |
parallel -j+0 'cat {} | parallel -I ./explore-bash.sh > {.}.out'
I do not know how to do this using GNU parallel. Please help.
Your solution seems reasonable. You just need to remove -I:
ls mydata_* | parallel -j+0 'cat {} | parallel ./explore-bash.sh > {.}.out'
Depending on your setup this may be faster as it will only run n jobs, where as the solution above will run n*n jobs in parallel (n = number of cores):
ls mydata_* | parallel -j1 'cat {} | parallel ./explore-bash.sh > {.}.out'
I have the following piece of code, which works as expected. It ensures that 2 processes are always spawned, and if any process fails, the script comes to a halt.
I have worked with GNU parallel earlier on simple one line scripts and they have worked really well.I'm sure the one below too can be made simpler.
The sleeper function in reality is MUCH more complex than one shown below.
The objective is that GNU parallel will call sleeper function in parallel and also do error handling
`sleeper(){
stat=$1
sleep 5
echo "Status is $1"
return $1
}
PROCS=2
errfile="errorfile"
rm "$errfile"
while read LINE && [ ! -f "$errfile" ]
do
while [ ! -f "$errfile" ]
do
NUM=$(jobs | wc -l)
if [ $NUM -lt $PROCS ]; then
(sleeper $LINE || echo "bad exit status" > "$errfile") &
break
else
sleep 2
fi
done
done<sleep_file
wait`
Thanks
What you are looking for is --halt (requires version 20150622):
sleeper(){
stat=$1
sleep 5
echo "Status is $1"
return $1
}
export -f sleeper
parallel -j2 --halt now,fail=1 -v sleeper ::: 0 0 0 1 0 1 0
If you do not want the sleeper to get killed (maybe you want it to finish so it cleans up), then use --halt soon,fail=1 to let the running jobs complete without starting new ones.
I'm starting the job by issuing this request
/POST /jenkins/job/jobName/build
but it returns nothing. I want to get the build number that just has started.
Although late , posting an answer so it can benefit any other seekers :
We can get the latest/current build no. using the following API :
<JENKINS_URI>/job/<JOB_NAME>/lastBuild/buildNumber
This will provide the current executing or the last successful build.
Example :
curl -X GET <JENKINS_URI>/<JOB_NAME>/lastBuild/buildNumber --user <user>:<key>
o/p : 20
This is how i do it, First trigger the Job with the rest api call. From this call's response, get the queue location. And using this queue location, fetch the build number. The build number will not be generated immediately, thats why the while loop.
JENKINS_EMAIL=<Email>
JENKINS_TOKEN=<API Key>
JENKINS_URL=<Jenkins Server URL>
JENKINS_JOB=<JOB>
# Trigger Job and get queue location
location=$(curl -X POST -s -I -u $JENKINS_EMAIL:$JENKINS_TOKEN "${JENKINS_URL}${JENKINS_JOB}/buildWithParameters?pass=ok" | grep location | awk '{ print $NF }')
location2=${location//[$'\t\r\n']}
# Wait till build number is generated
while true ; do
buildnumber=$(curl -X GET -s -u $JENKINS_EMAIL:$JENKINS_TOKEN "${location2}api/json" | jq '.executable.number')
if [[ $buildnumber != "null" ]]; then
echo "Build Started. Build number is : "$buildnumber"
break
else
echo "Still in Queue"
sleep 1
fi
done
Jenkins - How to access BUILD_NUMBER environment variable
use ${BUILD_NUMBER} to get the current build.
already extensive answers are available in stack overflow. kindly surf it before raising the question.
Suppose I am running N jobs with the following gnu parallel command:
seq $N | parallel -j 0 --progress ./job.sh
How can I invoke parallel to kill all running jobs and accept no more as soon as any one of them exits?
You can use --halt:
seq $N | parallel -j 0 --halt 2 './job.sh; exit 1'
A small problem with that solution is that you cannot tell if job.sh failed.
You may also use killall perl. It's not accurate way, but easy to remember