Trying to follow GNU Parallel as job queue with named pipes with GNU parallel 20201222, I run into issues of parallel not executing the last commands piped into it via tail -n+0 -f.
To demonstrate, I have 3 terminals open:
# terminal 1
true > jobqueue
tail -n+0 -f jobqueue | parallel
# terminal 2
tail -n+0 -f jobqueue | cat
Adding a single small test command to the queue:
# terminal 3
echo "echo test" >> jobqueue
Only terminal 2 prints "echo test", gnu parallel does not output anything.
# terminal 3
for i in `seq 10`; do echo "echo $i" >> jobqueue; done
Only terminal 2 prints "echo 1", ..., "echo 10" (one in each line), gnu parallel does not output anything.
# terminal 3
for i in `seq 100`; do echo "echo $i" >> jobqueue; done
Terminal 2 prints "echo 1", ..., "echo 100". Terminal 1 prints the lines "test", "1", ..., "10", "1", ..., "99", the last line "100" is missing.
Rerunning tail -n+0 -f jobqueue | parallel outputs all up to "99". Rerunning this with --resume --joblog log appended, outputs one more line ("100") but then also lags behind once new lines are added to joblog. For GNU parallel 20161222, the initial run only gets to line "84".
How can I force gnu parallel to flush its input queue on every line?
From man parallel:
There is a a small issue when using GNU parallel as queue system/batch manager: You have to submit JobSlot number of jobs before they will start, and after that you can submit one at a time, and job will start immediately if free slots are available. Output from the running or completed jobs are held back and will only be printed when JobSlots more jobs has been started (unless you use --ungroup or --line-buffer, in which case the output from the jobs are printed immediately). E.g. if you have 10 jobslots then the output from the first completed job will only be printed when job 11 has started, and the output of second completed job will only be printed when job 12 has started.
In other words: The jobs are running. Output is delayed. It is easier to see if you instead of using echo in your example use touch unique-file-name.
Related
I have a legacy project in Jenkins that hast to be pipelined (for
later parallelization), hence moving from simple tcsh script to
pipeline
running the script as
#!/bin/tcsh
source ./mysetting.sh
update
works but the same pipeline step fails due to missing alias expansion
stage ('update') {
steps {
//should be working but alias expansion fails
sh 'tcsh -c "source ./mysettings.sh; alias; update"'
//manually expanding the alias works fine
sh 'tcsh -c "source ./mysettings.sh; alias; python update.py;"'
}
}
calling alias in the steps properly lists all the set aliases, so I
can see them, but not use them.
I know in bash alias expansion has to be set
#enable shell option for alias_expansion
shopt -s expand_aliases
but in csh/tcsh that should be taken care of by source.
what am I missing?
found the solution:
sh '#!/bin/tcsh \n' +
'source ./mysettings.sh \n' +
'echo "Calling my alias" \n' +
'my_alias \n'
every line starting with sh launches a new shell, so it has to be in one line including line breaks.
further adding to the confusing was that documentation of jenkins says that it starts "a bash" but it launched /bin/sh which in my case pointed to something else
My original requirement is to check space and if enough space then proceed with the job or else abort the job. I am not failing the job as there has been another mail and process attached to failing of job. Thus I want something like:
if [ not enough space ]
then
... abort the job ....
fi
How can I abort the job through shell script or if any better option is there please help me.
Put this script in the pre build step (add shell build step before your job build step)
#max allowed usage .. maybe the whole disk
LIMIT = <put limit>
# you may replace the "." with your workspace dir
USED=`df . | awk '{print $5}' | sed -ne 2p | cut -d"%" -f1`
if [ $USED -gt $LIMIT ]
#If used space is bigger than LIMIT
then
exit 1
#This would end the build and report failure
fi
I have a Jenkins job for a db rollback script that uses a choice parameter for each environment (using NodeLabel Parameter Plugin).
I want the jobs to able to be run concurrently, but only for different environments.
"Execute concurrent builds if necessary" is enabled.
E.g. If the job is running for LIVE, allow someone to run the job again for TEST (this works). However, if LIVE is already running and someone runs the job for LIVE again, then do not run.
This plugin seems to suit my needs but is not shown on the list of available plugins in Manage Jenkins.
https://wiki.jenkins-ci.org/display/JENKINS/Concurrent+Run+Blocker+Plugin
Are there any other ways around this?
There's a solution with existing Jenkins plugins:
Create a Freestyle project named like Starter for concurrent builds exclusively on nodes.
☑ This build is parameterized
Node [NodeLabel Parameter Plugin]
Name: NODE
Choice Parameter
Name: JOB
Choices: ... the jobs' names you'd like to start with this ...
Build
Conditional step (single) [Conditional BuildStep Plugin]
Run?: Not
!: Execute Shell
Command:
#!/bin/bash +x -e
# Bash 4 needed for associative arrays
# From http://stackoverflow.com/questions/37678188/jenkins-stop-concurrent-job-with-same-parameter
echo ' Build --> Conditional step (single) --> Execute Shell'
echo " Checking whether job '$JOB' runs on node '$NODE'"
echo ' Creating array'
declare -A computers
# ------------------------------------------------------------------------
# Declare your nodes and their executors here as mentioned, for instance,
# in the API URI 'http://<jenkins>/computer/(master)/executors/0/api/xml':
computers=( # ^^^^^^ ^
[master]="0 1 2 3"
[slave]="0 1"
)
# Note: Executor indices DO NOT conform to the numbers in Jenkins'
# Build Executor Status UI.
# ------------------------------------------------------------------------
echo " Checking executors of node '$NODE'"
for computer in ${!computers[#]} ; do
for executorIdx in ${computers[$computer]} ; do
if [[ $computer == $NODE ]] ; then
if [[ "$computer" == "master" ]] ; then
node="(${computer})"
else
node=$computer
fi
url="${JENKINS_URL}/computer/${node}/executors/${executorIdx}/api/xml?tree=currentExecutable\[url\]"
echo " $url"
xml=$(curl -s $url)
#echo $computer, $executorIdx, $xml
if [[ "$xml" == *"/job/${JOB}"* ]] ; then
echo " Job '$JOB' is already building on '$computer' executor index '$executorIdx'"
echo ' Exiting with 1'
exit 1
fi
fi
done
done
echo ' Exiting with 0'
Builder: Set the build result
Result: Aborted
Conditional step (single)
Run?: Current build status
Builder: Trigger/call build on other projects
Build Triggers:
Projects to build: $JOB [ignore the error message]
Node Label parameter
Name: NODE [or how you call it in your downstream job(s)]
Node: $NODE
I have several (few hundreds) of files to run test on (each test takes few minutes).
Running sequentially is not acceptable and neither all together. So I am looking for something like a producer-consumer.
I tried pipeline jobs and parallel command the following way:
def files = findFiles glob: 'test_files/*'
def branches = [:]
files.each{
def test_command = "./test ${it}"
branches["${it}"] = { sh "${test_command} ${it}"}
}
stage name:'run', concurrency:2
parallel branches
Problem:
All the tasks are launch at the same time (OOM and all the fun)
Doesn't have the same introspection as the Jenkins parallel step, but since it seems not to support a fixed pool you can use xargs to achieve the same result:
def files = findFiles glob: 'test_files/*'
def branches = [:]
// there's probably a more efficient way to generate the list of commands
files.each{
sh "echo './test ${it}' >> tests.txt"
}
sh 'cat tests.txt | xargs -L 1 -I {} -P 2 bash -c "{}"'
The -P argument is the one that specifies a fixed number of 2 (or N) processes should always be running. Other tools like GNU Parallel offer even more tuning on how many processes should be used.
You can also try to use the lock step from the Lockable Resources plugin, the node step targeting a fixed number of executors. However this seems too much overhead to me unless your single tests are already taking tens of second each.
I want to capture the output on the bash shell when running the following command at the bash prompt.
rails runner "MyCloud::Deployment.checkuser"
Basically checkuser() returns a value of either 1 or 0, which I want to capture at the shell that launched the "rails runner" command.
Based on the written value at the shell, I would like to execute some other command. This checkuser() connects to the cloud, accesses the SQL server and returns the value, so I need to load the rails env.
I run this command in hudson "execute shell" window. How do I go about achieving this task?
Does rake help me with this?
Edit:
#to keep things simple:
module MyCloud
class Deployment
def self.checkuser
return 20
end
end
end
I am running the command from ubuntu box:
sreeni#ubuntu:~/work/co/RoR$ res="$(rails runner "MyCloud::Deployment.checkuser")"
sreeni#ubuntu:~/work/co/RoR$ echo $res
sreeni#ubuntu:~/work/co/RoR$
I want to capture the value that is returned from that check_user() to a variable on the bash shell.
Edit: You are confusing return values (objects returned to the same-language caller) with text output. You need to puts 20 to capture it as a string. You cannot capture return values from one language in another language without a foreign function interface or similar mechanism. A rare exception is the interface between *nix shells like Bash and Dash, where you can capture the return value from one in the other because they both implement something similar to a POSIX shell.
To get the exit code, test the command or $?:
rails runner "MyCloud::Deployment.checkuser"
if [ $? -eq 0 ]
then
echo "Success"
else
echo "Failure"
fi
Short form:
if rails runner "MyCloud::Deployment.checkuser"
then
echo "Success"
else
echo "Failure"
fi
To get string output, use a process substitution (and yes, those inner quotes are correct):
result="$(rails runner "MyCloud::Deployment.checkuser")"
if [ "$result" = '0' ]
then
echo "Success"
else
echo "Failure"
fi
Short form:
if [ "$(rails runner "MyCloud::Deployment.checkuser")" = '0' ]
then
echo "Success"
else
echo "Failure"
fi
To capture both output and exit code:
result="$(rails runner "MyCloud::Deployment.checkuser")"
exit_code=$?