Trying to follow GNU Parallel as job queue with named pipes with GNU parallel 20201222, I run into issues of parallel not executing the last commands piped into it via tail -n+0 -f.
To demonstrate, I have 3 terminals open:
# terminal 1
true > jobqueue
tail -n+0 -f jobqueue | parallel
# terminal 2
tail -n+0 -f jobqueue | cat
Adding a single small test command to the queue:
# terminal 3
echo "echo test" >> jobqueue
Only terminal 2 prints "echo test", gnu parallel does not output anything.
# terminal 3
for i in `seq 10`; do echo "echo $i" >> jobqueue; done
Only terminal 2 prints "echo 1", ..., "echo 10" (one in each line), gnu parallel does not output anything.
# terminal 3
for i in `seq 100`; do echo "echo $i" >> jobqueue; done
Terminal 2 prints "echo 1", ..., "echo 100". Terminal 1 prints the lines "test", "1", ..., "10", "1", ..., "99", the last line "100" is missing.
Rerunning tail -n+0 -f jobqueue | parallel outputs all up to "99". Rerunning this with --resume --joblog log appended, outputs one more line ("100") but then also lags behind once new lines are added to joblog. For GNU parallel 20161222, the initial run only gets to line "84".
How can I force gnu parallel to flush its input queue on every line?
From man parallel:
There is a a small issue when using GNU parallel as queue system/batch manager: You have to submit JobSlot number of jobs before they will start, and after that you can submit one at a time, and job will start immediately if free slots are available. Output from the running or completed jobs are held back and will only be printed when JobSlots more jobs has been started (unless you use --ungroup or --line-buffer, in which case the output from the jobs are printed immediately). E.g. if you have 10 jobslots then the output from the first completed job will only be printed when job 11 has started, and the output of second completed job will only be printed when job 12 has started.
In other words: The jobs are running. Output is delayed. It is easier to see if you instead of using echo in your example use touch unique-file-name.
I have a jenkins pipeline in which one stage is sonarqube analysis.I have a project and multiple subprojects in sonarqube.for ex projectkey=xyz.com subprojectkey=abc.xyz.com. My requirement is if quality gates didn't pass my build should fails . How i can define multiple project keys. I wrote a shell script to do the task but in that also I'm able to define only one project key.
PROJECTKEY="%teamcity.project.id%"
QGSTATUS=`curl -s -u SONAR_TOKEN: http://SONAR_URL:9000/api/qualitygates/project_status?projectKey=$PROJECTKEY | jq '.projectStatus.status' | tr -d '"'`
if [ "$QGSTATUS" = "OK" ]
then
exit 0
elif [ "$QGSTATUS" = "ERROR" ]
then
exit 1
fi
I have a Jenkins job for a db rollback script that uses a choice parameter for each environment (using NodeLabel Parameter Plugin).
I want the jobs to able to be run concurrently, but only for different environments.
"Execute concurrent builds if necessary" is enabled.
E.g. If the job is running for LIVE, allow someone to run the job again for TEST (this works). However, if LIVE is already running and someone runs the job for LIVE again, then do not run.
This plugin seems to suit my needs but is not shown on the list of available plugins in Manage Jenkins.
https://wiki.jenkins-ci.org/display/JENKINS/Concurrent+Run+Blocker+Plugin
Are there any other ways around this?
There's a solution with existing Jenkins plugins:
Create a Freestyle project named like Starter for concurrent builds exclusively on nodes.
☑ This build is parameterized
Node [NodeLabel Parameter Plugin]
Name: NODE
Choice Parameter
Name: JOB
Choices: ... the jobs' names you'd like to start with this ...
Build
Conditional step (single) [Conditional BuildStep Plugin]
Run?: Not
!: Execute Shell
Command:
#!/bin/bash +x -e
# Bash 4 needed for associative arrays
# From http://stackoverflow.com/questions/37678188/jenkins-stop-concurrent-job-with-same-parameter
echo ' Build --> Conditional step (single) --> Execute Shell'
echo " Checking whether job '$JOB' runs on node '$NODE'"
echo ' Creating array'
declare -A computers
# ------------------------------------------------------------------------
# Declare your nodes and their executors here as mentioned, for instance,
# in the API URI 'http://<jenkins>/computer/(master)/executors/0/api/xml':
computers=( # ^^^^^^ ^
[master]="0 1 2 3"
[slave]="0 1"
)
# Note: Executor indices DO NOT conform to the numbers in Jenkins'
# Build Executor Status UI.
# ------------------------------------------------------------------------
echo " Checking executors of node '$NODE'"
for computer in ${!computers[#]} ; do
for executorIdx in ${computers[$computer]} ; do
if [[ $computer == $NODE ]] ; then
if [[ "$computer" == "master" ]] ; then
node="(${computer})"
else
node=$computer
fi
url="${JENKINS_URL}/computer/${node}/executors/${executorIdx}/api/xml?tree=currentExecutable\[url\]"
echo " $url"
xml=$(curl -s $url)
#echo $computer, $executorIdx, $xml
if [[ "$xml" == *"/job/${JOB}"* ]] ; then
echo " Job '$JOB' is already building on '$computer' executor index '$executorIdx'"
echo ' Exiting with 1'
exit 1
fi
fi
done
done
echo ' Exiting with 0'
Builder: Set the build result
Result: Aborted
Conditional step (single)
Run?: Current build status
Builder: Trigger/call build on other projects
Build Triggers:
Projects to build: $JOB [ignore the error message]
Node Label parameter
Name: NODE [or how you call it in your downstream job(s)]
Node: $NODE
I have several (few hundreds) of files to run test on (each test takes few minutes).
Running sequentially is not acceptable and neither all together. So I am looking for something like a producer-consumer.
I tried pipeline jobs and parallel command the following way:
def files = findFiles glob: 'test_files/*'
def branches = [:]
files.each{
def test_command = "./test ${it}"
branches["${it}"] = { sh "${test_command} ${it}"}
}
stage name:'run', concurrency:2
parallel branches
Problem:
All the tasks are launch at the same time (OOM and all the fun)
Doesn't have the same introspection as the Jenkins parallel step, but since it seems not to support a fixed pool you can use xargs to achieve the same result:
def files = findFiles glob: 'test_files/*'
def branches = [:]
// there's probably a more efficient way to generate the list of commands
files.each{
sh "echo './test ${it}' >> tests.txt"
}
sh 'cat tests.txt | xargs -L 1 -I {} -P 2 bash -c "{}"'
The -P argument is the one that specifies a fixed number of 2 (or N) processes should always be running. Other tools like GNU Parallel offer even more tuning on how many processes should be used.
You can also try to use the lock step from the Lockable Resources plugin, the node step targeting a fixed number of executors. However this seems too much overhead to me unless your single tests are already taking tens of second each.
I wanted to show the user who triggered a Jenkins job in the post job email. This is possible by using the plugin Build User Vars Plugin and the env variable BUILD_USER.
But this variable do not get initialized when the job is triggered by a scheduler.
How can we achieve this? I know we have a plugin called - EnvInject Plugin, and that can be used...
But I just want to know how we can use this and achieve the solution...
Build user vars plugin wasn't working for me so I did a quick-and-dirty hack:
BUILD_CAUSE_JSON=$(curl --silent ${BUILD_URL}/api/json | tr "{}" "\n" | grep "Started by")
BUILD_USER_ID=$(echo $BUILD_CAUSE_JSON | tr "," "\n" | grep "userId" | awk -F\" '{print $4}')
BUILD_USER_NAME=$(echo $BUILD_CAUSE_JSON | tr "," "\n" | grep "userName" | awk -F\" '{print $4}')
SIMPLE SOLUTIONS (NO PLUGINS) !!
METHOD 1: Via Shell
BUILD_TRIGGER_BY=$(curl -k --silent ${BUILD_URL}/api/xml | tr '<' '\n' | egrep '^userId>|^userName>' | sed 's/.*>//g' | sed -e '1s/$/ \//g' | tr '\n' ' ')
echo "BUILD_TRIGGER_BY: ${BUILD_TRIGGER_BY}"
METHOD 2: Via Groovy
node('master') {
BUILD_TRIGGER_BY = sh ( script: "BUILD_BY=\$(curl -k --silent ${BUILD_URL}/api/xml | tr '<' '\n' | egrep '^userId>|^userName>' | sed 's/.*>//g' | sed -e '1s/\$/ \\/ /g'); if [[ -z \${BUILD_BY} ]]; then BUILD_BY=\$(curl -k --silent ${BUILD_URL}/api/xml | tr '<' '\n' | grep '^shortDescription>' | sed 's/.*user //g;s/.*by //g'); fi; echo \${BUILD_BY}", returnStdout: true ).trim()
echo "BUILD_TRIGGER_BY: ${BUILD_TRIGGER_BY}"
}
METHOD 3: Via Groovy
BUILD_TRIGGER_BY = "${currentBuild.getBuildCauses()[0].shortDescription} / ${currentBuild.getBuildCauses()[0].userId}"
echo "BUILD_TRIGGER_BY: ${BUILD_TRIGGER_BY}"
OUTPUT:
Started by user Admin / user#example.com
Note: Output will be both User ID and User Name
This can be done using the Jenkins Build User Vars Plugin which exposes a set of environment variables, including the user who started the build.
It gives environment variables like BUILD_USER_ID, EMAIL, etc.
When the build is triggered manually by a logged-in user, that user's userid is available in the BUILD_USER_ID environment variable.
However, this environment variable won't be replaced / initialized when the build is automatically triggered by a Jenkins timer / scheduler.
Attached a screenshot for details
This can be resolved by injecting a condition to the Job by using Conditional Build Step Plugin / Run Condition Plugin,where in to each job we can add a condition to initialize the variable BUILD_USER_ID only when the build is caused or triggered by the Timer or scheduler, by setting a condition using the regular expression..
Without Plugin ->
def cause = currentBuild.getBuildCauses('hudson.model.Cause$UserIdCause')
echo "userName: ${cause.userName}"
Install 'Build User Vars Plugin' and use like below:- [ See https://plugins.jenkins.io/build-user-vars-plugin ]
Be sure to check mark the Set jenkins user build variables checkbox under Build Environment for your Jenkins job's configuration.
I found similar but really working on Jenkins 2.1.x and easy for my understanding way.
And it works without any plugins.
if (currentBuild.getBuildCauses('hudson.model.Cause$UserIdCause')['userId']){
// Will be run only if someone user triggers build
// Because in other cases this contructions returns null
}
You can use in this construction any classes described here.
They will be returns maps with usable values.
This gets the username who clicked "Build Now" in a Jenkins pipeline job.
#NonCPS
def getBuildUser() {
return currentBuild.rawBuild.getCause(Cause.UserIdCause).getUserId()
}
I'm using a combination of the 'Execute Shell' and 'Env Inject' plugin as follows:
Create an 'Execute Shell' build step that uses shell parameter substitution to write default the value and echo that value into a file. Example highlighted in screen shot below.
Use the 'Env Inject' file to read that file as properties to set.
The token $BUILD_CAUSE from the email-ext plugin is what you are looking for.
You can see the full content token reference when you click the ? just after the Attach build log combobox at the email content configuration.
Some tokens get added by plugins, but this one should be aviable by default.
Edit: As pointed out by bishop in the comments, when using the EnvInject plugin, the $BUILD_CAUSE token gets changed to behave differently.
I have written a groovy script to extract the started by which would correctly get the source, regardless if user, scm or timer (could add more). It would recursively navigate the build tree to get the "original" 'started by' cause https://github.com/Me-ion/jenkins_build_trigger_cause_extractor
I wanted to trigger build initiator info to one of my slack/flock group so I used following way to get build initiator email and name by writing in Declarative fashion .
I am just printing here, you can use to store in some environment variable or write in one file giving file path according to your own convenience..
pipeline {
environment {
BRANCH_NAME = "${env.BRANCH_NAME}"
}
agent any
stages{
stage('Build-Initiator-Info'){
sh 'echo $(git show -s --pretty=%ae)'
sh 'echo $(git show -s --pretty=%an)'
}
}
}
Just to elaborate on Musaffir Lp's answer. The Conditional Build Step plugin now supports the Build Cause directly - it requires the Run Condition Plugin also.
If you wanted to detect when the build was started by a timer you can select a Run? value of Build Cause, with Build Cause of: TimerTrigger
This is a little simpler and more robust than using a regex. There are also other triggers you can detect, for example when the build was a result of Source Control Management commit, you can select: SCMTrigger.
This below is working for me.
Install "user build vars plugin"
Build Name = ${BUILD_NUMBER}_${TICKET}_${ENV,var="BUILD_USER_ID"}
I created a function that return the Triggered Job Name:
String getTriggeredJob(CURRENT_BUILD) {
if (CURRENT_BUILD.upstreamBuilds.size() > 0) {
TRIGGERED_JOB = CURRENT_BUILD.upstreamBuilds[0].projectName
if (!TRIGGERED_JOB.isEmpty()) {
return TRIGGERED_JOB
}
}
return "Self"
}
CURRENT_BUILD is env var currentBuild
How to return Username & UserId:
UserName: currentBuild.rawBuild.getCause(Cause.UserIdCause).getUserName()
UserId: currentBuild.rawBuild.getCause(Cause.UserIdCause).getUserId()
There is other way to get user_id, where you don't need to install anything.
BUILD_USER_ID = sh (
script: 'id -u',
returnStdout: true
).trim()
echo "bUILD USER: ${BUILD_USER_ID }"
For declarative pipeline syntax, here is a quick hack, base on #Kevin answer.
For declarative pipeline you need to enclose them in a node, else you will get an error/ build failure
node {
def BUILD_FULL = sh (
script: 'curl --silent '+buildURL+' | tr "{}" "\\n" | grep -Po \'"shortDescription":.*?[^\\\\]"\' | cut -d ":" -f2',
returnStdout: true
)
slackSend channel: '#ci-cd',
color: '#000000',
message: "The pipeline was ${BUILD_FULL} ${GIT_COMMIT_MSG} "
}
The output will be slack notification sent to your slack channel with the git short description