Currently I'm using Serena DIMENSIONS as configuration management with Jenkins for continuous Integration.
Once Developer check in new files in Project folder in Serena, The Jenkins job(which detect changes in Serena DIMENSIONS,download changed files and build the software) needs to be trigger with 15 minutes Delay(Delay is require to complete check in all necessary files.
Can you please tell me the solution?
With Jenkins Pipeline you can create a stage that uses sleep step. For example:
pipeline {
agent none
stages {
stage('Wait') {
agent { label 'wait-node' }
steps {
sleep time: 15, unit: "MINUTES"
}
}
}
}
There's one drawback - your executor is blocked for all the wait time. To work around this in elegant way you can define a dedicated node (wait-node) with large enough number of executors to handle waiting stages (note - other stages may run on different nodes). This way actual executors aren't blocked and you can see all waiting jobs on Jenkins Dashboard.
Related
I have a Jenkins pipeline job, which in turn starts dynamically several other pipeline jobs in parallel. At the moment I have only 1 node to execute these jobs.
All of these jobs start immediately. One of them actually does and starts its stage, and the other ones show in the log that they waiting for an executor:
Started by upstream project "parent project" build number 278
originally caused by:
Started by user user
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Still waiting to schedule task
Waiting for next available executor on win
I'd very much like to change that behavior and have them start when an executor is free. Until then they should stay in the build queue. The reason is that the durations will get shown all wrong, and in the stage view they all start showing "almost complete" when they haven't actually started.
This is the start of the child projects pipeline script, I do require a matching agent immediately:
pipeline {
agent {
label "win"
}
stages {
stage('[Jenkins] Setup') {
Is this achievable somehow? Would I need to switch to scripted pipeline maybe?
You can use throttle concurrent builds plugin and configure the properties "Maximum Total Concurrent Builds" in your job to required number. This way it will prevent the builds from running if it exceeds than the number you configured and you can see the builds in queue.
I was reading about the best practices of a Jenkins pipeline.
I have created a declarative pipeline which is not executing parallel jobs and I want to run everything on the same slave.
I use:
agent {
label 'xxx'
}
The rest of my pipeline looks like:
pipeline {
agent {
label 'xxx'
}
triggers {
pollSCM pipelineParams.polling
}
options {
buildDiscarder(logRotator(numToKeepStr: '3'))
}
stages {
stage('stage1') {
steps {
xxx
}
}
stage('stage2') {
steps {
xxx
}
}
}
post {
always {
cleanWs()
}
failure {
xxx"
}
success {
xxx
}
}
}
Now I read the best practices here.
Point 4 is telling:
Do: All Material Work Within a Node
Any material work within a pipeline should occur within a node block.
Why? By default, the Jenkinsfile script itself runs on the Jenkins
master, using a lightweight executor expected to use very few
resources. Any material work, like cloning code from a Git server or
compiling a Java application, should leverage Jenkins distributed
builds capability and run an agent node.
I suspect this is for scripted pipelines.
Now my questions are:
Do I ever have to create a node inside a stage in a declarative pipeline (it is possible) or do I have to use agent inside the stage when I want to run my stage on another specific agent?
My current pipeline has defined a label which is on 4 agents. But my whole pipeline is always executed on one agent (what I want) but I would suspect it's executing stage1 on slaveX and maybe stage2 on slaveY. Why is this not happening?
The documentation is quite misleading.
What the documentation is suggesting is to take advantage of distributed builds. Distributed builds activated either by using the agent or node block.
The agent should be used when you want to run the pipeline almost exclusively on one node. The node block allows for more flexibilty as it allows you to specify where a granular task should be executed.
If you running the pipeline on some agent and you encapsulate a step with node with the same agent, there won't be any effect execpt that a new executor will be allocated to the step encapsulated with node.
There is no obvious benefit in doing so. You will simply be consuming executors that you don't need.
In conclusion, you are already using distributed builds when using agent and this is what the documentation is vaguely recommending.
I have a BitBucket organization job configured in my Jenkins, which is configured to scan the whole organization every 20 minutes and if it identifies a commit in any of the organization's repositories it triggers an automatic build.
Sometimes, more than one branch is being changed at a certain time and this causes Jenkins to trigger more than one build of the same project.
One of these projects should never allow concurrent builds as it uses resources which are being locked when a build runs, this causes other branches where commits are being pushed to trigger but they always fail because their main resource is locked by the first instance of the build.
I'm aware to the Throttle Builds plugin and it looks perfect for freestyle/pipeline jobs but in the case of organization scanning I cannot configure anything in the repositories under the organization, just the organization itself, the same goes for Hudson Locks and Latches plugin.
Anyone knows any solution?
I had a similar problem, and wanted to make sure that each branch of my multibranch pipeline could only execute one build at a time. here's what I added to my pipeline script:
pipeline {
agent any
options {
disableConcurrentBuilds() //each branch has 1 job running at a time
}
...
...
}
https://jenkins.io/doc/book/pipeline/syntax/#options
[Update 09/30/2017]
You may also want to check out the lock & milestone steps of Declarative Pipeline.
Lock
Rather than attempt to limit the number of concurrent builds of a job using the stage, we now rely on the "Lockable Resources" plugin and the lock step to control this. The lock step limits concurrency to a single build and it provides much greater flexibility in designating where the concurrency is limited.
stage('Build') {
doSomething()
lock('myResource') {
echo "locked build"
}
}
Milestone
The milestone step is the last piece of the puzzle to replace functionality originally intended for stage and adds even more control for handling concurrent builds of a job. The lock step limits the number of builds running concurrently in a section of your Pipeline while the milestone step ensures that older builds of a job will not overwrite a newer build.
stage('Build') {
milestone()
echo "Building"
}
stage('Test') {
milestone()
echo "Testing"
}
https://jenkins.io/blog/2016/10/16/stage-lock-milestone/
https://jenkins.io/doc/pipeline/steps/pipeline-milestone-step/
https://jenkins.io/doc/pipeline/steps/lockable-resources/
I'm new to Jenkins and I'm trying to setup a project which will use few build executors.
The flow shall be as follows:
two build executors with webservice label return their IP addresses and wait for the third build executor to finish its job
third build executor with tester label collects those IP addresses and performs some long running job (e.x. sends HTTP requests to the webservices deployed on those two agents)
How to achieve such behavior in Jenkins?
I've found that when an build executor finishes its job it is immediately released and I don't know how to make it wait for other build executors to finish their jobs.
Edit:
I forgot to mention that I want the build executors with the webservice label to be reserved (not available for other jobs) till the build executor with the tester label will finish its long-running job.
Also all these build executors should be on separate slaves each. That means each slave has only one build executor.
I've finally managed to do this using Pipeline and below script:
node('webservice') {
def firstHostname = getHostname()
node('webservice') {
def secondHostname = getHostname()
node('tester') {
println 'Running tests against ' + firstHostname + ' and ' + secondHostname
// ...
}
}
}
def getHostname() {
sh 'hostname > output'
readFile('output').trim()
}
It acquires two build executors with webservice label. I'm getting their hostnames (I'm using them instead of the IP addresses) and pass them to the build executor with a tester label. Finally the tester runs some long-running tests.
Those two webservice build executors are blocked till the tester finishes its job, and no other project may use them during that time.
As Alex O mentioned, you can configure the master and slave relationship between the projects /executors inside the Jenkins projects /executors. There is option for that, "Build Triggers" -> Build after other projects are built
or use plugin to achieve it
https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin
or
https://wiki.jenkins-ci.org/display/JENKINS/Join+Plugin
What you actually want is probably that your job uses three slaves at the same time.
Re-thinking the setup in that way, it won't be necessary to consider the collection of IPs and the subsequent usage of the slaves as three different steps that must be aligned in some way.
Unfortunately, Jenkins does not support using multiple slaves for one build out-of-the box, but it will be possible to achieve what you want e.g. using the Multijob plugin and the Join plugin that Aaron mentioned already.
See also this question for information on how to use two slaves at the same time.
I have many Jenkins Jobs that I need to run on every Build,
At present time I have 4 slave servers.
I would like the jobs to run in parallel as much as possible, hence I defined the jobs as follow:
Execute concurrent builds if necessary - Disabled
Restrict where this project can be run - Enabled with the following values SalveLinux1HT||SalveLinux2HT||SalveLinux3HT||SalveLinux4HT
To my understanding if Job A and B are triggered at the same time, one should use 1HT and the other should use 2HT and they can run in parallel
however Jenkins build job A on all 4 slaves and only after it's finished he will build job B on all 4 slaves
This is the opposite of my goal
Any ideas?
Thanks in advance
You can use
Build Flow Plugin
You can find both installation and configuration instructions of this plugin at the above mentioned link.
If you want to run any jobs in parallel you can use following scripts:
parallel (
// job A and B will be scheduled in parallel.
{ build("jobA") },
{ build("jobB") }
)
// jobC will be triggered after jobs A and B are completed
build("jobC")