I'm trying to configure a pipeline with these steps:
Build
Copy artifacts
Integration test
I want them to be executed in this specific order on the same agent.
My setup is the server and one agent.
If I do this configuration
pipeline {
agent ???
options {
copyArtifactPermission(...);
}
stages
{
stage('BuildStage')
{
agent {label 'agent1'}
steps
{
build( [...] )
}
}
stage('Copy artifacts')
{
agent {label 'agent1'}
steps
{
copyArtifacts ...
}
}
stage('Integration Test')
{
agent {label 'agent1'}
steps
{
build( [...] )
}
}
}
}
It seems that the steps are not scheduled because the master node is busy executing the pipeline.
I'm thinking that I should force the master to execute the pipeline (hence the agent ???) and each stage on the agent?
I can get it to work by adding 'wait:false' to each step, but then I'm not sure the order is kept?
I have also set number of executors to 2, but it doesn't seem to solve the problem.
I'm coming from TeamCity, so maybe I've just misunderstood something basic about Jenkins?
If all the steps are meant to be run in the same agent, you can set it only once:
pipeline {
agent { label 'agent1' }
...
// Remove all other 'agent'
}
Don`t worry about the stages order (quote from https://www.jenkins.io/doc/book/pipeline/syntax/#sequential-stages)
Stages in Declarative Pipeline may have a stages section containing a
list of nested stages to be run in sequential order.
Related
i want to run two pipelines both on different agents to make both jobs run parallely. But i want one stage to execute asynchronously. I have added a global agent like this:
pipeline {
agent { label 'windows' }
stages {
stage('Initialize Pipeline') {
steps {
cleanWs()
printSuccess("Pipeline Started.")
callWebhook("Running","Pipeline Started.")
}
}
this label refers to two agents (Windows_Agent and Windows_2019)...either of which will run each of the two jobs.
But i want this stage to specifically run on a single agent:
stage('Remove tenant URL from maintenance window Cloudfront'){
agent { label 'Windows_2019' }
steps{
this stage picks up cloudfront distribution list and each pipeline removes a CNAME from it. this results in concurrency errors when this stage is executed parallely. Instead only this stage needs to execute on a single agent such that the other job executes when the specified agent Windows_2019 becomes available.
This should work
we are selecting agent for each stage.
pipeline {
agent none
stages {
stage('Initialize Pipeline') {
agent { label 'windows' }
steps {
cleanWs()
printSuccess("Pipeline Started.")
callWebhook("Running","Pipeline Started.")
}
stage('Remove tenant URL from maintenance window Cloudfront'){
agent { label 'Windows_2019' }
steps{}
}
}
}
I have a Jenkins declarative pipeline (Jenkins version 2.138.3)
On Jenkins screen why do i see that the same stage say 'compile' is being executed on the same executor....?
Image attached
Example: pipeline named 'multi-branch-pipeline-1' running on agent called 'agent-1' with three executors...here the stage 'stage-promotion' is being executed on the two different executors (2,3).
pipeline {
agent {label 'agent-1'}
stages{
stage('compile') {
agent {label 'agent-1'}
}
stage('stage-promotion') {
agent {label 'agent-1'}
}
}
}
Afaik, nested agent declarations work that way, the outer will keep its executor allocated and usable while the inner is running.
Since you use the same label (or rather direct addressing with agent name) it amounts to a 2nd executor on the same agent, there seems to be no logic for this rather unusual case, as you could just omit the agent declaration that amount to the same label.
Following will do the same as yours:
pipeline {
agent {label 'agent-1'}
stages{
stage('compile') {
// runs on agent-1
}
stage('stage-promotion') {
// runs on agent-1
}
}
}
Nested agents are very useful when you want to temporarily switch the machine in your pipeline:
pipeline {
agent {label 'A'}
stages{
stage('start server') {
// runs on machine x with label A
}
stage('test') {
agent {label 'B'}
// runs on machine y with label B
}
stage('stop server and archive logs') {
// runs on **the same machine as in stage start server**, same workspace etc.
}
}
}
Important part is, in the last stage, we can be sure to be on the same machine in the same workspace with no wait-time (i.e. executor contention) as in the first stage. If you use agent declarations inside the stages only, it could happen that we end up in a different machine than in the first stage if you have more than 1 agent with label A connected.
At the pipeline level I specify the agent and node (with both the label and custom workspace). When the pipeline kicks off it runs on the specified node, but when it hits the 'build job' picks the first available node. I tried playing with the NodeLabel plugin, but that didn't work either.
This is my Jenkinsfile:
pipeline {
agent {
node {
label "Make Build Server"
customWorkspace "$Workspace"
}
}
options {
skipDefaultCheckout()
}
stages {
stage('PreBuild'){
steps{
input 'Did you authenticate the server through all the firewalls?'
}
}
stage('Housekeeping'){
steps{
build job: 'Housekeeping'
}
}
}
}
When you use the build instruction in a Jenkinsfile, it's telling jenkins you want to build a completely separate job. It is that other job that will need to specify on what agent it will build. If it's a job based on a Jenkinsfile, then that other Jenkinsfile will indicate the agent. If it is a freestyle job, likewise. So the thing you were expecting--that the other job build on the agent you specified in the "parent Jenkinsfile"--is reasonable, but is not the way it works.
Hope this helps!
We use the jenkins pipeline dsl for our job descriptions. Now we have something like that:
pipeline {
agent none
options {
timestamps()
}
environment {
//SOME ENV VARS
}
stages {
// more stages
stage('stage1'){
when {
expression { env.BRANCH_NAME == 'master' }
}
agent { label 'slave1' }
steps{
//doing stuff
}
}
}
A stage in the build process that should only run when the master branch is build, you can think of a deploy job or something in that direction. The problem is, our resources of agents with that particular label are limited. When we build other branches the job still invoke a slave1 agent and than skips the stage after checking the that the branch is not master. This is bad, because when all slave1 agents are working on master branch jobs, the other jobs will have to wait for a slave1 agent becoming available just to check that it does need to run that stage.
Is there any way with the jenkins pipeline DSL to skip that stage without waiting for the slave1 agent to determine the branch?
The approach provided in the accepted answer works because you are not declaring an agent in the stage('stage1'). Instead you explicitly create a node within the steps and so the agent is not created when you check the condition.
This approach will work but it adds unnecessary complexity to your Jenkinsfile. As of Jenkins pipeline plugin version 1.2.6, the correct way to achieve this would be:
pipeline {
agent none
stages {
// more stages
stage('stage1'){
when {
beforeAgent true
branch 'master'
}
agent { label 'slave1' }
steps {
// do stuff
}
}
}
}
Check the syntax and available options available in the when tag documentation.
I found a solution that worked for me. Although I'm not quite sure why.
Introducing a parallel section and using nodes however fixed the issue and the stages are skipped without invoking the agent first. See the modified pipeline:
pipeline {
agent none
options {
timestamps()
}
environment {
//SOME ENV VARS
}
stages {
// more stages
stage('stage1'){
when { branch 'master' }
steps{
parallel(
'Job1': {
node( 'slave1' ){
//doing stuff
}
}
)
}
}
}
I'm trying to create my first Groovy script for Jenkins:
After looking here https://jenkins.io/doc/book/pipeline/, I created this:
node {
stages {
stage('HelloWorld') {
echo 'Hello World'
}
stage('git clone') {
git clone "ssh://git#mywebsite.example/myrepo.git"
}
}
}
However, I'm getting:
java.lang.NoSuchMethodError: No such DSL method "stages" found among steps
What am I missing?
Also, how can I pass my credentials to the Git Repository without writing the password in plain text?
You are confusing and mixing Scripted Pipeline with Declarative Pipeline, for complete difference see here. But the short story:
declarative pipelines is a new extension of the pipeline DSL (it is basically a pipeline script with only one step, a pipeline step with arguments (called directives), these directives should follow a specific syntax. The point of this new format is that it is more strict and therefor should be easier for those new to pipelines, allow for graphical editing and much more.
scripted pipelines is the fallback for advanced requirements.
So, if we look at your script, you first open a node step, which is from scripted pipelines. Then you use stages which is one of the directives of the pipeline step defined in declarative pipeline. So you can for example write:
pipeline {
...
stages {
stage('HelloWorld') {
steps {
echo 'Hello World'
}
}
stage('git clone') {
steps {
git clone "ssh://git#mywebsite.example/myrepo.git"
}
}
}
}
So if you want to use declarative pipeline that is the way to go.
If you want to scripted pipeline, then you write:
node {
stage('HelloWorld') {
echo 'Hello World'
}
stage('git clone') {
git clone "ssh://git#mywebsite.example/myrepo.git"
}
}
E.g.: skip the stages block.
A Jenkinsfile can be written using two types of syntax - Declarative and Scripted.
Declarative and Scripted Pipelines are constructed fundamentally differently. Declarative Pipeline is a more recent feature of Jenkins Pipeline which:
provides richer syntactical features over Scripted Pipeline syntax, and
is designed to make writing and reading Pipeline code easier.
Many of the individual syntactical components (or "steps") written into a Jenkinsfile, however, are common to both Declarative and Scripted Pipeline.
Example:
Declarative Pipeline fundamentals
In Declarative Pipeline syntax, the pipeline block defines all the work done throughout your entire Pipeline.
Jenkinsfile (Declarative Pipeline):
pipeline {
agent any 1
stages {
stage('Build') { 2
steps {
// 3
}
}
stage('Test') { 4
steps {
// 5
}
}
stage('Deploy') { 6
steps {
// 7
}
}
}
}
Execute this Pipeline or any of its stages, on any available agent.
Defines the "Build" stage.
Perform some steps related to the "Build" stage.
Defines the "Test" stage.
Perform some steps related to the "Test" stage.
Defines the "Deploy" stage.
Perform some steps related to the "Deploy" stage.
Scripted Pipeline fundamentals
In Scripted Pipeline syntax, one or more node blocks do the core work throughout the entire Pipeline. Although this is not a mandatory requirement of Scripted Pipeline syntax, confining your Pipeline's work inside of a node block does two things:
Schedules the steps contained within the block to run by adding an item to the Jenkins queue. As soon as an executor is free on a node, the steps will run.
Creates a workspace (a directory specific to that particular Pipeline) where work can be done on files checked out from source control.
Caution: Depending on your Jenkins configuration, some workspaces may not get automatically cleaned up after a period of inactivity. See tickets and discussion linked from JENKINS-2111 for more information.
Jenkinsfile (Scripted Pipeline):
node { 1
stage('Build') { 2
// 3
}
stage('Test') { 4
// 5
}
stage('Deploy') { 6
// 7
}
}
Execute this Pipeline or any of its stages, on any available agent.
Defines the "Build" stage. stage blocks are optional in Scripted Pipeline syntax. However, implementing stage blocks in a Scripted Pipeline provides clearer visualization of each `stage's subset of tasks/steps in the Jenkins UI.
Perform some steps related to the "Build" stage.
Defines the "Test" stage. 5
Perform some steps related to the "Test" stage.
Defines the "Deploy" stage.
Perform some steps related to the "Deploy" stage.
Pipeline example
Here is an example of a Jenkinsfile using Declarative and it's equivalent scriptive Pipeline syntax:
Jenkinsfile (Declarative Pipeline):
pipeline {
agent any
options {
skipStagesAfterUnstable()
}
stages {
stage('Build') {
steps {
sh 'make'
}
}
stage('Test'){
steps {
sh 'make check'
junit 'reports/**/*.xml'
}
}
stage('Deploy') {
steps {
sh 'make publish'
}
}
}
}
Jenkinsfile (Scripted Pipeline):
node {
stage('Build') {
sh 'make'
}
stage('Test') {
sh 'make check'
junit 'reports/**/*.xml'
}
if (currentBuild.currentResult == 'SUCCESS') {
stage('Deploy') {
sh 'make publish'
}
}
}