Here is an example of declarative pipeline where the agent is set for the pipeline but not set in the individual stages:
pipeline {
agent { node { label 'linux' } }
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage('Build') {
steps {
sh 'make'
}
}
}
}
Documentation I've found about scripted pipeline makes it clear that a single workspace will be used within a single node block but multiple node blocks might be allocated multiple workspaces, therefore it is necessary to stash between those steps, use the External Workspace Plugin, etc. if you want to be sure of what's in the workspace between steps.
I had a hard time finding documentation about workspace guarantees for declarative pipeline. What guarantees about workspaces exist for this example?
I believe I encountered two stages executing in different workspaces during testing of a similar pipeline but I'm not sure that's what was happening. I'd really like to avoid needing to stash my checkout prior to my build step or use the External Workspace plugin so I was hoping there'd be a way to force all my stages to run all in one workspace/on one node.
The Pipeline code presented should only create a single workspace and run all stages in it. Unless you create a new agent directive in any of your stages it will not utilize another node or workspace.
btw, checkout scm happens automatically at the beginning of the Pipeline with Declarative so you don't need to explicitly call that out.
i'm 70% sure--based on anecdotal evidence--that you will always get the same workspace on the same node in different stages of a declarative pipeline if you specify a node at the top level and never override it, the way you're doing.
i reserve the right to adjust my confidence level as i receive feedback on this answer. :D
Related
I was reading about the best practices of a Jenkins pipeline.
I have created a declarative pipeline which is not executing parallel jobs and I want to run everything on the same slave.
I use:
agent {
label 'xxx'
}
The rest of my pipeline looks like:
pipeline {
agent {
label 'xxx'
}
triggers {
pollSCM pipelineParams.polling
}
options {
buildDiscarder(logRotator(numToKeepStr: '3'))
}
stages {
stage('stage1') {
steps {
xxx
}
}
stage('stage2') {
steps {
xxx
}
}
}
post {
always {
cleanWs()
}
failure {
xxx"
}
success {
xxx
}
}
}
Now I read the best practices here.
Point 4 is telling:
Do: All Material Work Within a Node
Any material work within a pipeline should occur within a node block.
Why? By default, the Jenkinsfile script itself runs on the Jenkins
master, using a lightweight executor expected to use very few
resources. Any material work, like cloning code from a Git server or
compiling a Java application, should leverage Jenkins distributed
builds capability and run an agent node.
I suspect this is for scripted pipelines.
Now my questions are:
Do I ever have to create a node inside a stage in a declarative pipeline (it is possible) or do I have to use agent inside the stage when I want to run my stage on another specific agent?
My current pipeline has defined a label which is on 4 agents. But my whole pipeline is always executed on one agent (what I want) but I would suspect it's executing stage1 on slaveX and maybe stage2 on slaveY. Why is this not happening?
The documentation is quite misleading.
What the documentation is suggesting is to take advantage of distributed builds. Distributed builds activated either by using the agent or node block.
The agent should be used when you want to run the pipeline almost exclusively on one node. The node block allows for more flexibilty as it allows you to specify where a granular task should be executed.
If you running the pipeline on some agent and you encapsulate a step with node with the same agent, there won't be any effect execpt that a new executor will be allocated to the step encapsulated with node.
There is no obvious benefit in doing so. You will simply be consuming executors that you don't need.
In conclusion, you are already using distributed builds when using agent and this is what the documentation is vaguely recommending.
I have a CD pipeline that requires user confirmation at some stages, so I would like to free up server resources while the pipeline is waiting for the user input.
pipeline {
agent any
stages {
stage ('Build Stage') {
steps {
...
}
}
stage ('User validation stage') {
agent none
steps {
input message: 'Are you sure you want to deploy?'
}
}
stage ('Deploy Stage') {
steps {
...
}
}
}
}
You can see above that I have a global agent any but in the User Validation Stage I added agent none.
Can someone confirm that this doing what I want (no agent/node is waiting for the user input)? I don't see how to verify it, nothing different in the execution log...
If not, how could I do it?
This will not work as you expect. You cannot specify agent any on the entire pipeline and then expect agent none to not occupy the executor.
To prove this, you can run this code as you have it, and while it is waiting at the input stage, go to your main jenkins page and look at the Build Executor Status. You will see there is an executor still running your job.
Next, switch your pipeline to agent none and add agent any to all the other steps (besides your input step) and do the same test. You can see that while waiting at the input stage, none of the executors are occupied.
As to your question about different workspaces on different nodes... Assuming you are using code from an SCM, it will be checked out on every new node, so that isn't a concern. The only thing you need to worry about is artifacts you created in each stage.
It is not safe to "hope" that you will stay on the same node, though Jenkins will "try" to keep you there. But even then, there is not a guarantee that you will get the same workspace directory.
The correct way to handle this is to stash all the files that you may have created or modified that you will need in later stages. Then in the following stages, unstash the required files. Never assume files will make it between stages that have their own node declaration.
I am investigating the use of Jenkins Pipeline (specifically using Jenkinsfile). The context of my implementation is that I'm deploying a Jenkins instance using Chef. Part of this deployment may include some seed jobs, which will pull job configurations from source control (Jenkinsfile), to automate creation of our build jobs via Chef.
I've investigated the Jenkins documentation for both Pipeline as well as Jenkinsfile, and it seems to me that in order to use Jenkins Pipeline agents are required to be configured and set up in addition to Jenkins Master.
Am I understanding this correctly? Must Jenkins agents exist in order to use Jenkins Pipeline's Jenkinsfile? This specific line in the Jenkinsfile documentation leads me to believe this to be true:
Jenkinsfile (Declarative Pipeline)
pipeline {
agent any
stages {
stage('Build') {
steps {
echo 'Building..'
}
}
stage('Test') {
steps {
echo 'Testing..'
}
}
stage('Deploy') {
steps {
echo 'Deploying....'
}
}
}
}
The Declarative Pipeline example above contains the minimum necessary
structure to implement a continuous delivery pipeline. The agent
directive, which is required, instructs Jenkins to allocate an
executor and workspace for the Pipeline.
Thanks in advance for any Jenkins guidance!
The 'agent' part of the pipeline is required however this does not mean that you are required to have an external agent in addition to your master. If all you have is the master this pipeline will execute on the master. If you have additional agents available the pipeline would execute on whichever agent happens to be available when you run the pipeline.
If you go into
Manage Jenkins -> Manage Nodes and Clouds, you can see 'Master' itself is treated as one of the Default Nodes. With declarative format agent anyindicates any available agent which (including 'Master' as well from node configuration see below).
In case if you configure any New node, this can then be treated as New Agent in the pipeline agent any can be replaced by agent 'Node_Name'
You may can refer this LINK which give hint on Agent, Node and Slave briefly.
I'm trying to understand how to structure my Jenkins 2.7 pipeline groovy script. I've read through the pipeline tutorial, but feel that it could expand more on these topics.
I can understand that a pipeline can have many stages and each stage can have many steps. But what is the difference between a step(); and a method call inside a stage, say sh([script: "echo hello"]);. Should nodes be inside or outside of stages? Should the overall properties of a job be inside or outside a node?
Here is my current structure on an ubuntu master node:
#!/usr/bin/env groovy
node('master') {
properties([
[$class: 'BuildDiscarderProperty', strategy: [$class: 'LogRotator', numToKeepStr: '10']]
]);
stage 'Checkout'
checkout scm
stage 'Build'
sh([script: "make build"]);
archive("bin/*");
}
The concepts of node, stage and step are different:
node specifies where something shall happen. You give a name or a label, and Jenkins runs the block there.
stage structures your script into a high-level sequence. Stages show up as columns in the Pipeline Stage view with average stage times and colours for the stage result.
step is one way to specify what shall happen. sh is of a similar quality, it is a different kind of action. (You can also use build for things that are already specified as projects.)
So steps can reside within nodes, (if they don't, they are executed on the master), and nodes and steps can be structured into an overall sequence with stages.
It depends. Any node declaration allocates an executor (on Jenkins master or slave). This requires that you stash and unstash the workspace, as another executor does not have the checked out sources available.
Several steps of the Pipeline DSL run in a flyweight executor and thus do not require to be inside a node block.
This might be helpful for an example such as the following, where you need to allocate multiple nodes anyway:
stage("Checkout") {
checkout scm
}
stage("Build") {
node('linux') {
sh "make"
}
node('windows') {
bat "whatever"
}
}
stage("Upload") {
...
Another (maybe more realistic example) would be to allocate multiple nodes in parallel. Then there's no need to let the stage call be executed in another allocated executor (aka within node).
Your example looks good to me. There's no need to allocate multiple nodes within the single stages, as this would be only additional overhead.
Now that the Multibranch Pipeline job type has matured, is there any reason to use the simple Pipeline job type any longer? Even if you only have one branch today, it's probably wise to account for the possibility of multiple branches in the future, so what would the motivation be to use the Pipeline job type for your Jenkins Pipeline vs. always using the Multibranch Pipeline job type, assuming you are storing your Jenkinsfile in SCM? Is there feature parity between the two job types now?
In my experience with multibranch pipelines, the ONLY downside is that you can't see the last success/failure/duration columns on the Jenkins main page. They just show "NA" on the Jenkins front page since it's technically a 'folder' of sub-jobs.
Other than that I can't think of any other "cons" to using multibranch.
I disagree with the other answer.... that case was that multibranch sends changes for "any" branch. That's not necessarily true. If a Jenkinsfile exists on a random feature branch, but that branch is not defined in the pipeline then you can just not do anything with it using typical if/else conditionals.
For example:
node {
checkout scm
def workspace = pwd()
if (env.BRANCH_NAME == 'master') {
stage ('Some Stage 1 for master') {
sh 'do something'
}
stage ('Another Stage for Master') {
sh 'do something else here'
}
}
else if (env.BRANCH_NAME == 'stage') {
stage ('Some stage branch step') {
sh 'do something'
}
stage ('Deploy to stage target') {
sh 'do something else'
}
}
else {
sh 'echo "Branch not applicable to Jenkins... do nothing"'
}
}
Multibranch Pipeline works well if your Jenkins job deals with a single git repository. On the other hand, the pipeline job can be repository-neutral and branch-neutral and very flexible when working with multiple git repositories with a single Jenkins job.
For example, assume you have artifact-1 from repo-1, artifact-2 from repo-2, and integration tests from repo-3. And artifact-2 depends on artifact-1. A Jenkins job has to build artifact-1, then build artifact-2, and finally run integration tests from repo-3. And assume your code change goes to a feature-1 branch of repo-1 and feature-1 branch for new tests in repo-3. In this case, the Jenkins job builds feature-1 for artifact-1, then uses 'dev' branch as default from repo-2 (if feature-1 is not detected in repo-2), and runs 'feature-1' from repo-3 for new integration tests. As you can see, the job works well with three git repositories. A repo-neutral/branch-neutral pipeline job is ideal in this setting.
In a CI/CD situation, it may not be desirable to send every branch to the target environment. Using pipeline and specifying a single branch would allow you to filter, and send only /master to Staging or Production environments. Multibranch would be useful for sending any change on any branch specifically to a test environment.
On the other hand, if the QA/AutomatedTesting process is thorough enough, the risk with sending any branch to Production could be acceptable.
If you are still developing your flow, the simple pipeline has the added advantage of supporting parameterized projects. This feature is useful for developing the declarative pipelines in the jenkins gui, using the parameter to control what branch/repository you are targeting.