I am trying to have Jenkins pipline job run only using specific agents. For reasons I can not use any and I would prefer not to only use one agent via a label. The agents are named like: abc-a, abc-b, abc-c, xyz-a, xyz-b, xyz-c.
I would like only run the job using agents that are contain abc in their name. I was looking for something like:
pipeline {
agent {
contains: 'abc'
}
stages {
stage ("Example") {
steps{
sh 'really cool bash'
}
}
}
}
I read though the documentation here Jenkins Pipline but the best thing I can find is using a label, but I could be over looking something. Any ideas?
Related
I have multiple pipelines using Jenkinsfiles that retrieve a docker image from a private registry. I would like to be able to load the docker specific information into the pipelines from a file, so that I don’t have to modify all of my Jenkinsfiles when the docker label or credentials change. I attempted to do this using the example jenkinsfile below:
def common
pipeline {
agent none
options {
timestamps()
}
stages {
stage('Image fetch') {
steps{
script {
common = load('/home/jenkins/workspace/common/docker_image')
common.fetchImage()
}
}
}
}
With docker_image containing:
def fetchImage() {
agent {
docker {
label “target_node ”
image 'registry-url/image:latest'
alwaysPull true
registryUrl 'https://registry-url’
registryCredentialsId ‘xxxxxxx-xxxxxx-xxxx’
}
}
}
I got the following error when I executed the pipeline:
Required context class hudson.FilePath is missing Perhaps you forgot
to surround the code with a step that provides this, such as:
node,dockerNode
How can I do this using a declarative pipeline?
There are a few issues with this:
You can allocate a node only at top level
pipeline {
agent ...
}
Or you can use a per-stage node allocation like so:
pipeline {
agent none
....
stages {
stage("My stage") {
agent ...
steps {
// run my steps on this agent
}
}
}
}
You can check the docs here
The steps are supposed to be executed on the allocated node (or in some cases they can be executed without allocating a node at all).
Declarative Pipeline and Scripted Pipeline are two different things. Yes, it's possible to mix them, but scripted pipeline is meant to either abstract some logic into a shared library, or to provide you a way to be a "hard core master ninja" and write your own fully custom pipeline using the scripted pipeline and none of the declarative sugar.
I am not sure how your Docker <-> Jenkins connection is setup, but you would probably be better if you install a plugin and use agent templates to provide the agents you need.
If you have a Docker Swarm you can install the Docker Swarm Plugin and then in your pipeline you can just configure pipeline { agent { label 'my-agent-label' } }. This will automatically provision your Jenkins with an agent in a container which uses the image you specified.
If you have exposed /var/run/docker.sock to your Jenkins, then you could use Yet Another Docker Plugin, which has the same concept.
This way you can remove the agent configuration into the agent template and your pipeline will only use a label to have the agent it needs.
I was reading about the best practices of a Jenkins pipeline.
I have created a declarative pipeline which is not executing parallel jobs and I want to run everything on the same slave.
I use:
agent {
label 'xxx'
}
The rest of my pipeline looks like:
pipeline {
agent {
label 'xxx'
}
triggers {
pollSCM pipelineParams.polling
}
options {
buildDiscarder(logRotator(numToKeepStr: '3'))
}
stages {
stage('stage1') {
steps {
xxx
}
}
stage('stage2') {
steps {
xxx
}
}
}
post {
always {
cleanWs()
}
failure {
xxx"
}
success {
xxx
}
}
}
Now I read the best practices here.
Point 4 is telling:
Do: All Material Work Within a Node
Any material work within a pipeline should occur within a node block.
Why? By default, the Jenkinsfile script itself runs on the Jenkins
master, using a lightweight executor expected to use very few
resources. Any material work, like cloning code from a Git server or
compiling a Java application, should leverage Jenkins distributed
builds capability and run an agent node.
I suspect this is for scripted pipelines.
Now my questions are:
Do I ever have to create a node inside a stage in a declarative pipeline (it is possible) or do I have to use agent inside the stage when I want to run my stage on another specific agent?
My current pipeline has defined a label which is on 4 agents. But my whole pipeline is always executed on one agent (what I want) but I would suspect it's executing stage1 on slaveX and maybe stage2 on slaveY. Why is this not happening?
The documentation is quite misleading.
What the documentation is suggesting is to take advantage of distributed builds. Distributed builds activated either by using the agent or node block.
The agent should be used when you want to run the pipeline almost exclusively on one node. The node block allows for more flexibilty as it allows you to specify where a granular task should be executed.
If you running the pipeline on some agent and you encapsulate a step with node with the same agent, there won't be any effect execpt that a new executor will be allocated to the step encapsulated with node.
There is no obvious benefit in doing so. You will simply be consuming executors that you don't need.
In conclusion, you are already using distributed builds when using agent and this is what the documentation is vaguely recommending.
At the pipeline level I specify the agent and node (with both the label and custom workspace). When the pipeline kicks off it runs on the specified node, but when it hits the 'build job' picks the first available node. I tried playing with the NodeLabel plugin, but that didn't work either.
This is my Jenkinsfile:
pipeline {
agent {
node {
label "Make Build Server"
customWorkspace "$Workspace"
}
}
options {
skipDefaultCheckout()
}
stages {
stage('PreBuild'){
steps{
input 'Did you authenticate the server through all the firewalls?'
}
}
stage('Housekeeping'){
steps{
build job: 'Housekeeping'
}
}
}
}
When you use the build instruction in a Jenkinsfile, it's telling jenkins you want to build a completely separate job. It is that other job that will need to specify on what agent it will build. If it's a job based on a Jenkinsfile, then that other Jenkinsfile will indicate the agent. If it is a freestyle job, likewise. So the thing you were expecting--that the other job build on the agent you specified in the "parent Jenkinsfile"--is reasonable, but is not the way it works.
Hope this helps!
I am building Jenkins build pipeline and I was wondering if it is possible to somehow tag/visualize the build branch in Jenkins in the similar way as it is automatically possible in TeamCity.
I am using declarative pipeline defined in separate git repository and Jenkins 2.46.3.
From the picture it is not obvious that the last 2 builds were executed on a separate branch:
Thanks
You can modify the current build's display name and description using the following code:
currentBuild.displayName = env.BRANCH_NAME
currentBuild.description = 'Final Release'
This was recently highlighted in the BlueOcean 1.1 announcement, which shows both of them, in contrast to the regular interface, which only shows the displayName.
An example of a modified displayName from our public instance looks as follows:
You can find the code which generates this in our shared library here and here, essentially it is:
currentBuild.displayName = "#${currentBuild.getNumber()} - ${newVersion} (${increment})"
As you are mentioning Declarative Pipelines, let add that you have to wrap this code in a script block, of course. So probably (untested):
pipeline {
agent any
stages {
stage('Example') {
steps {
echo 'Hello World'
script {
currentBuild.displayName = env.BRANCH_NAME
}
}
}
}
}
Alternatively, you can extract it into a separate function.
Background
Let's say I have two jobs, one 'Pipeline Job' and one 'Build Job'. The 'Pipeline Job' runs on master and is of course a pipeline (using groovy). Then for a build part in the pipeline I use a slave running on Windows, the 'Build Job', which is responsible for building something I can't do on the master. The master is also running on Windows but lack some software needed for the specific build.
The Question
I have a groovy script that looks something like this:
#!groovy
node {
stage('Environment Preparation') {
// fetches stuff and sets up the environment on master
}
stage('Unit Testing') {
// some testing
}
stage('Build on Slave') {
def slaveJob = build job: 'BuildJob'
}
}
It works fine, where 'BuildJob' is "Restrict where this project can be run", i.e., on the slave.
My issue is that I want the output from 'BuildJob' to print in the pipeline logs. Do you have some clever ways of how this could be done? I'm open for everything, so if you know of more clever ways to start the 'BuildJob' etc. I'm eager to here it.
Thanks!
EDITED
You have to approve the things you want to access under script-approval. Not sure if you really neeed getRawBuild but it worked.
Search through console output of a Jenkins job
#!groovy
node {
stage('Environment Preparation') {
// fetches stuff and sets up the environment on master
}
stage('Unit Testing') {
// some testing
}
stage('Build on Slave') {
def slaveJob = build job: 'BuildJob'
println slaveJob.rawBuild.log
}
}
jenkinsurl/scriptApproval/ you approve the following:
method hudson.model.Run getLog
method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild
Well, sometimes stating a question makes you think from another perspective.
Hopefully someone will benefit from this.
I stumbled upon a Pipeline-Plugin tutorial where they showed how you could use node, to label where script code should run. The resulting groovy file looks something like this:
#!groovy
stage('Environment Preparation') {
node('master') {
// fetches stuff and sets up the environment on master
}
}
stage('Unit Testing') {
node('master') {
// some testing
}
}
stage('Build on Slave') {
node('remote') {
def out = bat script: 'C:\\Build\\build.bat', returnStdout: true
}
}
As you can see the tutorial made me refactor the script a bit. The node('remote') part is what defines that the upcoming stuff should be run on the slave machine.
I had to make some customizations in the batch script, so that everything important was printed to stdout.
You have to let Jenkins know which node is 'remote' by going in to Manage Jenkins > Manage Nodes, choose the slave agent in mind, Configure Node and add 'remote' or whatever suits you to the labels field.