I am writing a shared pipeline library for projects using either a Docker agent or agents specified via labels. I want the agent{} section to be configurable.
In a regular Jenkinsfile for a project using Docker the agent section looks like this:
agent
{
docker
{
label 'docker'
image 'my-image'
}
}
The agent section of projects not using Docker looks like this:
agent
{
node
{
label 'FOO'
label 'BAR'
}
}
What is the correct syntax for the agent section of the shared pipeline lib that would yield either the first or the second agent{} examples from above?
// /vars/sharedPipeline.groovy
def call(body) {
def pipelineParams= [:]
body.resolveStrategy = Closure.DELEGATE_FIRST
body.delegate = pipelineParams
body()
pipeline {
agent {
// <---- What goes here? What is the value that pipelineParams.buildAgent should have?
}
}
...
}
I want to avoid solutions that will force me to:
manually invoke 'docker run' from within a stage
check the label to decide what to do next
Update:
Based on what I have found so far it is not possible to dynamically choose between docker and non-docker agents at the top level of the pipeline. There are some workarounds available and they require specifying an agent for each individual stage or they use docker.image().inside() chains protected by some control flow constructs.
Related
I want to be able to use an AWS node to run our jenkins build stages or one of our PCs connected to hardware that can also test the systems. We have some PCs installed that have a recognised agent, and for the AWS node I want to run the steps in a docker container.
Based on this, I would like to decide whether to use the docker agent or the custom agent based on parameters passed in by the user at build time. My jenkinsfile looks like this:
#!/usr/bin/env groovy
#Library('customLib')
import groovy.transform.Field
pipeline {
agent none
parameters{
choice(name: 'NODE', choices: ['PC1', 'PC2', 'dockerNode', 'auto'], description: 'Select which node to run on? Select auto for auto assignment.')
}
stages {
stage('Initialise build environment'){
agent { // agent
docker {
image 'docker-image'
label 'docker-agent'
registryUrl 'url'
registryCredentialsId 'dockerRegistry'
args '--entrypoint=\'\''
alwaysPull true
}
}
steps {
script {
if (params.NODE != 'auto' && params.NODE != 'dockerNode'){
agentLabel = params.NODE + ' && ' + 'custom_string' // this will lead to running the stages on our PCs
}else {
agentLabel = 'docker-agent'
}
env.NODE = params.NODE
}
}
} // initialise build environment
stage('Waiting for available node') {
agent {label agentLabel} // if auto or AWSNode is chosen, I want to run in the docker container defined above; otherwise, use the PC.
post { cleanup { cleanWs() } }
stages {
stage('Import and Build project') {
steps {
script {
...
}
}
} // Build project
...
}
} // Waiting for available node
}
}
Assuming the PC option works fine if I don't add this docker option and the docker option also works fine on its own, how do I add a label to my custom docker agent and use it conditionally like above? Running this Jenkinsfile, I get the message 'There are no nodes with the label ‘docker-agent’ which seems to say that it can't find the label I defined.
Edit: I worked around this problem by adding two stages that run a when block that decides whether the stage runs or not, with a different agent.
Pros: I'm able to choose where the steps are run.
Cons:
The inside build and archive stages are identical, so they are simply repeated for each outer stage. (This may become a pro if I want to run a separate HW test stage only on one of the agents)
both outer stages always run, and that means that the docker container is always pulled from the registry, even if it is not used (i.e. if we choose to run the build process on a PC)
Seems like stages do not lend themselves easily to being encapsulated into functions.
I am new to Jenkins pipeline groovy script.
I have a job param named 'HOSTNAMES' which will take some host name values separated by comma as an input. I need to execute some scripts on all these hosts in parallel mode. How to achieve this using jenkins groovy script. Pls guide me.
Assuming all these nodes (hostnames) are connected to the Jenkins server and the HOSTNAMES parameter is a list of the node names (as they appear in the jenkins server) you can use the parallel step to achieve what you want.
You will have to transform each node name into a map entry representing the parallel execution branch and then run them using the parallel keyword.
something like:
def hosts = HOSTNAMES.split(',')
def executions = hosts.collectEntries { host ->
["Running on ${host}" : {
node(host) {
// The code to run on each node
stage("Execution") {
echo 'example code on #{host}'
...
}
}
}]
}
parallel executions
You can also run it in one line if you want:
parallel hosts.collectEntries { ...
I am trying to have Jenkins pipline job run only using specific agents. For reasons I can not use any and I would prefer not to only use one agent via a label. The agents are named like: abc-a, abc-b, abc-c, xyz-a, xyz-b, xyz-c.
I would like only run the job using agents that are contain abc in their name. I was looking for something like:
pipeline {
agent {
contains: 'abc'
}
stages {
stage ("Example") {
steps{
sh 'really cool bash'
}
}
}
}
I read though the documentation here Jenkins Pipline but the best thing I can find is using a label, but I could be over looking something. Any ideas?
I have multiple pipelines using Jenkinsfiles that retrieve a docker image from a private registry. I would like to be able to load the docker specific information into the pipelines from a file, so that I don’t have to modify all of my Jenkinsfiles when the docker label or credentials change. I attempted to do this using the example jenkinsfile below:
def common
pipeline {
agent none
options {
timestamps()
}
stages {
stage('Image fetch') {
steps{
script {
common = load('/home/jenkins/workspace/common/docker_image')
common.fetchImage()
}
}
}
}
With docker_image containing:
def fetchImage() {
agent {
docker {
label “target_node ”
image 'registry-url/image:latest'
alwaysPull true
registryUrl 'https://registry-url’
registryCredentialsId ‘xxxxxxx-xxxxxx-xxxx’
}
}
}
I got the following error when I executed the pipeline:
Required context class hudson.FilePath is missing Perhaps you forgot
to surround the code with a step that provides this, such as:
node,dockerNode
How can I do this using a declarative pipeline?
There are a few issues with this:
You can allocate a node only at top level
pipeline {
agent ...
}
Or you can use a per-stage node allocation like so:
pipeline {
agent none
....
stages {
stage("My stage") {
agent ...
steps {
// run my steps on this agent
}
}
}
}
You can check the docs here
The steps are supposed to be executed on the allocated node (or in some cases they can be executed without allocating a node at all).
Declarative Pipeline and Scripted Pipeline are two different things. Yes, it's possible to mix them, but scripted pipeline is meant to either abstract some logic into a shared library, or to provide you a way to be a "hard core master ninja" and write your own fully custom pipeline using the scripted pipeline and none of the declarative sugar.
I am not sure how your Docker <-> Jenkins connection is setup, but you would probably be better if you install a plugin and use agent templates to provide the agents you need.
If you have a Docker Swarm you can install the Docker Swarm Plugin and then in your pipeline you can just configure pipeline { agent { label 'my-agent-label' } }. This will automatically provision your Jenkins with an agent in a container which uses the image you specified.
If you have exposed /var/run/docker.sock to your Jenkins, then you could use Yet Another Docker Plugin, which has the same concept.
This way you can remove the agent configuration into the agent template and your pipeline will only use a label to have the agent it needs.
I am trying to use jenkins. But when I reading the Declarative Pipeline Syntax, I confused by the agent term
https://jenkins.io/doc/book/pipeline/syntax/#scripted-pipeline
What does the agent stand for?
Is that mean I can set the pipeline runtime folder path?
How to create an agent?
How set a label for agent?
I can feel you :-D.
Here are the answers:
The agent section specifies where the entire Pipeline, or a specific stage, will execute in the Jenkins environment depending on where the agent section is placed. The section must be defined at the top-level inside the pipeline block, but stage-level usage is optional. - Content copied from the agent section
NO, this has nothing to do with the pipeline runtime folder path.
You can for example Create an agent/node by the following tutorial:
How to Setup Jenkins Agent/Slave Using Password and ssh Keys. -
But there are many other ways to create an agent e.g. using a Docker-Container (...).
You can Set a label under the Configuration of the Node.
You can use a label in your pipeline like:
pipeline {
agent { label 'labelName' }
(...)
}
While #adbo covered questions asked, Jenkins glossary describes agent really well:
typically a machine, or container, which connects to a Jenkins controller and executes tasks when directed by the controller.
You can choose to run entire pipeline on any available agent (agent any at the top of the pipeline) or run a specific stage on the agent of choice e.g. run build stage in a specific environ by overriding agent in that stage:
agent { docker { image 'my image' } }