I have a Jenkinsfile where I am trying to:
Conditionally build a docker image to allow for testing
Conditionally run the tests using that docker image as my agent
The conditional build is working without a problem. It builds an image that I have assigned to the variable MYIMAGE shown below.
But the conditional running of the tests does not work because it tries to find an agent before it tests for the expression, but the agent is not there because the image has not been pushed.
In this case, I am triggering it with a PR comment of "test". But I cannot even get that far, because I cannot get the initial build to pass because it reaches the stage shown below -- which should be skipped since the trigger text is not present -- and it pauses there because it is trying to find a non-existent Docker image with which to create an agent. So, even though this step should be skipped entirely, it hangs here.
Here is that snippet of code:
stage('Test image') {
when {
expression {
return getTriggerText() == 'test'
}
}
agent {
docker {
image MYIMAGE
label MYLABEL
}
}
steps {
script {
sh 'python -m pytest tests/test_toyexample.py'
pullRequest.comment("Ran tests on image: $MYIMAGE")
}
}
}
And I am quite confident it is the agent portion that is messing things up: I have a very similar structure for skipping the image build except that it uses the standard agent, and that runs fine... it gets skipped.
Question
Is there a way to use a when expression like what I have above, even with an agent that does not exist? Keep in mind, this agent does exist if the step is not being skipped... it is only missing when the step is supposed to be skipped.
Related
I want to be able to use an AWS node to run our jenkins build stages or one of our PCs connected to hardware that can also test the systems. We have some PCs installed that have a recognised agent, and for the AWS node I want to run the steps in a docker container.
Based on this, I would like to decide whether to use the docker agent or the custom agent based on parameters passed in by the user at build time. My jenkinsfile looks like this:
#!/usr/bin/env groovy
#Library('customLib')
import groovy.transform.Field
pipeline {
agent none
parameters{
choice(name: 'NODE', choices: ['PC1', 'PC2', 'dockerNode', 'auto'], description: 'Select which node to run on? Select auto for auto assignment.')
}
stages {
stage('Initialise build environment'){
agent { // agent
docker {
image 'docker-image'
label 'docker-agent'
registryUrl 'url'
registryCredentialsId 'dockerRegistry'
args '--entrypoint=\'\''
alwaysPull true
}
}
steps {
script {
if (params.NODE != 'auto' && params.NODE != 'dockerNode'){
agentLabel = params.NODE + ' && ' + 'custom_string' // this will lead to running the stages on our PCs
}else {
agentLabel = 'docker-agent'
}
env.NODE = params.NODE
}
}
} // initialise build environment
stage('Waiting for available node') {
agent {label agentLabel} // if auto or AWSNode is chosen, I want to run in the docker container defined above; otherwise, use the PC.
post { cleanup { cleanWs() } }
stages {
stage('Import and Build project') {
steps {
script {
...
}
}
} // Build project
...
}
} // Waiting for available node
}
}
Assuming the PC option works fine if I don't add this docker option and the docker option also works fine on its own, how do I add a label to my custom docker agent and use it conditionally like above? Running this Jenkinsfile, I get the message 'There are no nodes with the label ‘docker-agent’ which seems to say that it can't find the label I defined.
Edit: I worked around this problem by adding two stages that run a when block that decides whether the stage runs or not, with a different agent.
Pros: I'm able to choose where the steps are run.
Cons:
The inside build and archive stages are identical, so they are simply repeated for each outer stage. (This may become a pro if I want to run a separate HW test stage only on one of the agents)
both outer stages always run, and that means that the docker container is always pulled from the registry, even if it is not used (i.e. if we choose to run the build process on a PC)
Seems like stages do not lend themselves easily to being encapsulated into functions.
No matter what I try, I seem to be unable toget a declerative pipeline to build my project inside a docker container, with the correct image.
I have verified the following:
Jenkins does build the correct image (based on messages in the log)
When I build the image manually, it is build correctly
When building the project inside a container with the correct image, the build succeeds
The Jenkins steps do run in a container with some image.
As far as I can tell, Jenkins simply uses the base image and not the correct one, resulting from the dockerfile I specify.
Things I've tried:
Let Jenkins figure it out
pipeline {
agent dockerfile
Using docker at the top level:
pipeline {
agent {
dockerfile {
filename 'Dockerfile'
reuseNode true
}
}
stages {
stage('configure') {
steps {
Use docker in each step
pipeline {
agent none
stages {
stage('configure') {
agent {
dockerfile {
filename 'Dockerfile'
reuseNode true
}
}
steps {
Abbreviations, due to the number of examples. Docker is not mentioned anywhere outside of the specified areas and simply removing the docker parts and using a regular agent works fine.
Logs
The logs are useless. They simply state that they build the image and verify that they exist and then fail to execute commands that have just been installed (meson in this case).
First of all, I suggest you to read:
The Pipeline syntax for the agent declaration and in particular the dockerfile section
This basic example on how to use dockerfile. Start with the minimal pipeline with agent { dockerfile true } and please show us the logs for example.
Without any logs or a more detailed explanation on your setup, it is difficult to help you.
I can certainly tell you that the second try is wrong because
reuseNode is valid for docker and dockerfile, and only has an effect when used on an agent for an individual stage.
Instead, I am not sure how the third try could ever work: with agent none you are forcing each stage to have an agent section, but in the stage's agent section you have the reuseNode option set to true. Isn't it a contradiction? How could you reuse a top-level node if this one does not exist?
I know it is not an answer, but it is also too long to stay in the comments in my opinion.
I always use it like this, with a pre-build image:
pipeline {
agent {
docker { image 'node:16-alpine' }
}
stages {
stage('Test') {
steps {
sh 'node --version'
}
}
}
}
But I can only guess what you want to do inside the docker environment.
I've created a Jenkinsfile in my Git repository that is defined as this:
pipeline {
//None parameter in the agent section means that no global agent will be allocated for the entire Pipeline’s
//execution and that each stage directive must specify its own agent section.
agent none
stages {
stage('Build') {
agent {
docker {
//This image parameter (of the agent section’s docker parameter) downloads the python:3.8
//Docker image and runs this image as a separate container. The Python container becomes
//the agent that Jenkins uses to run the Build stage of the Pipeline project.
image 'python:3.8.3'
}
}
steps {
//This sh step runs the Python command to compile the application
sh 'pip install -r requirements.txt'
}
}
}
}
When I tried to run the job with this Pipeline, I've got the following error:
I also tried to use image python:latest but this option didn't work either.
Can someone explain me :)?
Go to Computer Management -> Local Users and Groups and make sure the user used by jenkins is added to the docker-users group
I havent had much luck with google due to the phrasing of the question so I apologize for the very basic question.
If i have 3 build stages setup in my jenkinsfile and stages 1 and 3 always run, but 2 only runs if its a PR, do I make it keep the stage but not run the command, or just filter out the whole stage?
stage {
if (env.isPR) {
sh()
or more like
if (env.isPR) {
stage {
sh()
If you filter out the whole stage, then after a non-PR build, the stage visualization on the build page will remove that PR stage (for all builds, even the ones which in fact did run the PR stage).
So, from a visualization perspective, I would suggest that you retain the stage.
stage {
if (env.isPR) {
sh()
i've got a Jenkins job that should simply start a docker container using the Docker plugin.
If the container is stopped, the job runs correctly, but if the container is already running, the build step returns a failure due to an
com.github.dockerjava.api.exception.NotModifiedException
error.
This is basically the expected behavior of Jenkins but in my case, i want to set it to unstable to have a more meaningful response for the user.
I tried to add a conditional build step afterwards using TextFinder that scans the console output for the error, but it seems that it isn't executed after the docker build step fails.
Is there a way to change the build status just for this error?
In Jenkins, you can add a Groovy PostBuild Script for that job:
exceptionTextRegex = '.*com.github.dockerjava.api.exception.NotModifiedException.*'
if(manager.logContains(exceptionTextRegex)) {
manager.buildUnstable()
}
Thank you for pointing me in the right direction. Groovy PoistBuild was indeed the answer, but the script was a little bit bigger:
errpattern = ~/com.github.dockerjava.api.exception.NotModifiedException.*/;
manager.build.logFile.eachLine{ line ->
errmatcher=errpattern.matcher(line)
if (errmatcher.find()) {
manager.build.#result = hudson.model.Result.UNSTABLE
}
}