Among the possible steps one can use in a Jenkins pipeline, there is one with the name step, subtitled General Build Step. https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#step-general-build-step . I need to iterate on calling this step based on the contents of a file. I have created a groovy script to read the file and perform the iteration, but I am not sure how to create the equivalent of my step() in the groovy script. Here is the general format of the step I am trying to perform:
stage ('title') {
steps {
step([
$class: 'UCDeployPublisher',
siteName: 'literal string',
deploy: [
$class: 'com.urbancode.jenkins.plugins.ucdeploy.DeployHelper$DeployBlock',
param1: 'another literal string',
param2: 'yet another string'
]
])
}
}
The script step I have developed looks like this:
steps {
script {
def content = readFile(file:'data.csv', encoding:'UTF-8');
def lines = content.split('\n');
for (line in lines) {
// want to insert equivalent groovy code for the basic build step here
}
}
}
I'm expecting there is probably a trivial answer here. I'm just out of my element in the groovy/java world and I am not sure how to proceed. I have done extensive research, looked at source code for Jenkins, looked at plugins, etc. I am stuck!
Check the following, simply move your UCDeployPublisher to a new function and call that from your loop.
steps {
script {
def content = readFile(file:'data.csv', encoding:'UTF-8');
def lines = content.split('\n');
for (line in lines) {
runUCD(line)
}
}
}
// Groovy function
def runUCD(def n) {
stage ("title $n") {
steps {
step([
$class: 'UCDeployPublisher',
siteName: 'literal string',
deploy: [
$class: 'com.urbancode.jenkins.plugins.ucdeploy.DeployHelper$DeployBlock',
param1: 'another literal string',
param2: 'yet another string'
]
])
}
}
}
This is showing the code related to my comment on the accepted answer
pipeline {
stages {
stage ('loop') {
steps {
script {
... groovy to read/parse file and call runUCD
}
}
}
}
}
def runUCD(def param1, def param2) {
stage ("title $param1") {
step([
....
])
}
}
Related
This really helpful answer, got me 95% of the way there. Using this solution, I'm able to start n build stages in parallel. However, the map of parallel stages is essentially hardcoded. I want to be able to create it dynamically. The first step in this process is changing parallelStagesMap from a map, to a function that returns a map.
Unfortunatey, this small change causes my build to fail without any apparent error logs related to syntax.
How can I accomplish this? Am I using malformed Groovy syntax? I'd be grateful for any help.
def jobs = ["JobA", "JobB", "JobC"]
def parallelStagesMap() { // This is now a function that returns a map.
return jobs.collectEntries {
["${it}" : generateStage(it)]
}
}
def generateStage(job) {
return {
stage("stage: ${job}") {
echo "This is ${job}."
sh script: "sleep 15"
}
}
}
pipeline {
agent any
stages {
stage('parallel stage') {
steps {
script {
parallel parallelStagesMap() // I call the function here.
}
}
}
}
}
I got a working solution! It's not perfect, because I would like to extract the jobs.collectEntries part to my own function, but now I can define the contents of my parallel stages inline, instead of at the top of the file!
I tried writing a function matching the same signature as Map.collectEntries: ({ Closure -> Map }), but the Jenkins build fails without any logs once it hits my function. If someone's able to work that out, I'd be grateful.
def jobs = ["JobA", "JobB", "JobC"]
pipeline {
agent any
stages {
stage('parallel stage') {
steps {
script {
parallel jobs.collectEntries { j ->
["${j}" : { job -> return {
stage("stage: ${job}") {
echo "This is ${job}."
sh script: "sleep 15"
}
}}(j)]
}
}
}
}
}
}
My Jenkins pipeline is specified in a declarative way. However, in the place I'd like to introduce a parallel step in, I'm forced down into plain Groovy methods:
def call(Map parameters) {
runFirstStep(parameters)
// add second step that runs in parallel to runFirstStep
}
Unexperienced as I am with Groovy and Jenkins, I hoped to make something simple like this work:
def call(Map parameters) {
parallel(
'First step': {
runFirstStep(parameters)
},
'Second step': {
runSecondStep(parameters)
}
)
}
However, this seems to mix up declarative and imperative code in an unsuited way and leads to cryptic NullPointerExceptions.
Is there another, feasible solution for this? Or is overriding the declarative part that surrounds this the only option to squeeze in another parallel step?
def call(Map parameters) {
def steps = [:]
steps['First step'] = {
node('master') {
runFirstStep(parameters)
}
}
steps['Second step'] = {
node('master') {
runSecondStep(parameters)
}
}
parallel steps.plus([failFast: true])
}
Recently I have tried to build a jenkins pipeline with a large number of 'Tests' in one of its stages.
The thing is, at some point I got an error regarding my stages phase was too large, so I tried to solve it with function that will build all of my stages and I can run this function output(map of stages) in parallel.
Some of the stages have to run on agent(node) taken from a label, and others have some unique steps in them.
I am trying to understand in general, how can I write a function that will build a map to run in parallel - but was not successfull nor did I found any good example of it online.
I know the question is general, but if anyone can point me to some examples, or just write one, it will be great.
This is the snippet I am working on(not full JenkinsFile):
def getParallelBuilders(list_arr) {
def builders = [:]
builders['Test-1'] =
stage ('Test-1')
{
node('ci-nodes')
{
when {
environment name: 'TEST_NAME', value: 'true'
beforeAgent true
}
timeout(time: 1, unit: 'HOURS')
script { runtests() }
post {
success { onTestSuccess title: 'Temp', pytest: 'results.xml' }
cleanup { afterTestCleanup2("clean") }
}
}
}
return builders
}
The call to this function happens from my 'pipeline' block, after stages of build, configure etc:
stage('Testing') {
steps {
script { parallel getParallelBuilders(list_arr) }
}
}
Not sure if my approach to this problem is right at all,
hopefully someone can point me in the right direction.
After a while, here is the solution I got for my problem:
builders = [
'Test1':
{
stage ('Test1')
{
if (RUN_TESTS == 'true')
{
timeout(time: 30, unit: 'MINUTES')
{
node('ci-nodes')
{
try
{
runtests()
onTestSuccess title: 'Temp', pytest: 'results.xml'
}
catch (err)
{
onTestFailure testName: "Test1"
}
finally
{
afterTestCleanup()
}
}
}
}
}
}
The main issue was understanding the scripting pipeline syntax, which is really diffrent fron the declarative way.
When running a large Jenkins pipeline script, it can give the error:
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: General error during class generation: Method code too large!
java.lang.RuntimeException: Method code too large!
What is the reason for this error and how can it be fixed?
This is due to a limit between Java and Groovy, requiring that method bytecode be no larger than 64kb. It is not due to the Jenkins Pipeline DSL.
To solve this, instead of using a single monolithic pipeline script, break it up into methods and call the methods.
For example, instead of having:
stage foo
parallel([
... giant list of maps ...
])
Instead do:
stage foo
def build_foo() {
parallel([
...giant list of maps...
])}
build_foo()
If you are using declarative pipeline with shared library, you may need to refactor and externalize your global variables in the new methods. Here is a full example:
Jenkinsfile:
#Library("my-shared-library") _
myPipeline()
myPipeline.groovy:
def call() {
String SOME_GLOBAL_VARIABLE
String SOME_GLOBAL_FILENAME
pipeline {
stages() {
stage('stage 1') {
steps {
script {
SOME_GLOBAL_VARIABLE = 'hello'
SOME_GLOBAL_FILENAME = 'hello.txt'
...
}
}
}
stage('stage 2') {
steps {
script {
doSomething(fileContent: SOME_GLOBAL_VARIABLE, filename: SOME_GLOBAL_FILENAME)
sh "cat $SOME_GLOBAL_FILENAME"
}
}
}
}
}
}
def doSomething(Map params) {
String fileContent = params.fileContent
String filename = params.filename
sh "echo $fileContent > $filename"
}
I have the following Jenkins DSL file:
if (params["BUILD_SNAPSHOT"] == "true") {
parallel(
{
build("company-main-build-snapshot")
},
{
build("1-company-worker-build-snaphsot", WORKER_NAME: "sharding-worker")
}
)
}
parallel (
{
build("company-deployment-info",
API_KEY: "aaaaa5dd4cd58b94215f9cddd4441c391b4ddde226ede98",
APP: "company-Staging-App")
},
{
build("company-salt-role-deploy",
ENV: "staging",
ROLE: "app")
},
{
build("company-deployment-info",
API_KEY: "aaaaa5dd4cd58b94215f9cddd4441c391b4ddde226ede98",
APP: "company-Staging-Shardwork")
},
{
build("company-salt-workers-deploy",
ENVIRONMENT: "staging",
WORKER_TYPE: "shardwork")
}
)
if (params["REST_TEST"] == "true") {
build("company_STAGING_python_rest_test")
}
My task is to convert/rewrite this workflow file content to Jenkins pipeline Jenkinsfile.
I have some example files for reference but I'm having a hard time understanding how I should even begin...
Can anyone please shed some light on this subject?
First, have a good look at Jenkins pipeline documentation, it is a great start and it is providing a whole bunch of information such as Build Parameters usage or parallel steps.
Here are a few more hints for you to explore :
Parameters
Just use the parameter name as a variable such as :
if (BUILD_SNAPSHOT) {
...
}
Call other jobs
You can also use build step such as :
build job: '1-company-worker-build-snaphsot', parameters: [stringParam(name: 'WORKER_NAME', value: "sharding-worker")]
Use functions
Instead of calling downstream jobs using build steps each time, you might want to consider using pipeline functions from another Groovy script, either from your current project or even from an external, checked out Groovy script.
As an example, you could replace your second job call from :
build("1-company-worker-build-snaphsot", WORKER_NAME: "sharding-worker")
to :
git 'http://urlToYourGit/projectContainingYourScript'
pipeline = load 'global-functions.groovy'
pipeline.buildSnapshot("sharding-worker")
...of course the init phase (Git checkout and pipeline loading) is only needed once before you can call all your external scripts functions.
In short
To sum it up a little bit, your code could be converted to something along these lines :
node {
git 'http://urlToYourGit/projectContainingYourScript'
pipeline = load 'global-functions.groovy'
if(BUILD_SNAPSHOT) {
parallel (
phase1: { pipeline.buildMainSnapshot() },
phase2: { pipeline.buildWorkerSnapshot("sharding-worker") }
)
}
parallel (
phase1: { pipeline.phase1(params...) },
phase2: { pipeline.phase2(params...) },
phase3: { pipeline.phase3(params...) },
phase4: { pipeline.phase4(params...) }
)
if (REST_TEST) {
pipeline.finalStep()
}
}