Jenkins ver. 2.121.1
pipeline {
parameters {
string(
name: 'repo',
defaultValue:"foo",
description: "repo to build from")
}
agent any
stages {
stage('Checkout'){
steps {
echo params.repo
Expected:
echo prints "foo" or what ever value was set at "build with parameters" every time.
Actual:
The first time the pipeline is run the default value is printed.
Once the parameters are "installed" and visible in the GUI, every run after that the echo line prints empty string
Does anyone know what is causing this interference in the parameters being accessible?
Related
Our Jenkins setup consists of master nodes and different / dedicated worker nodes for running jobs in dev, test and prod environment. How do I go about creating a scripted pipeline code that allows users to select environment (possibly from master node) and depending upon the environment selected would execute the rest of the job in the node selected? Here is my initial thought:
stage('Select environment ') {
script {
def userInput = input(id: 'userInput', message: 'Merge to?',
parameters: [[$class: 'ChoiceParameterDefinition', defaultValue: 'strDef',
description:'describing choices', name:'Env', choices: "dev\ntest\nprod"]
])
println(userInput);
}
echo "Environment here ${params.Env}" // prints null here
stage("Build") {
node(${params.Env}) { // schedule job based upon the environment selected earlier
echo "My test here"
}
}
}
I am in the right path or should I be looking at something else?
Another follow up question is that the job that is running on the worker node also requires additional user input. Is there a way to combine the user input in one go such that the users would not be prompted with multiple user screens?
If you pass the environment as a build parameter when kicking off the job, and you have appropriate labels on your nodes, you could do something like:
agent = params.WHAT_NODE
agentLabels = "deploy && ${agent}"
pipeline {
agent { label agentLabels }
....
}
Ended up doing the following for scripted pipeline:
The code for selecting environment can be run on any node (whether master or slaves with agent running). The parameter can be injected into an environment variable: env..
node {
stage('Select Environment'){
env.Env = input(id: 'userInput', message: 'Select Environment',
parameters: [[$class: 'ChoiceParameterDefinition',
defaultValue: 'strDef',
description:'describing choices',
name:'Env',
choices: "jen-dev-worker\njen-test-worker\njen-prod-worker"]
])
println(env.Env);
}
stage('Display Environment') {
println(env.Env);
}
}
The following code snippet ensures that script would be executed on the environment selected in the last step. Requires Jenkins workers with labels: jen-dev-worker, jen-test-worker, jen-prod-worker) available.
node (env.Env) {
echo "Hello world, I am running on ${env.Env}"
}
I am trying to implement Machine learning in my jenkins pipeline.
For that I need output data of pipeline for each build.
Some parameters that i need are:
Which user triggered the pipeline
Duration of pipeline
Build number with its details
Pipeline pass/fail
If fail, at which stage it failed.
Error in the failed stage. (Why it failed)
Time required to execute each stage
Specific output of each stage (For. eg. : If a stage contains sonarcube execution then output be kind of percentage of codesmells or code coverage)
I need to fetch these details for all builds. How can get it?
There is jenkins api that can be implemented in python but i was able to get only JOB_NAME, Description of job, IS job Enabled.
These details werent useful.
There are 2 ways to get some of data from your list.
1. Jenkins API
For first 4 points from the list, you can use JSON REST API for a specific build to get those data. Example API endpoint:
https://[JENKINS_HOST]/job/[JOB_NAME]/[BUILD_NUMBER]/api/json?pretty=true
1. Which user triggered the pipeline
This will be under actions array in response, identyfi object in array by "_class": "hudson.model.CauseAction" and in it you will have shortDescription key which will have that information:
"actions": [
{
"_class": "hudson.model.CauseAction",
"causes": [
{
"_class": "hudson.triggers.SCMTrigger$SCMTriggerCause",
"shortDescription": "Started by an SCM change"
}
]
},
2. Duration of pipeline
It can be found under key: "duration". Example
"duration": 244736,
3. Build number with its details
I don't know what details you need, but for build number look for "number" key:
"number": 107,
4. Pipeline pass/fail
"result": "SUCCESS",
If you need to extract this information for all builds, run GET request for job API https://[JENKINS_HOST]/job/[JOB_NAME]/api/json?pretty=trueand extract all builds, then run above-mentioned request per build you have extracted.
I will write later a dummy python script to do just that.
2. Dump data in Jenkinsfile
There is also a possibility to dump some that information from Jenkinfile in post action.
pipeline {
agent any
stages {
stage('stage 1') {
steps {
sh 'echo "Stage 1 time: ${YOUR_TIME_VAR}" > job_data.txt'
}
}
}
post {
always {
sh 'echo "Result: ${result}" > job_data.txt'
sh 'echo "Job name: ${displayName}" > job_data.txt'
sh 'echo "Build number: ${number}" > job_data.txt'
sh 'echo "Duration: ${duration}" > job_data.txt'
archiveArtifacts artifacts: 'job_data.txt', onlyIfSuccessful: false
}
}
}
List of available global variables for pipeline job can be found:
https://[JENKINS_HOST]/pipeline-syntax/globals#env
For rest, you will need to implement your own logic in Jenkinsfile.
Ad. 5
Create a variable which holds information about current stage. At the beginning of each stage change its value to the ongoing stage. At the end dump to file like rest variables. If pipeline will fail let's say on stage foo in post action this variable will have exact same value because if pipeline fails it won't go to next stage.
Ad. 6
I'm not sure what you want, a traceback, error code?
I guess you will probably need to implement your own logging function.
Ad. 7
Make a function for measuring time for each stage and dump value at the end.
Ad. 8
Also not sure what you mean. Like, build artifacts?
At the end of each build this file job_data.txt will be archived as build artifact which can be later downloaded.
If i will find more elegant and simple solution I'll edit this post.
Hope it helps in any way
EDIT 1
Here is the script I've mentioned earlier.
import requests
username = "USERNAME"
password = "PASSWORD"
jenkins_host = "JENKINS_HOST"
jenkins_job = "JOBNAME"
request_url = "{0:s}/job/{1:s}/api/json".format(
jenkins_host,
jenkins_job,
)
job_data = requests.get(request_url, auth=(username, password)).json()
builds = []
for build in job_data.get('builds'):
builds.append(build.get('number'))
for build in builds:
build_url = "{0:s}/job/{1:s}/{2:d}/api/json".format(
jenkins_host,
jenkins_job,
build,
)
build_data = requests.get(build_url, auth=(username, password)).json()
build_name = build_data.get('fullDisplayName')
build_number = build_data.get('number')
build_status = build_data.get('result')
build_duration = build_data.get('duration')
for action in build_data.get('actions'):
if action.get("_class") == "hudson.model.CauseAction":
build_trigger = action.get('causes')
print(build_name)
print(build_status)
print(build_duration)
print(build_number)
print(build_trigger)
Please note you might need to authorize with API Token depending on your security settings.
A simplified pipeline will look something like:
1. build
2. unit test
3. deploy to dev
4. integration tests
5. deploy to prod
For step #5 I've setup a Jenkins pipeline input command. We won't be deploying to prod on every commit so if we abort all those jobs it will have a big list of grey builds. Is it possible to have a skip option so the build can still be shown as green blue?
There is a better solution I just found. You can access the result of the input like by using the return value. The user has to check the checkbox, to run the optional stage. Otherwise the steps of the stage are skipped. If you skipp the whole stage, the stage will disappear and that "cleans" the stage view history.
stage('do optional stuff?') {
userInput = input(
id: 'userInput', message: "Some important question?", parameters: [
booleanParam(defaultValue: false, description: 'really?', name: 'myValue')
])
}
stage('optional: do magic') {
if (userInput) {
echo "do magic"
} else {
// do what ever you want when skipping this build
currentBuild.result = "UNSTABLE"
}
}
How about:
stage('Deploy') {
when { branch 'master' }
steps {
sh '...'
}
}
}
the stage will be skipped for commits on other branches and will be green.
Can't you do something like this, it will be blue/green whatever you choose from input, and you can then run the deployment depending on it too?
def deployToProduction = true
try{
input 'Deploy to Production'
}catch(e){
deployToProduction = false
}
if(deployToProduction){
println "Deploying to production"
}
Instead of using pipeline as a code Jenkins2 feature, you can setup Jobs with downstream/upstream configuration.
Build -> Unit test -> Deploy to Dev -> Integration tests -> Promote to Prod -> Deploy to Prod
At present it gives more control to choose which version of pipeline you wish to Prod.
For greater visibility you can configure Delivery Pipeline using Delivery-Pipeline-Plugin.
Is it possible to save some values during a parallel execution and use these values during a final step?
In the following example, I would like to know which jenkins slave is used during the parallel execution and use it during the final step.
node {
stage 'Checkout'
checkout([...])
stash includes: '**', name: 'binary'
stage 'Running simulation'
parallel (
"stream 1" : {
node {
unstash "binary"
sh "echo \"\$(whoami)#\$(hostname):\$PWD\""
// How to save the previous result
// Run simulation on node first slave
...
}
},
"stream 2" : {
node {
unstash "binary"
sh "echo \"\$(whoami)#\$(hostname):\$PWD\""
// How to save the previous result
// Run simulation on node second slave
...
}
}
)
stage 'Gathering results files'
// use the values of the slaves to retrieve some files.
stage 'Generate report'
}
Thanks for your answer.
My bad, I was using the 2.3 version of Pipeline Nodes and Processes Plugin. It works fine with the version 2.5
hostname = sh (returnStdout: true, script: 'hostname')
println hostname
I am completely new to Jenkins 2 pipelines.
I had played with Jenkins 1 pipelines with the following view:
You could start a certain stage directly, let's say that I could choose to start running from the release stage, skipping Test.
I have a pretty simple Jenkins 2 pipeline definition:
stage('Preparation'){
echo """
Preparing
something
"""
}
stage('Greeting') {
parallel 'hello1':{
node{
echo 'hello world 1'
}
}, 'hello2':{
node{
echo 'hello world 2'
}
}
}
In the pipeline page I have "Build now" which runs all stages starting from Preparation.
My questions are:
How can I run the stage I prefer? For instance Greeting instead of starting from Preparation?
How do you define the dependencies between stages? I mean the stage called after another one completes
Is there a way to limit the stages that a certain user can start? Imagine that I only want a specific user to launch the Greeting stage.
How do you setup manual stages?
UPDATE: The real goal behind my questions is the modelling of a continuous delivery pipeline like the following with Jenkins 2 pipelines:
Build stage --> AUTO --> Acceptance Stage --> MANUAL --> Production Stage
--> MANUAL --> QA Stage
This is the behaviour I want:
Build Stage (any user can start it) when it finishes it triggers automatically the Acceptance Stage. This one can't be lauched manually, only automatically after succesfully finishing the Build Stage.
From Acceptance Stage I need that only authorized users can manually trigger QA Stage and Production Stage.
The business flow would be: a developer hits Build Stage, its code is built and packaged. Acceptance Stage begins, using the packaged code to run a bunch of automated tests.
At this point, when Acceptance Stage has finished OK, two things can happen:
Maybe QA Stage is needed to run more tests (Cucumber, manual, etc.). Some some authorized user would fire this stage.
When the product owner is happy, he can decice to launch the Production Stage to deploy the code in a production environment.
I am struggling to model this with Jenkins 2 pipelines.
There is no direct answer to some of your questions but they can be achieved with some additional coding. While certain people might find some other way to achieve but let me try with what I have in my mind:
1) How can I run the stage I prefer? For instance Greeting instead of starting from Preparation?
This could be achieved by adding a Boolean parameter FASTFORWARD_TO_GREETING and than using the value provided while executing build to manipulate the flow of your build. So your code will now look like :
if (FASTFORWARD_TO_GREETING == 'false') {
stage('Preparation'){
echo """
Preparing
something
"""
}
}
stage('Greeting') {
parallel 'hello1':{
node{
echo 'hello world 1'
}
}, 'hello2':{
node{
echo 'hello world 2'
}
}
}
2) How do you define the dependencies between stages? I mean the stage called after another one completes
Stages are executed serially, so if a stage is defined first it'll be started and completed first before moving to the next stage. However in parallel step this does not hold true, as all the steps will be executed in parallel. So in your example code the dependency you have defined is that stage "Preparation" will be executed first and than only "hello1" and "hello2" step will be executed in parallel. However there is no guarantee as to which "hello world1" or "hello world 2" would be printed.
3) Is there a way to limit the stages that a certain user can start? Imagine that I only want a specific user to launch the Greeting stage.
You can have an manual approval step just before some stage. For example, in your code you want stage Preparation to be executed and than you want it to go with manual approval before executing stage Greeting, your code will look something like this:
stage('Preparation'){
echo """
Preparing
something
"""
}
stage concurrency: 1, name: 'approve-greeting'
input id: 'greeting-deploy', message: 'Proceed to Greeting?', ok: 'Deploy'
stage('Greeting') {
parallel 'hello1':{
node{
echo 'hello world 1'
}
}, 'hello2':{
node{
echo 'hello world 2'
}
}
}
What will happen after this is when you execute the build the stage preparation will get executed but after that the job would wait for a manual approval to proceed. In the Jenkins Pipeline view the stage would be called "approve-greeting" and it'll wait until someone approves the build by clicking on it in the view.
4) How do you setup manual stages?
I believe this is answered in answer 3?
Please let me know in case you need further information/explanation.
EDIT:: Please find the further answers below:
Build Stage (any user can start it) when it finishes it triggers
automatically the Acceptance Stage.
Clearly Both the Build Stage and Acceptance Stage will be defined as normal stage in Jenkins pipeline. So your code would be simple like :
node {
//define any variable here
// Get source code from repo using checkout to directory say stackoverflow
// Get source code from repo for acceptance test using checkout to directory say stackoverflow-test
//Define any tool like Maven etc. location if required.
dir('stackoverflow') {
stage name: 'build'
//Do required steps
}
dir('stackoverflow-test') {
stage name: 'Acceptance'
//Do required steps here
}
At this point, when Acceptance Stage has finished OK, two things can
happen:
Maybe QA Stage is needed to run more tests (Cucumber, manual, etc.).
Some some authorized user would fire this stage.
When the product owner is happy, he can decide to launch the Production Stage to deploy the code in a production environment.
This you can do by having input option so after the above piece of code you could now write:
stage 'promotion'
def userInput = input(
id: 'userInput', message: 'Let\'s promote?', parameters: [
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Production', name: 'prod'],
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'ManualQA', name: 'qa']
])
echo ("Env: "+userInput['prod'])
echo ("Target: "+userInput['qa'])
Than you can take the value from above and manipulate the flow again. Like :
If the value of prod is true than proceed to Production stage,
If the value of qa is true than proceed to QA-Manual stage much like my above example code of FASTFORWARD_TO_GREETING.
EDIT 2
Further answering questions on the comment section:
1) How or where do I specify parameters like FASTFORWARD_TO_GREETING
Parameters like FASTFORWARD_TO_GREETING will be defined as Job level parameter
2) In the promotion stage you have to choose between ManualQA and
Production. If the user chooses ManualQA it runs that Stage skipping
Production. After it I want the user to be promted if he wants to
promote to production stage. If you could provide a full definition of
the pipeline it'd be great.
This you could manipulate after MaualQA stage with another input step but this time with only one parameter. So after stage Promotion, there would be stage ManualQA and than after that this below input step:
def userInput1 = input(
id: 'userInput', message: 'Let\'s promote?', parameters: [
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Production', name: 'prod']
])
3) How can I determine if a user has permissions to run a stage or
not. Ideally I would like to do it based on roles
I am not sure about how to do it with roles but I believe anyone with administrator access or running that job would have access to run/approve that stage, but I am not 100% sure if it could be somehow modified.
This is a full continuous delivery pipeline built with the indications I got from the accepted anwer:
node{
def user
def userInput
def mvnHome = tool 'M3'
wrap([$class: 'BuildUser']) {
user = env.BUILD_USER_ID
}
stage('Commit Stage'){
echo 'Downloading from Git...'
git 'https://github.com/codependent/spring-nio-rest.git'
'Building project...'
sh "${mvnHome}/bin/mvn clean install -DskipTests"
}
stage('Acceptance Stage') {
echo """
Getting image from Nexus...OK
Deploying image...OK
Executing tests...OK
"""
userInput = input(id: 'userInput', message: 'Select the next stage:', parameters: [
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Run QA tests', name: 'QA'],
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Run performance tests', name: 'performance']
])
}
if(userInput['QA']){
stage('QA Stage') {
echo """
Getting image from Nexus...OK
Deploying image...OK
Executing QA tests...OK
"""
}
}
if(userInput['performance']){
stage('Performance Stage') {
echo """
Getting image from Nexus...OK
Deploying image...OK
Executing Performance tests...OK
"""
}
}
stage('Production Stage') {
input message: 'Are you sure you want to deploy to Production?', submitter: 'codependent'
echo 'Deploying to Production...OK'
}
}