Our Jenkins setup consists of master nodes and different / dedicated worker nodes for running jobs in dev, test and prod environment. How do I go about creating a scripted pipeline code that allows users to select environment (possibly from master node) and depending upon the environment selected would execute the rest of the job in the node selected? Here is my initial thought:
stage('Select environment ') {
script {
def userInput = input(id: 'userInput', message: 'Merge to?',
parameters: [[$class: 'ChoiceParameterDefinition', defaultValue: 'strDef',
description:'describing choices', name:'Env', choices: "dev\ntest\nprod"]
])
println(userInput);
}
echo "Environment here ${params.Env}" // prints null here
stage("Build") {
node(${params.Env}) { // schedule job based upon the environment selected earlier
echo "My test here"
}
}
}
I am in the right path or should I be looking at something else?
Another follow up question is that the job that is running on the worker node also requires additional user input. Is there a way to combine the user input in one go such that the users would not be prompted with multiple user screens?
If you pass the environment as a build parameter when kicking off the job, and you have appropriate labels on your nodes, you could do something like:
agent = params.WHAT_NODE
agentLabels = "deploy && ${agent}"
pipeline {
agent { label agentLabels }
....
}
Ended up doing the following for scripted pipeline:
The code for selecting environment can be run on any node (whether master or slaves with agent running). The parameter can be injected into an environment variable: env..
node {
stage('Select Environment'){
env.Env = input(id: 'userInput', message: 'Select Environment',
parameters: [[$class: 'ChoiceParameterDefinition',
defaultValue: 'strDef',
description:'describing choices',
name:'Env',
choices: "jen-dev-worker\njen-test-worker\njen-prod-worker"]
])
println(env.Env);
}
stage('Display Environment') {
println(env.Env);
}
}
The following code snippet ensures that script would be executed on the environment selected in the last step. Requires Jenkins workers with labels: jen-dev-worker, jen-test-worker, jen-prod-worker) available.
node (env.Env) {
echo "Hello world, I am running on ${env.Env}"
}
Related
I have a pipeline parameterized by env which takes specific values shown below. The parameter is used in the script and cannot change. But the admin tells me that the labels for the agent have to depend on the parameter env and another fixed one (e.G. LABELX).
The problem I encounter is that, while the script requires exactly the values shown below, the label for the agent is not always ${params.env}, but in one case there's a mapping/translation to be made.
This is the extremely reduced groovy script:
pipeline {
agent {label "${params.env} && LABELX"}
parameters {
choice(
name: 'env',
choices: ['development', 'staging', 'production'],
)
}
stages {
stage('Process') {
steps {
sh """
# use ${params.env} in bash script
""""
}
}
}
}
The mapping I need is as follows:
env
label
development
development
staging
test
production
production
How can I replace the parameter staging with the label test before choosing an agent? I cannot do that in a script, because scripts are run by agents... I have to somehow do this before, inside the ${params.env} maybe. Or do I need an additional parameter (params.label)?
One way to solve it is to create a constant label mapping before your pipeline, and then use it in your pipeline to retrieve the needed value.
Something like:
LABELS = ['development':'development', 'staging':'test', 'production':'production']
pipeline {
agent {
label "${LABELS[params.env]} && LABELX"
}
parameters {
choice(
name: 'env',
choices: ['development', 'staging', 'production'],
)
}
stages {
stage('Process') {
steps {
sh """
# use ${params.env} in bash script
"""
}
}
}
}
By the way, it is not recommended to call your parameter env as it may override or collide in some cases with the default env map that contains all the environment parameters of the job, including those defined in the environment directive.
I am working on creating a single Jenkins job that allows you to pick the GitHub project and then select the branch you would like to run your SonarQube tests on.
So far I have been able to create a job that ONLY runs against the Master build of each project.
Does anyone have any experience creating something like this?
Thanks!
You need to parametrize your build.
You will have to make gitproject and gitBranch as a parameter this will make you select the project you want to run and select the branch too. Here is an example
pipeline {
agent {
node {
label any
}
}
parameters {
choice(
name: 'PLATFORM',
choices:"Test\nArt19-Data-Pipeline\nBrightcove-Report\nBrightcove-Video\nData-Delivery\nGlobal_Facebook_Engagement_Score\nGoogle-Analytics-Data-Pipeline\nInstagram-Data-Pipeline\nTwitter-Analytics\nTwitter-Data-Pipeline\nYoutube-Data",
description: "Choose the lambda function to deploy or rollback")
choice(
name: 'STAGE',
choices:"dev\nstag",
description: "Choose the lambda function to deploy or rollback")
}
stages {
stage("Git CheckOut") {
steps {
//CheckOut from the repository
//git credentialsId: 'svc.gitlab',branch:'master', url: 'git#git.yourProjectURL/yourProjectName.git'
echo " Parameters are ${PLATFORM}"
echo " STAGE IS ${STAGE}"
}
}
}
}
All you need is replace the 'master' with a a paramter and the 'yourProjectName' with another paramter instead of the one i used as example
I am building Jenkins with a Dockerfile, and during the Docker build I would like to have Jenkins pre-configured with a set of jobs. I find this works well with Jobs DSL, where jobs are seeded, but I have yet to preconfigure the "Pipeline" DSL. Given the direction of Jenkins and use of Jenkisfile, Pipeline, etc, I think there must be some way to allow Jenkins to automatically run with a set of jobs that were built using the Pipeline approach
Example Pipeline:
pipeline {
agent {
label 'cft'
}
parameters {
string(name: 'StackName', defaultValue: 'cft-stack', description: 'The name to give the CFT stack.')
string(name: 'KeyName', defaultValue: 'ACCOUNT', description: 'The account key to use for encryption.')
string(name: 'VpcId', defaultValue: 'vpc-1234', description: 'The VPC to assign to the cluster resources.')
string(name: 'SubnetID', defaultValue: 'subnet-1234, subnet-6789', description: 'The subnet(s) to assign to the cluster resources.')
stages {
stage('Build') {
steps {
s3Download(file:'cft.yaml'
, bucket:'cft-resources'
, path:'cft.yaml'
, force:true)
cfnUpdate(stack:"${params.StackName}"
, file:"cft.yaml"
, params:[
"SnapshotId=${params.SnapshotId}",
"KeyName=${params.KeyName}",
"VpcId=${params.VpcId}"
]
, timeoutInMinutes: 20
)
}
}
}
post {
failure {
echo 'FAILURE'
cfnDelete(stack:"${params.StackName}")
}
}
}
Dockerfile:
COPY ./groovy/*.groovy /usr/share/jenkins/ref/init.groovy.d/
Pipeline's Groovy files differ from the Groovy code that can be executed to configure Jenkins. You can't add pipelines the way you're trying to do.
Your options include
copy the XML file for the job definition (pointing to your repo, as the pipeline should be in the Jenkinsfile in the repo)
create a job using Groovy and configure it (not really practicable IMHO)
use JobDSL (again, with XML as starting point) to specify your Jenkins jobs. An example for automatically adding this can be found in tknerr/jenkins-pipes-infra.
I want to set some jenkins environment variables in run time based on my computation. How can i set this run-time in my jenkinsfile's step section.
for example: based on my calculation i get abc=1. How can i set this in real time in my jenkinsfile's step section so that i can use it later by calling $abc.
I am declaring my pipeline and environment variables as explained here:
https://jenkins.io/doc/pipeline/tour/environment/
i'm using Jenkins ver. 2.41
Here an example how to set variables and use it in the same Jenkinsfile.
The Variable versionToDeploy will be used by the build job step.
pipeline {
agent any
stages {
stage('Example') {
steps {
echo 'build the artifacts'
script {
versionToDeploy = '2.3.0'
}
}
}
}
post {
success {
echo 'start deploy job'
build job: 'pipeline-declarative-multi-job-deploy', parameters: [[$class: 'StringParameterValue', name: 'version', value: versionToDeploy]]
}
}
}
I am completely new to Jenkins 2 pipelines.
I had played with Jenkins 1 pipelines with the following view:
You could start a certain stage directly, let's say that I could choose to start running from the release stage, skipping Test.
I have a pretty simple Jenkins 2 pipeline definition:
stage('Preparation'){
echo """
Preparing
something
"""
}
stage('Greeting') {
parallel 'hello1':{
node{
echo 'hello world 1'
}
}, 'hello2':{
node{
echo 'hello world 2'
}
}
}
In the pipeline page I have "Build now" which runs all stages starting from Preparation.
My questions are:
How can I run the stage I prefer? For instance Greeting instead of starting from Preparation?
How do you define the dependencies between stages? I mean the stage called after another one completes
Is there a way to limit the stages that a certain user can start? Imagine that I only want a specific user to launch the Greeting stage.
How do you setup manual stages?
UPDATE: The real goal behind my questions is the modelling of a continuous delivery pipeline like the following with Jenkins 2 pipelines:
Build stage --> AUTO --> Acceptance Stage --> MANUAL --> Production Stage
--> MANUAL --> QA Stage
This is the behaviour I want:
Build Stage (any user can start it) when it finishes it triggers automatically the Acceptance Stage. This one can't be lauched manually, only automatically after succesfully finishing the Build Stage.
From Acceptance Stage I need that only authorized users can manually trigger QA Stage and Production Stage.
The business flow would be: a developer hits Build Stage, its code is built and packaged. Acceptance Stage begins, using the packaged code to run a bunch of automated tests.
At this point, when Acceptance Stage has finished OK, two things can happen:
Maybe QA Stage is needed to run more tests (Cucumber, manual, etc.). Some some authorized user would fire this stage.
When the product owner is happy, he can decice to launch the Production Stage to deploy the code in a production environment.
I am struggling to model this with Jenkins 2 pipelines.
There is no direct answer to some of your questions but they can be achieved with some additional coding. While certain people might find some other way to achieve but let me try with what I have in my mind:
1) How can I run the stage I prefer? For instance Greeting instead of starting from Preparation?
This could be achieved by adding a Boolean parameter FASTFORWARD_TO_GREETING and than using the value provided while executing build to manipulate the flow of your build. So your code will now look like :
if (FASTFORWARD_TO_GREETING == 'false') {
stage('Preparation'){
echo """
Preparing
something
"""
}
}
stage('Greeting') {
parallel 'hello1':{
node{
echo 'hello world 1'
}
}, 'hello2':{
node{
echo 'hello world 2'
}
}
}
2) How do you define the dependencies between stages? I mean the stage called after another one completes
Stages are executed serially, so if a stage is defined first it'll be started and completed first before moving to the next stage. However in parallel step this does not hold true, as all the steps will be executed in parallel. So in your example code the dependency you have defined is that stage "Preparation" will be executed first and than only "hello1" and "hello2" step will be executed in parallel. However there is no guarantee as to which "hello world1" or "hello world 2" would be printed.
3) Is there a way to limit the stages that a certain user can start? Imagine that I only want a specific user to launch the Greeting stage.
You can have an manual approval step just before some stage. For example, in your code you want stage Preparation to be executed and than you want it to go with manual approval before executing stage Greeting, your code will look something like this:
stage('Preparation'){
echo """
Preparing
something
"""
}
stage concurrency: 1, name: 'approve-greeting'
input id: 'greeting-deploy', message: 'Proceed to Greeting?', ok: 'Deploy'
stage('Greeting') {
parallel 'hello1':{
node{
echo 'hello world 1'
}
}, 'hello2':{
node{
echo 'hello world 2'
}
}
}
What will happen after this is when you execute the build the stage preparation will get executed but after that the job would wait for a manual approval to proceed. In the Jenkins Pipeline view the stage would be called "approve-greeting" and it'll wait until someone approves the build by clicking on it in the view.
4) How do you setup manual stages?
I believe this is answered in answer 3?
Please let me know in case you need further information/explanation.
EDIT:: Please find the further answers below:
Build Stage (any user can start it) when it finishes it triggers
automatically the Acceptance Stage.
Clearly Both the Build Stage and Acceptance Stage will be defined as normal stage in Jenkins pipeline. So your code would be simple like :
node {
//define any variable here
// Get source code from repo using checkout to directory say stackoverflow
// Get source code from repo for acceptance test using checkout to directory say stackoverflow-test
//Define any tool like Maven etc. location if required.
dir('stackoverflow') {
stage name: 'build'
//Do required steps
}
dir('stackoverflow-test') {
stage name: 'Acceptance'
//Do required steps here
}
At this point, when Acceptance Stage has finished OK, two things can
happen:
Maybe QA Stage is needed to run more tests (Cucumber, manual, etc.).
Some some authorized user would fire this stage.
When the product owner is happy, he can decide to launch the Production Stage to deploy the code in a production environment.
This you can do by having input option so after the above piece of code you could now write:
stage 'promotion'
def userInput = input(
id: 'userInput', message: 'Let\'s promote?', parameters: [
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Production', name: 'prod'],
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'ManualQA', name: 'qa']
])
echo ("Env: "+userInput['prod'])
echo ("Target: "+userInput['qa'])
Than you can take the value from above and manipulate the flow again. Like :
If the value of prod is true than proceed to Production stage,
If the value of qa is true than proceed to QA-Manual stage much like my above example code of FASTFORWARD_TO_GREETING.
EDIT 2
Further answering questions on the comment section:
1) How or where do I specify parameters like FASTFORWARD_TO_GREETING
Parameters like FASTFORWARD_TO_GREETING will be defined as Job level parameter
2) In the promotion stage you have to choose between ManualQA and
Production. If the user chooses ManualQA it runs that Stage skipping
Production. After it I want the user to be promted if he wants to
promote to production stage. If you could provide a full definition of
the pipeline it'd be great.
This you could manipulate after MaualQA stage with another input step but this time with only one parameter. So after stage Promotion, there would be stage ManualQA and than after that this below input step:
def userInput1 = input(
id: 'userInput', message: 'Let\'s promote?', parameters: [
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Production', name: 'prod']
])
3) How can I determine if a user has permissions to run a stage or
not. Ideally I would like to do it based on roles
I am not sure about how to do it with roles but I believe anyone with administrator access or running that job would have access to run/approve that stage, but I am not 100% sure if it could be somehow modified.
This is a full continuous delivery pipeline built with the indications I got from the accepted anwer:
node{
def user
def userInput
def mvnHome = tool 'M3'
wrap([$class: 'BuildUser']) {
user = env.BUILD_USER_ID
}
stage('Commit Stage'){
echo 'Downloading from Git...'
git 'https://github.com/codependent/spring-nio-rest.git'
'Building project...'
sh "${mvnHome}/bin/mvn clean install -DskipTests"
}
stage('Acceptance Stage') {
echo """
Getting image from Nexus...OK
Deploying image...OK
Executing tests...OK
"""
userInput = input(id: 'userInput', message: 'Select the next stage:', parameters: [
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Run QA tests', name: 'QA'],
[$class: 'BooleanParameterDefinition', defaultValue: false, description: 'Run performance tests', name: 'performance']
])
}
if(userInput['QA']){
stage('QA Stage') {
echo """
Getting image from Nexus...OK
Deploying image...OK
Executing QA tests...OK
"""
}
}
if(userInput['performance']){
stage('Performance Stage') {
echo """
Getting image from Nexus...OK
Deploying image...OK
Executing Performance tests...OK
"""
}
}
stage('Production Stage') {
input message: 'Are you sure you want to deploy to Production?', submitter: 'codependent'
echo 'Deploying to Production...OK'
}
}