We have a requirement to build a cucumber BDD Scenario where each step will trigger a Jenkins job. This is needed for a project which is data driven, data is created via Jenkins job and is published to a common location which can be accessed from other jobs. Could someone please advise if this can be achieved?
Many thanks in advance!
Example :
Given I create testdata in stage1 --> {this will trigger a Jenkins job to create the test data in stage1}
And I create testdata in stage2 --> {this will trigger a Jenkins job to create the test data in stage2}
Then I validate the response --> {this will trigger a Jenkins job to validate the response}
Related
What are seed jobs in Jenkins and how does it work ?
Can we create a new job from seed job without using github ?
That depends on context. Jenkins itself does not provide "seed jobs".
There's plugins that allow creating jobs from other jobs, like the excellent Job-DSL plugin. With that, you can create jobs where a groovy script creates a larger number of jobs for you.
The Job-DSL plugin refers to those jobs as "seed jobs" (but they're regular freestyle or pipeline jobs). The Job-DSL plugin does not require a github connection.
The seed job is a normal Jenkins job that runs the Job DSL script; in turn, the script contains instructions that create additional jobs. In short, the seed job is a job that creates more jobs. In this step, you will construct a Job DSL script and incorporate it into a seed job. The Job DSL script that you’ll define will create a single freestyle job that prints a 'Hello World!' message in the job’s console output.
A Job DSL script consists of API methods provided by the Job DSL plugin; you can use these API methods to configure different aspects of a job, such as its type (freestyle versus pipeline jobs), build triggers, build parameters, post-build actions, and so on. You can find all supported methods on the API reference site.
The jobs we used for creating new jobs are called Seed Jobs and this seed job generates new jobs using Jenkins files (using JobDSL plugin).
Here, we disabling this feature (Enable script security for Job DSL scripts)
Jenkins Dashboard→ Manage Jenkins → Configure Global Security
Way to create seed job :
JobDSL scripts for generating new jobs.
Job1.groovy
job("Job1"){
description("First job")
authenticationToken('secret')
label('dynamic')
scm {
github('Asad/jenkins_jobDSL1', 'master')
}
triggers {
gitHubPushTrigger()
}
steps {
shell ('''
echo "test"
''')
}
}
buildPipelineView('project-A') {
title('Project A CI Pipeline')
displayedBuilds(5)
selectedJob('Job1')
showPipelineParameters()
refreshFrequency(60)
}
and create same way others Job2.groovy and so on.
For Jenkins Job DSL documentation:-
Follow https://jenkinsci.github.io/job-dsl-plugin/
Think about a job - what is it actually ?
It is actually just a java/jre object that represents like this
How you generates such job/build ?
Configure Jenkins UI -> rest api to Jenkins url -> Jenkins service receive your call on the relevant endpoint -> calling to the relevant code/method and generate this new job
How Seed job will make it ?
Configure seed job on Jenkins UI only once -> run this seed job - > this code run against the internal Jenkins methods and skip all the manual process describes above
Now, when your code can talk directly to Jenkins code , things are much easier.just update your code on the relevant repo - and you are done
I understand this post is similar to:
Aggregating results of downstream is no test in Jenkins
and also to:
Aggregating results of downstream parameterised jobs in Jenkins
Nonetheless, I am not able to figure out for my case, how to make this working. I am currently using Jenkins 1.655.
I have jobs A, B, C - A being the upstream job. What I want to do is to have A call B and B call C. All needs to block and wait for completion of the next. If one fails, all fails. B and C generate unit test reports. So I want to aggregate these reports in A and then Publish that result in A. So, here's the current setup of the jobs:
Job A:
Build Steps
Execute shell: echo $(date) > aggregate
Trigger Parametrized Buid Job: Job B
Post Build Steps
Aggregate downstream test results
Record fingerprints of files to track usage: set Files to fingerprint to aggregate
Publish JUnit test result report (report files from B and C)
Job B:
Build Steps
Copy artifacts from another project: copy from upstream job aggregate file
Run tests to generate unit test reports
Trigger Parametrized Build Job: Job C
It ultimately fails here because aggregate is only archived in the
Post Build Steps of Job A. How can I archive an artifact in the Build Step?
Post Build Steps
Aggregate downstream test results (unit test.xml generated)
Record fingerprints of files to track usage: set Files to fingerprint to aggregate
I won't post Job C here for simplicity but it follows pretty much what B does.
So, summing it up, I want to have interlinked jobs that depend on each other and uses the parametrized plugin and the upstream job must aggregate the test results of all downstream.
Any help appreciated, thanks!
If you have no limitation on where to run your jobs you can always specify it to run on the same workspace\machine - this will solve all your issues.
If for some reason you can't run it on the same workspace, instead of using the copy artifact plugin you can use the link in Jenkins to the WS (guessing you're using Parameterized Trigger Plugin) so it'll be easy to wget the "aggregate" file from A job from the triggered job using the defined: TRIGGERED_BUILD_NUMBER_="Last build number triggered" from A. This will also help you to keep track of the jobs B and C you triggered to get the artifacts from there.
Hope it helps!
I am attempting to write a custom pre-send script for Jenkins email-ext plugin. I'm at the point where I can write a simple 'hello world' groovy script and run it with the "Email template testing" tool for the job I want to customize.
Alas, I'm having a bit of trouble navigating Jenkins build API. As a starting point, the job whose email I want to customize has N build steps. I only want to send an email if one of those steps faild, and I'd like to write a different message depending on which step failes.
How can determine the number of steps in the build (N), and which step failed?
Which build step caused the build result to change is only recorded in the console output, e.g.
Build step 'Execute shell' marked build as failure
This isn't otherwise stored as part of the build record, and so isn't available via the API.
So you would have to try scraping the console output to do what you want.
To determine how many build steps there are, you'd need to fetch the job configuration via /job/foo/config.xml and count how many items there are within the <project><builders> tag (assuming a freestyle project).
I have 4 jobs which needs to be executed in the following sequence
JOB A
|------> JOB B
|------> JOB C
|------> JOB D
In the above
A should trigger B & C parallely and C inturn triggers D.
A should hold the job as running till all 3 of them completed.
I tried the following plugins and couldn't achieve what I am looking for
Join Plugin
Multijob Plugin
Multi-Configuration Project
Paramterized Trigger Plugin
Is there any plugin which I haven't tried would help me in resolving this. Or is this can be achieved in a different way. Please advise.
Use DSL Script with Build Flow plugin.
try this Example for your execution:
build("job A")
parallel
(
{build("job B")}
{build("job C")}
)
build("job D")
Try the Locks and Latches plugin.
This may not be optimal way, but it should work. Use the Parameterized Trigger Plugin. To Job A, add a build step (NOT a Post Build Action) to start both Jobs B and C in the same build step AND block until they finish. In Job C, add a build step (NOT a Post Build Action) that starts Job D AND blocks until it is finished. That should keep Job A running for the full duration.
This isn't really optimal though: Job A is held open waiting for B and C to finish. Then C is held open until D is finished.
Is there some reason that Job A needs to remain running for the duration? Another possibility is to have Job A terminate after B and C are started, but have a Promotion on Job A that will execute your final actions after jobs B, C and D are successful.
I am trying to build a same system. I am building a certification pipeline where I need to run packager/build/deploy jobs and and corresponding test jobs. When all of them are successful, I want to aggregate the test results and trigger the release job that can do an automated maven release.
I selected Build pipeline plugin for visualization of the system. Initially tried with Parameterized trigger Plugin with blocking builds. I could not setup archiving the artifacts/fingerprinting and downstream build relationship this way since archiving the artifacts works only in postbuild. Then I put the Parameterized trigger in Post build activity. This way I was able to setup downstream builds, fingerprinting, aggregate test results but the build failures were not bubbling to upstream job chain and upstream jobs were non blocking
I was finally able to achieve this using these plugins-
Build Pipeline
MultiJob Plugin
FingerPrint Plugin
Copy Artifacts Plugin
Join Plugin
I'm using Jenkins 1.514
System looks like this
Trigger Job --> build (and deploy) Job (1..n) ---> Test Job (1..n)
Trigger Job -
Create as MultiJob and create a fingerprint file in shell exec
echo date +%s > fingerprint.txt
Trick is that file needs to be archived during the build, to do that execute this script-
ARCHIVEDIR=$JENKINS_HOME/jobs/$JOB_NAME/builds/$BUILD_ID/archive
mkdir $ARCHIVEDIR
cp fingerprint.txt $ARCHIVEDIR
Create MultiJob Phase consisting of build/deploy job.
Build/deploy job is itself a multijob
follow the same steps for creating build/deploy job as above relative
to fingerprinting.
Copy the fingerprint.txt artifact from upstream job
Setup MultiJob phase in deploy job that triggers the test job
create a new fingerprint file and force archive it similar to above step
Collect Junit results in the final test job.
In the trigger Job, use Join Plugin to execute the Release Job by choosing 'Run Post Build Actions at join' and execute the release project only on stable build of Trigger Job.
This way all the steps are showing up in Build Pipeline view and Trigger job is blocking for all downstream builds to finish and sets its status as the worst downstream build to give a decision point for release job.
Multijob Plugin
If you'd like to stop the mess with downstream / upstream jobs chains definitions. Or when you want to add a full hierarchy of Jenkins jobs that will be executed in sequence or in parallel. Add context to your buildflow implementing parameter inheritance from the MultiJob to all its Phases and Jobs. Phases are sequential while jobs inside each Phase are parallel.
https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin
In order to get the fastest feedback possible, we occasionally want Jenkins jobs to run in Parallel. Jenkins has the ability to start multiple downstream jobs (or 'fork' the pipeline) when a job finishes. However, Jenkins doesn't seem to have any way of making a downstream job only start of all branches of that fork succeed (or 'joining' the fork back together).
Jenkins has a "Build after other projects are built" button, but I interpret that as "start this job when any upstream job finishes" (not "start this job when all upstream jobs succeed").
Here is a visualization of what I'm talking about. Does anyone know if a plugin exists to do what I'm after?
Edit:
When I originally posted this question in 2012, Jason's answer (the Join and Promoted Build plugins) was the best, and the solution I went with.
However, dnozay's answer (The Build Flow plugin) was made popular a year or so after this question, which is a much better answer. For what it's worth, if people ask me this question today, I now recommend that instead.
Pipeline plugin
You can use the Pipeline Plugin (formerly workflow-plugin).
It comes with many examples, and you can follow this tutorial.
e.g.
// build
stage 'build'
...
// deploy
stage 'deploy'
...
// run tests in parallel
stage 'test'
parallel 'functional': {
...
}, 'performance': {
...
}
// promote artifacts
stage 'promote'
...
Build flow plugin
You can also use the Build Flow Plugin. It is simply awesome - but it is deprecated (development frozen).
Setting up the jobs
Create jobs for:
build
deploy
performance tests
functional tests
promotion
Setting up the upstream
in the upstream (here build) create a unique artifact, e.g.:
echo ${BUILD_TAG} > build.tag
archive the build.tag artifact.
record fingerprints to track file usage; if any job copies the same build.tag file and records fingerprints, you will be able to track the parent.
Configure to get promoted when promotion job is successful.
Setting up the downstream jobs
I use 2 parameters PARENT_JOB_NAME and PARENT_BUILD_NUMBER
Copy the artifacts from upstream build job using the Copy Artifact Plugin
Project name = ${PARENT_JOB_NAME}
Which build = ${PARENT_BUILD_NUMBER}
Artifacts to copy = build.tag
Record fingerprints; that's crucial.
Setting up the downstream promotion job
Do the same as the above, to establish upstream-downstream relationship.
It does not need any build step. You can perform additional post-build actions like "hey QA, it's your turn".
Create a build flow job
// start with the build
parent = build("build")
parent_job_name = parent.environment["JOB_NAME"]
parent_build_number = parent.environment["BUILD_NUMBER"]
// then deploy
build("deploy")
// then your qualifying tests
parallel (
{ build("functional tests",
PARENT_BUILD_NUMBER: parent_build_number,
PARENT_JOB_NAME: parent_job_name) },
{ build("performance tests",
PARENT_BUILD_NUMBER: parent_build_number,
PARENT_JOB_NAME: parent_job_name) }
)
// if nothing failed till now...
build("promotion",
PARENT_BUILD_NUMBER: parent_build_number,
PARENT_JOB_NAME: parent_job_name)
// knock yourself out...
build("more expensive QA tests",
PARENT_BUILD_NUMBER: parent_build_number,
PARENT_JOB_NAME: parent_job_name)
good luck.
There are two solutions that I have used for this scenario in the past:
Use the Join Plugin on your "deploy" job and specify "promote" as the targeted job. You would have to specify "Functional Tests" and "Performance Tests" as the joined jobs and start them via in some fashion, post build. The Parameterized Trigger Plugin is good for this.
Use the Promoted Builds Plugin on your "deploy" job, specify a promotion that works when downstream jobs are completed and specify Functional and Performance test jobs. As part of the promotion action, trigger the "promote" job. You still have to start the two test jobs from "deploy"
There is a CRITICAL aspect to both of these solutions: fingerprints must be correctly used. Here is what I found:
The "build" job must ORIGINATE a new fingerprinted file. In other words, it has to fingerprint some file that Jenkins thinks was originated by the initial job. Double check the "See Fingerprints" link of the job to verify this.
All downstream linked jobs (in this case, "deploy", "Functional Tests" and "Performance tests") need to obtain and fingerprint this same file. The Copy Artifacts plugin is great for this sort of thing.
Keep in mind that some plugins allow you change the order of fingerprinting and downstream job starting; in this case, the fingerprinting MUST occur before a downstream job fingerprints the same file to ensure the ORIGIN of the fingerprint is properly set.
The Multijob plugin works beautifully for that scenario. It also comes in handy if you want a single "parent" job to kick off multiple "child" jobs but still be able to execute each of the children manually, by themselves. This works by creating "phases", to which you add 1 to n jobs. The build only continues when the entire phase is done, so if a phase as multiple jobs they all must complete before the rest are executed. Naturally, it is configurable whether the build continues if there is a failure within the phase.
Jenkins recently announced first class support for workflow.
I believe the Workflow Plugin is now called the Pipeline Plugin and is the (current) preferred solution to the original question, inspired by the Build Flow Plugin. There is also a Getting Started Tutorial in GitHub.
Answers by jason & dnozay are good enough. But in case someone is looking for easy way just use JobFanIn plugin.
This diamond dependency build pipeline could be configured with
the DepBuilder plugin. DepBuilder is using its own domain
specific language, that would in this case look like:
_BUILD {
// define the maximum duration of the build (4 hours)
maxDuration: 04:00
}
// define the build order of the existing Jenkins jobs
Build -> Deploy
Deploy -> "Functional Tests" -> Promote
Deploy -> "Performance Tests" -> Promote
After building the project, the build visualization will be shown on the project dashboard page:
If any of the upstream jobs didn't succeed, the build will be automatically aborted. Abort behavior could be tweaked on a per job basis, for more info see the DepBuilder documentation.