Is there a way to change how Jenkins computes build health based on failing tests? I'd like Jenkins to treat any failing tests as something other than "sunny", but so far my searching for a way to do that has been fruitless. Does the standard setup allow this level of control, or are there perhaps plugins that can do this?
If you use the xUnit plugin, which supports several test report formats, you can add a post-build action to parse your build's test reports and declare a build as unstable or failed depending on the number of tests that fail or are skipped.
Related
target/surefire-reports/TEST*.xml, and
target/surefire-reports/junitreports/TEST*.xml
Above files only have the re-run tests from testng-failed.xml.
But I need complete test results to be published in Jenkins, the successful cases and failed cases, all.
Need a solution that does not needs code changes like implementing listeners etc.
You should find a testng-results.xml file somewhere in target/** which is what you are looking for.
It is recommended to use the TestNG Jenkins plugin too.
We are using Jenkins to run the selenium automation tests and my manager wants to see the list of failed builds and what percentage of the tests passed for the builds. We also have manual tests that get executed in JIRA. I need to combine both and derive the test metrics from them.
The way I think of proceeding is as follows:
Get the Jenkins data in JIRA first using the Jenkins plugin for JIRA.
Use the jira api to collect the testing results from Jenkins and manual tests run on jira.
Prepare a dashboard in JIRA to display all the metrics
Could you suggest if the above approach is correct and suggest something additional.
Thanks in advance!
Are you using cucumber? In that case you could use the cucumber reporting plugin for jenkins. If it doesn't suit your needs but you still use cucumber you can also generate reports in a format like JSON, which you could later parse and get your data.
I have the feeling what you want to do seems a bit complicated, and with not a big benefit. If the tests are failing it's likely you'll have to see what is happening. Having the percentage is sure nice, but I think you can spend some hours/days tailoring this just for having something cute that your manager wants but that has no specific purpose. I would opt for something simpler.
If the automated tests fail, create a jira issue automatically with jenkins. You could put the build number as a tag, or in the title. You can also create it always to indicate that build nr. ## was tested and everything went ok.
As a part of the manual testing process, report in jira what failed.
Create a dashboard and play a bit with tags and search to show which builds failed.
I would suggest AssertThat BDD & Test Management in Jira
Provides end-to-end integration - from features creation to manual and automated tests execution and reporting. Out of the box integration with test automation frameworks through plugins.
The plugin allows to download feature files stored in Jira before the run, execute the test in the usual way and then upload cucumber tests results back to Jira, which gives you a clear view on the testing progress in one place.
More info and usage examples on website https://www.assertthat.com/
I'm looking for a jenkins plugin that would allow selection of file(s)/folder(s) for a parameterized build.
The purpose is to be able to select different tests to execute, each test being defined as a .xml file in a svn repo.
eg of repo structure:
tests/business/cars/buy.xml
tests/business/cars/sell.xml
tests/system/core/stuff.xml
I'm not sure if these quite get at what you're looking for, but I found the following two possibilities:
Test In Progress which, based on their GitHub information, should allow you to run tests based on pattern matching and view the progress as they run.
RQM is an IBM tool made to allow you to run specific test cases in a test plan that have a custom property that you configured and provide at test execution time.
edit:
I just found Multi Module Test Publisher, which may be a little closer to what you're looking for. It acts as a replacement for the normal junit plugin and allows you to group junit tests into suites and view statistics on each separately.
Is it possible to set the build result for a build after that build is complete?
I could not find any plugins that do this already, and I was considering writing my own, but I wanted to see if this was even possible before going down that path.
(I have looked at existing code and how the "Fail The Build" plugin works as an example, but my understanding of the Jenkins code base is not advanced enough to understand what all the possibilities are.)
Use case: we have a build pipeline, and near the end of the pipeline there is a deploy-to-qa step that deploys the artifact to a QA environment. We have automated tests before this step to try to catch any problems with the artifact, but our test coverage is not very high in some areas so bugs could still slip through the cracks. I'd like to have the ability to mark a deploy-to-qa build as FAILED after the fact, to denote that that particular pipeline was invalid and is not a candidate for production release. (Basically the same as this Build Pipeline Plugin issue)
After some more investigation in the code, I believe that this is not possible.
From hudson.model.Run:
public void setResult(Result r) {
// state can change only when we are building
assert state==State.BUILDING;
// snip
...
}
So the build result cannot change except when in "building" state.
I could try to muck with the lastSuccessful and lastStable symlinks (as is done with the delete() function in hudson.model.AbstractBuild), but then those would be reset as soon as Jenkins reloaded the build results from jobs/JOBNAME/builds/.
I have an untested suggestion: Make a parametrized build, where the parameter determines if build will fail or not (for example simple bat / shell script testing the parameter from the environment variable it sets, and doing exit 0 or exit 1). This assumes that build pipelines manually triggered step will ask the parameters, and not use default values.
If it does not support interactive build parameters, then some other way is needed to tell this extra build step wether it should fail or not. Maybe editing upstream build description or display name to indicate failure, and then allowing build pipeline to continue to this extra build step, which probably has to use system groovy script to dig out upstream build description or display name.
I have seen several debates on this topic previously, and the outcome was always that it is theoretically possible to do so, but the codebase is not designed to allow this and it would have to be a very hacky workaround.
It's also been said that this is a bad practice in general, although I don't remember what the argument against it was.
I am facing the same requirement. I haven't found an appropriate plugin, changing the build status is not just a flag but has other impacts on links (eg latest successful build etc). So instead of changing the status of the build I looked for a possibility for qualifying the build. The Promoted Builds Plugin apply flags to build to define e.g. different quality stages. Build promotions can be performed manually or based on e.g. downstream project successful builds. Any successful build can be qualified, based on the promotion additional build and post build actions can be executed, e.g tagging or archiving.
Actually I was able to do it by changing the build.xml manually to <result>FAILURE</result>.
I've then played a little bit with mklink to create some symbolic links and also renamed the lastSuccessfulBuild to lastFailedBuild and it worked. If you are allowed to access the filesystem from within a Jenkins PlugIn, then it is possible to write one.
In case you are fine to delete the current build and start the same build using a version number and setting the next BUILD_NUMBER to the deleted one, then you could use this plugin to tell it to fail instead of succeed:
Fail The Build Plugin
I have a fairly complicated Jenkins job that builds, unit tests and packages a web application. Depending on the situation, I would like to do different things once this job completes. I have not found a re-usable/maintainable way to do this. Is that really the case or am I missing something?
The options I would like to have once my complicated job completes:
Do nothing
Start my low-risk-change build pipeline:
copies my WAR file to my artifact repository
deploys to production
Start my high-risk-change build pipeline:
copies my WAR file to my artifact repository
deploys to test
run acceptance tests
deploy to production
I have not found an easy way to do this. The simplest, but not very maintainable approach would be to make three separate jobs, each of which kicks off a downstream build. This approach scares me for a few reasons including the fact that changes would have to be made in three places instead of one. In addition, many of the downstream jobs are also nearly identical. The only difference is which downstream jobs they call. The proliferation of jobs seems like it would lead to an un-maintainable mess.
I have looked at using several approaches to keep this as one job, but none have worked so far:
Make the job a multi-configuration project (https://wiki.jenkins-ci.org/display/JENKINS/Building+a+matrix+project). This provides a way to inject the job with a parameter. I have not found a way to make the "build other projects" step respond to a parameter.
Use the Parameterized-Trigger plugin (https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Trigger+Plugin). This plugin lets you trigger downstream-jobs based on certain triggers. The triggers appear to be too restrictive though. They're all based on the state of the build, not arbitrary variables. I don't see any option provided here that would work for my use case.
Use the Flexible Publish plugin (https://wiki.jenkins-ci.org/display/JENKINS/Flexible+Publish+Plugin). This plugin has the opposite problem as the parameterized-trigger plugin. It has many useful conditions it can check, but it doesn't look like it can start building another project. Its actions are limited to publishing type activities.
Use Flexible Publish + Any Build Step plugin (https://wiki.jenkins-ci.org/display/JENKINS/Any+Build+Step+Plugin). The Any Build Step plugin allows making any build action available to the Flexible Publish plugin. While more actions were made available once this plugin was activated, those actions didn't include "build other projects."
Is there really not an easy way to do this? I'm surprised that I haven't found it and even more surprised that I haven't really seen any one else trying to do this? Am I doing something unusual? Is there something obvious that I am missing?
If I understood it correct you should be able to do this by following these Steps:
First Build Step:
Does the regular work. In your case: building, unit testing and packaging of the web application
Depending on the result let it create a file with a specific name.
This means if you want the low-risk-change to run afterwards create a file low-risk.prop
Second Build Step:
Create a Trigger/call builds on other projects Step from the Parameterized-Trigger
plugin.
Entery the name of your low-risk job into the Projects to build field
Click on: Add Parameter
Choose: Parameters from properties File
Enter low-risk.prop into the Use properties from file Field
Enable Don't trigger if any files are missing
Third Build Step:
Check if a low-risk.prop file exists
Delete the File
Do the same for the high-risk job
Now you should have the following Setup:
if a file called low-risk.prop occurs during the first Build Step the low-risk job will be started
if a file called high-risk.prop occurs during the first Build Step the high-risk job will be started
if there's no .prop File nothing happens
And that's what you wanted to achieve. Isn't it?
Have you looked at the Conditional Build Plugin? (https://wiki.jenkins.io/display/JENKINS/Conditional+BuildStep+Plugin)
I think it can do what you're looking for.
If you want a conditional post-build step, there is a plugin for that:
https://wiki.jenkins-ci.org/display/JENKINS/Post+build+task
It will search the console log for a RegEx you specify, and if found, will execute a custom script. You can configure fairly complex criteria, and you can configure multiple sets of criteria each executing different post build tasks.
It doesn't provide you with the usual "build step" actions, so you've got to write your own script there. You can trigger execution of the same job with different parameters, or another job with some parameters, in standard ways that jenkins supports (for example using curl)
Yet another alternative is Jenkins text finder plugin:
https://wiki.jenkins-ci.org/display/JENKINS/Text-finder+Plugin
This is a post-build step that allows to forcefully mark a build as "unstable" if a RegEx is found in console text (or even some file in workspace). So, in your build steps, depending on your conditions, echo a unique line into console log, and then do a RegEx for that line. You can then use "Trigger parameterized buids" and set the condition as "unstable". This has an added benefit of visually marking the build different (with a yellow ball), however you only have 1 conditional option with this method, and from your OP, looks like you need 2.
Try a combination of these 2 methods:
Do you use Ant for your builds?
If so, it's possible to do conditional building in ant by having a set of environment variables your build scripts can use to conditionally build. In Jenkins, your build will then be building all of the projects, but your actual build will decide whether it builds or just short-circuits.
I think the way to do it is to add an intermediate job that you put in the post-build step and pass to it all the parameters your downstream jobs could possibly need, and then within that job place conditional builds for the real downstream jobs.
The simplest approach I found is to trigger other jobs remotely, so that you can use Conditional Build Plugin or any other plugins to build other jobs conditionally.