On my Jenkins pipleline I run unit tests on both: debug and release configurations. Each test configuration generates separate JUnit XML results file. Test names on both: debug and release configuration are same. Currently I use the following junit command in order to show test results:
junit allowEmptyResults: true, healthScaleFactor: 0.0, keepLongStdio: true, testResults: 'Test-Dir/Artifacts/test_xml_reports_*/*.xml'
The problem is that on Jenkins UI both: debug and release tests results are shown together and it is not possible to know which test (from debug or release configuration) is failed.
Is it possible to show debug and release tests results separately? If yes, how can I do that?
We run the same integration tests against two different configurations with different DB types. We use maven and the failsafe plugin, so I take advantage of the -Dsurefire.reportNameSuffix so I can see the difference between the two runs.
The following is an example block of our Jenkinsfile:
stage('Integration test MySql') {
steps {
timeout(75) {
sh("mvn -e verify -DskipUnitTests=true -DtestConfigResource=conf/mysql-local.yaml " +
"-DintegrationForkCount=1 -DdbInitMode=migrations -Dmaven.test.failure.ignore=false " +
"-Dsurefire.reportNameSuffix=MYSQL")
}
}
post {
always {
junit '**/failsafe-reports/*MYSQL.xml'
}
}
}
In the report, the integration tests run against mysql then show up with MYSQL appended to their name.
It looks no solution for my question.
As a workaround I changed JUnit XML report format and included build variant name (debug/release) as a package name.
Related
I have test cases as executables ( windows exe ) and I am running all the test cases (exe) in Jenkins. Each test case produce output text file and at the end of the testcase, there is a formatted string like "TestCaseName PASSED/FAILED".
I can see from the build status if all the test cases were successful or not but if any of them fails, the build also says it fails. I want to use Jenkins Test Results Analyser or any other plugin to show as a table to see in which build which test case failed. The Jenkins Test Results Analyser works fine with jUnit or TestNg test reports. I was to thinking to generate manually such reports as post build step or if there is any other tool available?
I recently tried to add a "Publish test results" step within my Jenkins pipeline, but was wondering where to see/retrieve the TestResults.xml file that should be generated while deploying.
For reference, here's the code I'm using, straight from the documentation:
stage("Publish NUnit Test Report") {
steps {
nunit testResultsPattern: 'TestResult.xml'
}
}
Thanks a lot for your help!
I'm using this command to run my tests:
sh "${mvnHome}/bin/mvn clean test -e -Dgroups=categories.dbd"
but sometimes I want to run a specific test. How can I do it?
I read that I can use "This project is parameterized" but didn't understand how to use it.
I also saw this - https://plugins.jenkins.io/selected-tests-executor/ but it's not good enough since it required an external file.
If you use the maven-surefire-plugin you can simply run
sh "${mvnHome}/bin/mvn clean test -e -Dgroups=categories.dbd -Dtest=com.example.MyJavaTestClass"
or
sh "${mvnHome}/bin/mvn clean test -e -Dgroups=categories.dbd -Dtest=com.example.MyJavaTestClass#myTestMethod"
I suggest to add a parameter for the test class/method to your pipeline definition.
pipeline {
agent any
parameters {
string defaultValue: '', description: 'Test Name', name: 'TEST_NAME', trim: false
}
stages {
stage('run tests') {
steps {
script {
def optionalParameters = ""
if (params.TEST_NAME != null) {
optionalParameters += " -Dtest=" + params.TEST_NAME
}
sh "${mvnHome}/bin/mvn clean test -e -Dgroups=categories.dbd" + optionalParameters
}
}
}
...
}
...
}
Jenkins isn't really the tool for that. It's typical use case us running all tests. If you really want to run only one test, you should do that in your development environment
But the simplest thing to do, if you want the results for one of the tests, us to have Jenkins run all your tests and simply ignore the results for the other tests.
Generally, you should run (quick, cheap) unit tests in your development environment, and submit code for (expensive, slow) integration tests by Jenkins only once the code passes the unit tests (And Jenkins should run the unit tests again, just in case).
I suspect your real question is "how do I debug failure of an integration test run by Jenkins". Jenkins is a build and test tool not a debugging tool. It is not itself a suitable tool for debugging test failures. But the way you use Jenkins can help debugging.
Do not use integration tests as a substitute for unit tests.
If your software fails an integration test, but no unit tests, as usual with debugging a test failure you should be making hypotheses about the kinds of defect in your software that could cause the failure. Then check whether a unit test could be added that would detect that kind of defect. If so, add that unit test.
Ensure that your tests produce useful diagnostic messages on failure. Test assertions should have helpful messages. Test a should have descriptive names.
If an integration test checks a sequence of actions, ensure you also have tests for the individual actions.
Trying to use the nifty Jenkins Pipeline, I had problems finding out how to publish NUnit test results.
I am able to run the tests by specifying the following command in the pipeline script:
stage 'Test'
bat '"C:\\Program Files (x86)\\NUnit 2.6.4\\bin\\nunit-console-x86.exe" "ProjectName\\bin\\Release\\UnitTests.net.dll"'
But how to make Jenkins "publish" the test results is not obvious. The Snippet Generator only suggests junit, and that does not seem to work.
I used nunit plugin version 0.21 and was able to publish results using
stage("PublishTestReport"){
nunit testResultsPattern: 'TestResult.xml'
}
(TestResult.xml is at the root of jenkins workspace in this above example)
Investigating the NUnit plugin for Jenkins led me on to this issue, where I found the solution:
step([$class: 'NUnitPublisher', testResultsPattern: 'TestResult.xml', debug: false,
keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])
Adding this to the pipeline script worked for me!
However, it seemed the following should also work (but at the present, apparently, it does not): Using the Snippet Generator, select this:
step: General Build Step
Publish NUnit test result report
This generates the following in the Pipeline script:
step <object of type hudson.plugins.nunit.NUnitPublisher>
This fails!
It looks like support for test results analyzer was added to the pipeline plugin from the jira issue below. I'm having trouble figuring out how to acutally implement the plugin using a pipeline script.
https://issues.jenkins-ci.org/browse/JENKINS-30522
Regardless of the jira issue, how can I run the test results analyzer through my pipeline?
When you add test reports to your pipeline script this works automatically. The "Test Results Analyzer" button shows up right away for jobs that have tests, including those that use the pipeline plugin.
For example when using the standard "junit" report plugin like this, it should work out of the box:
stage('Unit tests') {
steps {
sh 'unit-tests.sh'
}
post {
always {
junit 'path/to/report.xml'
}
}
}