Trying to use the nifty Jenkins Pipeline, I had problems finding out how to publish NUnit test results.
I am able to run the tests by specifying the following command in the pipeline script:
stage 'Test'
bat '"C:\\Program Files (x86)\\NUnit 2.6.4\\bin\\nunit-console-x86.exe" "ProjectName\\bin\\Release\\UnitTests.net.dll"'
But how to make Jenkins "publish" the test results is not obvious. The Snippet Generator only suggests junit, and that does not seem to work.
I used nunit plugin version 0.21 and was able to publish results using
stage("PublishTestReport"){
nunit testResultsPattern: 'TestResult.xml'
}
(TestResult.xml is at the root of jenkins workspace in this above example)
Investigating the NUnit plugin for Jenkins led me on to this issue, where I found the solution:
step([$class: 'NUnitPublisher', testResultsPattern: 'TestResult.xml', debug: false,
keepJUnitReports: true, skipJUnitArchiver:false, failIfNoResults: true])
Adding this to the pipeline script worked for me!
However, it seemed the following should also work (but at the present, apparently, it does not): Using the Snippet Generator, select this:
step: General Build Step
Publish NUnit test result report
This generates the following in the Pipeline script:
step <object of type hudson.plugins.nunit.NUnitPublisher>
This fails!
Related
Within my freestyle Jenkins-job I´m executing unit-tests via the "execute Windows batch-command"-step:
call "C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" MyAssembly.dll
call SomeOtherProcess
As there are tests that fail, I´d expected the build to fail as well. However the test-publishing-step for NUnit markes the build as unstable:
Build step 'Publish NUnit test result report' changed build result to UNSTABLE
If I´d remove the SomeOtherProcess-line from my batch-script, everything works fine and the errors produced by nunit are reported as error in the build-process.
I read a similar issue for the JUnit-test-reporter (Jenkins JUnit Plugin reports a build as unstable even if test fails). Obviously that reporter does not even support failing the build. I´m not sure if the same applies to the NUnit-reporter as well.
The plugin set the result to UNSTABLE because the option, by default, failedTestsFailBuild is set to false.
You can control the behavior applies of NUnit, setting failedTestsFailBuild to true. When you call from a scripted or declarative pipeline.
The issue is the GUI doesn't reflect all the options available for this plugin. There is a PR opened to include this option inside the freestyle pipeline, you can vote up or ask the status of this PR.
To change to an error you need to catch the unstable result and set it to failure using a plugin or a scripted or declarative pipeline.
I have Jenkins building my C# .NET Core api project. I added some xUnit tests and included a powershell script inside of my Jenkins build with the "dotnet test" command to execute the tests.
That all works well and the tests are run and i can see the output in the Jenkins console.
The problem is that if i have failing tests nothing happens - jenkins goes merrily along and finished up the build process and reports it as a success.
How can i get it to fail the build?
Is there a response from the 'dotnet test' command?
I know there are xUnit Jenkins plugins but they all seem to revolve around "display the results of xUnit tests". Which is not really what i am after. I want to ACT on the results of the tests, not just see them in fancy html.
You should check for the return code from dotnet test command. It returns 0 if all tests were successful and 1 if any of the tests failed. Unfortunately it's not documented but was confirmed in this issue
I am using Jenkins MSBUILD plugin in a Freestyle project which is just fine. I have to specify the location of .TRX file and the test report gets generated.
Now I want to use groovy command so I can take advantage of Jenkins pipeline. can someone help me with batch/groovy command to use .TRX as a input and generate test report in Jenkins UI? I tried below but no success.
mstest.exe /publishresultsfile:"C:\Path_to_file\TestResult.TRX"
Below error occurs
Invalid switch "/publishresultsfile".
On my Jenkins pipleline I run unit tests on both: debug and release configurations. Each test configuration generates separate JUnit XML results file. Test names on both: debug and release configuration are same. Currently I use the following junit command in order to show test results:
junit allowEmptyResults: true, healthScaleFactor: 0.0, keepLongStdio: true, testResults: 'Test-Dir/Artifacts/test_xml_reports_*/*.xml'
The problem is that on Jenkins UI both: debug and release tests results are shown together and it is not possible to know which test (from debug or release configuration) is failed.
Is it possible to show debug and release tests results separately? If yes, how can I do that?
We run the same integration tests against two different configurations with different DB types. We use maven and the failsafe plugin, so I take advantage of the -Dsurefire.reportNameSuffix so I can see the difference between the two runs.
The following is an example block of our Jenkinsfile:
stage('Integration test MySql') {
steps {
timeout(75) {
sh("mvn -e verify -DskipUnitTests=true -DtestConfigResource=conf/mysql-local.yaml " +
"-DintegrationForkCount=1 -DdbInitMode=migrations -Dmaven.test.failure.ignore=false " +
"-Dsurefire.reportNameSuffix=MYSQL")
}
}
post {
always {
junit '**/failsafe-reports/*MYSQL.xml'
}
}
}
In the report, the integration tests run against mysql then show up with MYSQL appended to their name.
It looks no solution for my question.
As a workaround I changed JUnit XML report format and included build variant name (debug/release) as a package name.
I have a jenkins standalone job, that uses the MSTest plugin, it publishes the test result (.trx) on the jenkins UI. I want to use this feature of the plugin via the workflow script. How can i achieve this?
At the moment, i am using this batch file, but it need the extra utilities like the "msxls.exe" which doesn't comes with cloudbees jenkins out of the box.
stage name: 'Publish test result', concurrency: 1
bat 'C:\\bin\\msxsl.exe TestResult.trx "C:\\Jenkins\\plugins\\mstest\\WEB-INF\\mstest-to-junit_withOutput.xsl" -o JUnitLikeResultsOutputFile1.xml'
step([$class: 'JUnitResultArchiver', allowEmptyResults: true, testResults: 'JUnitLikeResultsOutputFile1.xml'])
If a plugin is compatible with the Pipeline plugin, then you can find out the appropriate Groovy DSL for it by enabling the "Snippet generator", choosing "step" and finding the desired build step in the "Build step" list.
It would look somewhat similar to the JUnitResultArchiver step you're already using.
However, unfortunately, the MSTest Plugin is not currently compatible with the Pipeline plugin, so it's not available in this list.
You would have to file a feature request to get this implemented.
The Pipeline plugin documentation also has some documentation for developers on how to make plugins compatible.