I'm using the Jenkins JUnit Plugin to collect my tests result and according to the documentation:
Allow empty results: If checked, the default behavior of failing a
build on missing test result files or empty test results is changed to
not affect the status of the build. Please note that this setting make
it harder to spot misconfigured jobs or build failures where the test
tool does not exit with an error code when not producing test report
files.
This is how I use it:
junit testResults: '**/reports/junit*.xml', testDataPublishers: [[$class: 'StabilityTestDataPublisher']]
So the behavior should fail if the test file is empty but I get "UNSTABLE" result with the message
Test report file /opt/workspace/integrations_develop/reports/junit_integrations.xml was length 0
How can I make it fail if the test file is empty?
Related
I have test cases as executables ( windows exe ) and I am running all the test cases (exe) in Jenkins. Each test case produce output text file and at the end of the testcase, there is a formatted string like "TestCaseName PASSED/FAILED".
I can see from the build status if all the test cases were successful or not but if any of them fails, the build also says it fails. I want to use Jenkins Test Results Analyser or any other plugin to show as a table to see in which build which test case failed. The Jenkins Test Results Analyser works fine with jUnit or TestNg test reports. I was to thinking to generate manually such reports as post build step or if there is any other tool available?
From time to time our Jenkins pipeline is marked as unstable, after researching it I found that it originates from the Junit plugin, which is publishing test results.
The weird thing is that all the tests are successfully passing (logs and also the whole pipeline proceeding), however the exported test results show that there are some errors.
Can anyone explain this to me?
All tests are passing (logs):
The test results exported by junit are showing some failures:
The whole build is marked as unstable (yellow):
I am using the Junit plugin for the tests report. My problem is most of the builds will mark as unstable which should be failed. Is there any way to mark the build as failed when having more than one test failed?
Example1:
Example2:
Example3:
As of now, there is no directly configurable threshold option for the JUnit plugin in Jenkins (Here is the open feature request). However, you can add an if condition after reporting the test results and set the build status.
def junitTestSummary = junit testResults: "**/*.xml"
// junitTestSummary.[failCount, totalCount, skipCount, passCount]
if(junitTestSummary.failCount>0){
error("Failing the pipeline. Because, there are ${junitTestSummary.failCount} tests failed")
}
Within my freestyle Jenkins-job I´m executing unit-tests via the "execute Windows batch-command"-step:
call "C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" MyAssembly.dll
call SomeOtherProcess
As there are tests that fail, I´d expected the build to fail as well. However the test-publishing-step for NUnit markes the build as unstable:
Build step 'Publish NUnit test result report' changed build result to UNSTABLE
If I´d remove the SomeOtherProcess-line from my batch-script, everything works fine and the errors produced by nunit are reported as error in the build-process.
I read a similar issue for the JUnit-test-reporter (Jenkins JUnit Plugin reports a build as unstable even if test fails). Obviously that reporter does not even support failing the build. I´m not sure if the same applies to the NUnit-reporter as well.
The plugin set the result to UNSTABLE because the option, by default, failedTestsFailBuild is set to false.
You can control the behavior applies of NUnit, setting failedTestsFailBuild to true. When you call from a scripted or declarative pipeline.
The issue is the GUI doesn't reflect all the options available for this plugin. There is a PR opened to include this option inside the freestyle pipeline, you can vote up or ask the status of this PR.
To change to an error you need to catch the unstable result and set it to failure using a plugin or a scripted or declarative pipeline.
I have multiple xml result files collected from different builds and I have to display a consolidated report on jenkins.
Is there any plugin available to perform the above action?
We also tried JUnit plugin, we got the following error:
ERROR: Step 'Publish JUnit test result report' failed: Test reports were found but none of them are new. Did leafNodes run? For example, D:\jenkins\workspace\test-1\testng-results.xml is 14 min old
try to check the timestamp when the junit report was generated and last execution time of jenkins job. Both timestamp should match.
If you see a difference, delete the report and re-generate the report.