From time to time our Jenkins pipeline is marked as unstable, after researching it I found that it originates from the Junit plugin, which is publishing test results.
The weird thing is that all the tests are successfully passing (logs and also the whole pipeline proceeding), however the exported test results show that there are some errors.
Can anyone explain this to me?
All tests are passing (logs):
The test results exported by junit are showing some failures:
The whole build is marked as unstable (yellow):
Related
I have test cases as executables ( windows exe ) and I am running all the test cases (exe) in Jenkins. Each test case produce output text file and at the end of the testcase, there is a formatted string like "TestCaseName PASSED/FAILED".
I can see from the build status if all the test cases were successful or not but if any of them fails, the build also says it fails. I want to use Jenkins Test Results Analyser or any other plugin to show as a table to see in which build which test case failed. The Jenkins Test Results Analyser works fine with jUnit or TestNg test reports. I was to thinking to generate manually such reports as post build step or if there is any other tool available?
Within my freestyle Jenkins-job I´m executing unit-tests via the "execute Windows batch-command"-step:
call "C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" MyAssembly.dll
call SomeOtherProcess
As there are tests that fail, I´d expected the build to fail as well. However the test-publishing-step for NUnit markes the build as unstable:
Build step 'Publish NUnit test result report' changed build result to UNSTABLE
If I´d remove the SomeOtherProcess-line from my batch-script, everything works fine and the errors produced by nunit are reported as error in the build-process.
I read a similar issue for the JUnit-test-reporter (Jenkins JUnit Plugin reports a build as unstable even if test fails). Obviously that reporter does not even support failing the build. I´m not sure if the same applies to the NUnit-reporter as well.
The plugin set the result to UNSTABLE because the option, by default, failedTestsFailBuild is set to false.
You can control the behavior applies of NUnit, setting failedTestsFailBuild to true. When you call from a scripted or declarative pipeline.
The issue is the GUI doesn't reflect all the options available for this plugin. There is a PR opened to include this option inside the freestyle pipeline, you can vote up or ask the status of this PR.
To change to an error you need to catch the unstable result and set it to failure using a plugin or a scripted or declarative pipeline.
I'm using the Jenkins JUnit Plugin to collect my tests result and according to the documentation:
Allow empty results: If checked, the default behavior of failing a
build on missing test result files or empty test results is changed to
not affect the status of the build. Please note that this setting make
it harder to spot misconfigured jobs or build failures where the test
tool does not exit with an error code when not producing test report
files.
This is how I use it:
junit testResults: '**/reports/junit*.xml', testDataPublishers: [[$class: 'StabilityTestDataPublisher']]
So the behavior should fail if the test file is empty but I get "UNSTABLE" result with the message
Test report file /opt/workspace/integrations_develop/reports/junit_integrations.xml was length 0
How can I make it fail if the test file is empty?
I have Jenkins building my C# .NET Core api project. I added some xUnit tests and included a powershell script inside of my Jenkins build with the "dotnet test" command to execute the tests.
That all works well and the tests are run and i can see the output in the Jenkins console.
The problem is that if i have failing tests nothing happens - jenkins goes merrily along and finished up the build process and reports it as a success.
How can i get it to fail the build?
Is there a response from the 'dotnet test' command?
I know there are xUnit Jenkins plugins but they all seem to revolve around "display the results of xUnit tests". Which is not really what i am after. I want to ACT on the results of the tests, not just see them in fancy html.
You should check for the return code from dotnet test command. It returns 0 if all tests were successful and 1 if any of the tests failed. Unfortunately it's not documented but was confirmed in this issue
I don't even know where to begin. JUnit plugin is installed and is running in some scripts (I couldn't find those, it's a large project, but the statistic page indicates 28 usages). But, the pipeline code junit "foo.xml" fails with
java.lang.NoSuchMethodError: No such DSL method 'junit' found among steps
Followed by 100,500 lines of usual nonsense.
Is there anything special I need to do to "enable" this plugin? Jenkins wiki lists it as "required" (whatever that means). The example Jenkinfile that illustrates the usage of this plugin never imports anything. Nonexistent debugging and ninja-style documentation don't really help getting to the culprit of this problem.
When I tried to replace junit 'foo.xml' with step([$class: 'JUnitResultArchiver', testResults: './foo.xml']). This "step" produced the following output:
Recording test results
And failed the build without any further messages. Neither in console nor in the logs collected by the pipeline.