I have test cases as executables ( windows exe ) and I am running all the test cases (exe) in Jenkins. Each test case produce output text file and at the end of the testcase, there is a formatted string like "TestCaseName PASSED/FAILED".
I can see from the build status if all the test cases were successful or not but if any of them fails, the build also says it fails. I want to use Jenkins Test Results Analyser or any other plugin to show as a table to see in which build which test case failed. The Jenkins Test Results Analyser works fine with jUnit or TestNg test reports. I was to thinking to generate manually such reports as post build step or if there is any other tool available?
Related
Within my freestyle Jenkins-job I´m executing unit-tests via the "execute Windows batch-command"-step:
call "C:\Program Files (x86)\NUnit.org\nunit-console\nunit3-console.exe" MyAssembly.dll
call SomeOtherProcess
As there are tests that fail, I´d expected the build to fail as well. However the test-publishing-step for NUnit markes the build as unstable:
Build step 'Publish NUnit test result report' changed build result to UNSTABLE
If I´d remove the SomeOtherProcess-line from my batch-script, everything works fine and the errors produced by nunit are reported as error in the build-process.
I read a similar issue for the JUnit-test-reporter (Jenkins JUnit Plugin reports a build as unstable even if test fails). Obviously that reporter does not even support failing the build. I´m not sure if the same applies to the NUnit-reporter as well.
The plugin set the result to UNSTABLE because the option, by default, failedTestsFailBuild is set to false.
You can control the behavior applies of NUnit, setting failedTestsFailBuild to true. When you call from a scripted or declarative pipeline.
The issue is the GUI doesn't reflect all the options available for this plugin. There is a PR opened to include this option inside the freestyle pipeline, you can vote up or ask the status of this PR.
To change to an error you need to catch the unstable result and set it to failure using a plugin or a scripted or declarative pipeline.
I'm using the Jenkins JUnit Plugin to collect my tests result and according to the documentation:
Allow empty results: If checked, the default behavior of failing a
build on missing test result files or empty test results is changed to
not affect the status of the build. Please note that this setting make
it harder to spot misconfigured jobs or build failures where the test
tool does not exit with an error code when not producing test report
files.
This is how I use it:
junit testResults: '**/reports/junit*.xml', testDataPublishers: [[$class: 'StabilityTestDataPublisher']]
So the behavior should fail if the test file is empty but I get "UNSTABLE" result with the message
Test report file /opt/workspace/integrations_develop/reports/junit_integrations.xml was length 0
How can I make it fail if the test file is empty?
I have Jenkins building my C# .NET Core api project. I added some xUnit tests and included a powershell script inside of my Jenkins build with the "dotnet test" command to execute the tests.
That all works well and the tests are run and i can see the output in the Jenkins console.
The problem is that if i have failing tests nothing happens - jenkins goes merrily along and finished up the build process and reports it as a success.
How can i get it to fail the build?
Is there a response from the 'dotnet test' command?
I know there are xUnit Jenkins plugins but they all seem to revolve around "display the results of xUnit tests". Which is not really what i am after. I want to ACT on the results of the tests, not just see them in fancy html.
You should check for the return code from dotnet test command. It returns 0 if all tests were successful and 1 if any of the tests failed. Unfortunately it's not documented but was confirmed in this issue
I have multiple xml result files collected from different builds and I have to display a consolidated report on jenkins.
Is there any plugin available to perform the above action?
We also tried JUnit plugin, we got the following error:
ERROR: Step 'Publish JUnit test result report' failed: Test reports were found but none of them are new. Did leafNodes run? For example, D:\jenkins\workspace\test-1\testng-results.xml is 14 min old
try to check the timestamp when the junit report was generated and last execution time of jenkins job. Both timestamp should match.
If you see a difference, delete the report and re-generate the report.
I am using Jenkins MSBUILD plugin in a Freestyle project which is just fine. I have to specify the location of .TRX file and the test report gets generated.
Now I want to use groovy command so I can take advantage of Jenkins pipeline. can someone help me with batch/groovy command to use .TRX as a input and generate test report in Jenkins UI? I tried below but no success.
mstest.exe /publishresultsfile:"C:\Path_to_file\TestResult.TRX"
Below error occurs
Invalid switch "/publishresultsfile".