i am using Hudson as CI server for Delphi 2010 projects. The XMLTestRunner.pas unit writes DUnit test result to a xml file that is used by the hudson xUnit plugin to report test results. The xUnit plugin shows failures but no messages:
Stacktrace
MESSAGE:
+++++++++++++++++++
STACK TRACE:
In the hudson project configuration at "Publish testing tools result report" i choose "NUnit-Version N/A (default)" as tesing tool, because there is no dunit option in the list and the xml files looks similar to nunit format:
<?xml version="1.0" encoding="ISO-8859-1" standalone="yes" ?>
<test-results total="123" notrun="0" date="20.12.2010" time="09:19:24">
<test-suite name="HelloWorldTestSuite" total="46" notrun="0">
<results>
<test-case name="TestCase.HelloWorldTest" execute="True" success="False" time="0,003" result="Failure">
<failure name="ETestFailure" location=""/>
<message>expected: < hello world! > but was: < hallo welt ></message>
</test-case>
...
In the hudson configuration there is also an "Custom Tool" option, where i have to specify a "Custom stylesheet", but i don't know how to write such a stylesheet (is there any documentation?).
On my C++ projets with boost test, the failures are reported nicely with all messages.
I had to change the XMLListener to get it work with hudson because the XML structure of a failure had to look like this:
<testcase name="GetFreeDirNameTest" classname="Test.exe.MyFiles.TTests" result="failed" time="0.000">
<failure message="Expected:
"-1"
But was:
"0"" type="failed">GetFreeDirNameTest: Expected:
"-1"
But was:
"0"
</failure>
</testcase>
What about using XMLTestRunner for NUnit compatible output from DUnit?
Related
I have python unit test report generated by junit in xml fomrat from the jenkins.
<?xml version="1.0" encoding="utf-8"?>
<testsuites>
<testsuite name="pytest" errors="0" failures="0" skipped="1" tests="3" time="0.023" timestamp="2021-08-29T21:42:01.852698" hostname="infinity">
<testcase classname="tests.SimpleTest" name="test_fail" time="0.000" />
<testcase classname="tests.SimpleTest" name="test_pass" time="0.000" />
<testcase classname="tests.SimpleTest" name="test_skipped" time="0.000">
<skipped type="pytest.skip" message="demonstrating skipping">
/var/lib/jenkins/workspace/GithubExample/tests.py:10: demonstrating skipping
</skipped>
</testcase>
</testsuite>
</testsuites>
Now I want to generate an html report which then I can use Publish HTML report plugin to publish to be accessible from confluence.
Now at my workplace I do not have junit macro and maybe not have other html generation plugin what I have is xsltproc ,
I tried to generated htmlreport using xsltproc but it throws an error
xsltproc test-result.xml stylesheet.xslt > index.html
warning: failed to load external entity "test-result.xml"
cannot parse test-result.xml
You can use junit2html. It is easy to use and provides a few options to print tests summary to terminal output or create a summary in the html too.
Even though it is a python module, you can execute it from shell.
Example: junit2html report.xml report.html
I have a test suite, where some of the XML files are generated after running unit tests (junit). And there are some other protractor automation tests which uses jasmine junit xml reporter (https://github.com/larrymyers/jasmine-reporters) to create XML file. I am generating both sets of XML files in same directory in my jenkins job, I have used junit plugin (Publish JUnit test result report) to publish report. So, now, after the job completes, 9 xmls are generated from unit tests and 2 XMLs are generated by jasmine-reporter, so as stated below from my jenkins job console output, it is able to recognize that there are 11 xml files from the pattern I have specified in job config.
22:49:52 [xUnit] [INFO] - [JUnit] - 11 test report file(s) were found with the
pattern '**/tests/target/failsafe-reports/junitreports/*.xml' relative to
'/home/jenkins/workspace/projectName' for the testing framework
'JUnit'.
But, problem is, when i click on "Test Result" link from jenkins job UI, it only shows all tests related to xml generated by unit test, and it doesn't show any result related to XML which was generated from jasmine report. Need help to know why jenkins is not showing any test result related to jasmine XML.
Below is a sample XML generated by unit test (which jenkins is showing in ui, when test result link is clicked).
<!--
Generated by org.testng.reporters.JUnitReportReporter
-->
<testsuite hostname="someHostName"
name="packageName.Class1" tests="22" failures="0" timestamp="16
Apr 2018 05:47:59 GMT" time="69.663" errors="0">
<testcase name="test1" time="2.488"
classname="packageName.Class1"/>
<testcase name="test2" time="5.808"
classname="packageName.Class1"/>
</testsuite>
Below is a sample XML generated by jasmine junit xml reporter (result of this is not shown by jenkins in it's UI, when clicking test result link)
<testsuites>
<testsuite name="chromeTestItemDetailsPage" timestamp="2018-04-16T05:48:43"
hostname="localhost" time="29.357" errors="0" tests="22" skipped="0"
disabled="0" failures="0">
<testcase classname="chromeTestItemDetailsPage" name="ItemTitleDisplayed"
time="0.895"/>
<testcase classname="chromeTestItemDetailsPage" name="ItemPriceDisplayed"
time="0.966"/>
</testsuite>
</testsuites>
I think you are encountering a Jenkins issue here.
Jenkins uses the attributes classname as well as testname of a <testcase> within a JUnit XML report to classify test results on the UI. Furthermore Jenkins expects the classname to be composed of a package name followed by a single dot and then a class name:
<testcase classname="packagename.classname" name="testname">
If no dot is present, Jenkins assumes the package "root". If multiple dots are present, only the last dot is recognized as separator. See this question for a better explanation.
If you look at your XML reports above you can notice, that the XML of the unit tests specify a package name in the classname attribute, while the XML of the jasmine reporter does not. I think the missing package name is the reason, why the test results are not showing up. Apparently Jenkins uses the package "root" only, if there are no packages present at all. If there is at least one test case with a package name, all other test cases without a package are "lost".
There are two possible solutions:
Report a bug for Jenkins since above behavior is unexpected
Modify the jasmine reporter to construct a classnameof the pattern package.class. There is a pull request at https://github.com/larrymyers/jasmine-reporters/pull/186/files that would allow modifications of the classname by a user function.
I have some xUnit tests running as part of a build and I need to post the results back to TFS 2010 so they show up in the build summary/log like normal tests would.
I execute the tests with the -nunit argument so that the output is an NUnit xml and not a xUnit one. I'm trying to use NUnit4TeamBuild to publish the results xml back to TFS and it looks like it's working but there's no test results anywhere at the end.
This is the detailed log for that step:
Publish xUnit Results
NUnitTfs.exe -n results.xml -t Project -b "Build_20171221.8" -v 2010
Loading NUnit Output from results.xml...
Transforming NUnit output to .trx file...
Adding build information to .trx file...
Publishing .trx file...
Checking test run does not exist...
Reading users identity...
Creating test run...
Creating test results...
Updating test run with details...
Uploading test run details...
Uploading assemblies...
Updating test results with details...
Uploading test result details...
The build finishes with 0 errors and 0 warnings but also says "No Test Results" in the build summary and there are no test results anywhere.
I have no idea why the tests aren't publishing.
Here's an example of the XML that it's trying to publish (note that I've changed the file paths and shortened the stack trace):
<?xml version="1.0" encoding="utf-8"?>
<test-results name="Test results" errors="0" inconclusive="0" ignored="0" invalid="0" not-run="0" date="2017-12-21" time="08:39:57" total="1" failures="1" skipped="0">
<environment os-version="unknown" platform="unknown" cwd="unknown" machine-name="unknown" user="unknown" user-domain="unknown" nunit-version="xUnit.net 2.3.1.3858" clr-version="64-bit .NET 4.0.30319.34209 [collection-per-class, non-parallel]" />
<culture-info current-culture="unknown" current-uiculture="unknown" />
<test-suite type="Assemblies" name="C:\Tests.DLL" executed="True" success="False" result="Failure" time="57.41">
<results>
<test-suite type="Assembly" executed="True" name="C:\Tests.DLL" result="Failure" success="False" time="57.410">
<results>
<test-suite type="TestCollection" executed="True" name="Tests" result="Failure" success="False" time="48.992">
<results>
<test-case name="Google_Search_Test" executed="True" result="Failure" success="False" time="48.9917024">
<failure>
<message>System.Exception : Test Failed. Exception has been thrown by the target of an invocation.</message>
<stack-trace>at Tests.Fixtures.Fixture.FormatException(Exception e) in... etc. </stack-trace>
</failure>
</test-case>
</results>
</test-suite>
</results>
</test-suite>
</results>
</test-suite>
</test-results>
TFS 2010 uses a proprietary results format with a “.trx” extension to publish results so we need to transpose the NUnit results before sending back through web services.
You can use NUnitTFS for xUnit, just try the workflow template NUnitAndPublishTemplate.xaml which provided by Ian Battersby.
Refer to Ian Battersby's blog for details : Running and publishing NUnit test results into TFS 2010
I have built a simple UITesting framework for one of my apps built in Xcode. 2 of the tests pass and one purposefully fails:
XCTAssertTrue(false)
I am using the Jenkins Xcode plugin and am using the post-built action: Publish JUnit test result report.
Jenkins successfully launches the simulator and runs all 3 tests. It also successfully picks up the failure in the logs:
Failing tests:
-[LightAlarmUITests testFailingTest()]
** TEST FAILED **
However, the Test Results Analyzer (plugin installed) shows all 3 tests are passing. When I inspect the test-results/*.xml file I see the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<testsuite failures="1" errors="0" hostname="Charlies-MacBook-Pro.local" name="LightAlarmUITests" tests="3" time="21.0" timestamp="2016-12-20T16:24:33.125Z">
<testcase classname="LightAlarmUITests" name="testFailingTest" time="6.372"/>
<testcase classname="LightAlarmUITests" name="testShowSettingsPage" time="7.167"/>
<testcase classname="LightAlarmUITests" name="testShowSoundAlarmsPage" time="7.594"/>
</testsuite>
As you can see in the xml, it is not marking individual test cases as pass or fail, but instead marking a failure against the entire test suite.
Does anyone know how to mark pass/fails against individual test cases?
There is a pull request open to fix this issue.
https://github.com/jenkinsci/xcode-plugin/pull/75
The reason behind this is Xcode 8 changed the way of XML of UI test report. JUnit plugin is not able to parse this XML to generate a test report. I have used XCPretty mediator that converts Xcode 8 XML report into JUnit supported XML. You can find more details at https://blog.talentica.com/2017/04/04/use-xcode-8-with-jenkins/
I have added a post-build action called Publish xUnit test result report, I have then added a JUnit sub-section and specified the paths to my xml files. I know that xUnit is finding all of the specified files ([xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern) however xUnit proceeds to base the success of the build on only one of the files and Latest Build Results also shows the breakdown of the tests from the same single file.
xUnit at no point logs an error or exception or indicates that it has had any sort of issue with any of the files when specified individually or all together.
If I set up a separate build for each file, xUnit correctly calculates the build success and reports the results for all of the files.
When I specify multiple files only the first file is used.
How do I get xUnit to report the results for multiple files in a single build?
I have tried:
Appending all test results into a single file.
This leads to the XML being badly formed and unparseable.
Adding multiple JUnit sub-sections (one for each file).
This is no different to normal, only the first specified file is used to report success and display results.
Upgrading xUnit to the latest version, from 1.96 to 1.101.
No change.
Enabling the xUnit logger for more information.
The logger contains no more useful information, only [xUnit] - [JUnit] - 3 test report file(s) were found with the pattern '*_test_output.xml' relative to...
Example output of one of the test results files:
<testsuites>
<testsuite name="all tests" errors="0" failures="2" tests="176468" hostname="tbd" time="1.79972" timestamp="tbd">
<testcase classname="Test Section" name="Test Name" time="0.000091"/>
<testcase classname="Test Section" name="Test Name" time="0.000221">
<failure message="2 <= 1" type="REQUIRE">
at path/path/path path/path/path/file.cpp:106
</failure>
</testcase>
<system-out/>
<system-err/>
</testsuite>
</testsuites>