I have added a post-build action called Publish xUnit test result report, I have then added a JUnit sub-section and specified the paths to my xml files. I know that xUnit is finding all of the specified files ([xUnit] [INFO] - [JUnit] - 3 test report file(s) were found with the pattern) however xUnit proceeds to base the success of the build on only one of the files and Latest Build Results also shows the breakdown of the tests from the same single file.
xUnit at no point logs an error or exception or indicates that it has had any sort of issue with any of the files when specified individually or all together.
If I set up a separate build for each file, xUnit correctly calculates the build success and reports the results for all of the files.
When I specify multiple files only the first file is used.
How do I get xUnit to report the results for multiple files in a single build?
I have tried:
Appending all test results into a single file.
This leads to the XML being badly formed and unparseable.
Adding multiple JUnit sub-sections (one for each file).
This is no different to normal, only the first specified file is used to report success and display results.
Upgrading xUnit to the latest version, from 1.96 to 1.101.
No change.
Enabling the xUnit logger for more information.
The logger contains no more useful information, only [xUnit] - [JUnit] - 3 test report file(s) were found with the pattern '*_test_output.xml' relative to...
Example output of one of the test results files:
<testsuites>
<testsuite name="all tests" errors="0" failures="2" tests="176468" hostname="tbd" time="1.79972" timestamp="tbd">
<testcase classname="Test Section" name="Test Name" time="0.000091"/>
<testcase classname="Test Section" name="Test Name" time="0.000221">
<failure message="2 <= 1" type="REQUIRE">
at path/path/path path/path/path/file.cpp:106
</failure>
</testcase>
<system-out/>
<system-err/>
</testsuite>
</testsuites>
Related
I have a test suite, where some of the XML files are generated after running unit tests (junit). And there are some other protractor automation tests which uses jasmine junit xml reporter (https://github.com/larrymyers/jasmine-reporters) to create XML file. I am generating both sets of XML files in same directory in my jenkins job, I have used junit plugin (Publish JUnit test result report) to publish report. So, now, after the job completes, 9 xmls are generated from unit tests and 2 XMLs are generated by jasmine-reporter, so as stated below from my jenkins job console output, it is able to recognize that there are 11 xml files from the pattern I have specified in job config.
22:49:52 [xUnit] [INFO] - [JUnit] - 11 test report file(s) were found with the
pattern '**/tests/target/failsafe-reports/junitreports/*.xml' relative to
'/home/jenkins/workspace/projectName' for the testing framework
'JUnit'.
But, problem is, when i click on "Test Result" link from jenkins job UI, it only shows all tests related to xml generated by unit test, and it doesn't show any result related to XML which was generated from jasmine report. Need help to know why jenkins is not showing any test result related to jasmine XML.
Below is a sample XML generated by unit test (which jenkins is showing in ui, when test result link is clicked).
<!--
Generated by org.testng.reporters.JUnitReportReporter
-->
<testsuite hostname="someHostName"
name="packageName.Class1" tests="22" failures="0" timestamp="16
Apr 2018 05:47:59 GMT" time="69.663" errors="0">
<testcase name="test1" time="2.488"
classname="packageName.Class1"/>
<testcase name="test2" time="5.808"
classname="packageName.Class1"/>
</testsuite>
Below is a sample XML generated by jasmine junit xml reporter (result of this is not shown by jenkins in it's UI, when clicking test result link)
<testsuites>
<testsuite name="chromeTestItemDetailsPage" timestamp="2018-04-16T05:48:43"
hostname="localhost" time="29.357" errors="0" tests="22" skipped="0"
disabled="0" failures="0">
<testcase classname="chromeTestItemDetailsPage" name="ItemTitleDisplayed"
time="0.895"/>
<testcase classname="chromeTestItemDetailsPage" name="ItemPriceDisplayed"
time="0.966"/>
</testsuite>
</testsuites>
I think you are encountering a Jenkins issue here.
Jenkins uses the attributes classname as well as testname of a <testcase> within a JUnit XML report to classify test results on the UI. Furthermore Jenkins expects the classname to be composed of a package name followed by a single dot and then a class name:
<testcase classname="packagename.classname" name="testname">
If no dot is present, Jenkins assumes the package "root". If multiple dots are present, only the last dot is recognized as separator. See this question for a better explanation.
If you look at your XML reports above you can notice, that the XML of the unit tests specify a package name in the classname attribute, while the XML of the jasmine reporter does not. I think the missing package name is the reason, why the test results are not showing up. Apparently Jenkins uses the package "root" only, if there are no packages present at all. If there is at least one test case with a package name, all other test cases without a package are "lost".
There are two possible solutions:
Report a bug for Jenkins since above behavior is unexpected
Modify the jasmine reporter to construct a classnameof the pattern package.class. There is a pull request at https://github.com/larrymyers/jasmine-reporters/pull/186/files that would allow modifications of the classname by a user function.
I have some xUnit tests running as part of a build and I need to post the results back to TFS 2010 so they show up in the build summary/log like normal tests would.
I execute the tests with the -nunit argument so that the output is an NUnit xml and not a xUnit one. I'm trying to use NUnit4TeamBuild to publish the results xml back to TFS and it looks like it's working but there's no test results anywhere at the end.
This is the detailed log for that step:
Publish xUnit Results
NUnitTfs.exe -n results.xml -t Project -b "Build_20171221.8" -v 2010
Loading NUnit Output from results.xml...
Transforming NUnit output to .trx file...
Adding build information to .trx file...
Publishing .trx file...
Checking test run does not exist...
Reading users identity...
Creating test run...
Creating test results...
Updating test run with details...
Uploading test run details...
Uploading assemblies...
Updating test results with details...
Uploading test result details...
The build finishes with 0 errors and 0 warnings but also says "No Test Results" in the build summary and there are no test results anywhere.
I have no idea why the tests aren't publishing.
Here's an example of the XML that it's trying to publish (note that I've changed the file paths and shortened the stack trace):
<?xml version="1.0" encoding="utf-8"?>
<test-results name="Test results" errors="0" inconclusive="0" ignored="0" invalid="0" not-run="0" date="2017-12-21" time="08:39:57" total="1" failures="1" skipped="0">
<environment os-version="unknown" platform="unknown" cwd="unknown" machine-name="unknown" user="unknown" user-domain="unknown" nunit-version="xUnit.net 2.3.1.3858" clr-version="64-bit .NET 4.0.30319.34209 [collection-per-class, non-parallel]" />
<culture-info current-culture="unknown" current-uiculture="unknown" />
<test-suite type="Assemblies" name="C:\Tests.DLL" executed="True" success="False" result="Failure" time="57.41">
<results>
<test-suite type="Assembly" executed="True" name="C:\Tests.DLL" result="Failure" success="False" time="57.410">
<results>
<test-suite type="TestCollection" executed="True" name="Tests" result="Failure" success="False" time="48.992">
<results>
<test-case name="Google_Search_Test" executed="True" result="Failure" success="False" time="48.9917024">
<failure>
<message>System.Exception : Test Failed. Exception has been thrown by the target of an invocation.</message>
<stack-trace>at Tests.Fixtures.Fixture.FormatException(Exception e) in... etc. </stack-trace>
</failure>
</test-case>
</results>
</test-suite>
</results>
</test-suite>
</results>
</test-suite>
</test-results>
TFS 2010 uses a proprietary results format with a “.trx” extension to publish results so we need to transpose the NUnit results before sending back through web services.
You can use NUnitTFS for xUnit, just try the workflow template NUnitAndPublishTemplate.xaml which provided by Ian Battersby.
Refer to Ian Battersby's blog for details : Running and publishing NUnit test results into TFS 2010
I have built a simple UITesting framework for one of my apps built in Xcode. 2 of the tests pass and one purposefully fails:
XCTAssertTrue(false)
I am using the Jenkins Xcode plugin and am using the post-built action: Publish JUnit test result report.
Jenkins successfully launches the simulator and runs all 3 tests. It also successfully picks up the failure in the logs:
Failing tests:
-[LightAlarmUITests testFailingTest()]
** TEST FAILED **
However, the Test Results Analyzer (plugin installed) shows all 3 tests are passing. When I inspect the test-results/*.xml file I see the following:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<testsuite failures="1" errors="0" hostname="Charlies-MacBook-Pro.local" name="LightAlarmUITests" tests="3" time="21.0" timestamp="2016-12-20T16:24:33.125Z">
<testcase classname="LightAlarmUITests" name="testFailingTest" time="6.372"/>
<testcase classname="LightAlarmUITests" name="testShowSettingsPage" time="7.167"/>
<testcase classname="LightAlarmUITests" name="testShowSoundAlarmsPage" time="7.594"/>
</testsuite>
As you can see in the xml, it is not marking individual test cases as pass or fail, but instead marking a failure against the entire test suite.
Does anyone know how to mark pass/fails against individual test cases?
There is a pull request open to fix this issue.
https://github.com/jenkinsci/xcode-plugin/pull/75
The reason behind this is Xcode 8 changed the way of XML of UI test report. JUnit plugin is not able to parse this XML to generate a test report. I have used XCPretty mediator that converts Xcode 8 XML report into JUnit supported XML. You can find more details at https://blog.talentica.com/2017/04/04/use-xcode-8-with-jenkins/
I am trying to get the code coverage with jenkins and jacoco plugins.
I have a jacoco agent jar on the machine where my testing is being executed. I then retrieve the dump and try to get the code coverage on jenkins.
However I keep getting the below error,
[JaCoCo plugin] Collecting JaCoCo coverage data...
[JaCoCo plugin] \**/coverage/jacoco.exec;\**/coverage/classes-cov;\**/application/; locations are configured
[JaCoCo plugin] Number of found exec files for pattern \**/coverage/jacoco.exec: 1
[JaCoCo plugin] Saving matched execfiles: /home/ec2-user/slave/workspace/Automation_Code_Coverage_POMS/coverage/jacoco.exec
[JaCoCo plugin] Saving matched class directories for class-pattern: \**/coverage/classes-cov: /home/ec2-user/slave/workspace/Automation_Code_Coverage_POMS/coverage/classes-cov
[JaCoCo plugin] Saving matched source directories for source-pattern: \**/application/:
[JaCoCo plugin] Loading inclusions files..
[JaCoCo plugin] inclusions: [\**/com/test/poms/\**]
[JaCoCo plugin] exclusions: [\**/poms/convertors/\**:\**/poms/scheduler/\**]
ERROR: Publisher 'Record JaCoCo coverage report' aborted due to exception:
java.io.IOException: Error while analyzing class /home/ec2-user/.jenkins/jobs/Automation_Code_Coverage_POMS/builds/43/jacoco/classes/com/test/poms/convertors/DtoToSroConverter.83f57acb46d004b5.class.
at org.jacoco.core.analysis.Analyzer.analyzerError(Analyzer.java:150)
at org.jacoco.core.analysis.Analyzer.analyzeClass(Analyzer.java:144)
at org.jacoco.core.analysis.Analyzer.analyzeAll(Analyzer.java:175)
at org.jacoco.core.analysis.Analyzer.analyzeAll(Analyzer.java:208)
at hudson.plugins.jacoco.ExecutionFileLoader.analyzeStructure(ExecutionFileLoader.java:126)
at hudson.plugins.jacoco.ExecutionFileLoader.loadBundleCoverage(ExecutionFileLoader.java:133)
at hudson.plugins.jacoco.JacocoReportDir.parse(JacocoReportDir.java:102)
at hudson.plugins.jacoco.JacocoBuildAction.loadRatios(JacocoBuildAction.java:291)
at hudson.plugins.jacoco.JacocoBuildAction.load(JacocoBuildAction.java:273)
at hudson.plugins.jacoco.JacocoPublisher.perform(JacocoPublisher.java:371)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
at hudson.model.Build$BuildExecution.post2(Build.java:185)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
at hudson.model.Run.execute(Run.java:1769)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:374)
Caused by: java.lang.IllegalStateException: Can't add different class with same name: com/test/poms/convertors/DtoToSroConverter
at org.jacoco.core.analysis.CoverageBuilder.visitCoverage(CoverageBuilder.java:106)
at org.jacoco.core.analysis.Analyzer$1.visitEnd(Analyzer.java:92)
at org.objectweb.asm.ClassVisitor.visitEnd(ClassVisitor.java:317)
at org.jacoco.core.internal.flow.ClassProbesAdapter.visitEnd(ClassProbesAdapter.java:98)
at org.objectweb.asm.ClassReader.accept(ClassReader.java:697)
at org.objectweb.asm.ClassReader.accept(ClassReader.java:506)
at org.jacoco.core.analysis.Analyzer.analyzeClass(Analyzer.java:107)
at org.jacoco.core.analysis.Analyzer.analyzeClass(Analyzer.java:142)
... 17 more
Notifying upstream projects of job completion
JaCoCo Can't add different class with same name: org/hamcrest/BaseDescription
The above link suggest to exclude the files, but if you look at the above logs, they are already being excluded but I still see this issue.
In my case see this:
12:35:12 [JaCoCo plugin] exclusions: [**/*koba*.class] in Jenkins logs while Jenkins Jacoco plugin performs the analysis. There's no backslashes as compared to what you get.
Secondly, the error that you are getting is due to either of the 2 reasons:
You have .java/.groovy files and after compilation, you create .class files.
It seems like there's class file (for ex: abc.java or com/test/poms/convertors/DtoToSroConverter in your case) which is present in the source folder that you have mentioned in the plugin's "source" field.
If you are creating any class files (for which you don't have a .java/.groovy file in source (src/main/java or src/main/groovy or src/test/java, src/test/groovy, src/xxx/java or src/xxx/groovy folders) then, jacoco analysis will error out with the same error i.e. it won't be able to find the respective source file (.java/.groovy) for the .class file it's analyzing.
Check, how many files are there with name starting with: DtoToSroConverter in your project.
Then, make sure the values that you mention in Jacoco plugin in Jenkins looks like this. NOTE: source code should NOT contains any test (unit/non-Unit test source) folders.
In my case, I'm saying, process all .exec files (anywhere in my project's workspace after the build/tests/jacoco process is complete) i.e. ****/*.exec**
Then, Path to class directories should always mention only the MAIN source class files (not test classes of either unit/non-unit types) i.e. I only used "build/classes/main" as these classes are generated against my source main code (src/main/java OR src/java). This value is the folder which contains your main source code classes only.
Path to source directories field should always contain the folder where the actual main source code exists (instead of including any test source code) i.e. I have used "src/java". I could have used "src/main/java" which is the Gradle/Maven standard folder structure for main source code. In my case, my main source code is in src/java folder.
Check if Path to source directories field and Path to class directories is set correctly? If yes, is there more than one file with the name: DtoToSroConverter
i am using Hudson as CI server for Delphi 2010 projects. The XMLTestRunner.pas unit writes DUnit test result to a xml file that is used by the hudson xUnit plugin to report test results. The xUnit plugin shows failures but no messages:
Stacktrace
MESSAGE:
+++++++++++++++++++
STACK TRACE:
In the hudson project configuration at "Publish testing tools result report" i choose "NUnit-Version N/A (default)" as tesing tool, because there is no dunit option in the list and the xml files looks similar to nunit format:
<?xml version="1.0" encoding="ISO-8859-1" standalone="yes" ?>
<test-results total="123" notrun="0" date="20.12.2010" time="09:19:24">
<test-suite name="HelloWorldTestSuite" total="46" notrun="0">
<results>
<test-case name="TestCase.HelloWorldTest" execute="True" success="False" time="0,003" result="Failure">
<failure name="ETestFailure" location=""/>
<message>expected: < hello world! > but was: < hallo welt ></message>
</test-case>
...
In the hudson configuration there is also an "Custom Tool" option, where i have to specify a "Custom stylesheet", but i don't know how to write such a stylesheet (is there any documentation?).
On my C++ projets with boost test, the failures are reported nicely with all messages.
I had to change the XMLListener to get it work with hudson because the XML structure of a failure had to look like this:
<testcase name="GetFreeDirNameTest" classname="Test.exe.MyFiles.TTests" result="failed" time="0.000">
<failure message="Expected:
"-1"
But was:
"0"" type="failed">GetFreeDirNameTest: Expected:
"-1"
But was:
"0"
</failure>
</testcase>
What about using XMLTestRunner for NUnit compatible output from DUnit?