Printing number of JUnit failures in Ant - ant

Question

<target name="test" depends="compile-test">
<junit failureProperty="test.failure">
<classpath refid="classpath.test" />
<formatter type="brief" usefile="false" />
<batchtest>
<fileset dir="${tst-dir}" includes="**/Test*.class" />
</batchtest>
</junit>
<fail message="test failed" if="test.failure" />
</target>
I want to print how many test cases are:
failed
error
passed
by making changes only in the build.xml file. How can I do that?

Answer 1

You can use the junitreport task to gather your test results.
If you need to print the summary metrics in your build file, you could use a filter chain to extract the information from the generated report.
There may (must?) be a simpler way to do this, but I didn't see it.
<target>
<junit failureProperty="test.failure">
<classpath refid="classpath.test" />
<!-- use XML formatter and let it output to file -->
<formatter type="xml" />
<!-- specify output dir for test result files -->
<batchtest todir="tmp/results">
<fileset dir="${tst-dir}" includes="**/Test*.class" />
</batchtest>
</junit>
<!-- generate report with junitreport -->
<junitreport todir="tmp">
<fileset dir="tmp/results" />
<report todir="tmp/report" />
</junitreport>
<!-- concat the report through a filter chain to extract what you want -->
<concat>
<fileset file="tmp/report/overview-summary.html" />
<filterchain>
<linecontainsregexp>
<regexp pattern='title="Display all tests"' />
</linecontainsregexp>
<tokenfilter>
<!-- escaped values of < and > are ">" and "<" -->
<replaceregex pattern='.*all tests.*>(\d+)<.*all failures.*>(\d+)<.*all errors.*>(\d+)<.*$' replace="Run: \1, Failed: \2, Errors: \3" />
</tokenfilter>
</filterchain>
</concat>
<fail message="test failed" if="test.failure" />
</target>
The output will be something like:
Buildfile: C:\\test\unit_test.xml
test:
[junit] Test MyUnitTest FAILED
[junit] Test MyUnitTest2 FAILED
[junitreport] Processing C:\\test\tmp\TESTS-TestSuites.xml to C:\DOCUME~1\xxx\LOCALS~1\Temp\1\null1075123857
[junitreport] Loading stylesheet jar:file:/C:/eclipse/eclipse-jee-ganymede-SR2-win32/eclipse/plugins/org.apache.ant_1.7.0.v200803061910/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 906ms
[junitreport] Deleting: C:\DOCUME~1\xxx\LOCALS~1\Temp\1\null1075123857
[concat] Run: 8, Failed: 4, Errors: 1
BUILD FAILED
C:\test\unit_test.xml:32: test failed
Total time: 1 second
If you are running a large number of tests, you will now have the overhead of report generation extraction.

Answer 2

Exactly the same answer as sudocode's but with this line to parse the report (works for RSA 8.5.1 / JUnit 4.11 - and I was not allowed to place this piece of code as comment in sudocodes code - nor am I allowed to comment... ):
<replaceregex pattern='<td><a title="Display all tests" href="all-tests.html">(\d+)</a></td><td><a title="Display all failures" href="alltests-fails.html">(\d+)</a></td><td><a title="Display all errors" href="alltests-errors.html">(\d+).*$' replace="Run: \1, Failed: \2, Errors: \3" />
Thanks to sudocode!

Answer 3

You can use junitreport to generate a combined XML report from a set of test runs.
To then generate a textual summary you can create an XSLT and use the ant XSLT target to format the file. This will produce an output file, but you can use ant to read this in and echo it to the console.
The XSLT should use something along the lines of to count the testcases, errors and failures.
count(//testsuites/testcase)
count(//testsuites/testcase/error)
count(//testsuites/testcase/error)
(If you really only want to modify your ant build file you could generate the XSLT to a temp folder at build time and remove it afterwards.)

Answer 4

As an alternative to sudocodes-approach: You could set the printsummary attribute to the junit-task. This will print a summary after each test-class. It's not an overall summary.
<junit failureProperty="test.failure" printsummary="yes">
<classpath refid="classpath.test" />
<formatter type="brief" usefile="false" />
<batchtest>
<fileset dir="${tst-dir}" includes="**/Test*.class" />
</batchtest>
</junit>

Answer 5

Try this attribute on your junit task:
printsummary="yes"
For a slick javadoc-like html report, change your formatter to:
<formatter type="xml" />
and then create the reports with a target that calls this:
<junitreport>
<fileset dir="${report.dir}/tmp">
<include name="TEST-*.xml" />
</fileset>
<report format="frames" styledir="${junitxslt.dir}" todir="${report.dir}/html" />
</junitreport>

Answer 6

Try to embed Jsoup into a custom task and use the task in your build to extract the data you need from the overview-summary.html.
My code snippet below -
import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
public class ExtractJunitSummaryTask extends Task {
protected String overviewSummaryHtml;
protected String outProperty;
public void execute() throws BuildException {
StringBuffer sb = new StringBuffer();
// TODO: read and put overviewSummaryHtml file content into a String
Document doc = Jsoup.parse(sb.toString());
String allTests = doc.getElementsByAttributeValueContaining("href", "all-tests").text();
String allFailures = doc.getElementsByAttributeValueContaining("href", "alltests-fails").text();
String allErrors = doc.getElementsByAttributeValueContaining("href", "alltests-errors").text();
String allSkippedTests = doc.getElementsByAttributeValueContaining("href", "alltests-skipped").text();