I've seen a couple of posts about this but none provide the solution I'm looking for. I have ant set up so when it runs our tests we get the following output (which I like):
[junit] Running net.windward.document.test.TestTab
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.59 sec
[junit] Running net.windward.document.test.TestTableAttributes
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.604 sec
... continues for 300+ tests ...
But then at the end I just get "some tests failed". Is there some way to have it then list out the filenames of the tests that have failed?
thanks - dave
build.xml:
<junit printsummary="yes" errorProperty="test.failed" failureProperty="test.failed" >
<formatter type="plain" />
<classpath path="${classpath.run.test}"/>
<batchtest fork="yes" todir="${reports}">
<fileset dir="${src}">
<include name="**/test/**/Test*.java" />
<exclude name="**/_*/**/*.java" />
</fileset>
</batchtest>
</junit>
<fail if="test.failed" message="one or more unit tests failed"/>
<echo>unit tests all passed</echo>
<antcall target="testinternal"/>
</target>
Read chapter 4 of "Java Development with Ant": http://www.manning-source.com/books/hatcher/hatcher_ch04.pdf
You can use
<formatter type="xml" />
to generate an xml output of the tests. This can be converted to html to view it in a browser after all tests have run using the ant task junitreport. You can also import the xml file into eclipse if you use it.
Related
I have a test suite which re-runs all the failed tests at the end of the suite execution. But even if the test passes in the second run, junit report displays the output of the first run, ie. it shows that the test failed.
Following is an extract from build.xml:
<junitreport tofile="./report/html/result.xml">
<fileset dir="./report">
<include name="result.xml"/>
</fileset>
<report format="noframes" styledir="./etc_time" todir="./report/html/">
</report>
</junitreport>
<copy file="report/html/junit-noframes.html" tofile="report/html/index.html" />
<fail message="Tests failed" if="test.failed"/>
Can someone please point to some tutorials on how to do it?
Thanks!
I was trying to integrate code coverage on my project by using jacoco, ant and teamcity. However, I realized that when jacoco task is around the junit task, teamcity does not catch the failing tests and everything is a success even with test failed.
Here are my 2 test tasks to test with and without jacoco and see teamcity bahaviours.
1- with jacoco activated
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
</jacoco:coverage>
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
2- without jacoco
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<!--<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">-->
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
<!--</jacoco:coverage>-->
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
Only jacoco task has been commented between the 2 releases of test.
Teamcity output
[CommonBuildTasks.-test] echo
[08:26:21]: [echo] JaCoCo activated
[08:26:21]: [CommonBuildTasks.-test] jacoco:coverage (4s)
[08:26:21]: [jacoco:coverage] Enhancing junit with coverage.
[08:26:22]: [jacoco:coverage] Running ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest
[08:26:25]: [jacoco:coverage] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 3.511 sec
[08:26:26]: [jacoco:coverage] Test ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest FAILED
[08:26:26]: [CommonBuildTasks.-test] copy
[08:26:26]: [copy] Copying 1 file to C:\TeamCity\buildAgent\work\cc10e09e43249f57\reports
As you can see, a test failed but teamcity has reported a successfull build.
Any idea why I got this behaviour?
Thanks
The answer is hidden in your call to the JUnit-Task:
<junit haltonfailure="no">...</junit>
With this configuration, the JUnit task does not fail the build on failing tests. This should lead to the desired behaviour:
<junit haltonfailure="yes">...</junit>
See the Ant documentation for the configuration of the JUnit task.
I solved this issue by using agent task instead of the coverage task. So, instead of
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
Use:
<jacoco:agent property="agentvmparam" destfile="${bin}/jacoco.exec"/>
<junit fork="yes"...
<jvmarg value="${agentvmparam}"/>
</junit>
Agent task uses the same properties as the coverage task. Then you can start your junit task without wrapping it in coverage task. That way teamcity is able to intercept junit task output.
There is part of ant script with junit task:
...
<target name="test">
<mkdir dir="path_to_report_dir">
<junit fork="true" printsummary="true" showoutput="true" maxmemory="1024M">
<classpath ... />
<batchtest todir="path_to_report_dir">
<formatter type="xml" />
<fileset ... />
</batchtest>
</junit>
</target>
...
This script works from Eclipse and from command line. But it doesn't work in TeamCity. The last informative message in TeamCity is:
[mkdir] Created dir: path_to_report_dir
Process exit code: 0
It looks like junit task doesn't work and also it stops performting aff all script. Where is trouble in?
The cause was in <fileset> file list. The TeamCity version of Ant doesn't work with strings like "/test/" (this mean select all files recursively); it only works with strings like "**/test/*.class". The local version of Ant supports both variants.
Thanks.
Don't know if this helps.... but here's my standard test target:
<target name="test" depends="compile-tests">
<junit printsummary="yes" haltonfailure="yes">
<classpath>
<path refid="test.path"/>
<pathelement path="${classes.dir}"/>
<pathelement path="${test.classes.dir}"/>
</classpath>
<formatter type="xml"/>
<batchtest fork="yes" todir="${test.reports.dir}">
<fileset dir="${test.src.dir}">
<include name="**/*Test*.java"/>
<exclude name="**/AllTests.java"/>
</fileset>
</batchtest>
</junit>
</target>
Build output:
test:
[junit] Running org.demo.AppTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.056 sec
Notes
Using Junit 4.10.
My test target looks as follows:
<target name="test" depends="compileTest">
<junit haltonfailure="yes">
<classpath>
<pathelement location="bin" />
<pathelement location="lib/xstream-1.4.2.jar" />
<pathelement location="lib/jettison-1.2.jar" />
</classpath>
<formatter type="plain" usefile="false" />
<batchtest>
<fileset dir="${test-src}">
<include name="**/*Test*" />
</fileset>
</batchtest>
</junit>
</target>
And output looks as follows:
test:
[junit] Testsuite: app.commons.error.test.ShellErrorMessageTest
[junit] Tests run: 9, Failures: 0, Errors: 0, Time elapsed: 0.006 sec
[junit] Testcase: testCreateNormal took 0.003 sec
[junit] Testcase: testGetSectionSeparatorLine took 0.001 sec
[junit] Testcase: testGetSectionSeparatorLineMultipleSymbols took 0 sec
[junit] Testcase: testGetSectionSeparatorLineEmptySymbol took 0 sec
[junit] Testcase: testGetSectionSeparatorLineEmptySymbolSpace took 0 sec
[junit] Testcase: testGetSectionSeparatorLineNull took 0 sec
How can i collapse this output to a 1 line summary?
Is there a way to receive 1 line output along the line of "Ran 500 tests. 0 Failures. 0 Errors"
You can set an unverbose junit-output by adding
<formatter type="brief" usefile="false"/>
within <junit>.
To further reduce it, the following options come to my mind:
use a standard unix/cygwin tool to filter the output,
add a <junitreport>-task within <junit>, or
check the free exemplary chapter "Testing with JUnit" from "Java Development with Ant, they probably have some advice for you in there.
I have made an Ant target that runs my JUnit 4 tests. Unfortunately all of them are executed twice!
Does anyone have an idea of what I have done wrong?
Here are my ant target:
<target name="junit" description="Execute unit tests" depends="compile">
<delete dir="rawtestoutput"/>
<delete dir="test-reports"/>
<mkdir dir="rawtestoutput"/>
<junit printsummary="on" failureproperty="junit.failure" fork="true">
<classpath refid="class.path.junit"/>
<formatter type="xml" usefile="true" />
<batchtest todir="rawtestoutput">
<fileset dir="src/test">
<include name="**/*.java"/>
<!-- Add util and testhelper classes here (to avoid "No tests in class" error) and add suite classes to avoid test being run twice -->
<exclude name="**/SessionHelper.java"/>
<exclude name="**/TestHelper.java"/>
<exclude name="**/AllTests.java"/>
<exclude name="**/AllEDITests.java"/>
</fileset>
</batchtest>
</junit>
<junitreport>
<fileset dir="rawtestoutput"/>
<report todir="test-reports"/>
</junitreport>
<fail if="junit.failure" message="Unit test(s) failed. See reports!"/>
</target>
My first idea was that it because of test suites. But I do not think that anymore. I have excluded the tests suites and furthermore, it not only the tests that are part of suites that are run twice. Its all my tests.
Below is a small sample of the test output of one of my testsclasses:
[20:24:53]: [junit] Running dk.gensam.gaia.business.ydelse.YdelsestypeBOTest
[20:24:53]: [junit] dk.gensam.gaia.business.ydelse.YdelsestypeBOTest (2s)
[20:24:54]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsevariationer
[20:24:55]: [loadYdelsevariationer] [Test Output] EMMA: collecting runtime coverage data ...
[20:24:55]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsestypeIndex_alleExisterendeErAnnullerede
[20:24:56]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsestypeIndex_ingenEksisterendeValgteRelationer
[20:24:56]: [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 3,077 sec
[20:24:56]: dk.gensam.gaia.business.ydelse.YdelsestypeBOTest (3s)
[20:24:56]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsevariationer
[20:24:56]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsestypeIndex_alleExisterendeErAnnullerede
[20:24:56]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsestypeIndex_ingenEksisterendeValgteRelationer
As you can see the tests in YdelsestypeBOTest are run twice...
From the line:
[20:24:55]: [loadYdelsevariationer] [Test Output] EMMA: collecting runtime coverage data
It looks like a different Ant target is invoking the code coverage tool Emma which is then re-running your tests. If you run your Ant script with this target i.e. ant junit, does it always do this?
It's hard to tell what exactly is happening here but try deleting all your test suites temporarily and recompile just to make sure they are not causing the issue. It looks like you may want to get rid of the test suites anyway if you are starting to move from test suites to using batchtest.