I have an Ant build target that performs some testing using jUnit4
<target name="integrationtest" depends="init, buildtests, deploytests">
<junit haltonfailure="false">
<sysproperty key="driver" value="org.openqa.selenium.firefox.FirefoxDriver" />
<sysproperty key="screenshotDir" value="${screenshotsDir}" />
<classpath>
<pathelement location="${interfaceTestJar}"/>
</classpath>
<batchtest>
<fileset dir="${interfaceTestClasses}">
<include name="**/tests/Test*.class" />
</fileset>
</batchtest>
</junit>
<junit haltonfailure="false">
<sysproperty key="driver" value="org.openqa.selenium.ie.InternetExplorerDriver" />
<classpath>
<pathelement location="${interfaceTestJar}"/>
</classpath>
<batchtest>
<fileset dir="${interfaceTestClasses}">
<include name="**/tests/Test*.class" />
</fileset>
</batchtest>
</junit>
<echo message="##teamcity[publishArtifacts '${artifactsDir}']" />
</target>
First junit task is always started, but if there is any failed tests in it, the second one isn't starts (exepected to start in any case, even if first one has failed tests)
EDIT: Seems like there is another problem. Second jUnit is not started in any case (if first is succeed or failed). In my TeamCity build log i see the following lines
[integrationtest] junit
[20:06:14]: [junit] ru.company.tests.TestDateField
[20:06:30]: [junit] Process exited with code 255
TestDateField is my first test suite. After it there are some more suites and they all succeed (and the first one too).
Is there a chance your test does something like System.exit?
Did you try adding fork="true" to your junit task, so it will run in a separate JVM?
I think you should use <junit haltonfailure="no"> instead of <junit haltonfailure="false">
As per the docs, haltonfailure should be yes or no.
Related
I was trying to integrate code coverage on my project by using jacoco, ant and teamcity. However, I realized that when jacoco task is around the junit task, teamcity does not catch the failing tests and everything is a success even with test failed.
Here are my 2 test tasks to test with and without jacoco and see teamcity bahaviours.
1- with jacoco activated
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
</jacoco:coverage>
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
2- without jacoco
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<!--<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">-->
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
<!--</jacoco:coverage>-->
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
Only jacoco task has been commented between the 2 releases of test.
Teamcity output
[CommonBuildTasks.-test] echo
[08:26:21]: [echo] JaCoCo activated
[08:26:21]: [CommonBuildTasks.-test] jacoco:coverage (4s)
[08:26:21]: [jacoco:coverage] Enhancing junit with coverage.
[08:26:22]: [jacoco:coverage] Running ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest
[08:26:25]: [jacoco:coverage] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 3.511 sec
[08:26:26]: [jacoco:coverage] Test ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest FAILED
[08:26:26]: [CommonBuildTasks.-test] copy
[08:26:26]: [copy] Copying 1 file to C:\TeamCity\buildAgent\work\cc10e09e43249f57\reports
As you can see, a test failed but teamcity has reported a successfull build.
Any idea why I got this behaviour?
Thanks
The answer is hidden in your call to the JUnit-Task:
<junit haltonfailure="no">...</junit>
With this configuration, the JUnit task does not fail the build on failing tests. This should lead to the desired behaviour:
<junit haltonfailure="yes">...</junit>
See the Ant documentation for the configuration of the JUnit task.
I solved this issue by using agent task instead of the coverage task. So, instead of
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
Use:
<jacoco:agent property="agentvmparam" destfile="${bin}/jacoco.exec"/>
<junit fork="yes"...
<jvmarg value="${agentvmparam}"/>
</junit>
Agent task uses the same properties as the coverage task. Then you can start your junit task without wrapping it in coverage task. That way teamcity is able to intercept junit task output.
There is part of ant script with junit task:
...
<target name="test">
<mkdir dir="path_to_report_dir">
<junit fork="true" printsummary="true" showoutput="true" maxmemory="1024M">
<classpath ... />
<batchtest todir="path_to_report_dir">
<formatter type="xml" />
<fileset ... />
</batchtest>
</junit>
</target>
...
This script works from Eclipse and from command line. But it doesn't work in TeamCity. The last informative message in TeamCity is:
[mkdir] Created dir: path_to_report_dir
Process exit code: 0
It looks like junit task doesn't work and also it stops performting aff all script. Where is trouble in?
The cause was in <fileset> file list. The TeamCity version of Ant doesn't work with strings like "/test/" (this mean select all files recursively); it only works with strings like "**/test/*.class". The local version of Ant supports both variants.
Thanks.
Don't know if this helps.... but here's my standard test target:
<target name="test" depends="compile-tests">
<junit printsummary="yes" haltonfailure="yes">
<classpath>
<path refid="test.path"/>
<pathelement path="${classes.dir}"/>
<pathelement path="${test.classes.dir}"/>
</classpath>
<formatter type="xml"/>
<batchtest fork="yes" todir="${test.reports.dir}">
<fileset dir="${test.src.dir}">
<include name="**/*Test*.java"/>
<exclude name="**/AllTests.java"/>
</fileset>
</batchtest>
</junit>
</target>
Build output:
test:
[junit] Running org.demo.AppTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.056 sec
Notes
Using Junit 4.10.
I am trying to use ant to run junit tests and generate reports.
I am able to successfully run the tests but the report files are empty.
What am I doing wrong ?
This is my build.xml :
<project name="JunitTest" default="test" basedir=".">
<property name="testdir" location="." />
<property name="srcdir" location="." />
<property name="full-compile" value="true" />
<property name="test.reports" value="./reports" />
<path id="classpath.base"/>
<path id="classpath.test">
<pathelement location="${testdir}" />
<pathelement location="${srcdir}" />
<path refid="classpath.base" />
</path>
<target name="clean" >
<delete verbose="${full-compile}">
<fileset dir="${testdir}" includes="**/*.class" />
</delete> `
</target>
<target name="compile" depends="clean">
<javac srcdir="${srcdir}" destdir="${testdir}" verbose="${full-compile}" >
<classpath refid="classpath.test"/>
</javac>
</target>
<target name="test" depends="compile">
<junit>
<classpath refid="classpath.test" />
<formatter type="brief" usefile="false" />
<test name="com.tests.nav1" />
</junit>
<junitreport todir="${test.reports}">
<fileset dir="${test.reports}">
<include name="TEST-*.xml" />
</fileset>
<report todir="${test.reports}" />
</junitreport>
</target>
</project>
and this is the output on the console :
[junit] Using CLASSPATH C:\eclipse\eclipse-java-helios-SR1-win32\eclipse\JunitWS\SeleniumTraining\src;C:\jars\junit.jar;C:\ant\lib\ant-launcher.jar;C:\ant\lib\ant.jar;C:\ant\lib\ant-junit.jar;C:\ant\lib\ant-junit4.jar
[junit] Testsuite: com.tests.nav1
[junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 48.187 sec
[junit] ------------- Standard Output ---------------
[junit] testnav2
[junit] ------------- ---------------- ---------------
[junitreport] Using class org.apache.tools.ant.taskdefs.optional.TraXLiaison
[junitreport] Processing C:\eclipse\eclipse-java-helios-SR1-win32\eclipse\JunitWS\SeleniumTraining\src\reports\TESTS-TestSuites.xml to C:\Users\pmahajan\AppData\Local\Temp\null236099757
[junitreport] Loading stylesheet jar:file:/C:/ant/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[junitreport] Transform time: 330ms
[junitreport] Deleting: C:\Users\pmahajan\AppData\Local\Temp\null236099757
BUILD SUCCESSFUL
Total time: 49 seconds
If you look at the ant snippet, there are a few issues:
You have set usefile=false, which means no output file is created
You have set formatter type=brief, which will print detailed information only for failed tests
You need to also specify the todir - the folder where the report has to go in the <test> tag - the default is current folder. This should match the folder you are using in <junitreport> task.
You can try with the following updated <junit> section...
<junit>
<classpath refid="classpath.test" />
<formatter type="xml"/>
<test name="com.tests.nav1" todir="${test.reports}"/>
</junit>
1 <target name="test" depends="compile">
2 <junit>
3 <classpath refid="classpath.test" />
4 <formatter type="brief" usefile="false" />
5 <test name="com.tests.nav1" />
6 </junit>
7 <junitreport todir="${test.reports}">
8 <fileset dir="${test.reports}">
9 <include name="TEST-*.xml" />
10 </fileset>
11 <report todir="${test.reports}" />
12 </junitreport>
13 </target>
The above segment of your coded needs following changes.
You have to specify the to directory option in the 5th line( for example todir = ${data.reports} )
In the 8th line the directory specified must me data.reports.
The 11th line must contain the option format with the value frames (format="frames").
The ant JUnit Task doc gives this example that might help you (as it apparently does exactly what you're trying to achieve):
<junit printsummary="yes" haltonfailure="yes">
<classpath>
<pathelement location="${build.tests}"/>
<pathelement path="${java.class.path}"/>
</classpath>
<formatter type="plain"/>
<test name="my.test.TestCase" haltonfailure="no" outfile="result">
<formatter type="xml"/>
</test>
<batchtest fork="yes" todir="${reports.tests}">
<fileset dir="${src.tests}">
<include name="**/*Test*.java"/>
<exclude name="**/AllTests.java"/>
</fileset>
</batchtest>
</junit>
Runs my.test.TestCase in the same VM, ignoring the given CLASSPATH; only a warning is printed if this test fails. In addition to the plain text test results, for this test a XML result will be output to result.xml. Then, for each matching file in the directory defined for ${src.tests} a test is run in a separate VM. If a test fails, the build process is aborted. Results are collected in files named TEST-name.txt and written to ${reports.tests}.
It is specified in the doc that printsummary can take values on and off, but they're using yes in the example which is on the same page, so I guess it's accepted too.
Please try formatter with "xml"
<formatter type="${junit.format}"/>
where junit.format is property with appropriate value.
I'd like to execute all tests in a /test directory without using annotations such as
#Suite.SuiteClasses( ....)
In the past i had a single class, which was calling many other classes to test them all. This approach is no longer acceptable.
I have a /test directory, underneath which i have a number of packages, each containing several tests.
In my current ANT script, i have:
<target name="compileTest" depends="compile" description="compile jUnit">
<javac srcdir="${test}" destdir="${bin}" includeantruntime="true" />
</target>
followed by
<target name="test" depends="compileTest">
<junit printsummary="yes" fork="no" haltonfailure="no">
<classpath location="${bin}" />
<formatter type="plain" />
</junit>
</target>
In the past, i had
<test name="MyCollectionOfTests" />
I'd rather not do this anymore.
What am i missing? Please advise.
You can use a nested batchtest. For instance:
<junit printsummary="on"
fork="on"
dir="${test.build}"
haltonfailure="false"
failureproperty="tests.failed"
showoutput="true">
<classpath>
<path refid="tests.classpath"/>
</classpath>
<batchtest todir="${test.report}">
<fileset dir="${test.gen}">
<include name="**/Test*.java"/>
</fileset>
<fileset dir="${test.src}">
<include name="**/Test*.java"/>
<exclude name="gen/**/*"/>
</fileset>
</batchtest>
</junit>
In its simplest form, you can simply add a nested:
<batchtest todir="report">
<fileset dir="test"/>
</batchtest>
to your junit call.
We have migrated to both JUnit 4 and ant 1.7
The tests runs fine in eclipse, but the annotations are ignored when running the tests using ant.
According to the Ant junit task documentation:
It also works with JUnit 4.0, including "pure" JUnit 4 tests using only annotations and no JUnit4TestAdapter.
But the documentation doesn't elaborate on how it should be configured.
Is there any special setting required for the junit task? Am I missing something?
We have both Tests that extends TestCase (i.e. 3.8 style) and "pure" Junit 4 tests, could that be the problem?
I am using pure JUnit4 tests with Ant.
Here is the interesting part of my build file:
<junit printsummary="yes" haltonfailure="yes">
<formatter type="xml"/>
<classpath refid="path.test"/>
<batchtest fork="yes" todir="${dir.report.unittests.xml}">
<fileset dir="src">
<include name="**/*Test*.java"/>
</fileset>
</batchtest>
</junit>
Make sure you have the latest version of the junit.jar file in the lib directory of Ant. As far as I know the required version is delivered with ant 1.7 or higher versions...
Ant ships with a version of JUnit 3 by default. JUnit 3 has no support for test annotations.
To use the JUnit 4 annotations from the junit task make sure that you provide the location of a JUnit 4 jar in a nested classpath element of the junit task (see this entry in the ant FAQ).
<junit showoutput="yes" fork="true">
<classpath>
<!-- The location of the JUnit version that you want to use -->
<pathelement location="lib/junit-4.9b1.jar"/>
</classpath>
<formatter type="plain" usefile="false" />
<batchtest>
<fileset dir="${tests.dir}"/>
</batchtest>
</junit>
This is a preferable solution to overwriting the ant-junit.jar in ANT_HOME/lib as it means you can keep your JUnit jar in source control alongside your code making upgrades to later versions straightforward.
Note that whilst I haven't specified any include pattern in my fileset above this does mean that the junit task will attempt to run JUnit against all the classes in that directory structure which might result in a number of classes being included that don't contain any tests depending on how you have structured your source files.
You can finally only find and execute tests with the skipNonTests parameter added in ant 1.9.3+!
This is the code snippet from the accepted answer above (except for the new skipNonTests parameter and getting rid of the "Test" in the filename requirement):
<junit printsummary="yes" haltonfailure="yes">
<formatter type="xml"/>
<classpath refid="path.test"/>
<batchtest skipNonTests="true" fork="yes" todir="${dir.report.unittests.xml}">
<fileset dir="src">
<include name="**/*.java"/>
</fileset>
</batchtest>
</junit>
This happened to me and it was because I was both using annotations and extending TestCase.
public class TestXXX extends TestCase {
#Test
public void testSimpleValidCase() {
// this was running
}
#Test
public void simpleValidCase() {
// this wasn't running
}
}
When you extend TestCase you are assuming JUnit3 style so JUnit4 annotations are ignored.
The solution is to stop extending TestCase.
Verify your classpath definition...
this solved my problem.
<path id="classpath" description="Classpath do Projeto">
<fileset dir="${LIB}">
<include name="**/*.jar" />
<exclude name="**/.SVN/*.*"/>
</fileset>
</path>
This is the relevant part of my generic ant script... not sure if that'll help you or not..
<junit fork="true"
forkmode="once"
haltonfailure="false"
haltonerror="false"
failureproperty="tests.failures"
errorproperty="tests.errors"
includeantruntime="true"
showoutput="true"
printsummary="true">
<classpath>
<path refid="path-id.test.classpath.run"/>
</classpath>
<formatter type="xml"/>
<batchtest fork="yes"
todir="${dir.build.testresults}">
<fileset dir="${dir.src.tests}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
Apply this annotation to the other classes org.junit.Ignore
I also tried to do tests with JUnit 4.0 without JUnit4TestAdapter, i.e. without method
public static junit.framework.Test suite() {
return new JUnit4TestAdapter(SomeTestClass.class);
}
I use ant 1.9.4.
Running ant test verbose (ant -v ) shows
[junit] Running multiple tests in the same VM
[junit] Implicitly adding /usr/share/java/junit.jar:/usr/sharejava/ant-launcher.jar:/usr/share/java/ant.jar:/usr/share/java/ant/ant-junit.jar to CLASSPATH
Aha, but still there is some ant-junit-task.
Downloading this shows in addition
/usr/share/java/ant/ant-junit4.jar which is not added implicitly.
I just added it explicitly:
<junit printsummary="yes"
fork="yes"
forkmode="once"
maxmemory="1023m"
showoutput="no">
...
<classpath>
<pathelement path="...:${junitJar}:${hamcrestJar}:/usr/share/java/ant/ant-junit4.jar" />
</classpath>
...
</junit>
and it worked. Without: no.
I am aware that this solution is not beautiful at all...
What I ended up doing was adding an Ant to one of my definitions that is used by the task>. Et voila.