I'm using ant to do my builds, and I'm using TestNG for testing.
Is it possible to not have the build fail whenever there is a test failure?
And, should I have a test method throw an Exception, or catch any exceptions that my be thrown in the method?
In case you are using junit task you can set haltonfailure property.
The quote from ant documentation:
haltonfailure - Stop the build process if a test fails (errors are
considered failures as well).
The sample:
<target name="test_skip_test">
<echo>before test call</echo>
<junit printsummary="yes" haltonfailure="no">
<classpath>
<pathelement location="${build.tests}"/>
<pathelement path="${java.class.path}"/>
</classpath>
<test name="UnitTest1" haltonfailure="no" outfile="result">
<formatter type="xml"/>
</test>
</junit>
<echo>after test call</echo>
</target>
Or you can use failureproperty property.
The quote from ant documentation:
failureproperty - The name of a property to set in the event of a
failure (errors are considered failures as well). Overrides value set
in <junit>.
Related
I was trying to integrate code coverage on my project by using jacoco, ant and teamcity. However, I realized that when jacoco task is around the junit task, teamcity does not catch the failing tests and everything is a success even with test failed.
Here are my 2 test tasks to test with and without jacoco and see teamcity bahaviours.
1- with jacoco activated
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
</jacoco:coverage>
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
2- without jacoco
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<!--<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">-->
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
<!--</jacoco:coverage>-->
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
Only jacoco task has been commented between the 2 releases of test.
Teamcity output
[CommonBuildTasks.-test] echo
[08:26:21]: [echo] JaCoCo activated
[08:26:21]: [CommonBuildTasks.-test] jacoco:coverage (4s)
[08:26:21]: [jacoco:coverage] Enhancing junit with coverage.
[08:26:22]: [jacoco:coverage] Running ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest
[08:26:25]: [jacoco:coverage] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 3.511 sec
[08:26:26]: [jacoco:coverage] Test ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest FAILED
[08:26:26]: [CommonBuildTasks.-test] copy
[08:26:26]: [copy] Copying 1 file to C:\TeamCity\buildAgent\work\cc10e09e43249f57\reports
As you can see, a test failed but teamcity has reported a successfull build.
Any idea why I got this behaviour?
Thanks
The answer is hidden in your call to the JUnit-Task:
<junit haltonfailure="no">...</junit>
With this configuration, the JUnit task does not fail the build on failing tests. This should lead to the desired behaviour:
<junit haltonfailure="yes">...</junit>
See the Ant documentation for the configuration of the JUnit task.
I solved this issue by using agent task instead of the coverage task. So, instead of
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
Use:
<jacoco:agent property="agentvmparam" destfile="${bin}/jacoco.exec"/>
<junit fork="yes"...
<jvmarg value="${agentvmparam}"/>
</junit>
Agent task uses the same properties as the coverage task. Then you can start your junit task without wrapping it in coverage task. That way teamcity is able to intercept junit task output.
I have an Ant build target that performs some testing using jUnit4
<target name="integrationtest" depends="init, buildtests, deploytests">
<junit haltonfailure="false">
<sysproperty key="driver" value="org.openqa.selenium.firefox.FirefoxDriver" />
<sysproperty key="screenshotDir" value="${screenshotsDir}" />
<classpath>
<pathelement location="${interfaceTestJar}"/>
</classpath>
<batchtest>
<fileset dir="${interfaceTestClasses}">
<include name="**/tests/Test*.class" />
</fileset>
</batchtest>
</junit>
<junit haltonfailure="false">
<sysproperty key="driver" value="org.openqa.selenium.ie.InternetExplorerDriver" />
<classpath>
<pathelement location="${interfaceTestJar}"/>
</classpath>
<batchtest>
<fileset dir="${interfaceTestClasses}">
<include name="**/tests/Test*.class" />
</fileset>
</batchtest>
</junit>
<echo message="##teamcity[publishArtifacts '${artifactsDir}']" />
</target>
First junit task is always started, but if there is any failed tests in it, the second one isn't starts (exepected to start in any case, even if first one has failed tests)
EDIT: Seems like there is another problem. Second jUnit is not started in any case (if first is succeed or failed). In my TeamCity build log i see the following lines
[integrationtest] junit
[20:06:14]: [junit] ru.company.tests.TestDateField
[20:06:30]: [junit] Process exited with code 255
TestDateField is my first test suite. After it there are some more suites and they all succeed (and the first one too).
Is there a chance your test does something like System.exit?
Did you try adding fork="true" to your junit task, so it will run in a separate JVM?
I think you should use <junit haltonfailure="no"> instead of <junit haltonfailure="false">
As per the docs, haltonfailure should be yes or no.
I have an ant task set up like so:
<target name="unit-test" description="unit tests" depends="compile-tests">
<mkdir dir="${build}/test"/>
<mkdir dir="${build}/test/raw"/>
<mkdir dir="${build}/test/reports"/>
<!-- set up scratch database for tests -->
<mkdir dir="${build.dbTest}" />
<junit printsummary="yes" haltonfailure="no" maxmemory="512m" >
<classpath>
<pathelement path="${java.class.path}"/>
<pathelement path="${build.classes}"/>
<pathelement path="${build.test-classes}"/>
<fileset dir="lib" includes="*.jar"/>
<fileset dir="lib-test" includes="*.jar"/>
</classpath>
<formatter type="xml"/>
<sysproperty key="derby.system.home" value="${build.dbTest}" />
<batchtest fork="yes" todir="${build}/test/raw">
<fileset dir="${src.test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
<junitreport todir="${build}/test">
<fileset dir="${build}/test/raw"/>
<report todir="${build}/test/reports"/>
</junitreport>
</target>
Which works pretty well for running all my tests, but running all my tests really slows down my TDD Fail-Pass-Refactor groove. My full test suite takes about six minutes to run, which is way too long for quick response changes during TDD, especially since most of the time I only care about results from one test. The work flow I'd like to have would be
create test for new feature/bug
run only the new test (or at most only the test class I just modified)
write some code
iterate 2-3 until the new tests are passing
run full set of tests to make sure nothing else broke
plug any broken test into the 2-3 cycle above and repeat full cycle
when all tests pass, declare victory.
TestNG seems to have capability for grouping tests, which seems ideal (I could have a "TDD" group for the tests I'm currently working with. Changing that when I start working on something is an acceptable level of manual configuration here), but I don't want to switch test frameworks unless I absolutely have to. Is there any way to do something similar, or another way to achieve my desired work flow, using JUnit?
I use in my scripts instead of
<include name="**/*Test.java"/>
the snippet
<include name="${test}"/>
and set the property test to **/*Test.java earlier in the script. Now I can start ant, setting the property to a different value:
ant test -Dtest=**/*AcceptanceTests.java
I'm running JUnit via Ant using a target something like this:
<target name="junit" depends="compile">
<mkdir dir="${report.dir}"/>
<junit printsummary="yes" haltonfailure="yes" showoutput="yes" >
<classpath>
<path refid="classpath"/>
<path location="${classes.dir}"/>
</classpath>
<formatter type="brief" usefile="false"/>
<formatter type="xml"/>
<batchtest fork="yes" todir="${report.dir}">
<fileset dir="${src.dir}" includes="**/*Tests.java" />
</batchtest>
</junit>
</target>
I have a class like this:
public class UserTests extends TestCase {
public void testWillAlwaysFail() {
fail("An error message");
}
public void testWillAlwaysFail2() {
fail("An error message2");
}
}
haltonfailure="yes" seems to cause the build to halt as soon as any single test has failed, logging only the first failed test. Setting it to "off" causes the entire build to succeed (even though test failure messages are written to the output).
What I want it for all tests to be run (even if one has failed), and then the build to be stopped if any tests have failed.
Is it possible to do this?
You can set the failureproperty attribute of the junit task, then test the property afterwards with the fail task:
<junit haltonfailure="no" failureproperty="test.failed" ... >
...
</junit>
<fail message="Test failure detected, check test results." if="test.failed" />
We have migrated to both JUnit 4 and ant 1.7
The tests runs fine in eclipse, but the annotations are ignored when running the tests using ant.
According to the Ant junit task documentation:
It also works with JUnit 4.0, including "pure" JUnit 4 tests using only annotations and no JUnit4TestAdapter.
But the documentation doesn't elaborate on how it should be configured.
Is there any special setting required for the junit task? Am I missing something?
We have both Tests that extends TestCase (i.e. 3.8 style) and "pure" Junit 4 tests, could that be the problem?
I am using pure JUnit4 tests with Ant.
Here is the interesting part of my build file:
<junit printsummary="yes" haltonfailure="yes">
<formatter type="xml"/>
<classpath refid="path.test"/>
<batchtest fork="yes" todir="${dir.report.unittests.xml}">
<fileset dir="src">
<include name="**/*Test*.java"/>
</fileset>
</batchtest>
</junit>
Make sure you have the latest version of the junit.jar file in the lib directory of Ant. As far as I know the required version is delivered with ant 1.7 or higher versions...
Ant ships with a version of JUnit 3 by default. JUnit 3 has no support for test annotations.
To use the JUnit 4 annotations from the junit task make sure that you provide the location of a JUnit 4 jar in a nested classpath element of the junit task (see this entry in the ant FAQ).
<junit showoutput="yes" fork="true">
<classpath>
<!-- The location of the JUnit version that you want to use -->
<pathelement location="lib/junit-4.9b1.jar"/>
</classpath>
<formatter type="plain" usefile="false" />
<batchtest>
<fileset dir="${tests.dir}"/>
</batchtest>
</junit>
This is a preferable solution to overwriting the ant-junit.jar in ANT_HOME/lib as it means you can keep your JUnit jar in source control alongside your code making upgrades to later versions straightforward.
Note that whilst I haven't specified any include pattern in my fileset above this does mean that the junit task will attempt to run JUnit against all the classes in that directory structure which might result in a number of classes being included that don't contain any tests depending on how you have structured your source files.
You can finally only find and execute tests with the skipNonTests parameter added in ant 1.9.3+!
This is the code snippet from the accepted answer above (except for the new skipNonTests parameter and getting rid of the "Test" in the filename requirement):
<junit printsummary="yes" haltonfailure="yes">
<formatter type="xml"/>
<classpath refid="path.test"/>
<batchtest skipNonTests="true" fork="yes" todir="${dir.report.unittests.xml}">
<fileset dir="src">
<include name="**/*.java"/>
</fileset>
</batchtest>
</junit>
This happened to me and it was because I was both using annotations and extending TestCase.
public class TestXXX extends TestCase {
#Test
public void testSimpleValidCase() {
// this was running
}
#Test
public void simpleValidCase() {
// this wasn't running
}
}
When you extend TestCase you are assuming JUnit3 style so JUnit4 annotations are ignored.
The solution is to stop extending TestCase.
Verify your classpath definition...
this solved my problem.
<path id="classpath" description="Classpath do Projeto">
<fileset dir="${LIB}">
<include name="**/*.jar" />
<exclude name="**/.SVN/*.*"/>
</fileset>
</path>
This is the relevant part of my generic ant script... not sure if that'll help you or not..
<junit fork="true"
forkmode="once"
haltonfailure="false"
haltonerror="false"
failureproperty="tests.failures"
errorproperty="tests.errors"
includeantruntime="true"
showoutput="true"
printsummary="true">
<classpath>
<path refid="path-id.test.classpath.run"/>
</classpath>
<formatter type="xml"/>
<batchtest fork="yes"
todir="${dir.build.testresults}">
<fileset dir="${dir.src.tests}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
Apply this annotation to the other classes org.junit.Ignore
I also tried to do tests with JUnit 4.0 without JUnit4TestAdapter, i.e. without method
public static junit.framework.Test suite() {
return new JUnit4TestAdapter(SomeTestClass.class);
}
I use ant 1.9.4.
Running ant test verbose (ant -v ) shows
[junit] Running multiple tests in the same VM
[junit] Implicitly adding /usr/share/java/junit.jar:/usr/sharejava/ant-launcher.jar:/usr/share/java/ant.jar:/usr/share/java/ant/ant-junit.jar to CLASSPATH
Aha, but still there is some ant-junit-task.
Downloading this shows in addition
/usr/share/java/ant/ant-junit4.jar which is not added implicitly.
I just added it explicitly:
<junit printsummary="yes"
fork="yes"
forkmode="once"
maxmemory="1023m"
showoutput="no">
...
<classpath>
<pathelement path="...:${junitJar}:${hamcrestJar}:/usr/share/java/ant/ant-junit4.jar" />
</classpath>
...
</junit>
and it worked. Without: no.
I am aware that this solution is not beautiful at all...
What I ended up doing was adding an Ant to one of my definitions that is used by the task>. Et voila.