I have a test suite which re-runs all the failed tests at the end of the suite execution. But even if the test passes in the second run, junit report displays the output of the first run, ie. it shows that the test failed.
Following is an extract from build.xml:
<junitreport tofile="./report/html/result.xml">
<fileset dir="./report">
<include name="result.xml"/>
</fileset>
<report format="noframes" styledir="./etc_time" todir="./report/html/">
</report>
</junitreport>
<copy file="report/html/junit-noframes.html" tofile="report/html/index.html" />
<fail message="Tests failed" if="test.failed"/>
Can someone please point to some tutorials on how to do it?
Thanks!
Related
I have run into this specific problem while executing the junit with jacoco:coverage for code coverage. Tried a few things but no luck yet.
I have this junit build script.
<target name="executeJunitMain" depends="createJunitLibs" description="Executes All Junit and prepare report.">
<junit fork="yes" haltonfailure="off" failureProperty="junit.failure" includeantruntime="true" maxmemory="256m">
<classpath refid="compile.class.path" />
<formatter type="xml" />
<jvmarg value="-javaagent:${external.junit.lib.dir}/jmockit.jar"/>
<sysproperty key="jacoco-agent.destfile" file="${coverage.dir}/jacoco.exec"/>
<batchtest fork="yes" todir="${report.dir}" >
<fileset dir="${dest.dir}">
<include name="**/Test*.class" />
<exclude name="**/AllTests.class" />
<exclude name="**/*$*.class"/>
</fileset>
</batchtest>
</junit>
<antcall target="report"/>
</target>
<!-- Execute the coverage report.-->
<target name="report" description="Collect Coverage Report">
<!-- Creates the Junit Report -->
<junitreport todir="${report.dir}">
<fileset dir="${report.dir}">
<include name="TEST-*.xml"/>
</fileset>
<report format="frames" todir="${report.dir}/html"/>
</junitreport>
<!-- Creates the coverage Report.-->
<mkdir dir="${report.dir}/coverage"/>
<jacoco:report>
<executiondata>
<file file="${coverage.dir}/jacoco.exec" />
</executiondata>
<structure name="Code Coverage">
<classfiles>
<zipfileset src="${sources.lib.dir}/${test.jar.name}.jar"/>
</classfiles>
</structure>
<html destdir="${report.dir}/coverage" />
</jacoco:report>
</target>
using this ant script sometimes the junit build is failing. Error as below
BUILD FAILED
C:\Users\user\Codebases\junit.xml:111: The following error occurred while executing this line:
C:\Users\user\Codebases\junit.xml:147: The following error occurred while executing this line:
C:\Users\user\Codebases\junit.xml:164: Unable to read execution data file C:\Users\user\CodeBases\ju
nit\coverage\jacoco.exec
But sometimes it works perfectly. I am not sure when it works and when it does not. Any help will be appreciated.
Thanks
I have found the issue that was causing the jacoco to fail. I had a few instrumented jars imported in the classpath which were not required for the junit execution or coverage. These instrumented jars were causing jacoco session to be incorrect as we also had jmockit integration as Mocking Framework. The below link was very helpful for finding out the issue
eclemma.org/jacoco/trunk/doc/classids.html
I was trying to integrate code coverage on my project by using jacoco, ant and teamcity. However, I realized that when jacoco task is around the junit task, teamcity does not catch the failing tests and everything is a success even with test failed.
Here are my 2 test tasks to test with and without jacoco and see teamcity bahaviours.
1- with jacoco activated
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
</jacoco:coverage>
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
2- without jacoco
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<!--<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">-->
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
<!--</jacoco:coverage>-->
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
Only jacoco task has been commented between the 2 releases of test.
Teamcity output
[CommonBuildTasks.-test] echo
[08:26:21]: [echo] JaCoCo activated
[08:26:21]: [CommonBuildTasks.-test] jacoco:coverage (4s)
[08:26:21]: [jacoco:coverage] Enhancing junit with coverage.
[08:26:22]: [jacoco:coverage] Running ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest
[08:26:25]: [jacoco:coverage] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 3.511 sec
[08:26:26]: [jacoco:coverage] Test ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest FAILED
[08:26:26]: [CommonBuildTasks.-test] copy
[08:26:26]: [copy] Copying 1 file to C:\TeamCity\buildAgent\work\cc10e09e43249f57\reports
As you can see, a test failed but teamcity has reported a successfull build.
Any idea why I got this behaviour?
Thanks
The answer is hidden in your call to the JUnit-Task:
<junit haltonfailure="no">...</junit>
With this configuration, the JUnit task does not fail the build on failing tests. This should lead to the desired behaviour:
<junit haltonfailure="yes">...</junit>
See the Ant documentation for the configuration of the JUnit task.
I solved this issue by using agent task instead of the coverage task. So, instead of
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
Use:
<jacoco:agent property="agentvmparam" destfile="${bin}/jacoco.exec"/>
<junit fork="yes"...
<jvmarg value="${agentvmparam}"/>
</junit>
Agent task uses the same properties as the coverage task. Then you can start your junit task without wrapping it in coverage task. That way teamcity is able to intercept junit task output.
There must be a simple setting I am missing so forgive me, but I've noticed on two occasions that my bad ant tasks do not cause the build to fail. For example:
Ant copy when source file does not exist ... BUILD SUCCESSFUL
Ant unzip, when task reports "can't write file" or similar message ... BUILD SUCCESSFUL
Ant exec error, invalid syntax ... BUILD SUCCESSFUL
How do I guarantee all ant task errors will result in a build failure?
<EXEC> tasks do no fail by default. You need to enable this with failonerror="true"
Failure of the Ant <COPY> task depends on what resource collection type is used. If you use a fileset or patternset, then all missing files are silently ignored. You can force a failure only by using the filelist type or the parameterized 'file` attribute is used.
Therefore what you want to use is either:
<copy todir="my_dir" file="foo" />
<copy todir="my_dir" flatten="true">
<filelist dir="" files="foo" />
</copy>
<copy todir="my_dir" flatten="true">
<filelist dir="">
<file name="foo" />
<file name="bar" />
<file name="zed" />
</filelist>
</copy>
Have you tried following:
<copy todir="your/path/details" failonerror="true">
</copy>
<zip destfile="your/path/details" whenempty="fail">
</zip>
<exec executable="your/path/details" failonerror="true">
</exec>
i have ant target which invokes simultaneously other 3 projects ant build target for junit. this build target executes all three projects even though any one of the project build fails. Problem here is, if any one of the build fails error message should display after compiling three projects build target but it is not happening, how can i solve it?
<target name="mainbuild">
<antcall target="junit-1">//in different project
<antcall target="junit-2">//in different project
<antcall target="junit-3">//in different project
<junitreport todir="./reports">
<fileset dir="./project-1/reports">
<include name="TEST-*.xml"/>
</fileset>
<fileset dir="./project-2/reports">
<include name="TEST-*.xml"/>
</fileset>
<fileset dir="./project-3/reports">
<include name="TEST-*.xml"/>
</fileset>
<report format="frames" todir="./report/html"/>
</junitreport>
</target>
<target name="junit-1">
.... do somethig
</target>
<target name="junit-2">
.... do somethig
</target>
<target name="junit-3">
.... do somethig
</target>
1) main build invokes 3 project,even though build fail in any one of the sub project, build successful message displays at the end, it shouldn't happen
2) if any one the sub project build fails, build report should generate, so that developer can analyze further on his failure.
Since you have clarified that your targets are in different projects, i.e. different build files, you will have to use the ant or subant tasks instead of antcall. The subant task has a parameter called failonerror which you can set to false so that a failure will not stop the top-level build. I don't know if it is possible to pass back to the top-level build the information that a sub-project build has actually failed its tests.
If your targets were all in the same project you could change your junit tasks so that they don't fail if a test fails, but instead set a property to indicate that a failure has occurred:
<junit failureProperty="test.failed"> ... </junit>
Then you can make your mainbuild target fail after generating the JUnit report:
<fail if="test.failed"> ... </fail>
I learned this from the book "Ant in Action".
I have an ant task set up like so:
<target name="unit-test" description="unit tests" depends="compile-tests">
<mkdir dir="${build}/test"/>
<mkdir dir="${build}/test/raw"/>
<mkdir dir="${build}/test/reports"/>
<!-- set up scratch database for tests -->
<mkdir dir="${build.dbTest}" />
<junit printsummary="yes" haltonfailure="no" maxmemory="512m" >
<classpath>
<pathelement path="${java.class.path}"/>
<pathelement path="${build.classes}"/>
<pathelement path="${build.test-classes}"/>
<fileset dir="lib" includes="*.jar"/>
<fileset dir="lib-test" includes="*.jar"/>
</classpath>
<formatter type="xml"/>
<sysproperty key="derby.system.home" value="${build.dbTest}" />
<batchtest fork="yes" todir="${build}/test/raw">
<fileset dir="${src.test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
<junitreport todir="${build}/test">
<fileset dir="${build}/test/raw"/>
<report todir="${build}/test/reports"/>
</junitreport>
</target>
Which works pretty well for running all my tests, but running all my tests really slows down my TDD Fail-Pass-Refactor groove. My full test suite takes about six minutes to run, which is way too long for quick response changes during TDD, especially since most of the time I only care about results from one test. The work flow I'd like to have would be
create test for new feature/bug
run only the new test (or at most only the test class I just modified)
write some code
iterate 2-3 until the new tests are passing
run full set of tests to make sure nothing else broke
plug any broken test into the 2-3 cycle above and repeat full cycle
when all tests pass, declare victory.
TestNG seems to have capability for grouping tests, which seems ideal (I could have a "TDD" group for the tests I'm currently working with. Changing that when I start working on something is an acceptable level of manual configuration here), but I don't want to switch test frameworks unless I absolutely have to. Is there any way to do something similar, or another way to achieve my desired work flow, using JUnit?
I use in my scripts instead of
<include name="**/*Test.java"/>
the snippet
<include name="${test}"/>
and set the property test to **/*Test.java earlier in the script. Now I can start ant, setting the property to a different value:
ant test -Dtest=**/*AcceptanceTests.java