I was trying to integrate code coverage on my project by using jacoco, ant and teamcity. However, I realized that when jacoco task is around the junit task, teamcity does not catch the failing tests and everything is a success even with test failed.
Here are my 2 test tasks to test with and without jacoco and see teamcity bahaviours.
1- with jacoco activated
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
</jacoco:coverage>
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
2- without jacoco
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<!--<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">-->
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
<!--</jacoco:coverage>-->
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
Only jacoco task has been commented between the 2 releases of test.
Teamcity output
[CommonBuildTasks.-test] echo
[08:26:21]: [echo] JaCoCo activated
[08:26:21]: [CommonBuildTasks.-test] jacoco:coverage (4s)
[08:26:21]: [jacoco:coverage] Enhancing junit with coverage.
[08:26:22]: [jacoco:coverage] Running ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest
[08:26:25]: [jacoco:coverage] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 3.511 sec
[08:26:26]: [jacoco:coverage] Test ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest FAILED
[08:26:26]: [CommonBuildTasks.-test] copy
[08:26:26]: [copy] Copying 1 file to C:\TeamCity\buildAgent\work\cc10e09e43249f57\reports
As you can see, a test failed but teamcity has reported a successfull build.
Any idea why I got this behaviour?
Thanks
The answer is hidden in your call to the JUnit-Task:
<junit haltonfailure="no">...</junit>
With this configuration, the JUnit task does not fail the build on failing tests. This should lead to the desired behaviour:
<junit haltonfailure="yes">...</junit>
See the Ant documentation for the configuration of the JUnit task.
I solved this issue by using agent task instead of the coverage task. So, instead of
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
Use:
<jacoco:agent property="agentvmparam" destfile="${bin}/jacoco.exec"/>
<junit fork="yes"...
<jvmarg value="${agentvmparam}"/>
</junit>
Agent task uses the same properties as the coverage task. Then you can start your junit task without wrapping it in coverage task. That way teamcity is able to intercept junit task output.
Related
I have run into this specific problem while executing the junit with jacoco:coverage for code coverage. Tried a few things but no luck yet.
I have this junit build script.
<target name="executeJunitMain" depends="createJunitLibs" description="Executes All Junit and prepare report.">
<junit fork="yes" haltonfailure="off" failureProperty="junit.failure" includeantruntime="true" maxmemory="256m">
<classpath refid="compile.class.path" />
<formatter type="xml" />
<jvmarg value="-javaagent:${external.junit.lib.dir}/jmockit.jar"/>
<sysproperty key="jacoco-agent.destfile" file="${coverage.dir}/jacoco.exec"/>
<batchtest fork="yes" todir="${report.dir}" >
<fileset dir="${dest.dir}">
<include name="**/Test*.class" />
<exclude name="**/AllTests.class" />
<exclude name="**/*$*.class"/>
</fileset>
</batchtest>
</junit>
<antcall target="report"/>
</target>
<!-- Execute the coverage report.-->
<target name="report" description="Collect Coverage Report">
<!-- Creates the Junit Report -->
<junitreport todir="${report.dir}">
<fileset dir="${report.dir}">
<include name="TEST-*.xml"/>
</fileset>
<report format="frames" todir="${report.dir}/html"/>
</junitreport>
<!-- Creates the coverage Report.-->
<mkdir dir="${report.dir}/coverage"/>
<jacoco:report>
<executiondata>
<file file="${coverage.dir}/jacoco.exec" />
</executiondata>
<structure name="Code Coverage">
<classfiles>
<zipfileset src="${sources.lib.dir}/${test.jar.name}.jar"/>
</classfiles>
</structure>
<html destdir="${report.dir}/coverage" />
</jacoco:report>
</target>
using this ant script sometimes the junit build is failing. Error as below
BUILD FAILED
C:\Users\user\Codebases\junit.xml:111: The following error occurred while executing this line:
C:\Users\user\Codebases\junit.xml:147: The following error occurred while executing this line:
C:\Users\user\Codebases\junit.xml:164: Unable to read execution data file C:\Users\user\CodeBases\ju
nit\coverage\jacoco.exec
But sometimes it works perfectly. I am not sure when it works and when it does not. Any help will be appreciated.
Thanks
I have found the issue that was causing the jacoco to fail. I had a few instrumented jars imported in the classpath which were not required for the junit execution or coverage. These instrumented jars were causing jacoco session to be incorrect as we also had jmockit integration as Mocking Framework. The below link was very helpful for finding out the issue
eclemma.org/jacoco/trunk/doc/classids.html
There is part of ant script with junit task:
...
<target name="test">
<mkdir dir="path_to_report_dir">
<junit fork="true" printsummary="true" showoutput="true" maxmemory="1024M">
<classpath ... />
<batchtest todir="path_to_report_dir">
<formatter type="xml" />
<fileset ... />
</batchtest>
</junit>
</target>
...
This script works from Eclipse and from command line. But it doesn't work in TeamCity. The last informative message in TeamCity is:
[mkdir] Created dir: path_to_report_dir
Process exit code: 0
It looks like junit task doesn't work and also it stops performting aff all script. Where is trouble in?
The cause was in <fileset> file list. The TeamCity version of Ant doesn't work with strings like "/test/" (this mean select all files recursively); it only works with strings like "**/test/*.class". The local version of Ant supports both variants.
Thanks.
Don't know if this helps.... but here's my standard test target:
<target name="test" depends="compile-tests">
<junit printsummary="yes" haltonfailure="yes">
<classpath>
<path refid="test.path"/>
<pathelement path="${classes.dir}"/>
<pathelement path="${test.classes.dir}"/>
</classpath>
<formatter type="xml"/>
<batchtest fork="yes" todir="${test.reports.dir}">
<fileset dir="${test.src.dir}">
<include name="**/*Test*.java"/>
<exclude name="**/AllTests.java"/>
</fileset>
</batchtest>
</junit>
</target>
Build output:
test:
[junit] Running org.demo.AppTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.056 sec
Notes
Using Junit 4.10.
I am not using an automated build tool. Just Checkstyle 5.5 and ANT 1.8.
I am trying to have Checkstyle run in my ANT script. The ANT script executes without error, but doesn't seem to call Checkstyle. I get no output except ANT reports BUILD SUCCESSFUL.
Here is my ant script:
<project name="ccu" xmlns:cs="antlib:com.puppycrawl.tools.checkstyle">
<target name="checkstyle" description="Generates a report of code convention violations.">
<cs:checkstyle config="custom_check.xml">
<fileset dir="src" casesensitive="yes">
<include name="**/*.java"/>
</fileset>
<!--
<fileset dir="src" includes="**\*.java"/>
-->
</cs:checkstyle>
</target>
</project>
what am i missing?
It was a classpath problem. For some reason I needed to direct the ANT classpath to the class files not the jar.
My final script looks like this:
<project name="ccu" xmlns:cs="antlib:com.puppycrawl.tools.checkstyle">
<taskdef resource="checkstyletask.properties">
<classpath>
<pathelement location="C:\myClasses\bin"/>
<pathelement location="C:\checkstyle-5.5\checkstyle-5.5-all.jar"/>
</classpath>
</taskdef>
<checkstyle config="custom_check.xml">
<fileset dir="src" includes="**/*.java"/>
</checkstyle>
</project>
I have an Ant build target that performs some testing using jUnit4
<target name="integrationtest" depends="init, buildtests, deploytests">
<junit haltonfailure="false">
<sysproperty key="driver" value="org.openqa.selenium.firefox.FirefoxDriver" />
<sysproperty key="screenshotDir" value="${screenshotsDir}" />
<classpath>
<pathelement location="${interfaceTestJar}"/>
</classpath>
<batchtest>
<fileset dir="${interfaceTestClasses}">
<include name="**/tests/Test*.class" />
</fileset>
</batchtest>
</junit>
<junit haltonfailure="false">
<sysproperty key="driver" value="org.openqa.selenium.ie.InternetExplorerDriver" />
<classpath>
<pathelement location="${interfaceTestJar}"/>
</classpath>
<batchtest>
<fileset dir="${interfaceTestClasses}">
<include name="**/tests/Test*.class" />
</fileset>
</batchtest>
</junit>
<echo message="##teamcity[publishArtifacts '${artifactsDir}']" />
</target>
First junit task is always started, but if there is any failed tests in it, the second one isn't starts (exepected to start in any case, even if first one has failed tests)
EDIT: Seems like there is another problem. Second jUnit is not started in any case (if first is succeed or failed). In my TeamCity build log i see the following lines
[integrationtest] junit
[20:06:14]: [junit] ru.company.tests.TestDateField
[20:06:30]: [junit] Process exited with code 255
TestDateField is my first test suite. After it there are some more suites and they all succeed (and the first one too).
Is there a chance your test does something like System.exit?
Did you try adding fork="true" to your junit task, so it will run in a separate JVM?
I think you should use <junit haltonfailure="no"> instead of <junit haltonfailure="false">
As per the docs, haltonfailure should be yes or no.
I'm running JUnit via Ant using a target something like this:
<target name="junit" depends="compile">
<mkdir dir="${report.dir}"/>
<junit printsummary="yes" haltonfailure="yes" showoutput="yes" >
<classpath>
<path refid="classpath"/>
<path location="${classes.dir}"/>
</classpath>
<formatter type="brief" usefile="false"/>
<formatter type="xml"/>
<batchtest fork="yes" todir="${report.dir}">
<fileset dir="${src.dir}" includes="**/*Tests.java" />
</batchtest>
</junit>
</target>
I have a class like this:
public class UserTests extends TestCase {
public void testWillAlwaysFail() {
fail("An error message");
}
public void testWillAlwaysFail2() {
fail("An error message2");
}
}
haltonfailure="yes" seems to cause the build to halt as soon as any single test has failed, logging only the first failed test. Setting it to "off" causes the entire build to succeed (even though test failure messages are written to the output).
What I want it for all tests to be run (even if one has failed), and then the build to be stopped if any tests have failed.
Is it possible to do this?
You can set the failureproperty attribute of the junit task, then test the property afterwards with the fail task:
<junit haltonfailure="no" failureproperty="test.failed" ... >
...
</junit>
<fail message="Test failure detected, check test results." if="test.failed" />