I am trying to generate the coverage report for my unit tests through cobertura.
Below are the steps I followed
1. Compile my code into a jar
2. Instrument the jar file
3. Run unit tests with the instrumented code
I get the error context already initialized, for all the unit test
Instead of the instrumented class files if i use the original jar the unit tests run but I get O% coverage in code.
below is my ant target for junit
<junit reloading="false" fork="yes" forkmode="perBatch" printSummary="yes" failureproperty="test.failure" includeantruntime="yes">
<jvmarg value="-XX:UseSplitVerifier" />
<sysproperty key="net.sourceforge.cobertura.datafile" value="{reports.dir}/cobertura.ser" />
<classpath>
<!-- Jar file of instrumented classes -->
<pathelement location="My_code_inst.jar">
<!-- Tests run with this jar but no coverage data -->
<!--pathelement location="My_code.jar" -->
<pathelement location="My_Test.jar"/>
</classpath>
<classpath refid="junit.classpath" />
<classpath refid="cobertura.classpath" />
<batchtest fork="yes" todir="${reports.dir}">
<resources>
<zipfileset src="My_Test.jar" includes="**/*Test.class"/>
</resources>
</batchtest>
</junit>
I did a binary comparison of My_code_inst.jar and My_code.jar. Did not see any orphans. The disassembled code revealed instrumented methods.
Related
I have run into this specific problem while executing the junit with jacoco:coverage for code coverage. Tried a few things but no luck yet.
I have this junit build script.
<target name="executeJunitMain" depends="createJunitLibs" description="Executes All Junit and prepare report.">
<junit fork="yes" haltonfailure="off" failureProperty="junit.failure" includeantruntime="true" maxmemory="256m">
<classpath refid="compile.class.path" />
<formatter type="xml" />
<jvmarg value="-javaagent:${external.junit.lib.dir}/jmockit.jar"/>
<sysproperty key="jacoco-agent.destfile" file="${coverage.dir}/jacoco.exec"/>
<batchtest fork="yes" todir="${report.dir}" >
<fileset dir="${dest.dir}">
<include name="**/Test*.class" />
<exclude name="**/AllTests.class" />
<exclude name="**/*$*.class"/>
</fileset>
</batchtest>
</junit>
<antcall target="report"/>
</target>
<!-- Execute the coverage report.-->
<target name="report" description="Collect Coverage Report">
<!-- Creates the Junit Report -->
<junitreport todir="${report.dir}">
<fileset dir="${report.dir}">
<include name="TEST-*.xml"/>
</fileset>
<report format="frames" todir="${report.dir}/html"/>
</junitreport>
<!-- Creates the coverage Report.-->
<mkdir dir="${report.dir}/coverage"/>
<jacoco:report>
<executiondata>
<file file="${coverage.dir}/jacoco.exec" />
</executiondata>
<structure name="Code Coverage">
<classfiles>
<zipfileset src="${sources.lib.dir}/${test.jar.name}.jar"/>
</classfiles>
</structure>
<html destdir="${report.dir}/coverage" />
</jacoco:report>
</target>
using this ant script sometimes the junit build is failing. Error as below
BUILD FAILED
C:\Users\user\Codebases\junit.xml:111: The following error occurred while executing this line:
C:\Users\user\Codebases\junit.xml:147: The following error occurred while executing this line:
C:\Users\user\Codebases\junit.xml:164: Unable to read execution data file C:\Users\user\CodeBases\ju
nit\coverage\jacoco.exec
But sometimes it works perfectly. I am not sure when it works and when it does not. Any help will be appreciated.
Thanks
I have found the issue that was causing the jacoco to fail. I had a few instrumented jars imported in the classpath which were not required for the junit execution or coverage. These instrumented jars were causing jacoco session to be incorrect as we also had jmockit integration as Mocking Framework. The below link was very helpful for finding out the issue
eclemma.org/jacoco/trunk/doc/classids.html
I'm using Cobertura 1.9.4 with NetBeans 7.2.1, Ant 1.8.3 and JDK 1.7.0_05 with JUnit 4.10.
My HTML coverage reports are being generated, but they all list 0% unit test coverage. This shouldn't be the case because I have unit test that cover my entire util class.
What's interesting is all my unit test start to fail if I include the instrumented classes in my test path by changing
run.test.classpath=\
${javac.test.classpath}:\
${build.test.classes.dir}
to
run.test.classpath=\
${build.instrumented.dir}:\
${javac.test.classpath}:\
${build.test.classes.dir}
Also, if I turn on the emma code coverage tool all my test cases will break as well.
NOTE: There is a nearly identical post Cobertura showing 0% coverage but since it is unanswered, slightly dated and my configuration differs I thought I'd capture my setup here. Here are two references that are similar to my config: 1)cobertura pdf 2)NetBeans CoberaturaAnt Wiki
Snipped from my build.xml
<!-- this strangely doesn't work so i'm temporarily using hardcoded path below
<taskdef classpath="cobertura.jar" resource="tasks.properties" />
-->
<path id="cobertura.class.path">
<pathelement location="libs/cobertura.jar" />
<pathelement location="libs/asm-3.0.jar" />
<pathelement location="libs/asm-tree-3.0.jar" />
<pathelement location="libs/log4j-1.2.9.jar" />
<pathelement location="libs/jakarta-oro-2.0.8.jar" />
</path>
<taskdef resource="tasks.properties" classpathref="cobertura.class.path" />
<target depends="-init-check,init" name="-init-check-cobertura">
<fail unless="build.classes.dir">Must set build.classes.dir</fail>
</target>
<target name="-post-compile" depends="-init-check-cobertura,init">
<delete dir="${build.instrumented.dir}" />
<delete dir="${build.test.coverage.dir}" />
<mkdir dir="${build.instrumented.dir}" />
<mkdir dir="${build.test.coverage.dir}" />
<cobertura-instrument todir="${build.instrumented.dir}"
datafile="${build.dir}/cobertura.ser">
<fileset dir="${build.classes.dir}">
<include name="**/*.class"/>
</fileset>
</cobertura-instrument>
</target>
<target name="coverage-report" depends="test">
<cobertura-report format="html" destdir="${build.test.coverage.dir}"
datafile="${build.dir}/cobertura.ser"/>
<delete file="${build.dir}/cobertura.ser" />
<delete dir="${build.instrumented.dir}" />
</target>
I've also defined these in my project.properties file
build.instrumented.dir=${build.dir}/instrumented
build.test.coverage.dir=${build.dir}/test/coverage
FYI I already tried adding these to my tag in build-impl.xml. still zero coverage.
<jvmarg value="-Dcobertura.use.java.nio=false"/>
<jvmarg value="-Dnet.sourceforge.cobertura.datafile=${basedir}/cobertura.ser"/>
The ant test target is from the netbeans generated build-impl.xml file
<target depends="init,compile-test,-pre-test-run,-do-test-run,test-report,-post-test-run,-test-browse"
description="Run unit tests." name="test"/>
I was trying to integrate code coverage on my project by using jacoco, ant and teamcity. However, I realized that when jacoco task is around the junit task, teamcity does not catch the failing tests and everything is a success even with test failed.
Here are my 2 test tasks to test with and without jacoco and see teamcity bahaviours.
1- with jacoco activated
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
</jacoco:coverage>
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
2- without jacoco
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<!--<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">-->
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
<!--</jacoco:coverage>-->
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
Only jacoco task has been commented between the 2 releases of test.
Teamcity output
[CommonBuildTasks.-test] echo
[08:26:21]: [echo] JaCoCo activated
[08:26:21]: [CommonBuildTasks.-test] jacoco:coverage (4s)
[08:26:21]: [jacoco:coverage] Enhancing junit with coverage.
[08:26:22]: [jacoco:coverage] Running ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest
[08:26:25]: [jacoco:coverage] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 3.511 sec
[08:26:26]: [jacoco:coverage] Test ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest FAILED
[08:26:26]: [CommonBuildTasks.-test] copy
[08:26:26]: [copy] Copying 1 file to C:\TeamCity\buildAgent\work\cc10e09e43249f57\reports
As you can see, a test failed but teamcity has reported a successfull build.
Any idea why I got this behaviour?
Thanks
The answer is hidden in your call to the JUnit-Task:
<junit haltonfailure="no">...</junit>
With this configuration, the JUnit task does not fail the build on failing tests. This should lead to the desired behaviour:
<junit haltonfailure="yes">...</junit>
See the Ant documentation for the configuration of the JUnit task.
I solved this issue by using agent task instead of the coverage task. So, instead of
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
Use:
<jacoco:agent property="agentvmparam" destfile="${bin}/jacoco.exec"/>
<junit fork="yes"...
<jvmarg value="${agentvmparam}"/>
</junit>
Agent task uses the same properties as the coverage task. Then you can start your junit task without wrapping it in coverage task. That way teamcity is able to intercept junit task output.
I have problem with ant sonar task. This task end with success but don't run unit tests and don't show code coverage.
Test task
<target name="test" depends=".....">
<path id="classpath">
<fileset dir="test/lib" includes="**/*.jar"/>
<fileset dir="lib" includes="**/*.jar"/>
<pathelement location="......."/>
</path>
<mkdir dir="build/tests"/>
<javac srcdir="test/src" destdir="build/tests" includes="**/*.java" debug="${debug}" deprecation="${deprecation}" optimize="${optimize}" nowarn="${nowarn}" fork="true">
<classpath refid="classpath"/>
</javac>
<copy todir="build/tests">
<fileset dir="test/src" excludes="**/*.java"/>
</copy>
<jacoco:coverage destfile="target/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
<junit printsummary="yes" fork="true" haltonfailure="false" showoutput="true" failureproperty="test.failed">
<formatter type="xml"/>
<classpath refid="classpath"/>
<classpath>
<pathelement location="build/tests"/>
</classpath>
<test name="com........MainTestSuite" todir="build"/>
</junit>
</jacoco:coverage>
<fail message="Test failure detected, check test results." if="test.failed"/>
</target>
and sonar task:
<target name="sonar" depends="build">
<property name="sonar.tests" value="test" />
<property name="sonar.libraries" value="" />
<property name="sonar.surefire.reportsPath" value="sonarWorkDir" />
<!-- The following properties are required to use JaCoCo: -->
<!-- 1. Tells Sonar to run the unit tests -->
<property name="sonar.dynamicAnalysis" value="true" />
<!-- 2. Tells Sonar which "tests" targets to run -->
<property name="sonar.jacoco.antTargets" value="test" />
<!-- 3. Tells Sonar to use JaCoCo as the code coverage engine -->
<property name="sonar.core.codeCoveragePlugin" value="jacoco" />
<!-- Execute Sonar -->
<sonar:sonar key="${JOB_NAME}" version="${VERSION}" xmlns:sonar="antlib:org.sonar.ant">
<sources>
<path location="...../src" />
</sources>
<binaries>
<path location="build/....." />
</binaries>
</sonar:sonar>
</target>
until sonar task runs i got a warning 10:00:20.290 WARN o.s.p.j.JaCoCoPlugin - Coverage information was not collected. Perhaps you forget to include debug information into compiled classes?
I would advise you to:
use the latest version of the Sonar Ant Task (from what I can see in your script, you're using the old syntax)
take a look at our sample Ant projects to find out where your issue is. Most probably, the following example is what you're looking for: https://github.com/SonarSource/sonar-examples/tree/master/projects/code-coverage/ut/ant/ut-ant-jacoco-runTests
Kind of a late response, but I just recently ran into this issue and couldn't find a simple solution on the internet. Instead, to solve it I simply took the advice of the stack trace. Coverage information was not collected. Perhaps you forget to include debug information into compiled classes?
And then went back to my ant file and made sure the source files (not test) were being compiled in debug mode. Apparently the jacoco plugin needs that extra info like line numbers in order to calculate the code coverage. Once you change your ant file you should finally see a code coverage percentage in sonar.
<javac srcdir="${source.dir}" target="1.6" source="1.6" destdir="${build.classes.dir}" debug="on">
...
</javac>
Also, make sure you include the following sonar properties :
sonar.binaries=build/classes
sonar.tests=junit/tests
sonar.dynamicAnalysis=reuseReports
sonar.junit.reportsPath=build/test-reports
sonar.java.coveragePlugin=jacoco
sonar.jacoco.reportPath=build/test-reports/jacoco.exec
So you probably want to remove sonar.antTargets and change sonar.dynamicAnalysis's value to reuseReports
I tried to get Cobertura running inside my ant script. All is successfull (source code building, junit tests, cobertura reports (xml / html); but in html reports, the code coverage is always at 0% ...
Ant Script : make-instrument
<!-- Make instrument for Cobertura engine -->
<target name="make-instrument">
<!-- Remove the coverage data file and any old instrumentation. -->
<delete file="${cobertura.ser}" />
<!-- Instrument the application classes, writing the instrumented classes into ${build.instrumented.dir}. -->
<cobertura-instrument todir="${report.cobertura.dir}">
<!-- The following line causes instrument to ignore any source line containing a reference to log4j,
for the purposes of coverage reporting. -->
<ignore regex="org.apache.log4j.*" />
<fileset dir="${webcontent.dir}/WEB-INF/classes">
<!-- Instrument all the application classes, but don't instrument the test classes. -->
<include name="**/*.class" />
<exclude name="**/*Test.class" />
</fileset>
</cobertura-instrument>
</target>
Ant Script : make-instrument
<target name="install-cobertura" if="is-hudson-env">
<path id="cobertura.classpath">
<fileset dir="${user.home.sharehunter.dir}/cobertura-${cobertura.rev}">
<include name="**/cobertura.jar" />
<include name="**/*.jar" />
</fileset>
</path>
<taskdef resource="tasks.properties" classpathref="cobertura.classpath" />
</target>
Ant Script : junit
<target name="run-tests" depends="make-instrument">
<path id="classpath.test">
<path path="${webcontent.dir}/WEB-INF/classes" />
<pathelement location="${webcontent.dir}/WEB-INF/classes" />
<fileset dir="${lib.dir}" includes="**/*.jar" />
<path location="${webcontent.dir}/WEB-INF/classes" />
<path location="${webcontent.dir}/WEB-INF" />
<path location="${webcontent.dir}" />
</path>
<junit fork="yes" failureProperty="test.failed">
<classpath refid="classpath.test" />
<classpath location="${user.home.dir}/junit-${junit.rev}.jar" />
<!-- Specify the name of the coverage data file to use.
The value specified below is the default. -->
<sysproperty key="net.sourceforge.cobertura.datafile" file="${cobertura.ser}" />
<!-- Note the classpath order: instrumented classes are before the original (uninstrumented) classes. -->
<classpath location="${report.cobertura.dir}" />
<!--
The instrumented classes reference classes used by the
Cobertura runtime, so Cobertura and its dependencies
must be on your classpath.
-->
<classpath refid="cobertura.classpath" />
<!-- Generate xml files for each junit tests runs -->
<formatter type="xml" />
<batchtest todir="${report.junit.dir}">
<fileset dir="${webcontent.dir}/WEB-INF/classes">
<include name="**/*Test.class" />
</fileset>
</batchtest>
</junit>
<!-- Generate Cobertura xml file containing the coverage data -->
<cobertura-report format="xml" srcdir="${src.main.java.dir}" destdir="${report.cobertura.dir}" datafile="${cobertura.ser}" />
<!-- Generate Cobertura html file report containing the coverage data -->
<cobertura-report format="html" srcdir="${src.main.java.dir}" destdir="${report.cobertura.dir}" datafile="${cobertura.ser}" />
</target>
This is what Cobertura FAQ Says
When I generate coverage reports, why do they always show 0% coverage everywhere?
Cobertura is probably using the wrong .ser file when generating the reports. When you instrument your classes, Cobertura generates a .ser file containing basic information about each class. As your tests run, Cobertura adds additional information to this same data file. If the instrumented classes can not find the data file when running then they will create a new one. It is important that you use the same cobertura.ser file when instrumenting, running, and generating reports.
The best way to do this is to specify the location of the data file when running your tests. You should pass the -Dnet.sourceforge.cobertura.datafile=${basedir}/cobertura.ser sysproperty to the JUnit task.
Another common problem is that the cobertura.ser file is deleted, but the previously instrumented classes are not also deleted. Any time you delete your coverage data file you should also deleted all instrumented classes.
Ok I found the problem. To be sure have this :
<!--
Note the classpath order: instrumented classes are before the
original (uninstrumented) classes. This is important.
-->
<classpath location="${instrumented.dir}" />
<classpath location="${classes.dir}" />
Instrumented classes must are before the original (uninstrumented) classes.
I tried similar way. I also used instrumented code before actual source code, but I am getting 0 % in the report file.
<macrodef name="coberturaTestMacro">
<attribute name="moduleName" />
<attribute name="classpath.module" />
<attribute name="classpath.junit" />
<attribute name="failOnCoverageFall" />
<attribute name="fileCoberturaData"/>
<sequential>
<path id="classpathCobertura">
<fileset dir="${homeCobertura}">
<include name="cobertura.jar" />
<include name="lib/**/*.jar" />
</fileset>
</path>
<taskdef classpathref="classpathCobertura" resource="tasks.properties" />
<property name="cob.instrumented.dir" value="target/cobertura/instrumented" />
<delete dir="target/cobertura" />
<cobertura-instrument todir="${cob.instrumented.dir}" datafile="#{fileCoberturaData}" >
<fileset dir="target/classes">
<include name="**/*.class" />
</fileset>
</cobertura-instrument>
<delete dir="target/reports/test" />
<mkdir dir="target/cobertura/reports" />
<junit printsummary="false" failureproperty="junit.failure"
maxmemory="512m" fork="true" forkmode="perTest">
<jvmarg value="-Djava.awt.headless=true" />
<classpath location="${homeCobertura}/cobertura.jar" />
<classpath location="${cob.instrumented.dir}" />
<classpath>
<path refid="#{classpath.module}" />
<path refid="#{classpath.junit}" />
</classpath>
<classpath path="target/test-classes" />
<batchtest todir="target/cobertura/reports/">
<fileset dir="src/test/java">
<include name="**/*Test.java" />
</fileset>
</batchtest>
</junit>
<cobertura-report srcdir="src/main/java" destdir="target/cobertura/reports/" />
<echo message="${line.separator}" />
<echo message="COVERAGE: #{moduleName} module..." />
<echo message="${line.separator}" />
<if>
<available file="target/cobertura/#{moduleName}-cobertura.properties" />
<then>
<var name="total.line-rate" file="target/cobertura/#{moduleName}-cobertura.properties" />
<cobertura-check haltonfailure="#{failOnCoverageFall}"
datafile="#{fileCoberturaData}" totallinerate="${total.line-rate}" />
</then>
</if>
<delete file="${dirBuild}/coverage-summary.properties" />
<cobertura-report datafile="#{fileCoberturaData}" destdir="target/cobertura/" format="summaryXml" />
<var name="total.line-rate" file="target/cobertura/coverage-summary.properties" />
<echo message="Total line coverage: ${total.line-rate}%" />
<propertyfile file="target/cobertura//#{moduleName}-cobertura.properties">
<entry key="total.line-rate" value="${total.line-rate}" type="int" />
</propertyfile>
</sequential>
</macrodef>
The surprising thing is that the generated report says total 2% coverage, but summary file says 0% coverage.
Where old cobertura task shows 8% coverage. I am totally confused :(
Probably it is not applicable to everybody but I had the similar issue where coverage was 0 for all classes.
There were 2 issues in my case
1) it was reading wrong jdk version 1.8 off of PATH. I updated PATH to read 1.6 jdk.
2) it was initially using version 1.8 of cobertura. I ran the build and it would generate coverage report but all the classes were 0% always. I updated the javac target to include
debug="true" debuglevel="vars,lines,source" reference: cobertura 0 coverage
Then ran the build again and saw that the there was an error when running tests and traced that back to an issue with version 1.8 of cobertura.
So, I upgraded
Cobertura 1.9.4
asm 3.1 from 2.2.1
asm-tree 3.1
other dependencies
1. jakarta-oro 2.0.8
2. log4j-1.2.9
After that ran the task again and the reports were alright.