I have made an Ant target that runs my JUnit 4 tests. Unfortunately all of them are executed twice!
Does anyone have an idea of what I have done wrong?
Here are my ant target:
<target name="junit" description="Execute unit tests" depends="compile">
<delete dir="rawtestoutput"/>
<delete dir="test-reports"/>
<mkdir dir="rawtestoutput"/>
<junit printsummary="on" failureproperty="junit.failure" fork="true">
<classpath refid="class.path.junit"/>
<formatter type="xml" usefile="true" />
<batchtest todir="rawtestoutput">
<fileset dir="src/test">
<include name="**/*.java"/>
<!-- Add util and testhelper classes here (to avoid "No tests in class" error) and add suite classes to avoid test being run twice -->
<exclude name="**/SessionHelper.java"/>
<exclude name="**/TestHelper.java"/>
<exclude name="**/AllTests.java"/>
<exclude name="**/AllEDITests.java"/>
</fileset>
</batchtest>
</junit>
<junitreport>
<fileset dir="rawtestoutput"/>
<report todir="test-reports"/>
</junitreport>
<fail if="junit.failure" message="Unit test(s) failed. See reports!"/>
</target>
My first idea was that it because of test suites. But I do not think that anymore. I have excluded the tests suites and furthermore, it not only the tests that are part of suites that are run twice. Its all my tests.
Below is a small sample of the test output of one of my testsclasses:
[20:24:53]: [junit] Running dk.gensam.gaia.business.ydelse.YdelsestypeBOTest
[20:24:53]: [junit] dk.gensam.gaia.business.ydelse.YdelsestypeBOTest (2s)
[20:24:54]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsevariationer
[20:24:55]: [loadYdelsevariationer] [Test Output] EMMA: collecting runtime coverage data ...
[20:24:55]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsestypeIndex_alleExisterendeErAnnullerede
[20:24:56]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsestypeIndex_ingenEksisterendeValgteRelationer
[20:24:56]: [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 3,077 sec
[20:24:56]: dk.gensam.gaia.business.ydelse.YdelsestypeBOTest (3s)
[20:24:56]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsevariationer
[20:24:56]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsestypeIndex_alleExisterendeErAnnullerede
[20:24:56]: [dk.gensam.gaia.business.ydelse.YdelsestypeBOTest] loadYdelsestypeIndex_ingenEksisterendeValgteRelationer
As you can see the tests in YdelsestypeBOTest are run twice...
From the line:
[20:24:55]: [loadYdelsevariationer] [Test Output] EMMA: collecting runtime coverage data
It looks like a different Ant target is invoking the code coverage tool Emma which is then re-running your tests. If you run your Ant script with this target i.e. ant junit, does it always do this?
It's hard to tell what exactly is happening here but try deleting all your test suites temporarily and recompile just to make sure they are not causing the issue. It looks like you may want to get rid of the test suites anyway if you are starting to move from test suites to using batchtest.
Related
I have a test suite which re-runs all the failed tests at the end of the suite execution. But even if the test passes in the second run, junit report displays the output of the first run, ie. it shows that the test failed.
Following is an extract from build.xml:
<junitreport tofile="./report/html/result.xml">
<fileset dir="./report">
<include name="result.xml"/>
</fileset>
<report format="noframes" styledir="./etc_time" todir="./report/html/">
</report>
</junitreport>
<copy file="report/html/junit-noframes.html" tofile="report/html/index.html" />
<fail message="Tests failed" if="test.failed"/>
Can someone please point to some tutorials on how to do it?
Thanks!
I was trying to integrate code coverage on my project by using jacoco, ant and teamcity. However, I realized that when jacoco task is around the junit task, teamcity does not catch the failing tests and everything is a success even with test failed.
Here are my 2 test tasks to test with and without jacoco and see teamcity bahaviours.
1- with jacoco activated
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
</jacoco:coverage>
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
2- without jacoco
<target name="-test">
<echo message="JaCoCo activated"/>
<!-- Import the JaCoCo Ant Task -->
<taskdef uri="antlib:org.jacoco.ant" resource="org/jacoco/ant/antlib.xml"/>
<!-- Run your unit tests, adding the JaCoCo agent -->
<!--<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">-->
<junit fork="yes" printsummary="yes" haltonfailure="no" showoutput="false" failureProperty="test.failed" errorProperty="test.failed">
<classpath>
<path location="${lib}/${projectName}.jar"/>
<path refid="project.classpath"/>
</classpath>
<formatter type="xml"/>
<batchtest todir="${reportingHome}">
<fileset dir="${test}">
<include name="**/*Test.java"/>
</fileset>
</batchtest>
</junit>
<!--</jacoco:coverage>-->
<copy todir="${completeReportDir}" overwrite="true">
<fileset dir="${reportingHome}">
<include name="*.xml"/>
</fileset>
</copy>
</target>
Only jacoco task has been commented between the 2 releases of test.
Teamcity output
[CommonBuildTasks.-test] echo
[08:26:21]: [echo] JaCoCo activated
[08:26:21]: [CommonBuildTasks.-test] jacoco:coverage (4s)
[08:26:21]: [jacoco:coverage] Enhancing junit with coverage.
[08:26:22]: [jacoco:coverage] Running ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest
[08:26:25]: [jacoco:coverage] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 3.511 sec
[08:26:26]: [jacoco:coverage] Test ca.thalesgroup.socialnetworkanalysisorchestrator.impl.client.SocialNetworkAnalysisOrchestratorServiceProviderTest FAILED
[08:26:26]: [CommonBuildTasks.-test] copy
[08:26:26]: [copy] Copying 1 file to C:\TeamCity\buildAgent\work\cc10e09e43249f57\reports
As you can see, a test failed but teamcity has reported a successfull build.
Any idea why I got this behaviour?
Thanks
The answer is hidden in your call to the JUnit-Task:
<junit haltonfailure="no">...</junit>
With this configuration, the JUnit task does not fail the build on failing tests. This should lead to the desired behaviour:
<junit haltonfailure="yes">...</junit>
See the Ant documentation for the configuration of the JUnit task.
I solved this issue by using agent task instead of the coverage task. So, instead of
<jacoco:coverage destfile="${bin}/jacoco.exec" xmlns:jacoco="antlib:org.jacoco.ant">
Use:
<jacoco:agent property="agentvmparam" destfile="${bin}/jacoco.exec"/>
<junit fork="yes"...
<jvmarg value="${agentvmparam}"/>
</junit>
Agent task uses the same properties as the coverage task. Then you can start your junit task without wrapping it in coverage task. That way teamcity is able to intercept junit task output.
I have a zend framework 2 project and i am trying to set up my jenkins so that unit tests can be executed. Jenkins is running on ubuntu and i am developing under Windows 7 with PHPStorm.
build.xml
<target name="phpunit" description="Run unit tests with PHPUnit">
<exec executable="phpunit" failonerror="true">
<arg value="${basedir}/module/Addressbook/test"/>
</exec>
</target>
Folder structure:
project
module
Addressbook
test
AddressbookTest
Controller
AddressbookControllerTest.php
Boostrap.php
phpunit.xml.dist
TestConfig.php
build.xml
Jenkins Output:
phpunit:
[exec] PHPUnit 3.7.13 by Sebastian Bergmann.
[exec]
[exec] PHP Fatal error: Class 'AddressbookTest\Bootstrap' not found in /var/lib/jenkins/workspace/Test/src/module/Addressbook/test/AddressbookTest/Controller/AddressbookControllerTest.php on line 28
PHPStorm on my local machine does this when running phpunit.xml.dist
D:\Zend\ZendServer\bin\php.exe -d auto_prepend_file=D:/Zend/Apache2/htdocs/demoshop/vendor/autoload.php C:\Users\ChristianB\AppData\Local\Temp\ide-phpunit.php --configuration D:/Zend/Apache2/htdocs/demoshop/module/Addressbook/test/phpunit.xml.dist
How can i use that for jenkins?
It looks like your include path isn't setup correctly, I wouldn't use exec directly with PHPUNIT when there's better options.
You should look into using PHING tasks with Jenkins, they work excellent together.
You then setup Jenking to trigger your PHING target to run the unit tests for you via the PHPUNIT task, an example phing target for PHPUNIT:
<target name="phpunit">
<phpunit bootstrap="${srcdir}/tests/bootstrap.php">
<formatter todir="${builddir}/reports" type="xml"/>
<batchtest>
<fileset dir="${srcdir}/tests">
<include name="**/*Test*.php"/>
<exclude name="**/Abstract*.php"/>
<exclude name="${srcdir}/vendor/**"/>
</fileset>
</batchtest>
</phpunit>
<!--
Generate a report from the XML data created..
note: error when using format="frames"
-->
<phpunitreport infile="${builddir}/reports/testsuites.xml"
format="noframes"
todir="${builddir}/reports/tests"
/>
</target>
I am using sonar to measure code quality. One thing that I do not know is the steps to measure code coverage using Cobertura.
I followed the steps from http://cobertura.sourceforge.net/anttaskreference.html and was able to generate xml files. How do I get these xml files into SONAR?
Is there an easier way to use Cobertura in SONAR?
I am running the code coverage (Cobertura) in a different server than my SONAR server. Both servers are running under LINUX.
Thanks for the help!
You configure the Sonar task to upload unit test and cobertura reports generated by other parts of your build logic.
This is in contrast to Maven which has a standard build life-cycle that Sonar is able to leverage.
Unit test and code coverage
The following logic runs the unit tests with cobertura instrumented classes. An XML coverage report is generated by cobertura at the end:
<target name="instrument-classes" depends="compile-tests">
<taskdef resource="tasks.properties" classpathref="test.path"/>
<cobertura-instrument todir="${instrumented.classes.dir}" datafile="${build.dir}/cobertura.ser">
<fileset dir="${classes.dir}"/>
</cobertura-instrument>
</target>
<target name="junit" depends="instrument-classes">
<junit printsummary="yes" haltonfailure="yes">
<classpath>
<path refid="test.path"/>
<pathelement path="${instrumented.classes.dir}"/>
<pathelement path="${test.classes.dir}"/>
</classpath>
<formatter type="xml"/>
<batchtest fork="yes" todir="${test.reports.dir}">
<fileset dir="${test.src.dir}">
<include name="**/*Test*.java"/>
<exclude name="**/AllTests.java"/>
</fileset>
</batchtest>
</junit>
</target>
<target name="test" depends="junit">
<cobertura-report format="xml" datafile="${build.dir}/cobertura.ser" destdir="${cobertura.reports.dir}"/>
</target>
Invoking Sonar
I normally use a very simple Sonar target:
<target name="sonar" depends="test">
<taskdef uri="antlib:org.sonar.ant" resource="org/sonar/ant/antlib.xml" classpathref="sonar.path"/>
<sonar:sonar key="${sonar.project.key}" version="${sonar.project.version}" xmlns:sonar="antlib:org.sonar.ant"/>
</target>
And use a properties file to control all aspects of Sonar's behaviour:
sonar.project.key=org.demo:demo
sonar.project.version=1.0-SNAPSHOT
sonar.projectName=Demo project
sonar.host.url=http://myserver:9000
sonar.jdbc.url=jdbc:mysql://myserver:3306/sonar?useUnicode=true&characterEncoding=utf8
sonar.jdbc.driverClassName=com.mysql.jdbc.Driver
sonar.jdbc.username=sonar
sonar.jdbc.password=sonar
sonar.sources=${src.dir}
sonar.tests=${test.src.dir}
sonar.binaries=${classes.dir}
sonar.dynamicAnalysis=reuseReports
sonar.surefire.reportsPath=${test.reports.dir}
sonar.java.coveragePlugin=cobertura
sonar.cobertura.reportsPath=${cobertura.reports.dir}/coverage.xml
Demonstrates how Sonar can be configured to pick up the unit test reports created by junit and the code coverage report generated by cobertura.
The build does not have to run on the same server as Sonar. In that case one must provide the remote Sonar URL and JDBC credentials.
You would have to add these properties to Sonar's pom.xml:
<properties>
<sonar.dynamicAnalysis>false</sonar.dynamicAnalysis>
<sonar.phase>generate-sources</sonar.phase>
<sonar.surefire.reportsPath>target/reports/test/</sonar.surefire.reportsPath>
<sonar.cobertura.reportPath>../project/target/reports/coverage/coverage.xml</sonar.cobertura.reportPath>
</properties>
(with paths appropriate to your environment)
And run:
mvn sonar:sonar
Check the user list for more details.
if you're using Maven, then you do not have anything special to specify in your POM file. Just run "mvn clean sonar:sonar" and Sonar will automatically compile your code, run your tests with Cobertura (which is the default coverage engine in Sonar) and push all the results in the DB.
Same if you're using Ant [1] or the simple java runner [2] instead of Maven.
I do insist on the fact that you do not have to manually run Cobertura (with an Ant task for instance) previously to running Sonar.
[1] http://docs.codehaus.org/display/SONAR/Analyzing+with+Sonar+Ant+Task
[2] http://docs.codehaus.org/display/SONAR/Analyse+with+a+simple+Java+Runner
Fabrice,
SonarSource
I've seen a couple of posts about this but none provide the solution I'm looking for. I have ant set up so when it runs our tests we get the following output (which I like):
[junit] Running net.windward.document.test.TestTab
[junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.59 sec
[junit] Running net.windward.document.test.TestTableAttributes
[junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 0.604 sec
... continues for 300+ tests ...
But then at the end I just get "some tests failed". Is there some way to have it then list out the filenames of the tests that have failed?
thanks - dave
build.xml:
<junit printsummary="yes" errorProperty="test.failed" failureProperty="test.failed" >
<formatter type="plain" />
<classpath path="${classpath.run.test}"/>
<batchtest fork="yes" todir="${reports}">
<fileset dir="${src}">
<include name="**/test/**/Test*.java" />
<exclude name="**/_*/**/*.java" />
</fileset>
</batchtest>
</junit>
<fail if="test.failed" message="one or more unit tests failed"/>
<echo>unit tests all passed</echo>
<antcall target="testinternal"/>
</target>
Read chapter 4 of "Java Development with Ant": http://www.manning-source.com/books/hatcher/hatcher_ch04.pdf
You can use
<formatter type="xml" />
to generate an xml output of the tests. This can be converted to html to view it in a browser after all tests have run using the ant task junitreport. You can also import the xml file into eclipse if you use it.