I'm running TestNG from Ant. I'm using my own test listeners. I'm refactoring the code and once a while I got
[testng] Total tests run: 7, Failures: 0, Skips: 7
[testng] Configuration Failures: 1, Skips: 2
What will be the best approach to fix configuration failures ?
The HTML reports will tell you which configuration methods failed.
Related
We have in our System SmokeTests on Jenkins. In the log, I see:
[ERROR] Tests run: 170, Failures: 1, Errors: 0, Skipped: 17
Jenkins give it out as successful. What can be the problem?
I have found the oroblem. The instance of the jenkins had not enogh memory to catch all. So i reduced the debugmode to write less informations. Know it run without a problem
For a continuous integration build, we use JcCoCo to minimise the number of tests to be run. However, on some commits it determines that there are no tests worth running at all. For example, if only an image was changed.
Here is a snippet from the build.xml:
<fileset id="int.tests" dir="${build.inttest.source}/java">
<include name="**/*Test.java"/>
</fileset>
<taskdef name="testng" classname="org.testng.TestNGAntTask"
classpath="${build.jars.test}/testng.jar"/>
<jacoco:coverage destfile="./inttest-jacoco.exec">
<testng outputDir="./reports/intTest" failureproperty="testNGFailed" haltonfailure="false"
verbose="2" workingDir="${build.dir}" classfilesetref="int.tests">
<classpath>
<path refid="build.inttest.classpath"/>
</classpath>
</testng>
</jacoco:coverage>
When no tests are run, property testNGFailed is set to true and the build subsequently fails.
The logging in this scenario looks like this:
13:16:49,116 INFO - [testng] ===============================================
13:16:49,116 INFO - [testng] Ant suite
13:16:49,116 INFO - [testng] Total tests run: 0, Failures: 0, Skips: 0
13:16:49,116 INFO - [testng] ===============================================
13:16:49,116 INFO -
13:16:49,191 WARN - [testng] [TestNG] No tests found. Nothing was run
How can I make the build pass when there are no test to be run, but fail when any test fails?
Can I get Jacoco to always run at least one test?
Can I get TestNG to only set the failureproperty when a test has failed?
First of all - JaCoCo does not execute your tests! Tests are executed by TestNG or JUnit or whatever testing framework - JaCoCo just collects coverage information from this execution.
According to http://testng.org/doc/ant.html :
classfilesetref - A reference to a ResourceCollection containing the test classes to be run
whereas in your case it points on directory containing .java files, so seems logical that TestNG can't find tests.
I followed contribution-guide for Apache Beam to setup my development environment. The project compiles fine, however I always get following failure when I run mvn verify:
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time
elapsed: 60.315 s <<< FAILURE! - in
org.apache.beam.runners.apex.examples.WordCountTest [ERROR]
testWordCountExample(org.apache.beam.runners.apex.examples.WordCountTest)
Time elapsed: 32.259 s <<< FAILURE! java.lang.AssertionError: result
files exist at org.junit.Assert.fail(Assert.java:88)
When I run test org.apache.beam.runners.apex.examples.WordCountTestvia IntelliJ, all assertions passes. I have a feeling there is a race condition in the assertion that is failing. Am I missing something to setup the environment?
I have a problem with TestNG. When I run tests in Eclipse everything it's OK, but when I run test in ANT i get errors:
TEST-RUN:
[testng] [TestNG] [Error]
[testng] Error creating object factory: class SomeClassTest
[testng] The tests failed.
In test I use:
#ObjectFactory
public IObjectFactory getObjectFactory() {
return new PowerMockObjectFactory();
}
You might have to use the 'objectFactory' attribute in the testng ant task, something like objectFactory="org.powermock.modules.testng.PowerMockObjectFactory". See https://groups.google.com/forum/#!topic/testng-dev/h68SIRxbeNI for more information. Hope that helps?
We're currently using JUnit 4.4 and Ant 1.7.1 for our builds. I have some test cases that are #Ignored. When I run them in Eclipse, the JUnit test runner reports them as ignored. I would like to see them listed in the XML output from Ant (so I can report on them), but they do not seem to be there.
Does anyone have this working? Is there a switch to turn them on? An upgrade I need to do?
It looks like this is a known Ant issue/bug.
This thread talks about the same issue, but it provides some additional information: you can get data on ignored tests when running the tests using maven surefire, and hudson is able to display that data.
http://jenkins.361315.n4.nabble.com/Is-it-possible-to-show-Ignore-JUnit-tests-td1565288.html
A fix for this issue has now been applied to the head of Ant core, scheduled for release as part of the upcoming version of Ant 1.9.0.
It should be possible to try this fix locally be replacing ant-junit.jar in your Ant distribution's lib directory with the version from the nightly builds, or by running the full nightly Ant distribution, or by building the Ant sources directly. Since the Ant team are currently voting on preparing a new release it may just be worth waiting for 1.9.0 to be officially packaged and pushed out for download.
Just tried Ant 1.9.0 with JUnit 4.11. If you use <junit printsummary="on"> you'll get output like:
[junit] Running com.example.IgnoredTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.01
[junit] Running com.example.PassingTest
[junit] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.01
[junit] Running com.example.FailingTest
[junit] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.01
I think it'd be preferrable if we could get output like this with printsummary=off:
[junit] Test com.example.IgnoredTest SKIPPED
[junit] Test com.example.FailingTest FAILED
but it seems the more verbose output above is the best we can do, unless I'm missing some obscure trick with one of the junit task arguments.