PITest skips tests in ant - ant

Question

I have the problem, that PITest skips some of my tests. It reports no line coverage for the tested class and all mutations in stay alive:
replaced return value with Collections.emptyList for ... → NO_COVERAGE
The list below is empty:
Tests examined
I did check the following:
the class file of class and test is supported in one of the directories in the classpath
the test is not excluded
tests in the same directory are run
source directory is provided correctly
no errors occur on the console, even when logging with verbose=true
I logged and checked all parameters
<pitest
pitClasspath="pit.path"
classPath="mutation.path"
targetClasses="my.package1.*,my.package2.*,my.package3.*"
targetTests="my.package1.Class1Test,my.package2.sub.Class2Test,my.package3.sub.Class3Test"
reportDir="pitest"
sourceDir="src,testsrc"
timestampedReports="false"
outputFormats="HTML,XML"
excludedClasses="my.package1.*Test,my.package2.*Test,my.package3.*Test"
verbose="true"
/>
I am using PITest version 1.4.11, but I tested as well with 1.6.2 and 1.4.3. Also I'm using Java 1.8.0_211 and ant 1.9.13.
I set up a test project at:
https://github.com/johannesn/pittestskippingtests
This is the log output for this sample project:
Buildfile: .../pitestskipingtests/build.xml
mutationCoverage:
[delete] Deleting directory .../pitestskipingtests/classes
[delete] Deleting directory .../pitestskipingtests/testclasses
[mkdir] Created dir: .../pitestskipingtests/classes
[mkdir] Created dir: .../pitestskipingtests/testclasses
[javac] .../pitestskipingtests/build.xml:30: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 3 source files to .../pitestskipingtests/classes
[javac] .../pitestskipingtests/build.xml:31: warning: 'includeantruntime' was not set, defaulting to build.sysclasspath=last; set to false for repeatable builds
[javac] Compiling 3 source files to .../pitestskipingtests/testclasses
[pitest] 19:24:08 PIT >> INFO : ---------------------------------------------------------------------------
[pitest] 19:24:08 PIT >> INFO : Enabled (+) and disabled (-) features.
[pitest] 19:24:08 PIT >> INFO : -----------------------------------------
[pitest] 19:24:08 PIT >> INFO : +FFBLOCK Filters mutations in code duplicated by finally block inlining
[pitest] 19:24:08 PIT >> INFO : +FSTATI Filters mutations in static initializers and code called only from them
[pitest] 19:24:08 PIT >> INFO : +FSEQUIVEQUALS Filters equivalent mutations that affect only performance in short cutting equals methods
[pitest] 19:24:08 PIT >> INFO : +FFEACH Filters mutations in compiler generated code that implements for each loops
[pitest] 19:24:08 PIT >> INFO : +FINFINC Filters mutations to increments that may cause infinite loops
[pitest] 19:24:08 PIT >> INFO : +FFLOOP Filters any mutations to increments in for loops as they may cause timeouts
[pitest] 19:24:08 PIT >> INFO : +FRETEQUIV Filters return vals mutants with bytecode equivalent to the unmutated class
[pitest] 19:24:08 PIT >> INFO : +FINULL Filters mutations in compiler generated code that checks for null by calling getClass
[pitest] 19:24:08 PIT >> INFO : +FTRYWR Filters mutations in code generated for try with resources statements
[pitest] 19:24:08 PIT >> INFO : +FKOTLIN Filters out junk mutations in bytecode created by compiler for kotlin language features
[pitest] 19:24:08 PIT >> INFO : +FSTATINIT Filters mutations in static initializers and code called only from them
[pitest] 19:24:08 PIT >> INFO : +FLOGCALL Filters mutations in code that makes calls to logging frameworks
[pitest] 19:24:08 PIT >> INFO : +FINFIT Filters mutations that may cause infinite loops by removing calls to iterator.next
[pitest] 19:24:08 PIT >> INFO : +FANN Filters mutations in classes and methods with matching annotations of class or runtime retention
[pitest] 19:24:08 PIT >> INFO : [annotation] Annotation to avoid (full package name not required)
[pitest] 19:24:08 PIT >> INFO : -CLASSLIMIT Limits the maximum number of mutations per class
[pitest] 19:24:08 PIT >> INFO : [limit] Integer value for maximum mutations to create per class
[pitest] 19:24:08 PIT >> INFO : -EXPORT Exports mutants bytecode and other details to disk
[pitest] 19:24:08 PIT >> INFO : ---------------------------------------------------------------------------
[pitest] 19:24:08 PIT >> FINE : Running report with ReportOptions [targetClasses=[my.package1.*, my.package2.*, my.package3.*], excludedMethods=[], excludedClasses=[my.package1.*Test, my.package2.*Test, my.package3.*Test], excludedTestClasses=[], codePaths=[], reportDir=pitest, historyInputLocation=null, historyOutputLocation=null, sourceDirs=[src, testsrc], classPathElements=[.../pitestskipingtests/resources/pitest/pitest-1.4.3.jar, .../pitestskipingtests/testclasses, .../pitestskipingtests/classes, .../pitestskipingtests/lib/junit-4.12.jar], mutators=[], features=[], dependencyAnalysisMaxDistance=-1, jvmArgs=[], numberOfThreads=1, timeoutFactor=1.25, timeoutConstant=4000, targetTests=[^my\.package1\.Class1Test$, ^my\.package2\.sub\.Class2Test$, ^my\.package3\.sub\.Class3Test$], loggingClasses=[], maxMutationsPerClass=0, verbose=true, failWhenNoMutations=true, outputs=[HTML, XML], groupConfig=TestGroupConfig [excludedGroups=[], includedGroups=[]], fullMutationMatrix=false, mutationUnitSize=0, shouldCreateTimestampedReports=false, detectInlinedCode=false, exportLineCoverage=false, mutationThreshold=0, coverageThreshold=0, mutationEngine=gregor, javaExecutable=null, includeLaunchClasspath=false, properties={}, maxSurvivors=-1, excludedRunners=[], includedTestMethods=[], testPlugin=junit, useClasspathJar=false]
[pitest] 19:24:08 PIT >> FINE : System class path is .../pitestskipingtests/resources/pitest/pitest-1.4.3.jar:.../pitestskipingtests/resources/pitest/pitest-ant-1.4.3.jar:.../pitestskipingtests/resources/pitest/pitest-entry-1.4.3.jar:.../pitestskipingtests/resources/pitest/xmlpull-1.1.3.1.jar:.../pitestskipingtests/resources/pitest/xstream-1.4.11.1.jar:.../pitestskipingtests/lib/junit-4.12.jar
[pitest] 19:24:08 PIT >> FINE : Maximum available memory is 3641 mb
[pitest] 19:24:08 PIT >> FINE : MINION : Installing PIT agent
[pitest]
[pitest] 19:24:09 PIT >> INFO : Sending 3 test classes to minion
[pitest] 19:24:09 PIT >> INFO : Sent tests to minion
[pitest] 19:24:09 PIT >> FINE : Coverage generator Minion exited ok
[pitest] 19:24:09 PIT >> INFO : Calculated coverage in 0 seconds.
[pitest] 19:24:09 PIT >> FINE : Used memory after coverage calculation 14 mb
[pitest] 19:24:09 PIT >> FINE : Free Memory after coverage calculation 231 mb
[pitest] 19:24:09 PIT >> FINE : According to coverage no tests hit the mutation MutationDetails [id=MutationIdentifier [location=Location [clazz=my.package1.Class1, method=method1, methodDesc=()Ljava/lang/String;], indexes=[4], mutator=org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator], filename=Class1.java, block=1, lineNumber=5, description=mutated return of Object value for my/package1/Class1::method1 to ( if (x != null) null else throw new RuntimeException ), testsInOrder=[], isInFinallyBlock=false, poison=NORMAL]
[pitest] 19:24:09 PIT >> FINE : According to coverage no tests hit the mutation MutationDetails [id=MutationIdentifier [location=Location [clazz=my.package2.sub.Class2, method=method1, methodDesc=()Ljava/lang/String;], indexes=[4], mutator=org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator], filename=Class2.java, block=1, lineNumber=5, description=mutated return of Object value for my/package2/sub/Class2::method1 to ( if (x != null) null else throw new RuntimeException ), testsInOrder=[], isInFinallyBlock=false, poison=NORMAL]
[pitest] 19:24:09 PIT >> FINE : According to coverage no tests hit the mutation MutationDetails [id=MutationIdentifier [location=Location [clazz=my.package3.sub.Class3, method=method1, methodDesc=()Ljava/lang/String;], indexes=[4], mutator=org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator], filename=Class3.java, block=1, lineNumber=6, description=mutated return of Object value for my/package3/sub/Class3::method1 to ( if (x != null) null else throw new RuntimeException ), testsInOrder=[], isInFinallyBlock=false, poison=NORMAL]
[pitest] 19:24:09 PIT >> INFO : Created 3 mutation test units
[pitest] 19:24:09 PIT >> FINE : Used memory before analysis start 23 mb
[pitest] 19:24:09 PIT >> FINE : Free Memory before analysis start 222 mb
[pitest] 19:24:09 PIT >> FINE : Running 3 units
[pitest] ================================================================================
[pitest] - Mutators
[pitest] ================================================================================
[pitest] > org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
[pitest] >> Generated 3 Killed 0 (0%)
[pitest] > KILLED 0 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
[pitest] > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
[pitest] > NO_COVERAGE 3
[pitest] --------------------------------------------------------------------------------
[pitest] ================================================================================
[pitest] - Timings
[pitest] ================================================================================
[pitest] > scan classpath : < 1 second
[pitest] > coverage and dependency analysis : < 1 second
[pitest] > build mutation tests : < 1 second
[pitest] > run mutation analysis : < 1 second
[pitest] --------------------------------------------------------------------------------
[pitest] > Total : < 1 second
[pitest] --------------------------------------------------------------------------------
[pitest] ================================================================================
[pitest] - Statistics
[pitest] ================================================================================
[pitest] >> Generated 3 mutations Killed 0 (0%)
[pitest] >> Ran 0 tests (0 tests per mutation)
[pitest] 19:24:09 PIT >> INFO : Completed in 0 seconds
BUILD SUCCESSFUL
Total time: 1 second

Answer 1

After a hell of a lot of debugging and trying things out, I found out that test methods were not found because the junit dependency hamcrest-core was missing.
For everyone else experiencing a problem like this, try debugging the following method:
org.pitest.junit.RunnerSuiteFinder#apply
Depending on your test setup (if you use junit e.g.) it might be a different subclass of
org.pitest.testapi.TestSuiteFinder
I experienced the following exception:
java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.junit.internal.builders.JUnit4Builder.runnerForClass(JUnit4Builder.java:10)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:37)
at org.pitest.junit.adapter.AdaptedJUnitTestUnit.createRunner(AdaptedJUnitTestUnit.java:107)
at org.pitest.junit.RunnerSuiteFinder.apply(RunnerSuiteFinder.java:40)
at org.pitest.junit.RunnerSuiteFinder.apply(RunnerSuiteFinder.java:35)
at org.pitest.extension.common.CompoundTestSuiteFinder.apply(CompoundTestSuiteFinder.java:20)
at org.pitest.extension.common.CompoundTestSuiteFinder.apply(CompoundTestSuiteFinder.java:9)
at org.pitest.testapi.execute.FindTestUnits.findTestUnits(FindTestUnits.java:47)
at org.pitest.testapi.execute.FindTestUnits.getTestUnits(FindTestUnits.java:40)
at org.pitest.testapi.execute.FindTestUnits.findTestUnitsForAllSuppliedClasses(FindTestUnits.java:29)
at org.pitest.coverage.execute.CoverageMinion.discoverTests(CoverageMinion.java:156)
at org.pitest.coverage.execute.CoverageMinion.getTestsFromParent(CoverageMinion.java:137)
at org.pitest.coverage.execute.CoverageMinion.main(CoverageMinion.java:83)
Caused by: java.lang.ClassNotFoundException: org.hamcrest.SelfDescribing
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 26 more
The exception is part of the runner variable. Print it using:
((ErrorReportingRunner) runner).causes.get(0).printStackTrace()