My project use Jenkins to report the conditionals of an assert statement.
I have XCTest that expects and AssertionError to be thrown, and it passes correctly. The problem is that Jenkins reports that the assert branch was not covered.
for (NSString * string in self.testArray) {
NSLog(#"result %#",string);
}
and Jenkins reports the coverages as:
Conditionals 80% (28/34)
I tried with testcase testArray = nil and testArray empty but the report isn't different. How can I pass this conditional test?
Related
I understand that when outputting the test coverage report from Pitest, you are presented with the following structure:
<block classname='classname' method='methodName' number='1'>
<tests>
<test name='testName1(file1)'/>
<test name='testName2(file2)'/>
</tests>
</block>
However, I am unsure of the meaning of the 'number' attribute. What is the significance of it?
Thanks
The number is the number of the block within the method.
Blocks are chunks of code that execute together. To calculate coverage, pitest creates a probe in each block. If the probe is executed, pitest then knows that all instructions within that block have been executed, unless an exception is thrown.
Most coverage systems place the probe at the end of the block, so in event of an exception the coverage will be underreported. Pitest places them at the beginning of each block. This causes overreporting, but this is desirable for pitest as the coverage data is used to target tests against mutants. It is better to run a test and it not to detect a mutant than not run one which would.
The precise definition of a block has changed between pitest releases. Originally, it roughly corresponded to branches in the code.
int foo(boolean b) {
System.out.println("hello"); // block 1
int i = 2; // block 1
if (b) { // block 1
System.out.println("boo"); // block 2
i = i + 42; // block 2
}
System.out.println("bye"); // block 3
return i; // block 3
}
So the code above would have three blocks as shown in the comments. Later releases have made the blocks smaller so tests can be targeted more accurately when exceptions occur.
Is there a way to mark a test run failed if the number of tests decreases in jenkins. We have a test report in JUnit.
For some reasons, sometimes the number of tests decreases. Often that are critical errors. Is there a way to say jenkins that in such case the test status should be red? (Maybe some plugin)
Thanks a lot for any hint!
There is a way.
Install the Groovy Postbuild Plugin and add a Post Build Step
Post-Build-Actions -> Add Step -> Groovy Postbuild
import jenkins.model.*
def currentBuild = manager.build
def totalCurrent
def totalPrevious
// evaluate test count of current Build
def result = currentBuild .getAction(hudson.tasks.junit.TestResultAction.class).result
if (result != null) {
totalCurrent = result.hasProperty( 'totalCount' ) ? result.totalCount : null
}
// evaluate test count of previous Build
result = currentBuild .previousBuild.getAction(hudson.tasks.junit.TestResultAction.class).result
if (result != null) {
totalPrevious = result.hasProperty( 'totalCount' ) ? result.totalCount : null
}
// fail the build if test count reduced
if(totalCurrent < totalPrevious) {
manager.buildFailure()
}
You'll may need to add some more nullsafe checks.
Even if there is a way to do this, it sounds like a bad idea considering the fact that your number of tests executed could actually decrease from build to build for legitimate reasons.
First I would try to find out why this is happening unexpectedly.
what is wrong with below code, comparing 2 strings in groovy
I am trying do the comparison between the 2 parameters in a single line to make it look tidier
if (params.dirname == ((params.path =~ ~/${params.dirname}/).with { matches() ? it[0] : null })) {
print success
}
Throwing Exception -
java.lang.NoSuchMethodError: No such DSL method 'matches' found among steps
There is no need to over-complicate your use case. According to:
params.dirname = hde, params.path = /usr/tmp/jenkins/hde/filename.txt or /usr/hde/jenkins/ing/filename.txt or any random path which has hde in it
you are trying to find if given string a contains substring b. It can be done using Java's method String.contains(String substring). Alternatively you can use regular expression for that, but String.contains() just looks a few times simpler to understand what is your intention. Consider following Groovy script:
def params = [
dirname: 'hde',
path: '/usr/tmp/jenkins/hde/filename.txt'
]
// Using String.contains()
if (params.path.contains(params.dirname)) {
println "Path '${params.path}' contains '${params.dirname}'"
}
// Using regular expression
if (params.path ==~ /(.*)${params.dirname}(.*)/) {
println "Path '${params.path}' contains '${params.dirname}'"
}
When you run it both if statements evaluates to true:
Path '/usr/tmp/jenkins/hde/filename.txt' contains 'hde'
Path '/usr/tmp/jenkins/hde/filename.txt' contains 'hde'
Statement coverage is said to make sure that every statement in the code is executed at least once.
Decision/branch coverage is said to test that each branch/output of a decisions is tested, i.e. all statements in both false/true branches will be executed.
But is it not the same? In Statement coverage I need to execute all statements so I guess it can be only done by running all possible ways. I know I am missing something here..
The answer by Paul isn't quite right, at least I think so (according to ISTQB's definitions). There's quite significant difference between statement, decision/branch and condition coverage.
I'll use the sample from the other answer but modified a bit, so I can show all three test coverage examples. Tests written here gives 100% test coverage for each type.
if(a || b)) {
test1 = true;
}
else {
if(c) {
test2 = true
}
}
We have two statements here - if(a||b) and if(c), to fully explain those coverage differences:
statement coverage have to test each statement at least once, so we need just two tests:
a=true b=false - that gives us path if(a||b) true -> test1 = true
a=false, b=false and c=true - that gives us path: if(a||b) false -> else -> if(c) -> test2=true.
This way we executed each and every statement.
branch/decision coverage needs one more test:
a=false, b=false, c=false - that leads us to that second if but we are executing false branch from that statement, that wasn't executed in statement coverage
That way we have all the branches tested, meaning we went through all the paths.
condition coverage needs another test:
a=false, b=true - that leads through the same path as first test but executes the other decision in OR statement (a||b) to go through it.
That way we have all conditions tested, meaning that we went through all paths (branches) and triggered it with each condition we could - first 'if' statement was true in first test because of a=true triggered it and in the last test because b=true triggered it. Of course someone can argue that case with a=true and b=true should be tested as well, but when we will check how 'or' works then we can see it isn't needed and also variable c can be of any value as in those tests it is not evaluated.
At least I interpreted it this way. If someone is still interested :)
EDIT: In most sources I found lately decision/branch coverage terms are equivalent and the term I described as decision coverage is in fact condition coverage hence that update of the answer.
If the tests have complete branch coverage then we can say it also has complete statement coverage, but not the vice versa.
100% branch coverage => 100% statement coverage
100% statement coverage does not imply 100% branch coverage
the reason is in branch coverage apart from executing all the statements, we should also verify if the tests executes all branches, which can be interpreted as covering all edges in the control flow branch
if(a){
if(b){
bool statement1 = true;
}
}
a = true, b = true will give 100% statement coverage, but not branch coverage
In the branch coverage we need to cover all the edges, which we missed in the statement coverage shown as red lines in the above image
Nice question. The explanation I often use is that an if-statement without an else-branch still has an invisible "empty" else-statement:
Plain statement coverage just insists that all statements that are actually there are really executed.
Branch coverage insists that even invisible else-branches are executed.
Similar situations occur with switch-statements without a default-case, and repeat-until loops. Branch coverage requires that a default-case is executed, and that a repeat-until is executed at least twice.
A code example:
if (passwordEnteredOK()) {
enterSystem();
}
/* Invisible else part
else {
// do nothing
}
*/
With statement coverage you just check that with a correct password you can use the system. With branch coverage you also test that with an incorrect password you will not enter the system.
You may have a statement like:
if(a || b || (c && d && !e)) {
test1 = true;
} else {
test2 = false;
}
If your code coverage says both the test1 and test2 lines are hit then you have statement coverage, but to get full branch coverage you will need to test when a is true, when a is false but b is true, when a and b are false but c and d are true and e is false, etc.
Branch coverage covers every potential combination of branch choices and so is harder to achieve 100% coverage.
I have a question for Code coverage with Sonar.
In this example :
public static void apply(Person person) {
if (person != null) {
callApply(person);
}
}
In my Sonar Code coverage, i have the following result :
if (person != null) => hit=(10) % = (75%)
callApply(person); => hit(10) % =()
I don't understand why with the same number of hit, the coverage is 75% ?, and why i don't have any value coverage when we call the method callApply
I think this version of Sonar use Jacoco for coverage code.
Thank you.
First, Sonar is using Cobertura by default (JaCoCo will be the default one in Sonar 3.2 that will ship during summer 2012).
Then, what makes you think that on the 1rst line you get 75% of coverage while you would have nothing on the 2nd line?