Is there a way to mark a test run failed if the number of tests decreases in jenkins. We have a test report in JUnit.
For some reasons, sometimes the number of tests decreases. Often that are critical errors. Is there a way to say jenkins that in such case the test status should be red? (Maybe some plugin)
Thanks a lot for any hint!
There is a way.
Install the Groovy Postbuild Plugin and add a Post Build Step
Post-Build-Actions -> Add Step -> Groovy Postbuild
import jenkins.model.*
def currentBuild = manager.build
def totalCurrent
def totalPrevious
// evaluate test count of current Build
def result = currentBuild .getAction(hudson.tasks.junit.TestResultAction.class).result
if (result != null) {
totalCurrent = result.hasProperty( 'totalCount' ) ? result.totalCount : null
}
// evaluate test count of previous Build
result = currentBuild .previousBuild.getAction(hudson.tasks.junit.TestResultAction.class).result
if (result != null) {
totalPrevious = result.hasProperty( 'totalCount' ) ? result.totalCount : null
}
// fail the build if test count reduced
if(totalCurrent < totalPrevious) {
manager.buildFailure()
}
You'll may need to add some more nullsafe checks.
Even if there is a way to do this, it sounds like a bad idea considering the fact that your number of tests executed could actually decrease from build to build for legitimate reasons.
First I would try to find out why this is happening unexpectedly.
I'm tryting to run our Cucumber JVM tests by few threads in parallel.
I'm using standart TastNG approach to do it (via suite XML file)
My xml file is:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="BDD" parallel="methods" thread-count="3" data-provider-thread-count="3">
<test name="BDD">
<classes>
<class name="com.tests.bdd.SimpleBDDTests"></class>
</classes>
</test>
</suite>
My test class is:
#CucumberOptions(features = "src/test/java/com/tests/bdd/simpleFeatures")
public class SimpleBDDTests {
private TestNGCucumberRunner tcr;
#BeforeClass(alwaysRun = true)
public void beforeClass() throws Exception {
tcr = new TestNGCucumberRunner(this.getClass());
}
#AfterClass(alwaysRun = true)
public void afterClass() {
tcr.finish();
}
#Test(dataProvider = "features")
public void feature(CucumberFeatureWrapper cucumberFeature) {
tcr.runCucumber(cucumberFeature.getCucumberFeature());
}
#DataProvider(parallel = true)
public Object[][] features() {
return tcr.provideFeatures();
}}
My feature files are like:
Feature: First test
#sanity
Scenario: First simple test
Given Base check step
I have 4 feature files, which are defines the same scenarios with only one step - Given Base check step
When these features are executed one by one, it works fine, but when i try to run them in parallel, everything gets broken.
Almost all of these featres marked as failed with the following exception:
A scoping block is already in progress
java.lang.IllegalStateException: A scoping block is already in progress
at cucumber.runtime.java.guice.impl.SequentialScenarioScope.checkState(SequentialScenarioScope.java:64)
at cucumber.runtime.java.guice.impl.SequentialScenarioScope.enterScope(SequentialScenarioScope.java:52)
at cucumber.runtime.java.guice.impl.GuiceFactory.start(GuiceFactory.java:34)
at cucumber.runtime.java.JavaBackend.buildWorld(JavaBackend.java:123)
at cucumber.runtime.Runtime.buildBackendWorlds(Runtime.java:141)
at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:38)
at cucumber.runtime.model.CucumberFeature.run(CucumberFeature.java:165)
at cucumber.api.testng.TestNGCucumberRunner.runCucumber(TestNGCucumberRunner.java:63)
I understand that it might be happen because of multi-thread calls to the same step - Given Base check step
So my question is how can i fix that? How can i run these tests in parallel?
PS: I know that it should be possible to do it by JUnit + Maven surefire plugin, but it is not applicable for current project, we need to achieve that goal by TestNG.
Thanks.
I have a grails 3.1.x project with lots of controllers and I use spock for testing the controllers.
I use a command pattern for my controllers so each action takes a command object.
An example for such a controller :
class PlaygroundController {
public Object pg(PgCO pgco) {
String s = null
if (pgco.one) {
s = pgco.one
}
if (pgco.two) {
s = pgco.two
}
render view: 'pg', model: [resultat: s]
}
}
I have a simple spock test case:
void "pg"() {
given:
PgCO pgCO = new PgCO()
pgCO.one = 'TEST'
when:
controller.pg(pgCO)
then:
view == '/playground/pg'
model
model.resultat == 'TEST'
}
This test runs fine, but when i run a jacocoTestReport it shows that my coverage is way below 50% , it's fine that it's not fully covered since I'm only covering one branch.
But the report contains loads of "missed instructions" on methods that I have no source code for.. specially there is a shadow pg() method that doesn't take any parameters. Grails injects this method, but I will never call that method from my tests since I use the command object version of the same method. this method but also a lot of other methods coming from grails controller framework.
Isn't there anyw ay to tell jacoco that it's only the "source code" that needs to be checked for coverage ?
I've tried specifying exact source dirs, but this doesn't help
Here's an image of the coverage report:
coverage report
I have a daily simple project just enabling the jacoco plugin, and running latest jacoco :
jacoco {
toolVersion = "0.7.5.201505241946"
}
Have anyone else a solution to this ?
My project use Jenkins to report the conditionals of an assert statement.
I have XCTest that expects and AssertionError to be thrown, and it passes correctly. The problem is that Jenkins reports that the assert branch was not covered.
for (NSString * string in self.testArray) {
NSLog(#"result %#",string);
}
and Jenkins reports the coverages as:
Conditionals 80% (28/34)
I tried with testcase testArray = nil and testArray empty but the report isn't different. How can I pass this conditional test?
Statement coverage is said to make sure that every statement in the code is executed at least once.
Decision/branch coverage is said to test that each branch/output of a decisions is tested, i.e. all statements in both false/true branches will be executed.
But is it not the same? In Statement coverage I need to execute all statements so I guess it can be only done by running all possible ways. I know I am missing something here..
The answer by Paul isn't quite right, at least I think so (according to ISTQB's definitions). There's quite significant difference between statement, decision/branch and condition coverage.
I'll use the sample from the other answer but modified a bit, so I can show all three test coverage examples. Tests written here gives 100% test coverage for each type.
if(a || b)) {
test1 = true;
}
else {
if(c) {
test2 = true
}
}
We have two statements here - if(a||b) and if(c), to fully explain those coverage differences:
statement coverage have to test each statement at least once, so we need just two tests:
a=true b=false - that gives us path if(a||b) true -> test1 = true
a=false, b=false and c=true - that gives us path: if(a||b) false -> else -> if(c) -> test2=true.
This way we executed each and every statement.
branch/decision coverage needs one more test:
a=false, b=false, c=false - that leads us to that second if but we are executing false branch from that statement, that wasn't executed in statement coverage
That way we have all the branches tested, meaning we went through all the paths.
condition coverage needs another test:
a=false, b=true - that leads through the same path as first test but executes the other decision in OR statement (a||b) to go through it.
That way we have all conditions tested, meaning that we went through all paths (branches) and triggered it with each condition we could - first 'if' statement was true in first test because of a=true triggered it and in the last test because b=true triggered it. Of course someone can argue that case with a=true and b=true should be tested as well, but when we will check how 'or' works then we can see it isn't needed and also variable c can be of any value as in those tests it is not evaluated.
At least I interpreted it this way. If someone is still interested :)
EDIT: In most sources I found lately decision/branch coverage terms are equivalent and the term I described as decision coverage is in fact condition coverage hence that update of the answer.
If the tests have complete branch coverage then we can say it also has complete statement coverage, but not the vice versa.
100% branch coverage => 100% statement coverage
100% statement coverage does not imply 100% branch coverage
the reason is in branch coverage apart from executing all the statements, we should also verify if the tests executes all branches, which can be interpreted as covering all edges in the control flow branch
if(a){
if(b){
bool statement1 = true;
}
}
a = true, b = true will give 100% statement coverage, but not branch coverage
In the branch coverage we need to cover all the edges, which we missed in the statement coverage shown as red lines in the above image
Nice question. The explanation I often use is that an if-statement without an else-branch still has an invisible "empty" else-statement:
Plain statement coverage just insists that all statements that are actually there are really executed.
Branch coverage insists that even invisible else-branches are executed.
Similar situations occur with switch-statements without a default-case, and repeat-until loops. Branch coverage requires that a default-case is executed, and that a repeat-until is executed at least twice.
A code example:
if (passwordEnteredOK()) {
enterSystem();
}
/* Invisible else part
else {
// do nothing
}
*/
With statement coverage you just check that with a correct password you can use the system. With branch coverage you also test that with an incorrect password you will not enter the system.
You may have a statement like:
if(a || b || (c && d && !e)) {
test1 = true;
} else {
test2 = false;
}
If your code coverage says both the test1 and test2 lines are hit then you have statement coverage, but to get full branch coverage you will need to test when a is true, when a is false but b is true, when a and b are false but c and d are true and e is false, etc.
Branch coverage covers every potential combination of branch choices and so is harder to achieve 100% coverage.