Whenever I am trying to run the tagged test cases in jenkins, they are getting skipped with the message
WARN org.testng.internal.Utils - The test method 'testRunner.TestRunner.feature' will be skipped since its data provider 'features' returned an empty array or iterator.
but the same test cases are getting executed and passed in my local machine. I have integrated cucumber with maven and testNG
Below is my test runner file
#CucumberOptions(features = { "src/test/java/com/test_Cases_FeatureFiles" },
glue = {"stepDefs"},
plugin = { "pretty","html:target/cucumber-html-report","com.aventstack.extentreports.cucumber.adapter.ExtentCucumberAdapter:" },
monochrome = true,
tags = "#FileDownload_Tests")
public class TestRunner extends AbstractTestNGCucumberTests {
}
Please suggest a way out for this issue.
Related
For the reasons I cannot influence, there is the following mechanism on TestNG launching on the project.
In a word, it creates a new instance of TestNG, adds listeners, specifies classes and runs the tests. Then, all this dirty code is called from Gradle run task (which is actually empty and as far as I understood, simply calls the TestManager.main() method).
I removed the part of code just to show the main direction:
class TestManager {
static void main(String[] args) {
try {
runTests(args[0])
} catch (Exception e) {
e.printStackTrace()
System.in.read()
}
}
private static void runTests(Application app) {
TagsConfig.runs.each { run ->
if (run.execute) {
List<TagsSuite> suites = TagsConfig.suites
suites.each { suite ->
if (suite.execute) {
Reflections reflections = new Reflections("${app.packageName}.${suite.name}")
def classes = reflections.getSubTypesOf(${suite.name})
if (classes.size() > 0) {
TestNG testNG = new TestNG()
testNG.testClasses = classes
testNG.groupByInstances = true
testNG.outputDirectory = "testng-output"
testNG.addListener(new TestListenerAdapter())
testNG.addListener(new ExceptionListener())
testNG.addListener(new AllureTestListener())
if (TagsConfig.isSmoke) {
testNG.setGroups("smoke")
} else if (TagsConfig.isExtendedSmoke) {
testNG.setGroups("extended_smoke")
}
testNG.run()
}
}
}
}
}
}
So the test launch looks like this:
gradle clean
gradle run
I can not change the way the tests are started now and one of the problems is that the build in Jenkins is always Sucessful even if there are failed/broken/skipped tests.
So, I can get the number of failed tests from TestListenerAdapter, but how can I let Jenkins know that there were failed tests?
Maybe by returning an exit code from a Gradle run task or by installing some plugin that will check the count of failed tests in TestListenerAdapter?
For now, I'm setting a "FAILED_TESTS_COUNT" system property in onFinish() event and change the build status in pipeline if it is not 0, but this looks really dirty.
Jenkins 2.89.3
Gradle 3.5.1
TestNG 6.9.8
I personally use the Jenkins Text Finder post-build action. In the above example, it looks for a pattern in the standard output and set the build to "unstable" (orange) when it's found.
In your case, just uncheck both checkboxes and find the according matching pattern.
I'm tryting to run our Cucumber JVM tests by few threads in parallel.
I'm using standart TastNG approach to do it (via suite XML file)
My xml file is:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="BDD" parallel="methods" thread-count="3" data-provider-thread-count="3">
<test name="BDD">
<classes>
<class name="com.tests.bdd.SimpleBDDTests"></class>
</classes>
</test>
</suite>
My test class is:
#CucumberOptions(features = "src/test/java/com/tests/bdd/simpleFeatures")
public class SimpleBDDTests {
private TestNGCucumberRunner tcr;
#BeforeClass(alwaysRun = true)
public void beforeClass() throws Exception {
tcr = new TestNGCucumberRunner(this.getClass());
}
#AfterClass(alwaysRun = true)
public void afterClass() {
tcr.finish();
}
#Test(dataProvider = "features")
public void feature(CucumberFeatureWrapper cucumberFeature) {
tcr.runCucumber(cucumberFeature.getCucumberFeature());
}
#DataProvider(parallel = true)
public Object[][] features() {
return tcr.provideFeatures();
}}
My feature files are like:
Feature: First test
#sanity
Scenario: First simple test
Given Base check step
I have 4 feature files, which are defines the same scenarios with only one step - Given Base check step
When these features are executed one by one, it works fine, but when i try to run them in parallel, everything gets broken.
Almost all of these featres marked as failed with the following exception:
A scoping block is already in progress
java.lang.IllegalStateException: A scoping block is already in progress
at cucumber.runtime.java.guice.impl.SequentialScenarioScope.checkState(SequentialScenarioScope.java:64)
at cucumber.runtime.java.guice.impl.SequentialScenarioScope.enterScope(SequentialScenarioScope.java:52)
at cucumber.runtime.java.guice.impl.GuiceFactory.start(GuiceFactory.java:34)
at cucumber.runtime.java.JavaBackend.buildWorld(JavaBackend.java:123)
at cucumber.runtime.Runtime.buildBackendWorlds(Runtime.java:141)
at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:38)
at cucumber.runtime.model.CucumberFeature.run(CucumberFeature.java:165)
at cucumber.api.testng.TestNGCucumberRunner.runCucumber(TestNGCucumberRunner.java:63)
I understand that it might be happen because of multi-thread calls to the same step - Given Base check step
So my question is how can i fix that? How can i run these tests in parallel?
PS: I know that it should be possible to do it by JUnit + Maven surefire plugin, but it is not applicable for current project, we need to achieve that goal by TestNG.
Thanks.
I have a grails 3.1.x project with lots of controllers and I use spock for testing the controllers.
I use a command pattern for my controllers so each action takes a command object.
An example for such a controller :
class PlaygroundController {
public Object pg(PgCO pgco) {
String s = null
if (pgco.one) {
s = pgco.one
}
if (pgco.two) {
s = pgco.two
}
render view: 'pg', model: [resultat: s]
}
}
I have a simple spock test case:
void "pg"() {
given:
PgCO pgCO = new PgCO()
pgCO.one = 'TEST'
when:
controller.pg(pgCO)
then:
view == '/playground/pg'
model
model.resultat == 'TEST'
}
This test runs fine, but when i run a jacocoTestReport it shows that my coverage is way below 50% , it's fine that it's not fully covered since I'm only covering one branch.
But the report contains loads of "missed instructions" on methods that I have no source code for.. specially there is a shadow pg() method that doesn't take any parameters. Grails injects this method, but I will never call that method from my tests since I use the command object version of the same method. this method but also a lot of other methods coming from grails controller framework.
Isn't there anyw ay to tell jacoco that it's only the "source code" that needs to be checked for coverage ?
I've tried specifying exact source dirs, but this doesn't help
Here's an image of the coverage report:
coverage report
I have a daily simple project just enabling the jacoco plugin, and running latest jacoco :
jacoco {
toolVersion = "0.7.5.201505241946"
}
Have anyone else a solution to this ?
Ideally I'd like to be able to invoke the script with some kind of unit test before I have it execute on a Jenkins.
Is there any way to test a Job DSL script other than having jenkins run it?
Besides the examples in job-dsl-gradle-example, you can also go a step further and write tests for individual files or jobs. For example let's assume you have a job configuration file located in jobs/deployJob.groovy
import javaposse.jobdsl.dsl.DslScriptLoader
import javaposse.jobdsl.dsl.MemoryJobManagement
import javaposse.jobdsl.dsl.ScriptRequest
import spock.lang.Specification
class TestDeployJobs extends Specification {
def 'test basic job configuration'() {
given:
URL scriptURL = new File('jobs').toURI().toURL()
ScriptRequest scriptRequest = new ScriptRequest('deployJob.groovy', null, scriptURL)
MemoryJobManagement jobManagement = new MemoryJobManagement()
when:
DslScriptLoader.runDslEngine(scriptRequest, jobManagement)
then:
jobManagement.savedConfigs.each { String name, String xml ->
with(new XmlParser().parse(new StringReader(xml))) {
// Make sure jobs only run manually
triggers.'hudson.triggers.TimerTrigger'.spec.text().isEmpty()
// only deploy every environment once at a time
concurrentBuild.text().equals('false')
// do a workspace cleanup
buildWrappers.'hudson.plugins.ws__cleanup.PreBuildCleanup'
// make sure masked passwords are active
!buildWrappers.'com.michelin.cio.hudson.plugins.maskpasswords.MaskPasswordsBuildWrapper'.isEmpty()
}
}
}
}
This way you are able to go through every XML node you want to make sure to have all the right values set.
Have a look at the job-dsl-gradle-example. The repo contains a test for DSL scripts.
Doing it in the same way as crasp but using Jenkins test harness as explained in Jenkins Unit Test page, which is slower but would work with auto-generated DSL giving syntax errors as explained here.
After setting the code as explained here, you can just do a test like this one:
#Unroll
void 'check descriptions #file.name'(File file) {
given:
JobManagement jobManagement = new JenkinsJobManagement(System.out, [:], new File('.'))
Jenkins jenkins = jenkinsRule.jenkins
when:
GeneratedItems items = new DslScriptLoader(jobManagement).runScript(file.text)
then:
if (!items.jobs.isEmpty()) {
items.jobs.each { GeneratedJob generatedJob ->
String text = getItemXml(generatedJob, jenkins)
with(new XmlParser().parse(new StringReader(text))) {
// Has some description
!description.text().isEmpty()
}
}
}
where:
file << TestUtil.getJobFiles()
}
I have a problem while running fixtureLoader.load in BootStrap.groovy.
import grails.plugin.fixtures.FixtureLoader
class BootStrap {
def fixtureLoader
def init = { servletContext ->
environments {
test {
fixtureLoader.load {
build {
device1(Device, name: "device1")
device2(Device, name: "device2")
device3(Device, name: "device3")
}
}
}
}
}
def destroy = {
}
}
When Grails starts integration test phase, the load is executed. Then when Grails starts functional test phase, the load is executed another time without cleaning previous execution.
Thats means:
If I run with "test-app functional:" or "test-app integration:" everything is all right.
If I run with "test-app" both functional and integration tests are executed.
This is the JSON representation of data at functional test phase (running with "test-app"):
[[name:device3], [name:device2], [name:device1],[name:device3], [name:device2], [name:device1]]
This is same JSON representation of data at functional test phase (running with "test-app functional:")
[[name:device3], [name:device2], [name:device1]]
How to avoid this duplicates?
Thanks in advance
The default test database is a non-persistant in-memory hsqldb that gets discarded at the end of your tests, but changes to it will carry over between test phases. Also, integration testing rolls back changes after each test, but this doesn't apply to database changes made in Bootstrap.groovy.
One simple way to fix this is to simple check for the existence of the fixtures before trying to create them. For example:
environments {
test {
if (Device.count() == 0) {
// build fixtures
}
}
}
Another possible solution is to use separate databases for integration and functional tests. There's an example of how to do it at http://www.redtoad.ca/ataylor/2011/02/setting-grails-functional-test-database/