I'm tryting to run our Cucumber JVM tests by few threads in parallel.
I'm using standart TastNG approach to do it (via suite XML file)
My xml file is:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="BDD" parallel="methods" thread-count="3" data-provider-thread-count="3">
<test name="BDD">
<classes>
<class name="com.tests.bdd.SimpleBDDTests"></class>
</classes>
</test>
</suite>
My test class is:
#CucumberOptions(features = "src/test/java/com/tests/bdd/simpleFeatures")
public class SimpleBDDTests {
private TestNGCucumberRunner tcr;
#BeforeClass(alwaysRun = true)
public void beforeClass() throws Exception {
tcr = new TestNGCucumberRunner(this.getClass());
}
#AfterClass(alwaysRun = true)
public void afterClass() {
tcr.finish();
}
#Test(dataProvider = "features")
public void feature(CucumberFeatureWrapper cucumberFeature) {
tcr.runCucumber(cucumberFeature.getCucumberFeature());
}
#DataProvider(parallel = true)
public Object[][] features() {
return tcr.provideFeatures();
}}
My feature files are like:
Feature: First test
#sanity
Scenario: First simple test
Given Base check step
I have 4 feature files, which are defines the same scenarios with only one step - Given Base check step
When these features are executed one by one, it works fine, but when i try to run them in parallel, everything gets broken.
Almost all of these featres marked as failed with the following exception:
A scoping block is already in progress
java.lang.IllegalStateException: A scoping block is already in progress
at cucumber.runtime.java.guice.impl.SequentialScenarioScope.checkState(SequentialScenarioScope.java:64)
at cucumber.runtime.java.guice.impl.SequentialScenarioScope.enterScope(SequentialScenarioScope.java:52)
at cucumber.runtime.java.guice.impl.GuiceFactory.start(GuiceFactory.java:34)
at cucumber.runtime.java.JavaBackend.buildWorld(JavaBackend.java:123)
at cucumber.runtime.Runtime.buildBackendWorlds(Runtime.java:141)
at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:38)
at cucumber.runtime.model.CucumberFeature.run(CucumberFeature.java:165)
at cucumber.api.testng.TestNGCucumberRunner.runCucumber(TestNGCucumberRunner.java:63)
I understand that it might be happen because of multi-thread calls to the same step - Given Base check step
So my question is how can i fix that? How can i run these tests in parallel?
PS: I know that it should be possible to do it by JUnit + Maven surefire plugin, but it is not applicable for current project, we need to achieve that goal by TestNG.
Thanks.
Related
Whenever I am trying to run the tagged test cases in jenkins, they are getting skipped with the message
WARN org.testng.internal.Utils - The test method 'testRunner.TestRunner.feature' will be skipped since its data provider 'features' returned an empty array or iterator.
but the same test cases are getting executed and passed in my local machine. I have integrated cucumber with maven and testNG
Below is my test runner file
#CucumberOptions(features = { "src/test/java/com/test_Cases_FeatureFiles" },
glue = {"stepDefs"},
plugin = { "pretty","html:target/cucumber-html-report","com.aventstack.extentreports.cucumber.adapter.ExtentCucumberAdapter:" },
monochrome = true,
tags = "#FileDownload_Tests")
public class TestRunner extends AbstractTestNGCucumberTests {
}
Please suggest a way out for this issue.
]I am a junior dev trying to lear about Jenkins, I have been learning on my own for a couple of months. Currently I have a pipeline (just for learning purposes) which runs static analysis on a folder, and then publish it, I have been able to send a report through email using jelly templates, from there I realized it is posbile to instantiate the classes of a plugin to use its methods so I went to the cppcheck javadoc and did some trial and error so I can get some values of my report and then do something else with them, so I had something like this in my pipeline:
pipeline {
agent any
stages {
stage('analysis') {
steps {
script{
bat'cppcheck "E:/My_project/Source/" --xml --xml-version=2 . 2> cppcheck.xml'
}
}
}
stage('Test'){
steps {
script {
publishCppcheck pattern:'cppcheck.xml'
for (action in currentBuild.rawBuild.getActions()) {
def name = action.getClass().getName()
if (name == 'org.jenkinsci.plugins.cppcheck.CppcheckBuildAction') {
def cppcheckaction = action
def totalErrors = cppcheckaction.getResult().report.getNumberTotal()
println totalErrors
def warnings = cppcheckaction.getResult().statistics.getNumberWarningSeverity()
println warnings
}
}
}
}
}
}
}
which output is:
[Pipeline] echo
102
[Pipeline] echo
4
My logic (wrongly) tells me that if I can access to the report and statistics classes like that and uses their methods getNumberTotal() and getNumberWarningSeverity() respectively, therefore I should be able to also access the DiffState class in the same way and use the valueOf() method to get an enum of the new errors. But adding this to my pipeline:
def nueva = cppcheckaction.getResult().diffState.valueOf(NEW)
println nueva
Gives me an error:
org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: No such field found: field org.jenkinsci.plugins.cppcheck.CppcheckBuildAction diffState
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.unclassifiedField(SandboxInterceptor.java:425)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onGetProperty(SandboxInterceptor.java:409)
...
I can see in the javadoc there is a diffState class with a valueOf() method, but I cannot access to it is therre any other way to get the new errors between the last build and the current one?
I see 2 issues that could be causing this:
CppcheckResult doesn't have a member variable diffState so you can't access it obviously
If you check the javadoc of CppcheckResult the class does have:
private CppcheckReport report;
public CppcheckStatistics getReport()
and
private CppcheckStatistics statistics;
public CppcheckStatistics getStatistics()
there is no member (and getter method) for diffState so maybe try to call:
/**
* Get differences between current and previous statistics.
*
* #return the differences
*/
public CppcheckStatistics getDiff(){
my suggestion: cppcheckaction.getResult().getDiff().valueOf(NEW). Furthermore CppcheckWorkspaceFile does have a method getDiffState().
Please have a look at the script approval of your Jenkins (see here).
The syntax error might appear because Jenkins (Groovy Sandbox) blocks the execution of an (for the Jenkins) "unknown" and potential dangerous method.
Jenkins settings - Script Approval - Approve your blocked method
For the reasons I cannot influence, there is the following mechanism on TestNG launching on the project.
In a word, it creates a new instance of TestNG, adds listeners, specifies classes and runs the tests. Then, all this dirty code is called from Gradle run task (which is actually empty and as far as I understood, simply calls the TestManager.main() method).
I removed the part of code just to show the main direction:
class TestManager {
static void main(String[] args) {
try {
runTests(args[0])
} catch (Exception e) {
e.printStackTrace()
System.in.read()
}
}
private static void runTests(Application app) {
TagsConfig.runs.each { run ->
if (run.execute) {
List<TagsSuite> suites = TagsConfig.suites
suites.each { suite ->
if (suite.execute) {
Reflections reflections = new Reflections("${app.packageName}.${suite.name}")
def classes = reflections.getSubTypesOf(${suite.name})
if (classes.size() > 0) {
TestNG testNG = new TestNG()
testNG.testClasses = classes
testNG.groupByInstances = true
testNG.outputDirectory = "testng-output"
testNG.addListener(new TestListenerAdapter())
testNG.addListener(new ExceptionListener())
testNG.addListener(new AllureTestListener())
if (TagsConfig.isSmoke) {
testNG.setGroups("smoke")
} else if (TagsConfig.isExtendedSmoke) {
testNG.setGroups("extended_smoke")
}
testNG.run()
}
}
}
}
}
}
So the test launch looks like this:
gradle clean
gradle run
I can not change the way the tests are started now and one of the problems is that the build in Jenkins is always Sucessful even if there are failed/broken/skipped tests.
So, I can get the number of failed tests from TestListenerAdapter, but how can I let Jenkins know that there were failed tests?
Maybe by returning an exit code from a Gradle run task or by installing some plugin that will check the count of failed tests in TestListenerAdapter?
For now, I'm setting a "FAILED_TESTS_COUNT" system property in onFinish() event and change the build status in pipeline if it is not 0, but this looks really dirty.
Jenkins 2.89.3
Gradle 3.5.1
TestNG 6.9.8
I personally use the Jenkins Text Finder post-build action. In the above example, it looks for a pattern in the standard output and set the build to "unstable" (orange) when it's found.
In your case, just uncheck both checkboxes and find the according matching pattern.
I have a grails 3.1.x project with lots of controllers and I use spock for testing the controllers.
I use a command pattern for my controllers so each action takes a command object.
An example for such a controller :
class PlaygroundController {
public Object pg(PgCO pgco) {
String s = null
if (pgco.one) {
s = pgco.one
}
if (pgco.two) {
s = pgco.two
}
render view: 'pg', model: [resultat: s]
}
}
I have a simple spock test case:
void "pg"() {
given:
PgCO pgCO = new PgCO()
pgCO.one = 'TEST'
when:
controller.pg(pgCO)
then:
view == '/playground/pg'
model
model.resultat == 'TEST'
}
This test runs fine, but when i run a jacocoTestReport it shows that my coverage is way below 50% , it's fine that it's not fully covered since I'm only covering one branch.
But the report contains loads of "missed instructions" on methods that I have no source code for.. specially there is a shadow pg() method that doesn't take any parameters. Grails injects this method, but I will never call that method from my tests since I use the command object version of the same method. this method but also a lot of other methods coming from grails controller framework.
Isn't there anyw ay to tell jacoco that it's only the "source code" that needs to be checked for coverage ?
I've tried specifying exact source dirs, but this doesn't help
Here's an image of the coverage report:
coverage report
I have a daily simple project just enabling the jacoco plugin, and running latest jacoco :
jacoco {
toolVersion = "0.7.5.201505241946"
}
Have anyone else a solution to this ?
Ideally I'd like to be able to invoke the script with some kind of unit test before I have it execute on a Jenkins.
Is there any way to test a Job DSL script other than having jenkins run it?
Besides the examples in job-dsl-gradle-example, you can also go a step further and write tests for individual files or jobs. For example let's assume you have a job configuration file located in jobs/deployJob.groovy
import javaposse.jobdsl.dsl.DslScriptLoader
import javaposse.jobdsl.dsl.MemoryJobManagement
import javaposse.jobdsl.dsl.ScriptRequest
import spock.lang.Specification
class TestDeployJobs extends Specification {
def 'test basic job configuration'() {
given:
URL scriptURL = new File('jobs').toURI().toURL()
ScriptRequest scriptRequest = new ScriptRequest('deployJob.groovy', null, scriptURL)
MemoryJobManagement jobManagement = new MemoryJobManagement()
when:
DslScriptLoader.runDslEngine(scriptRequest, jobManagement)
then:
jobManagement.savedConfigs.each { String name, String xml ->
with(new XmlParser().parse(new StringReader(xml))) {
// Make sure jobs only run manually
triggers.'hudson.triggers.TimerTrigger'.spec.text().isEmpty()
// only deploy every environment once at a time
concurrentBuild.text().equals('false')
// do a workspace cleanup
buildWrappers.'hudson.plugins.ws__cleanup.PreBuildCleanup'
// make sure masked passwords are active
!buildWrappers.'com.michelin.cio.hudson.plugins.maskpasswords.MaskPasswordsBuildWrapper'.isEmpty()
}
}
}
}
This way you are able to go through every XML node you want to make sure to have all the right values set.
Have a look at the job-dsl-gradle-example. The repo contains a test for DSL scripts.
Doing it in the same way as crasp but using Jenkins test harness as explained in Jenkins Unit Test page, which is slower but would work with auto-generated DSL giving syntax errors as explained here.
After setting the code as explained here, you can just do a test like this one:
#Unroll
void 'check descriptions #file.name'(File file) {
given:
JobManagement jobManagement = new JenkinsJobManagement(System.out, [:], new File('.'))
Jenkins jenkins = jenkinsRule.jenkins
when:
GeneratedItems items = new DslScriptLoader(jobManagement).runScript(file.text)
then:
if (!items.jobs.isEmpty()) {
items.jobs.each { GeneratedJob generatedJob ->
String text = getItemXml(generatedJob, jenkins)
with(new XmlParser().parse(new StringReader(text))) {
// Has some description
!description.text().isEmpty()
}
}
}
where:
file << TestUtil.getJobFiles()
}