I have a scenario written as below in specflow.
#tag
Scenario: Test scenario 1
Step in Gherkin
Step in Gherkin
Step in Gherkin
#tag
Scenario: Test scenario 2
Step in Gherkin
Step in Gherkin
Step in Gherkin
#tag
Scenario: Test scenario 2
Step in Gherkin
Step in Gherkin
Step in Gherkin
I also have below hooks for the above feature and scenarios.
[BeforeScenario("tag")]
public static void BeforeS()
{
}
[BeforeFeature("tag")]
public static void BeforeF()
{
}
[BeforeTestRun("tag")]
public static void BeforeTR()
{
}
[AfterScenario("tag")]
public static void AfterS()
{
}
[AfterFeature("tag")]
public static void AfterF()
{
}
[AfterTestRun("tag")]
public static void AfterTR()
{
}
When I run all the tests together if 1st test case fails, how I do I start the next test all from the beginning?
Each test scenario is independent and it will execute seperately. It means even if one of your scenarios failed other scenarios will execute fine.
Related
For the reasons I cannot influence, there is the following mechanism on TestNG launching on the project.
In a word, it creates a new instance of TestNG, adds listeners, specifies classes and runs the tests. Then, all this dirty code is called from Gradle run task (which is actually empty and as far as I understood, simply calls the TestManager.main() method).
I removed the part of code just to show the main direction:
class TestManager {
static void main(String[] args) {
try {
runTests(args[0])
} catch (Exception e) {
e.printStackTrace()
System.in.read()
}
}
private static void runTests(Application app) {
TagsConfig.runs.each { run ->
if (run.execute) {
List<TagsSuite> suites = TagsConfig.suites
suites.each { suite ->
if (suite.execute) {
Reflections reflections = new Reflections("${app.packageName}.${suite.name}")
def classes = reflections.getSubTypesOf(${suite.name})
if (classes.size() > 0) {
TestNG testNG = new TestNG()
testNG.testClasses = classes
testNG.groupByInstances = true
testNG.outputDirectory = "testng-output"
testNG.addListener(new TestListenerAdapter())
testNG.addListener(new ExceptionListener())
testNG.addListener(new AllureTestListener())
if (TagsConfig.isSmoke) {
testNG.setGroups("smoke")
} else if (TagsConfig.isExtendedSmoke) {
testNG.setGroups("extended_smoke")
}
testNG.run()
}
}
}
}
}
}
So the test launch looks like this:
gradle clean
gradle run
I can not change the way the tests are started now and one of the problems is that the build in Jenkins is always Sucessful even if there are failed/broken/skipped tests.
So, I can get the number of failed tests from TestListenerAdapter, but how can I let Jenkins know that there were failed tests?
Maybe by returning an exit code from a Gradle run task or by installing some plugin that will check the count of failed tests in TestListenerAdapter?
For now, I'm setting a "FAILED_TESTS_COUNT" system property in onFinish() event and change the build status in pipeline if it is not 0, but this looks really dirty.
Jenkins 2.89.3
Gradle 3.5.1
TestNG 6.9.8
I personally use the Jenkins Text Finder post-build action. In the above example, it looks for a pattern in the standard output and set the build to "unstable" (orange) when it's found.
In your case, just uncheck both checkboxes and find the according matching pattern.
I'm tryting to run our Cucumber JVM tests by few threads in parallel.
I'm using standart TastNG approach to do it (via suite XML file)
My xml file is:
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="BDD" parallel="methods" thread-count="3" data-provider-thread-count="3">
<test name="BDD">
<classes>
<class name="com.tests.bdd.SimpleBDDTests"></class>
</classes>
</test>
</suite>
My test class is:
#CucumberOptions(features = "src/test/java/com/tests/bdd/simpleFeatures")
public class SimpleBDDTests {
private TestNGCucumberRunner tcr;
#BeforeClass(alwaysRun = true)
public void beforeClass() throws Exception {
tcr = new TestNGCucumberRunner(this.getClass());
}
#AfterClass(alwaysRun = true)
public void afterClass() {
tcr.finish();
}
#Test(dataProvider = "features")
public void feature(CucumberFeatureWrapper cucumberFeature) {
tcr.runCucumber(cucumberFeature.getCucumberFeature());
}
#DataProvider(parallel = true)
public Object[][] features() {
return tcr.provideFeatures();
}}
My feature files are like:
Feature: First test
#sanity
Scenario: First simple test
Given Base check step
I have 4 feature files, which are defines the same scenarios with only one step - Given Base check step
When these features are executed one by one, it works fine, but when i try to run them in parallel, everything gets broken.
Almost all of these featres marked as failed with the following exception:
A scoping block is already in progress
java.lang.IllegalStateException: A scoping block is already in progress
at cucumber.runtime.java.guice.impl.SequentialScenarioScope.checkState(SequentialScenarioScope.java:64)
at cucumber.runtime.java.guice.impl.SequentialScenarioScope.enterScope(SequentialScenarioScope.java:52)
at cucumber.runtime.java.guice.impl.GuiceFactory.start(GuiceFactory.java:34)
at cucumber.runtime.java.JavaBackend.buildWorld(JavaBackend.java:123)
at cucumber.runtime.Runtime.buildBackendWorlds(Runtime.java:141)
at cucumber.runtime.model.CucumberScenario.run(CucumberScenario.java:38)
at cucumber.runtime.model.CucumberFeature.run(CucumberFeature.java:165)
at cucumber.api.testng.TestNGCucumberRunner.runCucumber(TestNGCucumberRunner.java:63)
I understand that it might be happen because of multi-thread calls to the same step - Given Base check step
So my question is how can i fix that? How can i run these tests in parallel?
PS: I know that it should be possible to do it by JUnit + Maven surefire plugin, but it is not applicable for current project, we need to achieve that goal by TestNG.
Thanks.
I have a test integration test as shown below.
package bibs
import static org.junit.Assert.*
import groovy.util.GroovyTestCase;
import org.junit.*
class BibsServiceTests extends GroovyTestCase{
BibsService service
#Before
void setUp() {
// Setup logic here
}
#After
void tearDown() {
// Tear down logic here
}
#Test
void testSomething() {
assertEquals service.convertRangeStringToRangeList("asd"), "asd"
}
}
When i run the integration test, the error i get is
Running 1 integration test... 1 of 1
| Failure: testSomething(bibs.BibsServiceTests)
| java.lang.NullPointerException: Cannot invoke method convertRangeStringToRangeList() on null object
at bibs.BibsServiceTests.testSomething(BibsServiceTests.groovy:30)
| Completed 1 integration test, 1 failed in 292ms
| Tests FAILED - view reports in
If i change the name of the service to something other than service like
BibsService bibsService
Then the test passes. I am wondering why this is the case? Thanks!
I think you have some confusion between Unit and Integration tests.
When writing a unit test against a service (e.g. BibsService) you can use the #grails.test.mixin.TestFor which injects the service into the test under the variable "service".
Example:
import grails.test.mixin.TestFor
#TestFor(BibsService)
BibsServiceSpec extends spock.lang.Specification {
void "Test service exists"() {
expect:
null != service
}
}
When writing an Integration test, Grails uses spring injection to make the service beans available. This means you need the name of the service bean (e.g. bibsService) when using it in an integration test. For Grails services this is typically the first character lower-cased followed by the rest of the name.
Example
BibsServiceSpec extends spock.lang.Specification {
// Injected via spring.
BibsService bibsService
... Do test here ...
}
I'm not able to run an integration test in IntelliJ IDEA. Here is a test template generated by grails create-integration-test
#Integration
#Rollback
class TestServiceIntSpec extends Specification{
void "test something"() {
//some code here
}
}
Here is the output when I'm trying to run it from junit configuration :
Process finished with exit code 0
Empty test suite.
Also seems like grails using development env if I'm running this test from IDE, I have to specify env explicitly via -Dgrails.env=test
Spock tests ('Specification') identify which methods are tests by the presence of when:, then:, or expect:, etc.
HypeMK's answer is correct. To elaborate, the following test may not run because it does not have the presence of the spock keywords that outline the specification nature of the test (expect, when, then, etc):
#TestFor(BeanFormTagLib)
class BeanFormTagLibSpec extends Specification {
def setup() {}
void "address setup"() {
assertOutputEquals ('Hello World', '<g:beanFormTagLib domainName="com.myapp.Address" />');
}
}
Here we correct the issue by adding the "expect" keyword:
#TestFor(BeanFormTagLib)
class BeanFormTagLibSpec extends Specification {
def setup() {}
void "address setup"() {
expect:
assertOutputEquals ('Hello World', '<g:beanFormTagLib domainName="com.myapp.Address" />');
}
}
I have a Grails 2.3.11 project using the Quartz plugin, with a job defined in grails-app/jobs/mypkg
package mypkg
class DoStuffJob {
static triggers = {
simple repeatInterval: 5000l // execute job once in 5 seconds
}
def execute() {
/* do stuff */
}
}
To test this job, I created an integration test in test/integration
package mypkg
...
#TestMixin(IntegrationTestMixin)
class DoStuffJobTests {
def doStuffJob
#Before
void setup() {
doStuffJob = new DoStuffJob()
}
#After
void cleanup() {
}
#Test
void "DoStuffJob"() {
doStufFJob.execute()
}
}
This test runs correctly and passes, but the Spring Tool Suite IDE has an error "Groovy:unable to resolve class DoStuffJob" in my test's setup() method.
This causes STS to prompt me about errors in the workspace whenever I launch the project, and also causes ugly red X's in Project Explorer.
Why is STS showing this error, and how do I fix it?
Add grails-app/jobs as a source folder by right-clicking on it and selecting Build Path | Use as Source Folder.