Let me start by stating I am very wet behind the ear with gherkins and cucumber.
I've put together a PoC for my company of an integration a Jenkins projects that will build and execute tests when there is a check in a Git repository. When the tests have completed Jenkins will then update the test managed in Xray for Jira.
The tests are cucumber written using gherkins. I have in vain attempted to cause a single test to produce a failure just to be able to add that to the demo I am going to be giving to upper management.
Here is the contents of my file HelloWorld.feature:
Feature: First Hello World
#firsth #hello #XT-93
Scenario Outline: First Hello World
Given I have "<task>" task
And Step from "<scenario>" in "<file>" feature file
When I attempt to solve it
Then I surely succeed
Examples:
| task | scenario | file |
| first | First Hello | First Feature |
Currently all the tests I have pass. I have attempted to modify that test so that it would fail but thus far have only been able to get it to show in Xray as EXECUTING or TO DO.
I have searched to see if it was possible to create a test that would always result in a test failure but have not been able to find anything.
I know do not know gherkins, I'm only using what was given to me to work with, so please forgive my question.
Thank you for any guidance anyone might be able to provide.
Cucumber assumes a step passes if no exception is thrown. Causing a test to fail is easy. Just throw an exception in the step definition.
Most unit testing frameworks give you an explicit way to fail a test. You haven't mentioned the tech stack in use, but MS Test for .NET gives you Assert.Fail("reason for failure goes here.");
Or simply throw an explicit exception: throw new Exception("fail test on purpose");
As long as the step throws an exception the entire scenario should fail.
Related
Here is the Sample feature file, has #ignore examples.
#ChildTest
Scenario: Sub Report
Given I have clicked on EmpId: '<EmpId>' to view Report
When Loading mask is hidden
Then I have clicked on 'Back to Results' link.
#ignore
Examples:
| EmpId | Date |
| CHILD_TEST_SKIPPED | dynamic |
I would like TestGenerator to AVOID Unit test method generation for #ignore examples
You cannot get the test generator to ignore those tests. SpecFlow tags become [Test category("ignore")] attributes above the test methods that get generated.
You will need to filter out those tests in Test Explorer. Enter -trait:ignore in the Test Explorer search bar to exclude those tests.
An alternative is to set the test to "pending":
Scenario: ...
Given pending
And in the step definition for Given pending call: Assert.Inconclusive("This test is temporarily disabled.");
Then the tests get executed, but report that they are neither passing nor failing. I do this quite a bit when implementing new features so I can write the tests ahead of time.
I am using Spec Runner to run my test cases, the scenrios are getting called twice.
What might be the issue?
Please find the below scenario and test results attached
ScenarioTestResults
Are they failing scenarios? In the standard configuration SpecFlow+Runner retries failing tests.
To disable the retry of a scenario, you have to set the retryCount parameter in the execution element to 0. See http://www.specflow.org/plus/documentation/SpecFlowPlus-Runner-Profiles/#Execution
Full disclosure: I am one of the developers of the SpecFlow+Runner.
I have a weird situation with Jenkins... We've just started using Gradle for a project at my job and when I run the tests locally with JUnit everything is fine. But when these tests are run by jenkins for the builds of branch "A", only one test fails because of an assert(always the same test).
org.junit.ComparisonFailure: expected: "E[ZZ0530]Z" but was:"E[SY5654]Z"
It looks like the mock is not injected or the mock is ignoring the "when" mocking statement.
Here is the test :
#Test
public void testEvent() {
Date eventDateTime = TimeUtils.parseDate("2013-05-30 00:00:00");
event.setEventDatetime(eventDateTime);
//Mocking the prefix return
Mockito.when(eventCodeHelperMock.getEventCodePrefixFromEvent(event)).thenReturn("EZZ");
//Tested methode
eventWrapper.setSuffix("Z");
// Event code = prefix + date + suffix
assertEquals("EZZ0530Z", event.getEventCode());
}
What is a lot stranger is that when I create a branch "B" from branch "A" all the tests succeeds when the build is created on jenkins.
I've made some research and tried to force an other build, wipe out the current workspace and recreating the job but it didn't work.
Thanks for your help!
I have had similar problems in the past and it has been due to the order in which the junits tests are run. For example, one test modifies the state of an object but you dont see the effects of this till the tests run in a different order, and tests unexpectedly fail. There is not sufficient code in the question you have posted to tell whether this is definitely the case, but I would recommend checking the order in which the tests are being run, and also look at the objects that you are using to determine if there is a problem with the state of those objects being 'dirtied'.
Hello i am new to jbehave and thucydides and the issue i am facing is all the steps are executed in the .java file but it only skips the #when step due to which my test gets skipped. I tried several options but it always marks when as pending when i run the test.
After executing tests case, check your console or report file for story/step error annotation.
Tests that contain no steps are considered to be pending. If one of steps (the "given-when-then" structure) gets PENDING while executing, then whole test gets labelled as SKIPPED. http://www.wakaleo.com/thucydides-one-page/thucydides.html#_defining_high_level_tests_in_junit - 6.2.1
From my experience, most pending steps ("given-when-then") are from bad spelling of a step name/title. Step from .story file and implementation file of a story (your_story.java depending on language) are different. Like "xx yy" =/= "xx yv"
From time to time I run into the issue that Grails integration tests the name of which ends in "IntegrationTests" don't work coming up with exceptions that show that GORM methods have not been added to domain classes. After renaming those tests to "*IntegrationTest" (no s at the end) they work fine.
A short example:
class MyIntegrationTests {
#Test
void myTest() {
assert MyDomainClass.count() == 0
}
}
Will fail with the following exception:
Failure: myTest(de.myproject.MyIntegrationTests)
groovy.lang.MissingMethodException: No signature of method: de.myproject.MyDomainClass.count() is applicable for argument types: () values: []
Possible solutions: count(), ident(), print(java.io.PrintWriter), print(java.lang.Object), getCount(), wait()
at de.myproject.MyIntegrationTests.myTest(MyIntegrationTests.groovy:9)
After renaming MyIntegrationTests to MyIntegrationTest the test passes.
Is there some kind of magic happening according to the test's name? All I found in Grails documentation is: "Tests can also use the suffix of Test instead of Tests. " Any ideas?
I eventually found the cause for the different behaviour of "*Test" and "*Tests" myself: Different postfixes change the order in which the tests are being run. To make things worse, the exact order is platform-dependent. Thus, my tests ran locally (OSX) in a different order than on my CI machine (Linux), and thereby produced different results.
Why the exception occurrs in some order is a totally different problem, though, which I haven't figured out (yet).
This should work how you had it originally as long as the file is in the integration folder. Are you sure you didn't have it in the unit test folder and then move it into the integration folder on the rename? Or possibly that you're using intellij and you did a "junit" test run rather than a "grails" one?
The error you're getting seems to imply that grails didn't start up when running your test.
Your test won't be executed if it does not has the suffix Tests.
Copied From Grails documentation home page (http://grails.org/doc/latest/guide/testing.html):
The default class name suffix is Tests but as of Grails 1.2.2, the
suffix of Test is also supported.
j-