Currently we are running around 2000 spec flow test cases
When we run spec flow we get object reference error or actual count = 0
This issue occurs for few test cases randomly
If we manually run that test case then it gets passed.
The exact root cause is not detected as it is intermitent
So in order to resolve the issue we kept retry so that if a test fails we run that test again
This issue did not occur again for some time. However now since the number of test cases have increased we are facing the similar issue again
One of the resolution is to increase retry count. However we would like to know exact root cause so that we can apply proper fix for this issue.
Related
I saw that theres an option to force a Stage to fail if it takes too much time to run, but my question is if theres an option to force it to fail if it runs less than a certain amount of time?
I need to run tests over some code that a programmer wrote, and I know that it must run for at least 3 hours in order to pass. The problem is that he wrote in the code that if theres an error in it, it fails but return success and the Stage ends in few second and marked as success.
I want to make the stage fail if it runs only few seconds.
Thanks
When running our test suite we perform a re-run which gives us 2 HTML reports at the end. What I am looking to do is have one final report so that I can then share it with stakeholders etc.
Can I merge the 2 reports so that in the first run a test had failed but in the second run it had passed, the report shows the test has passed?
I basically want to merge the reports to show a final outcome of the test run. Thanks
By only showing the report that passed you'd be throwing away a valuable piece of information: that there is an issue with the test suite making it flaky during execution. It can be something to do with the architecture or design of a particular test, or maybe the wait/sleep periods for some elements. Or, in some cases, the application we're testing has some sort of issue that a lot of times goes unchecked.
You should treat a failing report with as much respect as a passing one. I'd share with the stakeholders both reports and a short analysis of why the tests are failing in the first one(s), or why do they usually fail, and a proposal/strategy to fix the failure.
Regarding the merging of the reports, it can be done. You can, via a script that takes both reports, maybe extract the body of each, and, element by element only do a copy the passing one if the other is failing, or if both are failing, copy a failing one. But it looks like that would be an effort to hide a possible problem, and not to fix it from the ground up.
Edit:
There is at least one lib that can help you achieve this,
ReportBuilder, or the Java equivalent:
ReportBuilderJava.
We have some feature specs which fails randomly. We don't have too much time to fix them and we don't really know for now how to do this. Because of that we must rerun builds on cicrcle ci until they are green. Is it possible to run some spec, and if it fails rerun this few times, until it's green?
Try to have a look at following gems:
https://github.com/dblock/rspec-rerun
https://github.com/y310/rspec-retry
(taken from discussion in https://github.com/rspec/rspec-core/issues/456)
Personally I think having flickering tests is worse then having no tests in the first place because the are adding hassle and they destroy the trust in tests in general, which you need for swift refactoring.
Best would be
delete them since they don't provide the value they should
take your time to rewrite them
For getting the time to do so try to convince management that the investment in time on fixing these issues saves a lot of developer time in the long run (best with quick example calculation: x fails a day, result in yyy extra minutes with devs waiting for the built to be green) ;)
What's Happening
In our Rspec + Capybara + selenium (FF) test suite we're getting A LOT of inconsistent "Capybara::ElementNotFound" errors.
The problem is they only happen sometimes. Usually they won't happen locally, they'll happen on CircleCi, where I expect the machines are much beefier (and so faster)?
Also the same errors usually won't happen when the spec is run in isolation, for example by running rspec with a particular line number:42.
Bare in mind however that there is no consistency. The spec won't consistently fail.
Our current workaround - sleep
Currently the only thing we can do is to litter the specs with 'sleeps'. We add them whenever we get an error like this and it fixes it. Sometimes we have to increase the sleep times which is making out tests very slow as you can imagine.
What about capybara's default wait time?
Doesn't seem to be kicking in I imagine as the test usually fails under the allocated wait time (5 seconds currently)
Some examples of failure.
Here's a common failure:
visit "/#/things/#{#thing.id}"
find(".expand-thing").click
This will frequently result in:
Unable to find css ".expand-thing"
Now, putting a sleep in between those two lines fixes it. But a sleep is too brute force. I might put a second, but the code might only need half a second.
Ideally I'd like Capybara's wait time to kick in because then it only waits as long as it needs to, and no longer.
Final Note
I know that capybara can only do the wait thing if the selector doesn't exist on the page yet. But in the example above you'll notice I'm visiting the page and the selecting, so the element is not on the page yet, so Capybara should wait.
What's going on?
Figured this out. SO, when looking for elements on a page you have a few methods available to you:
first('.some-selector')
all('.some-selector') #returns an array of course
find('.some-selector')
.first and .all are super useful as they let you pick from non unique elements.
HOWEVER .first and .all don't seem to auto-wait for the element to be on the page.
The Fix
The fix then is to always use .find(). .find WILL honour the capybara wait time. Using .find has almost completely fixed my tests (with a few unrelated exceptions).
The gotcha of course is that you have to use more unique selectors as .find MUST only return a single element, otherwise you'll get the infamous Capybara::Ambiguous exception.
Ember works asynchronously. This is why Ember generally recommends using Qunit. They've tied in code to allow the testing to pause/resume while waiting for the asynchronous functions to return. Your best bet would be to either attempt to duplicate the pause/resume logic that's been built up for qunit, or switch to qunit.
There is a global promise used during testing you could hook up to: Ember.Test.lastPromise
Ember.Test.lastPromise.then(function(){
//continue
});
Additionally visit/click return promises, you'll need some manner of telling capybara to pause testing before the call, then resume once the promise resumes.
visit('foo').then(function(){
click('.expand-thing').then(function(){
assert('foobar');
})
})
Now that I've finished ranting, I'm realizing you're not running these tests technically from inside the browser, you're having them run through selenium, which means it's not technically in the browser (unless selenium has made some change since last I used it, possible). Either way you'll need to watch that last promise, and wait on it before you can continue, testing after an asynchronous action.
I have many cucumber feature files, each consists of many scenarios.
When run together, some of them fails.
When I run each single test file, they passes.
I think my database is not correctly clean after each scenario.
What is the correct process to determine what is causing this behavior ?
By the sound of it your tests are depening upon one another. You should be trying to get each indervidual test to do what ever set up is required for that indervidual test to run.
The set up parts should be done during the "Given" part of your features.
Personally, to stop the features from becoming verbose and to keep them close to the business language that they where written in, i sometimes add additional steps that are required to do the setup and call them from the steps that are in the feature file.
If this makes sence to you
This happens to me for different reasons and different times.
Sometimes its that a stub or mock is invoked in one scenario that screws up another, but only when they are both run (each is fine alone).
The only way I've been able to solve these is debugging while running enough tests to get a failure. You can drop the debugger line in step_definitions or call it as a step itself (When I call the debugger) and match that up to a step definition that just says 'debugger' as the ruby code.