We were used to running our grails integration test against in memory HSQLDB database, but at the failure point it was difficult to investigate as the data was lost. We migrated to running the test against the physical database(postgres) and all is well when the tests passes. At any point if the tests fail we want the data to be committed in the database for postmortem analysis as to why the test failed.
To summarize, we want the tests to run in rollback mode as long as the test passes so that one test doesn't affect the other test and on the first failure of a test, commit the data at that point and stop.
We spend considerable amount of time investigating the integration test failure and would like to know if we have any option in grails to stop at first integration test failure with data preserved in the database for investigation. I searched little and didn't find any suitable pointers. If you follow any other practice to troubleshoot integration test and if it is worth sharing please let us know.
Simple hack you could try:
set a global flag on failure, test for the flag in each test. if flag is set exit the test
Recently I came across with Grails Guard Plugin and I think it can be useful in this case, because besides running integration tests faster, it preserves the data saved after tests are run.
Related
We currently use ruby and cucumber setup. There are some steps failing in the tests(end to end regression tests) due to known bugs. Developers takes sometime to fix them according to their work load and bug severity. How best to deal with these failing tests?
Should we tag them with bug ticket numbers and let the those specific tests skip when it runs on CI?
Let them fail and mark the build unstable until the dev fixes them how many ever days they take?
Is there any other way in cucumber to say these specific tests have a different state other than pass or fail to indicate that its under control?
I managed to find a better solution to suit this need on searching further online. we could mark the test as "pending" so that it wont fail but goes yellow and indicates pending.
https://phabricator.wikimedia.org/T58243
The beauty of this is, in future if the bug is fixed and the step doesn't fail anymore, it will indicate about that so we can remove the pending status.
Expected pending 'bug jira-195' to fail. No Error was raised. No longer pending? (Cucumber::Pending exception)
I've been working on a project that I want to add automated tests. I already added some unit tests, but I'm not confident with the process that I've been using, I do not have a great experience with automated tests so I would like to ask for some advice.
The project is integrated with our web API, so it has a login process. According to the logged user the API provides a configuration file which will allow / disallow the access to some modules and permissions within the mobile application. We also have a sync process where the app will access several methods from the API to download files (PDFs, html, videos, etc) and also receive a lot of data through JSON files. The user basically doesn't have to insert data, just use the information received in the sync process.
What I did to add unit tests in this scenario so far was to simulate a logged user, then I added some fixture objects to the user and test them.
I was able to test the web service integration, I used Nocilla to return fake JSONs and assert the result. So far I was only able to test individual request, but I still don't know how should I test the sync process.
I'm having a hard time to create unit tests for my view controllers. Should I unit test just the business logic and do all the rest with tools like KIF / Calabash?
Is there an easy way to setup the fixture data and files?
Thanks!
Everybody's mileage may vary but here's what we settled on and why.
Unit tests: We use a similar strategy. Only difference is we use OHTTPStubs instead of Nocilla because we saw some more flexibility there that we needed and were happy to trade off the easier syntax of Nocilla.
Doing more complicated (non-single query) test cases quickly lost its luster because we were essentially rebuilding whole HTTP request/response flows and that wasn't very "unit". For functional tests, we did end up adopting KIF (at least for dev focused efforts, assuming you don't have a seaparte QA department) for a few reasons:
We didn't buy/need the multi-language abstraction layer afforded by
Appium.
We wanted to be able to run tests on many devices per
build server.
We wanted more whitebox testing and while
Subliminal was attractive, we didn't want to build hooks in our main
app code.
Testing view controller logic (anything that's not unit-oriented) is definitely much more useful using KIF/Calbash or something similar so that's the path I would suggest.
For bonus points, here are some other things we did. Goes to show what you could do I guess:
We have a basic proof of concept that binds KIF commands to a JSON RPC server. So you can run a test target on a device and have that device respond to HTTP requests, which will then fire off test cases or KIF commands. One of the advantage of this is that you can reuse some of the test code you wrote for single device for multiple device test cases.
Our CI server builds integration tests as a downstream build of our main build (which includes unit tests). When the build starts we use XCTool to precompile tests, and then we have some scripts that starts recording a quicktime screen recording, runs the KIF tests, exports the result, and then archive it on our CI server so we can see a live test run along with test logs.
Not really part of this answer but happy to share that if you ping me.
I've been getting my feet wet with Geb on Grails, but there's not a lot of documentation about how it behaves. For instance, how does geb handle rollbacks? From what I'm observing, it runs the app and runs the test on the browser itself without turning it off in between tests.
What happens to the database data when one spec (spec A) alters an object (object Z), and a few tests later, another spec(spec B) alters the same object? Does geb rollback the database to it's virgin state every time a spec is ran? I'm trying to confirm because I have geb tests that ran fine when executed individually, but when I ran them as a suite, some of them fails, and the best reason I could come up with is the data isn't in pristine condition when a second test was run on it. Any thoughts?
Geb tests and functional tests in general are quite different from unit and integration tests. Unit and integration tests run in the same JVM, and the test runner starts a transaction before each test and rolls it back after the test runs, which has the effect of resetting the database but in reality it's just keeping tests from changing the database. But any data inserted into the database before the tests start (e.g. from BootStrap) will be there for each test.
But functional tests are typically run in one JVM but they make remote calls to your app running in a second JVM. This limits what you can do during tests, for example you can't manipulate metaclasses, or change Spring bean instance variables, and you can't start and rollback transactions to isolate data changes between tests. You can do any of those things, but they would affect the local JVM only.
Geb could remotely make these changes of course, but that would require modifying your application to add a controller or some other way of making remote calls, but it doesn't.
In general tests shouldn't be ordered and should be independent, but I find that when doing functional tests that it makes sense to break that rule and order them, where an earlier test does some inserts or other changes and later tests do further work and/or checks based on earlier changes. I've also added test-only controller actions that can be used to roll back changes (via a transaction or by deleting inserted data, undoing updates and deletes, etc.) and make other changes to assist the tests, but this has to be done carefully to ensure that it is only available during testing and doesn't become a significant security risk.
I am building a educational service, and it is a rather process heavy application. I have a lot of user actions triggering various actions, some present, some future.
For example, when a student completes a lesson for his day, the following should happen:
updating progress count for his user-module record
checking if he has completed a particular module and progressing him to the next one (which in turn triggers more actions)
triggering current emails to other users
triggering FUTURE emails to himself (ongoing lesson plans)
creating range of other objects (grading todos by teachers)
any other special case events
All these triggers are built into the observers of various objects, and the execution delayed using Sidekiq.
What is killing me is the testing, and the paranoia that I might breaking something whenever I push something. In the past, I do a lot of assertion and validations checks, and they were sufficient. For this project, I think this is not enough, given the elevated complexity.
As such, I would like to implement a testing framework, but after reading through the various options (Rspec, Cucumber), it is not immediately clear what I should be investing my effort into, given my rather specific needs, especially for the observers and scheduled events.
Any advice and tips on what approach and framework would be the most appropriate? Would probably save my ass in the very near future ;)
Not that it matters, but I am using Rails 3.2 / Mongoid. Happy to upgrade if it works.
Testing can be a very subjective topic, with different approaches depending on the problems at hand.
I would say that given your primary need for testing end-to-end processes (often referred to as acceptance testing), you should definitely checkout something like cucumber or steak. Both allow you to drive a headless browser and run through your processes. This kind of testing will catch any big show stoppers and allow you to modify the system and be notified of breaks caused by your changes.
Unit testing, although very important, and should always be used in parallel with acceptance tests, isn't for doing end-to-end testing, Its primarily for testing the output of specific methods in isolation
A common pattern to use is called Test Driven Development (TDD). In this, you write your acceptance tests first, in the "outer" test loop, and then code your app with Unit tests as part of the "inner" test loop. The idea being, when you've finished the code in the inner loop, then the outer loop should also pass, and you should have built up enough test coverage to have confidence that any future changes to the code will either pass/fail the test depending on if the original requirements are still met.
And lastly, a test suite is something that should grow and change as your app does. You may find that whole sections of your test suite can (and maybe should) be rewritten depending on how the requirements of the system change.
Unit Testing is a must. you can use Rspec or TestUnit for that. It will give you atleast 80% confidence.
Enable "render views" for controller specs. You will catch syntax errors and simple logical errors faster that way.There are ways to test sidekiq jobs. Have a look at this.
Once you are confident that you have enough unit tests, you can start looking into using cucumber/capybara or rspec/capybara for feature testing.
We have a service to pick up custom tests in XML and convert those to CodedUI tests. We then start a process for MSTest to load the tests into the Test Controller which then distributes the tests across various Agents. We run regression tests at night, so no one is around to fix a system if something goes wrong. When certain exceptions occur in the test program, it pops open an error window and no more test can run on the system. Subsequent tests are loaded into the agent and fail immediately because they can not perform their assigned tasks. Thousands of tests that should take all night on multiple systems now fail in minutes.
We can detect that an error occurred by how quickly a test is returned, but we don't know how to disable the agent so as not to allow it to pick up any more tests.
addendum:
If the test has failed so miserably that no more tests can attempt a successful run (as noted, we may not have an action to handle some, likely new, popup), then we want to disable that agent as no more tests need to run on it: they will all fail. As we have many agents running concurrently, if one fails (and gets disabled), the load can still be distributed without a long string of failures. These other regression tests can still have a chance to succeed (everything works) or fail (did we miss another popup, or is this an actual regression failure).
2000 failures in 20 seconds doesn't say anything except 1 system had an problem that no one realized it would have and now we wasted a whole night of tests. 2 failures (1 natural, 1 caused by issue from previous failure) and 1 system down means the total nights run might be extended by an hour or two and we have useful data on how to start the day: fix 1 test and rerun both failures.
One would need to abort the testrun in that case. If you are running mstest yourself, you would need to inject a ^c into the command line process. But: if no-one is around to fix it, why does it matter that the consequenting test fail ? if its just to see which test was the cause of the error quickly, why not generate a code ui check to see if the message box is there and mark the test inconclusive with Assert.inconclusive. The causing test would stand out like a flag.
If you can detect the point at which you want to disable the agent then you can disable the agent by running the "TestAgentConfig.exe delete" which will reset the agent to unconfigured state.