I'm in the phase of integrating cucumber tests for an existing application. The app interacts heavily with a backend. The simplest case being the registration of an user.
In order to make the tests repeatable, I cannot go against the (real) backend (for several reasons, one not being able to register two users with the same e-mail).
I'd like to know which options do I have to mock the backend behavior and deliver mocked responses. For unit testing I know Nocilla and several other frameworks. Integrating them of course does not work for calabash tests.
One alternative would be to have a real server running which I'd control from the calabash tests to control the networking. But that seems a bit overkill. Another option would be to have mock objects in the calabash target which I could control via the calabash backdoor mechanism.
I doubt that aren't the only options. So my question is: is there a more elegant way to do so?
As I received no other feedback until today, I'll answer my own question.
We opted for the way of creating a very simple mock for our backend. We used Sinatra, but a node.js or similar technique would have led to the same result. The mock can be controlled via a simple REST API.
In the step definitions we configure our mock backend appropriately for the running scenario. It's a bit of overhead, because the mock has to evolve with the real backend, but until today it feels like a robust solution.
Related
It seems that in Xcode there are two kinds of tests, UI-tests which test an app running in another process using Automation, and Unit tests, where small units of code are tested in isolation. I wonder if there's a middle ground somewhere...
For instance, if I run unit tests there's a full app launched in memory that I don't use, because I am instantiating the classes that I need for testing in my test case classes. (I only know it's there because I see it in the simulator window and because my test classes sometimes accidentally communicate with it using notification center.)
It would be nice though, if I could use this app for integration tests. They would be similar to UI-tests, but much faster, and I would be able to access the implementation. I could for instance do that in order to install mocks so I can simulate behaviour of remote endpoints and see what the effects of certain responses are from the view point of the user.
The question is: Is there a good (documented) way to start up an app with mocks in the XCTest context and find that middle ground between unit tests and UI-tests?
funny thing, the other day we were also having this kind of problem. I'm not sure if this info might help you.
So the short answer to your specific question is that there is no written documentation in the apple docs(AFAIK) that specifically states how to make mock classes inside your tests but it doesn't necessarily mean you cant make one. There are a lot of ways on which you can make a mock up scenario to work with your code
manual way : making protocols/interfaces to be implemented by a working class and a mock class, overriding specific methods of working classes to somehow mock results, swizzling and etc.
3rd party way : notables (OCMock, Mockito)
in the case of objective c, there's a lot of 'em 3rd parties out there but with swift there are a few out there that are still in development because of the support that swift has with its runtime library (swift is yet to fully expose its runtime library)
here's a helpful link : https://academy.realm.io/posts/tryswift-veronica-ray-real-world-mocking-swift/ (great article)
Cheers
I'm currently looking at various integration/functional/acceptance testing frameworks for iOS and they all seem to rely heavily on the UI and accessibility labels. To me a lot of these seem more based towards UI tests as opposed to integration testing. Are there any dedicated frameworks that don't rely on this, especially when working with modules/frameworks that aren't built with a UI?
Is XCTest the way to go here when testing this kind of functionality? I've seen fitnesse and this seems to be the only other option I've found so far.
So in this instance I could have something like an API framework that is purely responsible for making network requests, parsing the response and creating relevant objects. Whilst I would write unit tests to test the individual pieces of functionality, I would also like to test an end to end flow.
Are there any suitable testing frameworks for this type of thing? Preferably swift based
thanks
Belonging to the world of web development, I am used to powerful testing techniques and tools like Chai, Sinon, Mocha, Jasmine, to name a few, as well as to wide support and community, well-written books with deep insights. There are all possible cases I can imagine that are covered with existing tools.
Now, learning iOS development with Swift, I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
I find the community is not so wide in iOS development, especially, Swift, as a result of the language being not so long on the landscape.
Please share the best practices for testing that can help me to create solid predictably behaving apps.
When developing iOS apps you have two possible test types you can use, one for unit testing, and one for acceptance testing. Those are referred in Xcode respectively as "Test Target" and "UI Test Target".
Within the tests target you can test all your app code by importing its Swift module using #testable import MyAppName, and then writing unit tests using XCTest, the framework provided by Apple.
On top on the tools provided by Apple, the FOSS community has developed other great testing libraries. In particular Quick is an testing framework that provides a syntax similar to Mocha.
When it comes to the UI tests target there is an important thing to be said: you can only test your app behaviour via those. There is no way to access the application code, everything is done through proxies that simulate the user's interaction with the UI. That is, in my opinion, for a good reason. The kind of tests one should write via tools that only allow UI interaction should be acceptance/funcitonal tests, which should only verify the app's behaviour. Not having access to the code is a good way to avoid misusing this tool.
I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
Going deeper:
using stubs and mocks
Because of Swift's nature doing real mocking is quite hard. It's better to use fakes and doubles. There is also a great deal of literature online on why using mocks and stubs is seldom a good idea, for example this, this, this, and this.
imitating connecting to web services asynchronously
After having said all that it is true that sometimes stubbing is useful, such in the case you highlight. A good library to do this is OHHTTPStubs.
get myself sure tests can be written in a clear manner
I believe there are enough tools today to support writing clear and effective tests with little effort. The rest is up to you, your team, and the way you architect your app. As it is for any other programming language after all :D
If I can put a shameless plug in here, at the start of the year I wrote a post covering the state of testing in the iOS plaform, a lot of new things have happened since then, but it could be a good starting point if you want to dig deeper.
Background
I am looking for a way to implement a scheme similar to what Frank library uses to implement "Automated acceptance tests for native iOS apps", but I want this scheme rely on native iOS/MacOSX technologies. Sorry for the following TLDR but it deserves verbose explanation.
1. Here is a short overview of how Frank works:
it has Client and Server parts.
Server part is embedded into an application that we want to run acceptance tests against. Frank tutorials show us how to create a duplicate target of an app's main target and embed Frank HTTP server to it.
Client part - is a mainly a Cucumber that runs plain text scenarios: each scenario contains directives that should be run against an app (fill text field, touch button, ensure that a specific element exists on a page etc...). Also each scenario launches its own instance of app by this means providing a fresh state every time we enter a new scenario.
Client (Cucumber and Ruby-to-Objective-C bridge) communicates with Server (HTTP server embedded into an app) via HTTP protocol. It uses special conventions so client can tell server what an app should do so the specific scenario could be performed.
2. Recently I've found the following article written by the author of Frank Pete Hodgson:
http://blog.thepete.net/blog/2012/11/18/writing-ios-acceptance-tests-using-kiwi/
In which he suggests more simple way of writing acceptance tests for the developers who don't like to rely on external tools like Cucumber and Ruby. Let me quote the author himself:
Before I start, let me be clear that I personally wouldn’t use this approach to writing acceptance tests. I much prefer using a higher-level language like ruby to write these kinds of tests. The test code is way less work and way more expressive, assuming you’re comfortable in ruby. And that’s why I wanted to try this experiment. I’ve spoken to quite a few iOS developers over time who are not comfortable writing tests in ruby. They are more comfortable in Objective-C than anything else, and would like to write their tests in the same language they use for their production code. Fair enough.
This blog post inspired me to quickly roll my own very raw and primitive tool that does exactly what Pete described in his blog post: NativeAutomation.
Indeed, like it was described by Pete, it is possible to run acceptance tests by using just Kiwi/PublicAutomation setup placed in a simple OCTests target. I really liked it because:
It is just pure C/Objective-C. It was very easy to build initial bunch of C helpers, that look like Capybara helpers:
tapButtonWithTitle, fillTextFieldWithPlaceholder, hasLabelWithTitle and so on...
It does not require external tools and languages: no need in Cucumber/Ruby or anything else. NativeAutomation itself uses just PublicAutomation that Frank also uses. PublicAutomation is needed to simulate user interactions on an app's screen: touches, fills, gestures...
It is very handy to run these tests right from Xcode by just running a Cocoa Unit Tests bundle. (Though command-line builds are easy as well).
Problem
The problem Kiwi/PublicAutomation approach has is that the whole testing suite is embedded into the app's bundle. It means that after each scenario is run, it is not possible to reset application to force it to be in a fresh state before next scenario begins execution. The only way to workaround this problem is to write Kiwi's beforeEach hook with a method that performs a soft-reset of an application like:
+ (void)resetApplication {
[Session invalidateSession];
[LocationManager resetInstance];
[((NavigationController *)[UIApplication sharedApplication].delegate.window.rootViewController) popToRootViewControllerAnimated:NO];
[OHHTTPStubs removeAllStubs];
cleanDatabase();
shouldEventuallyBeTrue(^BOOL{
return FirstScreen.isCurrentScreen;
});
but in case of application which involve networking, asynchronous jobs, core data, file operations it becomes hard to perform a real teardown of a stuff left by previous scenario.
Question
The problem described above made me thinking about if it is possible to implement a more complex approach similar to Frank's approach, with a second app that works apart from main app's bundle and does not rely on external tools like Cucumber(Ruby).
Here is how I see the way it could be done.
Besides a main app (MainApp) there is a second iOS (or Mac OS X) app (AcceptanceTestsRunnerApp) that contains the whole acceptance tests suite and runs this suite against a main app bundle:
It does fire up new simulator instance before it enters each new scenario and executes current scenario against current simulator's app's instance.
The question is:
I am not well awared about Mac OSX or iOS technologies that would allow me to do that: I don't know if it is even possible to setup a Mac OS X / iOS application (AcceptanceTestsRunnerApp) that could take a control over a main app (MainApp) and run acceptance tests scenarios against it.
I will be thankful for any thoughts/suggestions that people who feel more comfortable with having native Objective-C tool for writing acceptance tests for iOS applications might have.
UPDATED LATER
...I did read some documentation about XPC services but the irony about it is that scheme I am looking for should be quite opposite to the scheme XPC documentation suggests:
Ideally I want my AcceptanceTestsRunnerApp dominate over MainApp: run it and control it (user interactions, assertions about view hierarchy) via some object proxying to MainApp's application delegate while XPC services setup would assume XPC service (AcceptanceTestsRunnerApp) to be subordinate to an app (MainApp) and will require XPC service to live inside app's bundle which I want to avoid by all means.
...My current reading is Distributed Objects Programming Topics. It seems to me that I will find my answer there. If no one provides me with guides or directions I will post an answer about my own researches and thoughts.
...This is the next step in my quest: Distributed Objects on iOS.
KIF and the port of Jasmine - Cedar to iOS work fairly well from my experience.
However, I also have had lots of good usage out of calabash, which like Frank, is powered by gherkin.
I'm trying to sift through the myriad of test solutions out there, and I'm not even sure if I'm headed the right direction. The story is: we're running a RESTful Web service, implemented as a Rails app, which backs our mobile clients. We're unit testing the Web service (of course), but that involves mocking out many parts of the application, for instance, the search stack (Apache SOLR).
Moreover, our tests don't (i.e. cannot!) cover critical routes such as the mobile sign-in/sign-on process, since that involves communication between the API application and the mobile website, where the user can enter credentials, e.g. for SSO (Janrain Engage). Hence, a standard Rails integration test won't do.
I realize that in theory, if the test suite is very well designed, where mocking only happens strictly at the those join points where the tests for the next layer start, then by unit or functional testing the service API and the mobile website separately, one could get the same test coverage. I find that in practice though, this is an illusion if you have several developers working on the test suite independently; I just admit that our unit tests are simply not that well designed. Particularly when exercising TDD I found that while the tests can drive the application code, the test code design is only tailored to the unit under test, resulting in a rather wildly grown test suite.
Another thing I found is that sometimes we didn't detect regressions purely using unit tests, where e.g. bad queries were sent to the SOLR server due to knock-on effects. That's why I thought the only true way to ensure that the entire stack works along the critical routes is to automatically end-to-end test it on a staging server before every deployment, i.e. having actual HTTP requests sent to the app.
My questions would be:
do you think this is a reasonable thing to do at all? I found very little information about end-to-end testing a live API on the Web, leaving me wondering whether I'm making any sense at all
which tools/setup would you suggest? We use Watir to run acceptance tests for our website, but it seems to be overkill for a Web service (no browser environment needed, no JS or anything UI-ish). Something as simple as a Ruby script even?
any general best practices or advice you can give me w.r.t. to designing such tests?
This might be of interest to you: http://groups.google.com/group/ruby-capybara/browse_thread/thread/5c27bf866eb7fad3
What you might want to to try is combining Cucumber (or similar) with one of the tools mentioned in the link above, so you can do something like
Given I have 2 posts
When I send "DELETE" to post with id 1
Then I should have 1 post
This way you can test the API's full stack and check the result.
We've faced similar problems lately, like testing only on the unit level was fast but not enough to spot integration related anomalies.
I really like Cucumber's gherkin syntax, the value for me in it is that we can run the exercise once and then make several expectations after with separate titles, like if they would be separate RSpec it blocks.
There is a gem called Turnip, which makes possible to write RSpec tests with the gherkin syntax. We have been using it for a while and I'm quite happy with it, we even started having less integration level defects.
I collected my experience regarding in a blog post: https://blog.kalina.tech/2019/10/end-to-end-testing-for-api-only-rails-apps-with-cucumber-syntax.html