Belonging to the world of web development, I am used to powerful testing techniques and tools like Chai, Sinon, Mocha, Jasmine, to name a few, as well as to wide support and community, well-written books with deep insights. There are all possible cases I can imagine that are covered with existing tools.
Now, learning iOS development with Swift, I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
I find the community is not so wide in iOS development, especially, Swift, as a result of the language being not so long on the landscape.
Please share the best practices for testing that can help me to create solid predictably behaving apps.
When developing iOS apps you have two possible test types you can use, one for unit testing, and one for acceptance testing. Those are referred in Xcode respectively as "Test Target" and "UI Test Target".
Within the tests target you can test all your app code by importing its Swift module using #testable import MyAppName, and then writing unit tests using XCTest, the framework provided by Apple.
On top on the tools provided by Apple, the FOSS community has developed other great testing libraries. In particular Quick is an testing framework that provides a syntax similar to Mocha.
When it comes to the UI tests target there is an important thing to be said: you can only test your app behaviour via those. There is no way to access the application code, everything is done through proxies that simulate the user's interaction with the UI. That is, in my opinion, for a good reason. The kind of tests one should write via tools that only allow UI interaction should be acceptance/funcitonal tests, which should only verify the app's behaviour. Not having access to the code is a good way to avoid misusing this tool.
I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
Going deeper:
using stubs and mocks
Because of Swift's nature doing real mocking is quite hard. It's better to use fakes and doubles. There is also a great deal of literature online on why using mocks and stubs is seldom a good idea, for example this, this, this, and this.
imitating connecting to web services asynchronously
After having said all that it is true that sometimes stubbing is useful, such in the case you highlight. A good library to do this is OHHTTPStubs.
get myself sure tests can be written in a clear manner
I believe there are enough tools today to support writing clear and effective tests with little effort. The rest is up to you, your team, and the way you architect your app. As it is for any other programming language after all :D
If I can put a shameless plug in here, at the start of the year I wrote a post covering the state of testing in the iOS plaform, a lot of new things have happened since then, but it could be a good starting point if you want to dig deeper.
Related
It seems that in Xcode there are two kinds of tests, UI-tests which test an app running in another process using Automation, and Unit tests, where small units of code are tested in isolation. I wonder if there's a middle ground somewhere...
For instance, if I run unit tests there's a full app launched in memory that I don't use, because I am instantiating the classes that I need for testing in my test case classes. (I only know it's there because I see it in the simulator window and because my test classes sometimes accidentally communicate with it using notification center.)
It would be nice though, if I could use this app for integration tests. They would be similar to UI-tests, but much faster, and I would be able to access the implementation. I could for instance do that in order to install mocks so I can simulate behaviour of remote endpoints and see what the effects of certain responses are from the view point of the user.
The question is: Is there a good (documented) way to start up an app with mocks in the XCTest context and find that middle ground between unit tests and UI-tests?
funny thing, the other day we were also having this kind of problem. I'm not sure if this info might help you.
So the short answer to your specific question is that there is no written documentation in the apple docs(AFAIK) that specifically states how to make mock classes inside your tests but it doesn't necessarily mean you cant make one. There are a lot of ways on which you can make a mock up scenario to work with your code
manual way : making protocols/interfaces to be implemented by a working class and a mock class, overriding specific methods of working classes to somehow mock results, swizzling and etc.
3rd party way : notables (OCMock, Mockito)
in the case of objective c, there's a lot of 'em 3rd parties out there but with swift there are a few out there that are still in development because of the support that swift has with its runtime library (swift is yet to fully expose its runtime library)
here's a helpful link : https://academy.realm.io/posts/tryswift-veronica-ray-real-world-mocking-swift/ (great article)
Cheers
I'm currently looking at various integration/functional/acceptance testing frameworks for iOS and they all seem to rely heavily on the UI and accessibility labels. To me a lot of these seem more based towards UI tests as opposed to integration testing. Are there any dedicated frameworks that don't rely on this, especially when working with modules/frameworks that aren't built with a UI?
Is XCTest the way to go here when testing this kind of functionality? I've seen fitnesse and this seems to be the only other option I've found so far.
So in this instance I could have something like an API framework that is purely responsible for making network requests, parsing the response and creating relevant objects. Whilst I would write unit tests to test the individual pieces of functionality, I would also like to test an end to end flow.
Are there any suitable testing frameworks for this type of thing? Preferably swift based
thanks
We would like to integrate a tests suite in our iOS app. To have an overall idea, this app is using web services, is saving files on the device and has some complex navigation parts. The requirements:
- run the tests suite with Jenkins
- being able to launch the test suite on a set of devices and iOS versions
- we don't expressly need something readable by non devs like Calabash proposes
We were thinking about doing some workflow testing (interaction bringing from one screen to another) and unit testing.
We googled a bit but articles are often at least 2 years old which is like the Paleolithic period in the mobile world. But still, it gives some nice first inputs.
For Unit Testing, we were thinking about:
GHUnit
XCTest
Kiwi
For workflow testing, we were thinking about:
Zucchini
Calabash
Regarding all the mentionned requirements, does anybody see any advice to chose one framework or the other (or another proposition)?
Thank you for any feedback.
ps: by the way, some interesting articles we have found on the subject that can help:
http://blog.lesspainful.com/2012/03/07/Calabash-iOS/
http://iosunittesting.com/faq/
I recommend Kiwi for unit testing for several reasons:
It supports nested contexts that can each have their own setup and teardown blocks. This allows your tests to be more DRY. XCTest only has one setup and teardown method which is used for all the tests in a file. With Kiwi's nested contexts, you can have setup code that is performed before some but not all of your tests based on how you define the contexts.
Kiwi has great support for mocking/stubbing dependencies when needed. I have found it to be more robust than OCMock (mocked class methods with OCMock stuck around after the test if you forgot to call -stopMocking whereas Kiwi always makes sure to teardown mocked class methods after each test).
I am not experienced in workflow/application testing but I plan to eventually check out KIF and UIAutomation.
I would highly recommend looking at iOS Tests/Specs TDD/BDD and Integration & Acceptance Testing for a more comprehensive discussion of testing libraries.
Background
I am looking for a way to implement a scheme similar to what Frank library uses to implement "Automated acceptance tests for native iOS apps", but I want this scheme rely on native iOS/MacOSX technologies. Sorry for the following TLDR but it deserves verbose explanation.
1. Here is a short overview of how Frank works:
it has Client and Server parts.
Server part is embedded into an application that we want to run acceptance tests against. Frank tutorials show us how to create a duplicate target of an app's main target and embed Frank HTTP server to it.
Client part - is a mainly a Cucumber that runs plain text scenarios: each scenario contains directives that should be run against an app (fill text field, touch button, ensure that a specific element exists on a page etc...). Also each scenario launches its own instance of app by this means providing a fresh state every time we enter a new scenario.
Client (Cucumber and Ruby-to-Objective-C bridge) communicates with Server (HTTP server embedded into an app) via HTTP protocol. It uses special conventions so client can tell server what an app should do so the specific scenario could be performed.
2. Recently I've found the following article written by the author of Frank Pete Hodgson:
http://blog.thepete.net/blog/2012/11/18/writing-ios-acceptance-tests-using-kiwi/
In which he suggests more simple way of writing acceptance tests for the developers who don't like to rely on external tools like Cucumber and Ruby. Let me quote the author himself:
Before I start, let me be clear that I personally wouldn’t use this approach to writing acceptance tests. I much prefer using a higher-level language like ruby to write these kinds of tests. The test code is way less work and way more expressive, assuming you’re comfortable in ruby. And that’s why I wanted to try this experiment. I’ve spoken to quite a few iOS developers over time who are not comfortable writing tests in ruby. They are more comfortable in Objective-C than anything else, and would like to write their tests in the same language they use for their production code. Fair enough.
This blog post inspired me to quickly roll my own very raw and primitive tool that does exactly what Pete described in his blog post: NativeAutomation.
Indeed, like it was described by Pete, it is possible to run acceptance tests by using just Kiwi/PublicAutomation setup placed in a simple OCTests target. I really liked it because:
It is just pure C/Objective-C. It was very easy to build initial bunch of C helpers, that look like Capybara helpers:
tapButtonWithTitle, fillTextFieldWithPlaceholder, hasLabelWithTitle and so on...
It does not require external tools and languages: no need in Cucumber/Ruby or anything else. NativeAutomation itself uses just PublicAutomation that Frank also uses. PublicAutomation is needed to simulate user interactions on an app's screen: touches, fills, gestures...
It is very handy to run these tests right from Xcode by just running a Cocoa Unit Tests bundle. (Though command-line builds are easy as well).
Problem
The problem Kiwi/PublicAutomation approach has is that the whole testing suite is embedded into the app's bundle. It means that after each scenario is run, it is not possible to reset application to force it to be in a fresh state before next scenario begins execution. The only way to workaround this problem is to write Kiwi's beforeEach hook with a method that performs a soft-reset of an application like:
+ (void)resetApplication {
[Session invalidateSession];
[LocationManager resetInstance];
[((NavigationController *)[UIApplication sharedApplication].delegate.window.rootViewController) popToRootViewControllerAnimated:NO];
[OHHTTPStubs removeAllStubs];
cleanDatabase();
shouldEventuallyBeTrue(^BOOL{
return FirstScreen.isCurrentScreen;
});
but in case of application which involve networking, asynchronous jobs, core data, file operations it becomes hard to perform a real teardown of a stuff left by previous scenario.
Question
The problem described above made me thinking about if it is possible to implement a more complex approach similar to Frank's approach, with a second app that works apart from main app's bundle and does not rely on external tools like Cucumber(Ruby).
Here is how I see the way it could be done.
Besides a main app (MainApp) there is a second iOS (or Mac OS X) app (AcceptanceTestsRunnerApp) that contains the whole acceptance tests suite and runs this suite against a main app bundle:
It does fire up new simulator instance before it enters each new scenario and executes current scenario against current simulator's app's instance.
The question is:
I am not well awared about Mac OSX or iOS technologies that would allow me to do that: I don't know if it is even possible to setup a Mac OS X / iOS application (AcceptanceTestsRunnerApp) that could take a control over a main app (MainApp) and run acceptance tests scenarios against it.
I will be thankful for any thoughts/suggestions that people who feel more comfortable with having native Objective-C tool for writing acceptance tests for iOS applications might have.
UPDATED LATER
...I did read some documentation about XPC services but the irony about it is that scheme I am looking for should be quite opposite to the scheme XPC documentation suggests:
Ideally I want my AcceptanceTestsRunnerApp dominate over MainApp: run it and control it (user interactions, assertions about view hierarchy) via some object proxying to MainApp's application delegate while XPC services setup would assume XPC service (AcceptanceTestsRunnerApp) to be subordinate to an app (MainApp) and will require XPC service to live inside app's bundle which I want to avoid by all means.
...My current reading is Distributed Objects Programming Topics. It seems to me that I will find my answer there. If no one provides me with guides or directions I will post an answer about my own researches and thoughts.
...This is the next step in my quest: Distributed Objects on iOS.
KIF and the port of Jasmine - Cedar to iOS work fairly well from my experience.
However, I also have had lots of good usage out of calabash, which like Frank, is powered by gherkin.
I want to do unit testing in iOS. By default test classes are created using XCTest framework classes in Xcode 5. I have used SenTestingKit in earlier versions of iOS. What are the differences between these two frameworks? What are the advantages of XCTest framework. I googled for related documentation but I did not find any. Only thing I found is the interface classes to use inside the kit. Can anyone point me to the related resources.
Apple's documentation is notably lacking with regards to testing. It's a shame, because spending five minutes with XCTest made me fall for it. So, here are some docs that you may find useful.
First, stop and watch the WWDC video about testing: https://developer.apple.com/wwdc/videos/?include=409#409 (must be a registered developer)
Then, read about Bots. They're the real magic in XCTest. I'm currently setting up an environment of a Mac Mini running Mavericks Server connected to a small array of iPod Touches, iPhones and iPads that will simultaneously run my unit tests and report back. It's damn near magic.
https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/000-About_Continuous_Integration/about_continuous_integration.html#//apple_ref/doc/uid/TP40013292
All that said, I don't know SenTest well enough to compare the two. But I am learning XCTest, so here are a few thoughts on what I'm really digging about it so far.
It's already there. There's nothing to download, and you can make your test class objects at the same time as you make that which you wish to test. Rad.
Because it's integrated so deeply with Xcode, you can see test coverage in the gutter. You can also use Companion View to see the testers from source, or the source from tester methods. Also, it will always be up to date - no waiting to fix integration with Xcode because Apple "broke" something.
Continuous Integration. The bots are just magical, and they're so easy. You also get Git hosting, which is a small fringe perk, but more so if you prefer your Git internally hosted but don't want to pay for GitHub Enterprise.
And, a few cons for measure:
Lack of Documentation. Asking me to read the headers for a list of assertions is a ballsy move. I have no doubt Apple will fix this soon, but not soon enough.
You are tied to Xcode a little more deeply. Not really a con for development, but scripting XCTest isn't as elegant as other options may be.
Continuous Integration requires OS X Server. It can be on the same machine if you really want, but it's $20 to buy, even if you owned Mountain Lion Server. I am not at all bitter about this one (ha).
My two cents. Hope it helps. I'd really recommend watching that video I linked to - they do live test creation and make a really compelling case for unit testing in general, regardless of if you use XCTest.
XCTest are included by default with ios7 and integrate better with the new continuous integration functionality.
https://developer.apple.com/technologies/tools/
It seems the advantages are presently few, but that will hopefully change over time, so starting a community wiki to document some:
The XCTAssert… family of macros can handle you leaving out the format parameter (used for commenting why a test should pass) entirely. This is handy for keeping simple tests (where it's obvious from the test itself what it's doing) clean in the editor. STAssert… required us to make the format parameter nil or some such instead.
There's a generic XCTAssert() macro, which feels neater than bodging some types of check into STAssertTrue.