Typically I find that nearly all my most important test cases for iPhone development revolve around UI testing rather than business logic or data testing. I'm not very familiar with automated UI testing in the XCode environment. Can someone point me to a good tutorial or book?
UPDATE
This question was written several years ago and UI testing has come a long way since then. Using the UI Automation is still an option, but the KIF Framework is a much better solution for functional testing now, IMO.
From KIF's github page:
KIF, which stands for Keep It Functional, is an iOS integration test
framework. It allows for easy automation of iOS apps by leveraging the
accessibility attributes that the OS makes available for those with
visual disabilities.
KIF builds and performs the tests using a standard XCTest testing
target. Testing is conducted synchronously in the main thread (running
the run loop to force the passage of time) allowing for more complex
logic and composition. This also allows KIF to take advantage of the
Xcode 5 Test Navigator, command line build tools, and Bot test
reports.
Your best bet will be to use the UI Automation instrument that debuted with iOS 4.0. This can be scripted to test out many aspects of your user interface.
Apple provides a great introduction to this tool in the video for the WWDC 2010 session 306 - "Automating User Interface Testing with Instruments". I demonstrate how to set up scripts and do testing in the video for the "Testing" session of my advanced iOS development course on iTunes U. My notes on UI Automation from that class, including sample scripts, can be found here.
Additionally, you might want to look at James Turner's article "How to use UIAutomation to create iPhone UI tests" and Alex Vollmer's "Working with UIAutomation".
What are you really trying to test? I agree that testing view interaction is important but find that I can usually construct a unit or integration test which tests the view or the view and it's controller to validate this behavior without attempting to drive the entire app via the UI.
For example http://blog.carbonfive.com/2010/03/10/testing-view-controllers/ has examples of tests which test that UIKit views are bound to IBActions or have appropriate delegate's and that a controller responds to those action or delegate messages appropriately. There's no need to actually drive a UIButton or UITableView since I'm willing to trust that those classes actually work and instead only need to test my use and configuration of them.
For custom view components you can send UIResponder messages directly in your tests and assert that they behave as expected, again without driving the entire UI.
Can you provide an example of the sort of behavior you want to test?
This link could be helpful take a look https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html
Related
It seems that in Xcode there are two kinds of tests, UI-tests which test an app running in another process using Automation, and Unit tests, where small units of code are tested in isolation. I wonder if there's a middle ground somewhere...
For instance, if I run unit tests there's a full app launched in memory that I don't use, because I am instantiating the classes that I need for testing in my test case classes. (I only know it's there because I see it in the simulator window and because my test classes sometimes accidentally communicate with it using notification center.)
It would be nice though, if I could use this app for integration tests. They would be similar to UI-tests, but much faster, and I would be able to access the implementation. I could for instance do that in order to install mocks so I can simulate behaviour of remote endpoints and see what the effects of certain responses are from the view point of the user.
The question is: Is there a good (documented) way to start up an app with mocks in the XCTest context and find that middle ground between unit tests and UI-tests?
funny thing, the other day we were also having this kind of problem. I'm not sure if this info might help you.
So the short answer to your specific question is that there is no written documentation in the apple docs(AFAIK) that specifically states how to make mock classes inside your tests but it doesn't necessarily mean you cant make one. There are a lot of ways on which you can make a mock up scenario to work with your code
manual way : making protocols/interfaces to be implemented by a working class and a mock class, overriding specific methods of working classes to somehow mock results, swizzling and etc.
3rd party way : notables (OCMock, Mockito)
in the case of objective c, there's a lot of 'em 3rd parties out there but with swift there are a few out there that are still in development because of the support that swift has with its runtime library (swift is yet to fully expose its runtime library)
here's a helpful link : https://academy.realm.io/posts/tryswift-veronica-ray-real-world-mocking-swift/ (great article)
Cheers
Belonging to the world of web development, I am used to powerful testing techniques and tools like Chai, Sinon, Mocha, Jasmine, to name a few, as well as to wide support and community, well-written books with deep insights. There are all possible cases I can imagine that are covered with existing tools.
Now, learning iOS development with Swift, I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
I find the community is not so wide in iOS development, especially, Swift, as a result of the language being not so long on the landscape.
Please share the best practices for testing that can help me to create solid predictably behaving apps.
When developing iOS apps you have two possible test types you can use, one for unit testing, and one for acceptance testing. Those are referred in Xcode respectively as "Test Target" and "UI Test Target".
Within the tests target you can test all your app code by importing its Swift module using #testable import MyAppName, and then writing unit tests using XCTest, the framework provided by Apple.
On top on the tools provided by Apple, the FOSS community has developed other great testing libraries. In particular Quick is an testing framework that provides a syntax similar to Mocha.
When it comes to the UI tests target there is an important thing to be said: you can only test your app behaviour via those. There is no way to access the application code, everything is done through proxies that simulate the user's interaction with the UI. That is, in my opinion, for a good reason. The kind of tests one should write via tools that only allow UI interaction should be acceptance/funcitonal tests, which should only verify the app's behaviour. Not having access to the code is a good way to avoid misusing this tool.
I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
Going deeper:
using stubs and mocks
Because of Swift's nature doing real mocking is quite hard. It's better to use fakes and doubles. There is also a great deal of literature online on why using mocks and stubs is seldom a good idea, for example this, this, this, and this.
imitating connecting to web services asynchronously
After having said all that it is true that sometimes stubbing is useful, such in the case you highlight. A good library to do this is OHHTTPStubs.
get myself sure tests can be written in a clear manner
I believe there are enough tools today to support writing clear and effective tests with little effort. The rest is up to you, your team, and the way you architect your app. As it is for any other programming language after all :D
If I can put a shameless plug in here, at the start of the year I wrote a post covering the state of testing in the iOS plaform, a lot of new things have happened since then, but it could be a good starting point if you want to dig deeper.
We would like to integrate a tests suite in our iOS app. To have an overall idea, this app is using web services, is saving files on the device and has some complex navigation parts. The requirements:
- run the tests suite with Jenkins
- being able to launch the test suite on a set of devices and iOS versions
- we don't expressly need something readable by non devs like Calabash proposes
We were thinking about doing some workflow testing (interaction bringing from one screen to another) and unit testing.
We googled a bit but articles are often at least 2 years old which is like the Paleolithic period in the mobile world. But still, it gives some nice first inputs.
For Unit Testing, we were thinking about:
GHUnit
XCTest
Kiwi
For workflow testing, we were thinking about:
Zucchini
Calabash
Regarding all the mentionned requirements, does anybody see any advice to chose one framework or the other (or another proposition)?
Thank you for any feedback.
ps: by the way, some interesting articles we have found on the subject that can help:
http://blog.lesspainful.com/2012/03/07/Calabash-iOS/
http://iosunittesting.com/faq/
I recommend Kiwi for unit testing for several reasons:
It supports nested contexts that can each have their own setup and teardown blocks. This allows your tests to be more DRY. XCTest only has one setup and teardown method which is used for all the tests in a file. With Kiwi's nested contexts, you can have setup code that is performed before some but not all of your tests based on how you define the contexts.
Kiwi has great support for mocking/stubbing dependencies when needed. I have found it to be more robust than OCMock (mocked class methods with OCMock stuck around after the test if you forgot to call -stopMocking whereas Kiwi always makes sure to teardown mocked class methods after each test).
I am not experienced in workflow/application testing but I plan to eventually check out KIF and UIAutomation.
I would highly recommend looking at iOS Tests/Specs TDD/BDD and Integration & Acceptance Testing for a more comprehensive discussion of testing libraries.
Background
I am looking for a way to implement a scheme similar to what Frank library uses to implement "Automated acceptance tests for native iOS apps", but I want this scheme rely on native iOS/MacOSX technologies. Sorry for the following TLDR but it deserves verbose explanation.
1. Here is a short overview of how Frank works:
it has Client and Server parts.
Server part is embedded into an application that we want to run acceptance tests against. Frank tutorials show us how to create a duplicate target of an app's main target and embed Frank HTTP server to it.
Client part - is a mainly a Cucumber that runs plain text scenarios: each scenario contains directives that should be run against an app (fill text field, touch button, ensure that a specific element exists on a page etc...). Also each scenario launches its own instance of app by this means providing a fresh state every time we enter a new scenario.
Client (Cucumber and Ruby-to-Objective-C bridge) communicates with Server (HTTP server embedded into an app) via HTTP protocol. It uses special conventions so client can tell server what an app should do so the specific scenario could be performed.
2. Recently I've found the following article written by the author of Frank Pete Hodgson:
http://blog.thepete.net/blog/2012/11/18/writing-ios-acceptance-tests-using-kiwi/
In which he suggests more simple way of writing acceptance tests for the developers who don't like to rely on external tools like Cucumber and Ruby. Let me quote the author himself:
Before I start, let me be clear that I personally wouldn’t use this approach to writing acceptance tests. I much prefer using a higher-level language like ruby to write these kinds of tests. The test code is way less work and way more expressive, assuming you’re comfortable in ruby. And that’s why I wanted to try this experiment. I’ve spoken to quite a few iOS developers over time who are not comfortable writing tests in ruby. They are more comfortable in Objective-C than anything else, and would like to write their tests in the same language they use for their production code. Fair enough.
This blog post inspired me to quickly roll my own very raw and primitive tool that does exactly what Pete described in his blog post: NativeAutomation.
Indeed, like it was described by Pete, it is possible to run acceptance tests by using just Kiwi/PublicAutomation setup placed in a simple OCTests target. I really liked it because:
It is just pure C/Objective-C. It was very easy to build initial bunch of C helpers, that look like Capybara helpers:
tapButtonWithTitle, fillTextFieldWithPlaceholder, hasLabelWithTitle and so on...
It does not require external tools and languages: no need in Cucumber/Ruby or anything else. NativeAutomation itself uses just PublicAutomation that Frank also uses. PublicAutomation is needed to simulate user interactions on an app's screen: touches, fills, gestures...
It is very handy to run these tests right from Xcode by just running a Cocoa Unit Tests bundle. (Though command-line builds are easy as well).
Problem
The problem Kiwi/PublicAutomation approach has is that the whole testing suite is embedded into the app's bundle. It means that after each scenario is run, it is not possible to reset application to force it to be in a fresh state before next scenario begins execution. The only way to workaround this problem is to write Kiwi's beforeEach hook with a method that performs a soft-reset of an application like:
+ (void)resetApplication {
[Session invalidateSession];
[LocationManager resetInstance];
[((NavigationController *)[UIApplication sharedApplication].delegate.window.rootViewController) popToRootViewControllerAnimated:NO];
[OHHTTPStubs removeAllStubs];
cleanDatabase();
shouldEventuallyBeTrue(^BOOL{
return FirstScreen.isCurrentScreen;
});
but in case of application which involve networking, asynchronous jobs, core data, file operations it becomes hard to perform a real teardown of a stuff left by previous scenario.
Question
The problem described above made me thinking about if it is possible to implement a more complex approach similar to Frank's approach, with a second app that works apart from main app's bundle and does not rely on external tools like Cucumber(Ruby).
Here is how I see the way it could be done.
Besides a main app (MainApp) there is a second iOS (or Mac OS X) app (AcceptanceTestsRunnerApp) that contains the whole acceptance tests suite and runs this suite against a main app bundle:
It does fire up new simulator instance before it enters each new scenario and executes current scenario against current simulator's app's instance.
The question is:
I am not well awared about Mac OSX or iOS technologies that would allow me to do that: I don't know if it is even possible to setup a Mac OS X / iOS application (AcceptanceTestsRunnerApp) that could take a control over a main app (MainApp) and run acceptance tests scenarios against it.
I will be thankful for any thoughts/suggestions that people who feel more comfortable with having native Objective-C tool for writing acceptance tests for iOS applications might have.
UPDATED LATER
...I did read some documentation about XPC services but the irony about it is that scheme I am looking for should be quite opposite to the scheme XPC documentation suggests:
Ideally I want my AcceptanceTestsRunnerApp dominate over MainApp: run it and control it (user interactions, assertions about view hierarchy) via some object proxying to MainApp's application delegate while XPC services setup would assume XPC service (AcceptanceTestsRunnerApp) to be subordinate to an app (MainApp) and will require XPC service to live inside app's bundle which I want to avoid by all means.
...My current reading is Distributed Objects Programming Topics. It seems to me that I will find my answer there. If no one provides me with guides or directions I will post an answer about my own researches and thoughts.
...This is the next step in my quest: Distributed Objects on iOS.
KIF and the port of Jasmine - Cedar to iOS work fairly well from my experience.
However, I also have had lots of good usage out of calabash, which like Frank, is powered by gherkin.
I am trying to find out if i can use UIAutomation (Instruments) for all my automation need. So far one of the show stopper i see is, UIAutomation is unable to switch the application while my application require to share a document between two application. Guys - What's the basic approach differentiation between UIAutomation & PefectoMobile.
BTW, I can switch between applications using Pefecto Mobile solution.
Thanks & Regards -
Perfecto Mobile is for testers that won't probably see the code, for example. They have access to devices that have the app/apps. From there, they will create automated scripts that will then perform almost the same thing as UI automation. The thing is that with UI Automation, you have to give accessibility to your elements in order to test something. I would say that UI Automation has a better performance overall. Think of it in this way:
UI Automation - Developers or persons that will have a closer relationship with the code.
Perfecto Mobile - Testers, that sometimes don't even know how iOS works internally. This is useful when you want to really separate things: developers develop, and testers test it.
Separating things is good, because a tester that never seen code, or how the app is suppose to work, gives a fresh perspective to the project. You can also argue that, normally, a tester ( a person that tests for living) will do a better job a testing than a developer.
I don't know anything about PerfectoMobile, but I can answer questions about UI Automation. UI Automation is just a tool, there is no service built behind it like it looks that PerfectoMobile provides. UI Automation comes as part of Apple's developer tools so you don't need to download or license anything else to use it. Once you have tests written in UI Automation, you can use a service like cisimple.com to run them in the cloud as part of a distributed continuous integration process, but that's not set up for you out of the box.
I've written some tutorials on UI Automation that might help you understand what it does so you can make you decision: http://cocoamanifest.net/articles/2011/05/uiautomation-an-introduction.html