I'm currently looking at various integration/functional/acceptance testing frameworks for iOS and they all seem to rely heavily on the UI and accessibility labels. To me a lot of these seem more based towards UI tests as opposed to integration testing. Are there any dedicated frameworks that don't rely on this, especially when working with modules/frameworks that aren't built with a UI?
Is XCTest the way to go here when testing this kind of functionality? I've seen fitnesse and this seems to be the only other option I've found so far.
So in this instance I could have something like an API framework that is purely responsible for making network requests, parsing the response and creating relevant objects. Whilst I would write unit tests to test the individual pieces of functionality, I would also like to test an end to end flow.
Are there any suitable testing frameworks for this type of thing? Preferably swift based
thanks
Related
Belonging to the world of web development, I am used to powerful testing techniques and tools like Chai, Sinon, Mocha, Jasmine, to name a few, as well as to wide support and community, well-written books with deep insights. There are all possible cases I can imagine that are covered with existing tools.
Now, learning iOS development with Swift, I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
I find the community is not so wide in iOS development, especially, Swift, as a result of the language being not so long on the landscape.
Please share the best practices for testing that can help me to create solid predictably behaving apps.
When developing iOS apps you have two possible test types you can use, one for unit testing, and one for acceptance testing. Those are referred in Xcode respectively as "Test Target" and "UI Test Target".
Within the tests target you can test all your app code by importing its Swift module using #testable import MyAppName, and then writing unit tests using XCTest, the framework provided by Apple.
On top on the tools provided by Apple, the FOSS community has developed other great testing libraries. In particular Quick is an testing framework that provides a syntax similar to Mocha.
When it comes to the UI tests target there is an important thing to be said: you can only test your app behaviour via those. There is no way to access the application code, everything is done through proxies that simulate the user's interaction with the UI. That is, in my opinion, for a good reason. The kind of tests one should write via tools that only allow UI interaction should be acceptance/funcitonal tests, which should only verify the app's behaviour. Not having access to the code is a good way to avoid misusing this tool.
I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
Going deeper:
using stubs and mocks
Because of Swift's nature doing real mocking is quite hard. It's better to use fakes and doubles. There is also a great deal of literature online on why using mocks and stubs is seldom a good idea, for example this, this, this, and this.
imitating connecting to web services asynchronously
After having said all that it is true that sometimes stubbing is useful, such in the case you highlight. A good library to do this is OHHTTPStubs.
get myself sure tests can be written in a clear manner
I believe there are enough tools today to support writing clear and effective tests with little effort. The rest is up to you, your team, and the way you architect your app. As it is for any other programming language after all :D
If I can put a shameless plug in here, at the start of the year I wrote a post covering the state of testing in the iOS plaform, a lot of new things have happened since then, but it could be a good starting point if you want to dig deeper.
We would like to integrate a tests suite in our iOS app. To have an overall idea, this app is using web services, is saving files on the device and has some complex navigation parts. The requirements:
- run the tests suite with Jenkins
- being able to launch the test suite on a set of devices and iOS versions
- we don't expressly need something readable by non devs like Calabash proposes
We were thinking about doing some workflow testing (interaction bringing from one screen to another) and unit testing.
We googled a bit but articles are often at least 2 years old which is like the Paleolithic period in the mobile world. But still, it gives some nice first inputs.
For Unit Testing, we were thinking about:
GHUnit
XCTest
Kiwi
For workflow testing, we were thinking about:
Zucchini
Calabash
Regarding all the mentionned requirements, does anybody see any advice to chose one framework or the other (or another proposition)?
Thank you for any feedback.
ps: by the way, some interesting articles we have found on the subject that can help:
http://blog.lesspainful.com/2012/03/07/Calabash-iOS/
http://iosunittesting.com/faq/
I recommend Kiwi for unit testing for several reasons:
It supports nested contexts that can each have their own setup and teardown blocks. This allows your tests to be more DRY. XCTest only has one setup and teardown method which is used for all the tests in a file. With Kiwi's nested contexts, you can have setup code that is performed before some but not all of your tests based on how you define the contexts.
Kiwi has great support for mocking/stubbing dependencies when needed. I have found it to be more robust than OCMock (mocked class methods with OCMock stuck around after the test if you forgot to call -stopMocking whereas Kiwi always makes sure to teardown mocked class methods after each test).
I am not experienced in workflow/application testing but I plan to eventually check out KIF and UIAutomation.
I would highly recommend looking at iOS Tests/Specs TDD/BDD and Integration & Acceptance Testing for a more comprehensive discussion of testing libraries.
I'm in the phase of integrating cucumber tests for an existing application. The app interacts heavily with a backend. The simplest case being the registration of an user.
In order to make the tests repeatable, I cannot go against the (real) backend (for several reasons, one not being able to register two users with the same e-mail).
I'd like to know which options do I have to mock the backend behavior and deliver mocked responses. For unit testing I know Nocilla and several other frameworks. Integrating them of course does not work for calabash tests.
One alternative would be to have a real server running which I'd control from the calabash tests to control the networking. But that seems a bit overkill. Another option would be to have mock objects in the calabash target which I could control via the calabash backdoor mechanism.
I doubt that aren't the only options. So my question is: is there a more elegant way to do so?
As I received no other feedback until today, I'll answer my own question.
We opted for the way of creating a very simple mock for our backend. We used Sinatra, but a node.js or similar technique would have led to the same result. The mock can be controlled via a simple REST API.
In the step definitions we configure our mock backend appropriately for the running scenario. It's a bit of overhead, because the mock has to evolve with the real backend, but until today it feels like a robust solution.
There are two third-party frameworks for dealing with the iOS UI Automation instrument: Alex Volmer's tuneup.js and the Jasmine framework.
What are the differences between these two frameworks and what are their advantages and disadvantages?
tuneup.js and Jasmine both provide nice wrappers around UIAutomation's rather verbose and arduous assertion API.
A plus of tuneup.js is that it not only includes the test/assertion abstractions, but also a number of handy convenience methods for performing various common tasks in UIAutomation (typing text into a textfield, for example). The downside to tuneup.js (in comparison to Jasmine), is that it lacks some of the nice delineation that Jasmine provides via nested describe / it spec definitions.
As noted above, Jasmine gives a rich way to define various specs of your app. You can demarkate various tests quite easily. Additionally, if you've used Jasmine in regular ol' Javascript testing, then it'll be very familiar.
In either case, if your app has even a remotely deep element hierarchy, I'd recommend pairing either tuneup or Jasmine with a framework of my own, mechanic.js. Mechanic makes traversing your app's 'DOM' and interacting with elements really easy. Plain ol' UIAutomation can get a bit verbose; mechanic is sort of the JQuery of UIAutomation.
Anyways, my personal choice would be Jasmine + Mechanic in most cases. I prefer the cadence of Jasmine's spec definition API, and paired with Mechanic, makes for terse, readable UI tests.
Jasmine and tuneup.js both provide a similar simple syntax for you to write your iOS automation tests with.
One advantage of Jasmine is that it has a good reporting structure so if you plan on running your automation tests from a CI server such as Jenkins or Hudson, it'll be much easier for your CI server to parse the XML test results from Jasmine versus the .plist format you get when using tuneup.js
Typically I find that nearly all my most important test cases for iPhone development revolve around UI testing rather than business logic or data testing. I'm not very familiar with automated UI testing in the XCode environment. Can someone point me to a good tutorial or book?
UPDATE
This question was written several years ago and UI testing has come a long way since then. Using the UI Automation is still an option, but the KIF Framework is a much better solution for functional testing now, IMO.
From KIF's github page:
KIF, which stands for Keep It Functional, is an iOS integration test
framework. It allows for easy automation of iOS apps by leveraging the
accessibility attributes that the OS makes available for those with
visual disabilities.
KIF builds and performs the tests using a standard XCTest testing
target. Testing is conducted synchronously in the main thread (running
the run loop to force the passage of time) allowing for more complex
logic and composition. This also allows KIF to take advantage of the
Xcode 5 Test Navigator, command line build tools, and Bot test
reports.
Your best bet will be to use the UI Automation instrument that debuted with iOS 4.0. This can be scripted to test out many aspects of your user interface.
Apple provides a great introduction to this tool in the video for the WWDC 2010 session 306 - "Automating User Interface Testing with Instruments". I demonstrate how to set up scripts and do testing in the video for the "Testing" session of my advanced iOS development course on iTunes U. My notes on UI Automation from that class, including sample scripts, can be found here.
Additionally, you might want to look at James Turner's article "How to use UIAutomation to create iPhone UI tests" and Alex Vollmer's "Working with UIAutomation".
What are you really trying to test? I agree that testing view interaction is important but find that I can usually construct a unit or integration test which tests the view or the view and it's controller to validate this behavior without attempting to drive the entire app via the UI.
For example http://blog.carbonfive.com/2010/03/10/testing-view-controllers/ has examples of tests which test that UIKit views are bound to IBActions or have appropriate delegate's and that a controller responds to those action or delegate messages appropriately. There's no need to actually drive a UIButton or UITableView since I'm willing to trust that those classes actually work and instead only need to test my use and configuration of them.
For custom view components you can send UIResponder messages directly in your tests and assert that they behave as expected, again without driving the entire UI.
Can you provide an example of the sort of behavior you want to test?
This link could be helpful take a look https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html