I am trying to find out if i can use UIAutomation (Instruments) for all my automation need. So far one of the show stopper i see is, UIAutomation is unable to switch the application while my application require to share a document between two application. Guys - What's the basic approach differentiation between UIAutomation & PefectoMobile.
BTW, I can switch between applications using Pefecto Mobile solution.
Thanks & Regards -
Perfecto Mobile is for testers that won't probably see the code, for example. They have access to devices that have the app/apps. From there, they will create automated scripts that will then perform almost the same thing as UI automation. The thing is that with UI Automation, you have to give accessibility to your elements in order to test something. I would say that UI Automation has a better performance overall. Think of it in this way:
UI Automation - Developers or persons that will have a closer relationship with the code.
Perfecto Mobile - Testers, that sometimes don't even know how iOS works internally. This is useful when you want to really separate things: developers develop, and testers test it.
Separating things is good, because a tester that never seen code, or how the app is suppose to work, gives a fresh perspective to the project. You can also argue that, normally, a tester ( a person that tests for living) will do a better job a testing than a developer.
I don't know anything about PerfectoMobile, but I can answer questions about UI Automation. UI Automation is just a tool, there is no service built behind it like it looks that PerfectoMobile provides. UI Automation comes as part of Apple's developer tools so you don't need to download or license anything else to use it. Once you have tests written in UI Automation, you can use a service like cisimple.com to run them in the cloud as part of a distributed continuous integration process, but that's not set up for you out of the box.
I've written some tutorials on UI Automation that might help you understand what it does so you can make you decision: http://cocoamanifest.net/articles/2011/05/uiautomation-an-introduction.html
Related
Belonging to the world of web development, I am used to powerful testing techniques and tools like Chai, Sinon, Mocha, Jasmine, to name a few, as well as to wide support and community, well-written books with deep insights. There are all possible cases I can imagine that are covered with existing tools.
Now, learning iOS development with Swift, I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
I find the community is not so wide in iOS development, especially, Swift, as a result of the language being not so long on the landscape.
Please share the best practices for testing that can help me to create solid predictably behaving apps.
When developing iOS apps you have two possible test types you can use, one for unit testing, and one for acceptance testing. Those are referred in Xcode respectively as "Test Target" and "UI Test Target".
Within the tests target you can test all your app code by importing its Swift module using #testable import MyAppName, and then writing unit tests using XCTest, the framework provided by Apple.
On top on the tools provided by Apple, the FOSS community has developed other great testing libraries. In particular Quick is an testing framework that provides a syntax similar to Mocha.
When it comes to the UI tests target there is an important thing to be said: you can only test your app behaviour via those. There is no way to access the application code, everything is done through proxies that simulate the user's interaction with the UI. That is, in my opinion, for a good reason. The kind of tests one should write via tools that only allow UI interaction should be acceptance/funcitonal tests, which should only verify the app's behaviour. Not having access to the code is a good way to avoid misusing this tool.
I wonder, how I can thoroughly test methods of my app using stubs and mocks, imitating connecting to web services asynchronously and get myself sure tests can be written in a clear manner, creating peace of mind for my aging body.
Going deeper:
using stubs and mocks
Because of Swift's nature doing real mocking is quite hard. It's better to use fakes and doubles. There is also a great deal of literature online on why using mocks and stubs is seldom a good idea, for example this, this, this, and this.
imitating connecting to web services asynchronously
After having said all that it is true that sometimes stubbing is useful, such in the case you highlight. A good library to do this is OHHTTPStubs.
get myself sure tests can be written in a clear manner
I believe there are enough tools today to support writing clear and effective tests with little effort. The rest is up to you, your team, and the way you architect your app. As it is for any other programming language after all :D
If I can put a shameless plug in here, at the start of the year I wrote a post covering the state of testing in the iOS plaform, a lot of new things have happened since then, but it could be a good starting point if you want to dig deeper.
Background
I am looking for a way to implement a scheme similar to what Frank library uses to implement "Automated acceptance tests for native iOS apps", but I want this scheme rely on native iOS/MacOSX technologies. Sorry for the following TLDR but it deserves verbose explanation.
1. Here is a short overview of how Frank works:
it has Client and Server parts.
Server part is embedded into an application that we want to run acceptance tests against. Frank tutorials show us how to create a duplicate target of an app's main target and embed Frank HTTP server to it.
Client part - is a mainly a Cucumber that runs plain text scenarios: each scenario contains directives that should be run against an app (fill text field, touch button, ensure that a specific element exists on a page etc...). Also each scenario launches its own instance of app by this means providing a fresh state every time we enter a new scenario.
Client (Cucumber and Ruby-to-Objective-C bridge) communicates with Server (HTTP server embedded into an app) via HTTP protocol. It uses special conventions so client can tell server what an app should do so the specific scenario could be performed.
2. Recently I've found the following article written by the author of Frank Pete Hodgson:
http://blog.thepete.net/blog/2012/11/18/writing-ios-acceptance-tests-using-kiwi/
In which he suggests more simple way of writing acceptance tests for the developers who don't like to rely on external tools like Cucumber and Ruby. Let me quote the author himself:
Before I start, let me be clear that I personally wouldn’t use this approach to writing acceptance tests. I much prefer using a higher-level language like ruby to write these kinds of tests. The test code is way less work and way more expressive, assuming you’re comfortable in ruby. And that’s why I wanted to try this experiment. I’ve spoken to quite a few iOS developers over time who are not comfortable writing tests in ruby. They are more comfortable in Objective-C than anything else, and would like to write their tests in the same language they use for their production code. Fair enough.
This blog post inspired me to quickly roll my own very raw and primitive tool that does exactly what Pete described in his blog post: NativeAutomation.
Indeed, like it was described by Pete, it is possible to run acceptance tests by using just Kiwi/PublicAutomation setup placed in a simple OCTests target. I really liked it because:
It is just pure C/Objective-C. It was very easy to build initial bunch of C helpers, that look like Capybara helpers:
tapButtonWithTitle, fillTextFieldWithPlaceholder, hasLabelWithTitle and so on...
It does not require external tools and languages: no need in Cucumber/Ruby or anything else. NativeAutomation itself uses just PublicAutomation that Frank also uses. PublicAutomation is needed to simulate user interactions on an app's screen: touches, fills, gestures...
It is very handy to run these tests right from Xcode by just running a Cocoa Unit Tests bundle. (Though command-line builds are easy as well).
Problem
The problem Kiwi/PublicAutomation approach has is that the whole testing suite is embedded into the app's bundle. It means that after each scenario is run, it is not possible to reset application to force it to be in a fresh state before next scenario begins execution. The only way to workaround this problem is to write Kiwi's beforeEach hook with a method that performs a soft-reset of an application like:
+ (void)resetApplication {
[Session invalidateSession];
[LocationManager resetInstance];
[((NavigationController *)[UIApplication sharedApplication].delegate.window.rootViewController) popToRootViewControllerAnimated:NO];
[OHHTTPStubs removeAllStubs];
cleanDatabase();
shouldEventuallyBeTrue(^BOOL{
return FirstScreen.isCurrentScreen;
});
but in case of application which involve networking, asynchronous jobs, core data, file operations it becomes hard to perform a real teardown of a stuff left by previous scenario.
Question
The problem described above made me thinking about if it is possible to implement a more complex approach similar to Frank's approach, with a second app that works apart from main app's bundle and does not rely on external tools like Cucumber(Ruby).
Here is how I see the way it could be done.
Besides a main app (MainApp) there is a second iOS (or Mac OS X) app (AcceptanceTestsRunnerApp) that contains the whole acceptance tests suite and runs this suite against a main app bundle:
It does fire up new simulator instance before it enters each new scenario and executes current scenario against current simulator's app's instance.
The question is:
I am not well awared about Mac OSX or iOS technologies that would allow me to do that: I don't know if it is even possible to setup a Mac OS X / iOS application (AcceptanceTestsRunnerApp) that could take a control over a main app (MainApp) and run acceptance tests scenarios against it.
I will be thankful for any thoughts/suggestions that people who feel more comfortable with having native Objective-C tool for writing acceptance tests for iOS applications might have.
UPDATED LATER
...I did read some documentation about XPC services but the irony about it is that scheme I am looking for should be quite opposite to the scheme XPC documentation suggests:
Ideally I want my AcceptanceTestsRunnerApp dominate over MainApp: run it and control it (user interactions, assertions about view hierarchy) via some object proxying to MainApp's application delegate while XPC services setup would assume XPC service (AcceptanceTestsRunnerApp) to be subordinate to an app (MainApp) and will require XPC service to live inside app's bundle which I want to avoid by all means.
...My current reading is Distributed Objects Programming Topics. It seems to me that I will find my answer there. If no one provides me with guides or directions I will post an answer about my own researches and thoughts.
...This is the next step in my quest: Distributed Objects on iOS.
KIF and the port of Jasmine - Cedar to iOS work fairly well from my experience.
However, I also have had lots of good usage out of calabash, which like Frank, is powered by gherkin.
I want to do unit testing in iOS. By default test classes are created using XCTest framework classes in Xcode 5. I have used SenTestingKit in earlier versions of iOS. What are the differences between these two frameworks? What are the advantages of XCTest framework. I googled for related documentation but I did not find any. Only thing I found is the interface classes to use inside the kit. Can anyone point me to the related resources.
Apple's documentation is notably lacking with regards to testing. It's a shame, because spending five minutes with XCTest made me fall for it. So, here are some docs that you may find useful.
First, stop and watch the WWDC video about testing: https://developer.apple.com/wwdc/videos/?include=409#409 (must be a registered developer)
Then, read about Bots. They're the real magic in XCTest. I'm currently setting up an environment of a Mac Mini running Mavericks Server connected to a small array of iPod Touches, iPhones and iPads that will simultaneously run my unit tests and report back. It's damn near magic.
https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/000-About_Continuous_Integration/about_continuous_integration.html#//apple_ref/doc/uid/TP40013292
All that said, I don't know SenTest well enough to compare the two. But I am learning XCTest, so here are a few thoughts on what I'm really digging about it so far.
It's already there. There's nothing to download, and you can make your test class objects at the same time as you make that which you wish to test. Rad.
Because it's integrated so deeply with Xcode, you can see test coverage in the gutter. You can also use Companion View to see the testers from source, or the source from tester methods. Also, it will always be up to date - no waiting to fix integration with Xcode because Apple "broke" something.
Continuous Integration. The bots are just magical, and they're so easy. You also get Git hosting, which is a small fringe perk, but more so if you prefer your Git internally hosted but don't want to pay for GitHub Enterprise.
And, a few cons for measure:
Lack of Documentation. Asking me to read the headers for a list of assertions is a ballsy move. I have no doubt Apple will fix this soon, but not soon enough.
You are tied to Xcode a little more deeply. Not really a con for development, but scripting XCTest isn't as elegant as other options may be.
Continuous Integration requires OS X Server. It can be on the same machine if you really want, but it's $20 to buy, even if you owned Mountain Lion Server. I am not at all bitter about this one (ha).
My two cents. Hope it helps. I'd really recommend watching that video I linked to - they do live test creation and make a really compelling case for unit testing in general, regardless of if you use XCTest.
XCTest are included by default with ios7 and integrate better with the new continuous integration functionality.
https://developer.apple.com/technologies/tools/
It seems the advantages are presently few, but that will hopefully change over time, so starting a community wiki to document some:
The XCTAssert… family of macros can handle you leaving out the format parameter (used for commenting why a test should pass) entirely. This is handy for keeping simple tests (where it's obvious from the test itself what it's doing) clean in the editor. STAssert… required us to make the format parameter nil or some such instead.
There's a generic XCTAssert() macro, which feels neater than bodging some types of check into STAssertTrue.
Typically I find that nearly all my most important test cases for iPhone development revolve around UI testing rather than business logic or data testing. I'm not very familiar with automated UI testing in the XCode environment. Can someone point me to a good tutorial or book?
UPDATE
This question was written several years ago and UI testing has come a long way since then. Using the UI Automation is still an option, but the KIF Framework is a much better solution for functional testing now, IMO.
From KIF's github page:
KIF, which stands for Keep It Functional, is an iOS integration test
framework. It allows for easy automation of iOS apps by leveraging the
accessibility attributes that the OS makes available for those with
visual disabilities.
KIF builds and performs the tests using a standard XCTest testing
target. Testing is conducted synchronously in the main thread (running
the run loop to force the passage of time) allowing for more complex
logic and composition. This also allows KIF to take advantage of the
Xcode 5 Test Navigator, command line build tools, and Bot test
reports.
Your best bet will be to use the UI Automation instrument that debuted with iOS 4.0. This can be scripted to test out many aspects of your user interface.
Apple provides a great introduction to this tool in the video for the WWDC 2010 session 306 - "Automating User Interface Testing with Instruments". I demonstrate how to set up scripts and do testing in the video for the "Testing" session of my advanced iOS development course on iTunes U. My notes on UI Automation from that class, including sample scripts, can be found here.
Additionally, you might want to look at James Turner's article "How to use UIAutomation to create iPhone UI tests" and Alex Vollmer's "Working with UIAutomation".
What are you really trying to test? I agree that testing view interaction is important but find that I can usually construct a unit or integration test which tests the view or the view and it's controller to validate this behavior without attempting to drive the entire app via the UI.
For example http://blog.carbonfive.com/2010/03/10/testing-view-controllers/ has examples of tests which test that UIKit views are bound to IBActions or have appropriate delegate's and that a controller responds to those action or delegate messages appropriately. There's no need to actually drive a UIButton or UITableView since I'm willing to trust that those classes actually work and instead only need to test my use and configuration of them.
For custom view components you can send UIResponder messages directly in your tests and assert that they behave as expected, again without driving the entire UI.
Can you provide an example of the sort of behavior you want to test?
This link could be helpful take a look https://developer.apple.com/library/mac/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/UsingtheAutomationInstrument/UsingtheAutomationInstrument.html
I'm building a test harness for my Delphi 2009 app. Testing the logic is fairly simple. Making sure the forms work properly is proving a bit more complicated. I'd like a way to simulate real user input, to open a form, make it think there's a user typing certain things and clicking in certain places, and make sure it reacts correctly. I'm sure there's a way to do this, I just don't know what it is. Does anyone know how to do it?
DUnit has GUITesting.pas whicih extends testing so you can send clicks, keys and text to controls on form, but that is about it.
Last year there where mentions of Zombie GUI testing framework that was used internaly by CodeGear developers, but nothing since Steve left for Falafel.
TestComplete is a good choice. Another commercial option for GUI testing is SmarteScript:
well for .net there's NUnitForms for GUI testing a win application.
don't know any open source for delphi though.
Test Complete can test delphi forms but it's not free.
There's 2 parts to this, firstly how do you automate the GUI, and secondly how do I then 'test' whether its working/not working.
Firtsly: To automate the GUI on windows try using AutoIT. Its a free tool for controlling windows interfaces, sending keyboard input events etc. http://www.autoitscript.com/autoit3/
Secondly: Testing is a big field, and I won't try and give you a whirlwind tour. But the mechanics of driving the GUI and testing the results could be handled using AutoIT's built in Basic like language or by using it in conjunction with a language like Ruby and TestUnit (rubys built-in unit testing framework).
If there is nothing Deliphi-specic out there and you need a quick solution try some easy to learn scripting solutions like AutoIt.
For a bit more sophisticated scripting, you might have a look on Scripted GUI Testing with Ruby.
But be aware, that you should not test too much functionality via the GUI, because such tests are very likely to break. If you end up with too much GUI testing you may need to rethink the design: Decouple logic from the GUI and test the logic directly with some xUnit framework.
Also have a look on a similar question about windows forms test automation.
It seems like DUnit has some gui-testing functionality: delphiextreme.com
Not exactly an answer to your question, but there is a very good page (IMHO of course) about GUI Architectures by Martin Fowler, featuring the "Humble View" architecture as the last entry, which is geared specifically towards test-driven software development. Worth looking into.
This will of course not help you with the task of testing whether all controls are wired correctly and handle all necessary events, but it should help to minimize the amount of GUI code that does need testing.
OpenCTF is good for you.
Quote:
OpenCTF is a test framework add-on for Embarcadero Delphi® which
performs automatic checks of all components in Forms (or DataModules).
It provides an easy way to build automatic quality checks for large
projects where many components have to pass repeated tests.
Adding OpenCTF tests to a DUnit test suite requires only a few lines of code.
Writing your own custom component tests needs only a few seconds.
OpenCTF is based on the DUnit open source test framework and extends
it by specialized test classes and helper functions.
Please head here to download.