I am using UIAutomation to test an app, and I would like to find out my code coverage. But since javascript has no preprocessor, that means that gcov and similar are not an option. Has anyone come up with a solution for this?
For Xcode version 4.5 and newer:
Set the “Generate Test Coverage Files” build setting to Yes.
Set the “Instrument Program Flow” build setting to Yes.
This will generate code coverage files every time you run your application in the simulator and exit the application. A detailed explanation of these two steps can be found at the beginning of http://qualitycoding.org/xcode-code-coverage/.
For any unit tests, code coverage files will be generated every time you hit the test button and the tests complete. For UIAutomation, it is a little bit more tricky. You have to ensure the application exits at the conclusion of your tests. The easiest way I found to do this is to turn off multitasking. Add UIApplicationExitsOnSuspend in your MyAppName-Info.plist file and set this option to 'YES'. Run your UI automation test and at the end of it you can exit the app either by manually pressing the home button in the simulator or using the UIATarget.localTarget().deactivateAppForDuration() method.
Note: if your app has any UI Automation tests that rely on the deactivateAppForDuration() method, the tests will terminate upon running the command.
Code Coverage is only used for Unit Testing, there is no Code Coverage for UIAutomation because there is no way to tell how many elements on screen has been "touched" by UIAutoamtion
Related
I have an iOS app in Xcode 8.2. It has a test target / scheme, for which “Gather coverage data” is checked in the scheme’s Test / Info settings. Coverage data is not gathered. I see how many times a line was iterated over in the gutter as usual, but the Report navigator’s test runs don’t indicate any coverage at all.
I’m wondering if this is because I’ve set the tests to run hostless, i.e. without needing to actually fire up my app – they’re pure logic tests.
Is this possible?
Yes a hostless XCTest target should gather code coverage data.
'iOS Unit Testing' (aka XCTest) bundles, which test a dynamic framework or something else which doesn't require the application environment to exist to run, should happily collect code coverage data and display it in Xcode. Even of the Host Application is set to None. This works either when running Xcode > Product > Test on the Scheme for the framework under test or on the Scheme for the unit tests themselves (if the test bundle is listed in the Test pane of the Scheme editor).
Your problem must be somewhere else, sorry. Its hard for me to guess what the problem is, I suggest you try making a fresh project and see if you can reproduce the problem.
Code Coverage is commonly used with tests in Xcode. I would like to use it for manually executed app. Can I do it, possibly with third-party tools?
For example: I build and launch the app on device, perform some actions with it and then look at code coverage results.
The solution was suggested by someone here: https://github.com/SlatherOrg/slather/issues/209
You could try having a XCUITest that sleeps forever, then manually use the app, and see if on termination the coverage files are generated.
I simply tried the solution and it worked perfectly for me:
class FooTests: XCTestCase {
override func setUp() {
super.setUp()
let app = XCUIApplication()
app.launch()
}
func testBalances() {
sleep(30)
XCTAssert(true)
}
}
After the test succeeded on XCTAssert(true), I could see the code coverage for the manual use cases. You can play around with the sleep(30) to match your requirement.
Probably you might have already figured it out, but it was possible before Xcode7. The way we achieved this was, to set the "Instrument Program Flow" and "Generate Test Coverage files" flags to Yes, on your project, and later add a "flush" code somewhere inside you application, to write the coverage data files. This "flushing" part actually writes the coverage data files, which can be used later by other tools such as gcovr or lcov, to get your coverage data. Once you interact with the app, either manually or through automated tests, the coverage data gets written.
However, with Xcode7, looks like the coverage data is limited to only Xcode unit tests. I am still trying to figure out, if there is any way to gather the coverage data, by interacting with the application manually, or through automated tests.
Code coverage will record which parts of your code your test have run. But you could Build some UITests which would preform some actions as you said. UI can record UI tests to repeat the actions you perform on the simulator then when you run the test it will repeat what you did. The coverage will then show which parts of the code where excited during the UITests.
Look for some info on UITesting in Xcode 7. There is a good demo in one of the developer sessions from wwdc15
https://developer.apple.com/videos/wwdc/2015/?id=406
In my iOS app, I have a webview.
In the project, there is a specific target that runs integration test for javascript and native code communication. These tests are written in javascript, and ran inside the WebView.
I can detect when javascript is done testing. What I'd like to do is exit the application, so that the xcodebuild -scheme MyJSTests would run and exit properly.
What's the best way to exit an iOS application when the tests are done?
EDIT:
I have no choice on the test framework whatsoever, as we receive the test files from another project. All I can do is implement a method that's responsible for reporting and exiting.
The tests are executed in full javascript, inside the webview. The test scripts are loaded through a custom URLProtocol. At the end of the tests, a method call reportResults() is done, that is forwarded to native code.
It is against apple's guidelines to end a program through code. Exiting a program by their guidelines should be done only by the user, otherwise it can be interpreted as an undesired action by the user.
I have enabled test coverage with no problem using the Generate Test Coverage Files and Instrument Program Flow with the fopen$UNIX2003 and fwrite$UNIX2003 hack.
But the problem is, when you use XCode to run tests it ends up launching the simulator which launches your app. When that happens the output for the test coverage is not truly correct because it thinks certain parts of the code are touched because they are executed when the app launches, not because a test touched them.
Is there a better way to see what code was actually touched by a test?
EDIT:
So not really an answer to my question, but a somewhat "solution" that will at least get better coverage numbers can be found here: Run logic tests in Xcode 4 without launching the simulator
I have a set of automated test cases set up in instruments using tuneup.js to test an app. I decided to use tuneup.js as it allowed me to separate my tests into individual test cases and run the whole set from one individual script, this works fine if all the tests run ok, however if one fails, all the tests fail as the simulator is left in an unknown state (I have written my tests so they all start and end on the same login screen) Is there a way to reset the simulator, or restart the app between test cases?
Try to launch tests from the command line. UI Automation allows to execute only one test in one run. After the test will be completed (does not matter if it was failed or passed) - application will be kicked by the system (UIAutomation). At least it works with real devices.
Your command line launch script will work in the following manner:
1. Reads configuration file (can be any file txt or xml) with path to your tests. At this point you will have an array with path to your tests and total tests count.
2. Then using simple 'for' loop (from 1 to 'testcount') it will launch UIAutomation with required parameters. One of the parameters will be the path to your test script that was read from the configuration file.
You can also put the path to the 'configuration file' as a parameter for your command line launch script. This will allow you to run any test set simply calling the launch script with required configuration file.
I wrote a script that will reset the contents & settings of all versions and devices for the iOS Simulator. It grabs the device names and version numbers from the menu, so it will include any new devices or iOS versions that Apple releases simulators for.
It's easy to run manually or use in a build-script. I would suggest adding it as a Pre-Action Run Script before the build.
https://github.com/michaelpatzer/ResetAllSimulators
Having failed tests leave your app in an unknown state is one of the main problems with using Apple's instruments tool as-is. We solved this in a framework called Illuminator (on GitHub, and inspired by tuneup.js) in two ways.
First, we wrote an automation bridge -- a channel for RPC with the app being tested, which allows us to reset our app before each test.
In cases where that's not sufficient, the Illuminator test runner has an option to re-run each failed test in its own pristine launch of the simulator (e.g. with --retest 1x,solo).