I have a large Xcode 10 project for an iOS app that has nearly 40 screens and associated view controllers (all are Objective-C). Some of the code is a decade old and some new. The project has no UI or unit tests, but I'd like to enable code coverage and manually exercise the app. Is there a way to add, say, one test that will allow me to run the entire app manually and track which methods and functions are or are not used?
It was simpler than I expected, just not easy because I'd never done any tests before.
I made a UI test that keeps running until some testable event occurs that I can trigger, such as quitting the app (putting it into the background).
Run this test and exercise the app. When done, press Home. The test will exit and the code coverage for whatever you did while running will be shown in the Code Coverage column of the editor.
Add a new target for UI testing.
Edit Scheme, Test, Options: Code Coverage for: all targets.
Add this code as a UI test:
- (void)testCoverage
{
// UI tests must launch the application that they test.
XCUIApplication *app = [[XCUIApplication alloc] init];
[app launch];
// Code Coverage is turned on in the Test scheme
// (a) loop until Home button pushed...
while ( [app waitForState:XCUIApplicationStateRunningBackground timeout:60] == NO ) {
// still in the foreground
};
// now in background
// (b) or, sleep(20);
}
Must be a UI test. As a unit test, the line XCUIApplication *app = [[XCUIApplication alloc] init]; causes an exception.
Related
I'm writing an iOS app, that uses GoogleMaps. I've written some unit tests to test separate business logic modules. They look like this:
func testIfStationsExistNearPoint1() {
let comMangager = CommunicationManager()
comMangager.getListOfStations(coordinates: Coordinates(longitude: Constants.Point1CoordinateLongitude, latitude: Constants.Point1CoordinateLatitude), distance: 50, completion: { (json) in
if (json != nil) {
let stations = StationWrapper.sharedStationWrapper.jsonToStations(json: json!)
assert(stations.count > 0, String(format:"%ld stations", stations.count))
}
else {
XCTFail("json is nil")
}
}) { (error) in
XCTFail(String(format:"List of stations returned error: %#", error))
}
}
Note, that those unit tests are completely separated from UI (Where GoogleMaps are used. Coordinates is a custom class, not GoogleMaps CLLocationCoordinate2D).
Also note, that application itself works ok. And I'm 100% sure, that in main app, all code, that works with map drawing, is called in main thread.
Yet somehow I receive this error, when trying any of my unit tests:
com.google.Maps.LabelingBehavior (15): -[UIApplication
applicationState] must be used from main thread only
When I set a breakpoint in the very first line of this unit test, it's not even hit. Same thing for both simulator and real device.
I wonder, how's that even possible, since I'm not using Google Maps anywhere in unit tests? I've tried to search for similar cases, yet haven't found any, that was related unit testing.
Any ideas what's wrong here because I feel completely stuck?
XCode launches your application before executing unit tests.
So this error is produced by your application before Xcode starts executing unit tests. That's why your breakpoint at the beginning of test is not hit.
Placing exception breakpoint should lead you to source of the issue.
I'm new to UI Testing and am trying to integrate the feature into an existing project. I'm trying the most basic of tests to just see the framework in action but am having some difficulty. I've added a new UI Testing Bundle target to my project and am able to run my basic test. However I keep getting a "Failure attempting to launch" error that always occurs when I'm trying to actually launch the app in my test. It also says something about a "nil token for current process reference".
Here's the code that I'm testing out:
#implementation SKYPracticeUITesting
- (void)setUp {
[super setUp];
// Put setup code here. This method is called before the invocation of each test method in the class.
// In UI tests it is usually best to stop immediately when a failure occurs.
self.continueAfterFailure = NO;
// UI tests must launch the application that they test. Doing this in setup will make sure it happens for each test method.
XCUIApplication *app = [[XCUIApplication alloc] init];
// In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this.
[app launch];
}
- (void)tearDown {
// Put teardown code here. This method is called after the invocation of each test method in the class.
[super tearDown];
}
- (void)testExample {
// Use recording to get started writing UI tests.
// Use XCTAssert and related functions to verify your tests produce the correct results.
NSLog(#"Something is happening");
}
#end
Any help would be greatly appreciated, especially considering the process in Objective-C isn't very well documented.
How can I stop Xcode iOS unit tests if a fatalerror is hit?
That is in case I have 10 unit tests, but it happens that the code it calls for unit test number 5 has a coding problem (** coding issue in this case is in the test case and setup code **) and is throwing a fatalError. So in this case the unit testing stops there and does not continue to other test cases in that test class.
(not sure if this is the intended operational / process for good unit testing or not? )
Try
override func setUp() {
super.setUp()
// Put setup code here. This method is called before the invocation of each test method in the class.
continueAfterFailure = false
}
A related problem I had was stopping the unit test with a breakpoint when it fails, so that I can see the issue, without having to click or scroll through tons of test output.
Making a unit test failure breakpoint is simple, but wasn't easy to find.
Xcode 8+ there is a new breakpoint that you can add called a "Test Failure Breakpoint".
Click on Breakpoints (Left Panel ... Command + 7)
Click on + (Bottom left corner)
Click on "Test Failure Breakpoint"
References
Debugging Tests with Xcode
There is no easy fix for this. In case of an uncaught (objc) exception or a failed assert the process that runs the unit tests did receive either a mach exception or a unix signal.
Matt Gallagher has a nice solution for this, which he presents in this blog post.
It is good practice to write safe code everywhere, including your unit tests.
Regarding your problem, as Oleg Danu replied, you should set continueAfterFailure = false.
The next step is to add following before your test can crash.
var optionalVariable: Int?
XCTAssertNotNil(optionalVariable)
I recommend to add it into setUp()
In this way your test will stop before crashing Xcode, and you don't need to set any breakpoints.
In Xcode, is there a way for me run a single test case n times automatically?
Reason for doing this is that some of my beta testers are encountering random crashes in my app. I see the crash logs in TestFlight, along with the stack trace, but I can't reproduce the crash.
The crash happens infrequently but when it does, it always happens when users are trying to create a DB record, which then gets uploaded to a server. The problem with the crash logs is that my code does not make an appearance in their stack traces (all UIKit & CoreFoundation stuff - and different each time).
My solution is to run the test for that part of the app 100s of times, with the exception breakpoint set, to try to trigger the bug in my dev environment. But I don't know how to do this automatically.
It took 7 years, but as of Xcode 13, support for test repetition is now built-in.
From the Xcode 13 release notes:
Enable test repetition in your test plan, xcodebuild, or by running your test from the test diamond by Control-clicking and selecting Run Repeatedly to bring up the test repetition dialog.
You can read my answer here.
Basically you need to override invokeTest method
override func invokeTest() {
for time in 0...15 {
print("this test is being invoked: \(time) times")
super.invokeTest()
}
}
In Xcode as such, no.
You can create an XCTestCase class that hooks into the test-running methods it inherits to return multiple runs, but that tends to be annoying and mostly undocumented.
It's probably easier to instead have a "meta-test" that calls out to your other test method repeatedly:
func testOnce() {}
func testManyTimes() {
for _ in 0..<1000 { testOnce() }
}
You might need to call out to some per-test setup/teardown methods. You could perhaps work around that by instead making the loop body be something like:
let test = XCTestCase(selector: #selector(testOnce))
test.invokeTest()
This would lean on the XCTest machinery that your standard tests use, but it might gripe about not being wired into an XCTestCaseRun (or not).
So I had this idea to test the implementation of my screen tracking (with Google Analytics) on my app using UI automation.
The original idea was to build a UI script to go through the screens while checking if the tracking events are being sent accordingly. I need this as sometimes I'm not able to compose everything out of view controllers or the events are not forwarded in the expected order. Regardless of that, I should test this aspect of my app as well and I thought that UI automation was the answer.
I have implemented a script to go through the screens using the UI automation instrument and this is working correctly. I even went so far as using tuneup js to make the code more streamlined and easier to follow.
I was expecting to have something like (in general terms, the syntax is only a simplification):
Being on screen X
Tap button A
Expect screen Y and tracking event for the screen Y
However, as far as I was able to check, testing the screen tracking is something that is not possible with the UI automation.
Or am I missing something?
I thought of creating an invisible view that stays on top of all the view hierarchy and changing its name every time a new screen is loaded to allow me to test it with UI automation but the idea sounded a little over the top...
What do you people suggest? Look for another UI automation tool? Do it with unit testing instead?
Thanks in advance for any help
You could use a UIAlertView and inspect those alerts. Instead of sending the analytics events you can pop up the alert so you can check on it in UIAutomation.
Analytics abstraction frameworks like AnalyticsKit provide an easy way to change the analytics provider. And AnalyticsKit even has an example for that (take a look at the AnalyticsKitDebugProvider class). So the changes to your production code are minimal.
You could use a build configuration where you set a build variable to control the initialization of your analytics
id<AnalyticsKitProvider> provider
#ifdef USE_UI_AUTOMATION_ANALYTICS
provider = [[TestAutomationProvider alloc] init];
#else
provider = [[RealProvider alloc] initWithApiKey:API_KEY];
#endif
[AnalyticsKit initializeLoggers:#[provider]];
In UIAutomation you can test for the alert coming up. You can utilize assertions.js out of the tuneup.js package to write a function like this
function checkForAlert()
{
var alert = null;
retry( function() {
log("wait until alert appaers");
alert = UIATarget.localTarget().frontMostApp().alert();
assertNotNull(alert, "No alert found");
assertTrue("The name you can choose for the alert" == alert.name());
}, 5, 1.0);
return alert;
};
This combines waiting for the alert and testing if it finally appear. If the alert not appears, the test will fail.
In your test you use this in the following way:
var analyticAlert = checkForAlert() // if alert appears it will be in the var, otherwise the test fails at this point.
analyticAlert.buttons()["OK"].tap(); // dismiss the alert
To make this work you also need to set an onAlert handler. Otherwise UIAutomation would try to dismiss your alert immediately. This has to be done before your tests code. Alert handling is explained in the UIAutomation docs.
function MyOnAlertHandler(alert)
{
if("The name you choose"==alert.name()) // filter all alerts created by analytics provider
{
return true; // handle alert in your test
}
return false // automaticly dismiss all other
}
UIATarget.onAlert = MyOnAlertHandler; // set the alert handler