I am trying to learn automated UI testing with KIF in sample app. My simple test is failing continuously.
My code :
#import <XCTest/XCTest.h>
#import <KIF/KIF.h>
#interface AutomatedUITestsSampleUITests : KIFTestCase
#end
#implementation AutomatedUITestsSampleUITests
- (void)setUp {
[super setUp];
// Put setup code here. This method is called before the invocation of each test method in the class.
// In UI tests it is usually best to stop immediately when a failure occurs.
self.continueAfterFailure = NO;
// UI tests must launch the application that they test. Doing this in setup will make sure it happens for each test method.
[[[XCUIApplication alloc] init] launch];
// In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this.
}
- (void)tearDown {
// Put teardown code here. This method is called after the invocation of each test method in the class.
[super tearDown];
}
- (void)testExample {
[tester waitForViewWithAccessibilityLabel:#"LOGIN - Button"];
[tester tapViewWithAccessibilityLabel:#"LOGIN - Button" traits:UIAccessibilityTraitButton];
}
I have set Accessibility Label LOGIN - Button to a UIButton in my storyboard also Accessibility is enabled.
I am getting this in console :
Test Case '-[AutomatedUITestsSampleUITests testExample]' started.
t = 0.00s Start Test at 2017-06-16 13:45:07.071
t = 0.00s Set Up
t = 0.04s Launch com.UITests.AutomatedUITestsSample
t = 4.63s Waiting for accessibility to load
t = 8.77s Wait for app to idle
t = 10.48s Tear Down
Test Case '-[AutomatedUITestsSampleUITests testExample]' failed (10.868 seconds).
Test Suite 'AutomatedUITestsSampleUITests' failed at 2017-06-16 13:45:17.939.
Executed 1 test, with 1 failure (1 unexpected) in 10.868 (12.735) seconds
Test Suite 'AutomatedUITestsSampleUITests.xctest' failed at 2017-06-16 13:45:17.940.
Executed 1 test, with 1 failure (1 unexpected) in 10.868 (12.739) seconds
Test Suite 'All tests' failed at 2017-06-16 13:45:17.941.
Executed 1 test, with 1 failure (1 unexpected) in 10.868 (12.743) seconds
It's a very simple test. Why is it failing? Thanks in advance.
If you have access to the XCUIApplication there is something wrong with your project configuration. KIF tests target should be "Unit Test Target", not "UI Test Target". With correct configuration there is no need to launch the app (and no possibility of app launch manipulation) - it is launched when the unit tests begin. When you use UI test target your tests are running in separate process and KIF doesn't have access to your app.
I suggest to configure your test target again with this guide
Related
I have a large Xcode 10 project for an iOS app that has nearly 40 screens and associated view controllers (all are Objective-C). Some of the code is a decade old and some new. The project has no UI or unit tests, but I'd like to enable code coverage and manually exercise the app. Is there a way to add, say, one test that will allow me to run the entire app manually and track which methods and functions are or are not used?
It was simpler than I expected, just not easy because I'd never done any tests before.
I made a UI test that keeps running until some testable event occurs that I can trigger, such as quitting the app (putting it into the background).
Run this test and exercise the app. When done, press Home. The test will exit and the code coverage for whatever you did while running will be shown in the Code Coverage column of the editor.
Add a new target for UI testing.
Edit Scheme, Test, Options: Code Coverage for: all targets.
Add this code as a UI test:
- (void)testCoverage
{
// UI tests must launch the application that they test.
XCUIApplication *app = [[XCUIApplication alloc] init];
[app launch];
// Code Coverage is turned on in the Test scheme
// (a) loop until Home button pushed...
while ( [app waitForState:XCUIApplicationStateRunningBackground timeout:60] == NO ) {
// still in the foreground
};
// now in background
// (b) or, sleep(20);
}
Must be a UI test. As a unit test, the line XCUIApplication *app = [[XCUIApplication alloc] init]; causes an exception.
I am running a few unit tests inside Xcode with XCTests. When I run them, all the tests either pass and fail and show the little green check or red cross. The main issue seems to be that my tests are prematurely canceling and they are not being run properly.
Here's some more information:
When the tests are run, there is no console output whatsoever which was not happening before. By that I mean I did have the expected console output before, now I don't.
When the tests are run, it does not require me to unlock my iPhone so that the app can run. Moreover, even if my iPhone is unlocked, it doesn't run the application which it would do before.
When running my tests, it finishes compiling and then very quickly says "Canceling..." in the information bar in Xcode (the top and centered bar that shows warnings and "Building " when you build your project).
I am using the .xcworkspace file (becasue I am using Pods)
I have one build scheme which is the project.
I had these same issues plus the issues of the green checks and red crosses not showing up on my tests after they ran and the sometimes my tests not showing up at all in the testing tab. I fixed these issues by deleting the Podfile.lock file, Pods directory, and the .xcworkspace directory and then running Pod install. After doing this, I had the same issues, except those two issues.
I tried running new tests on a different branch, same issues.
Before running into any of these issues, I was just writing tests and trying different methods of writing tests. I've reverted all those changes and I still get these errors.
I have also found the same issues when running the tests no the simulator.
When I compile and run the app from Xcode, it runs fine.
Any advice on how to solve this would be great! Please ask for more details if needed.
Here's a video of me running my tests. I set up a new project and new unit test target to show what's going on. Note: There's an empty test in there and a test that should always pass:
As you can see, the issue is the tests are not running properly. The green and red crosses are not showing up anymore, the test that's supposed to always pass is failing, there is no console output, and in the top information bar it's saying "canceling..." almost immediately. Any advice on why this canceling is happening would be great!
Here are some screenshots of how the code is set up:
Here's the code as well (it's just the basic code after creating a new unit test, except I added in one assert that should always pass):
import XCTest
#testable import example
class exampleTests: XCTestCase {
override func setUp() {
// Put setup code here. This method is called before the invocation of each test method in the class.
}
override func tearDown() {
// Put teardown code here. This method is called after the invocation of each test method in the class.
}
func testExample() {
// This is an example of a functional test case.
// Use XCTAssert and related functions to verify your tests produce the correct results.
XCTAssertTrue(true)
}
func testPerformanceExample() {
// This is an example of a performance test case.
self.measure {
// Put the code you want to measure the time of here.
}
}
}
Edit:
So, as per #CPR's suggestion, I tried it again on the simulator and the errors were basically all gone. I tried it again on the physical device, but the errors persisted. Note: I had tried it before on the simulator and got the same errors. I didn't change anything since then so I am unsure of why it worked now. Here's the console output from running the tests on the sim (not sure if that will help, but I'll add it anyway):
Test Suite 'All tests' started at 2018-12-06 03:59:53.045
Test Suite 'exampleTests.xctest' started at 2018-12-06 03:59:53.046
Test Suite 'exampleTests' started at 2018-12-06 03:59:53.046
Test Case '-[exampleTests.exampleTests testExample]' started.
Test Case '-[exampleTests.exampleTests testExample]' passed (0.001 seconds).
Test Case '-[exampleTests.exampleTests testPerformanceExample]' started.
/Users/akashkundu/Documents/example/exampleTests/exampleTests.swift:30: Test Case '-[exampleTests.exampleTests testPerformanceExample]' measured [Time, seconds] average: 0.000, relative standard deviation: 115.251%, values: [0.000004, 0.000001, 0.000001, 0.000001, 0.000001, 0.000001, 0.000001, 0.000001, 0.000001, 0.000001], performanceMetricID:com.apple.XCTPerformanceMetric_WallClockTime, baselineName: "", baselineAverage: , maxPercentRegression: 10.000%, maxPercentRelativeStandardDeviation: 10.000%, maxRegression: 0.100, maxStandardDeviation: 0.100
Test Case '-[exampleTests.exampleTests testPerformanceExample]' passed (0.348 seconds).
Test Suite 'exampleTests' passed at 2018-12-06 03:59:53.396.
Executed 2 tests, with 0 failures (0 unexpected) in 0.348 (0.350) seconds
Test Suite 'exampleTests.xctest' passed at 2018-12-06 03:59:53.396.
Executed 2 tests, with 0 failures (0 unexpected) in 0.348 (0.350) seconds
Test Suite 'All tests' passed at 2018-12-06 03:59:53.397.
Executed 2 tests, with 0 failures (0 unexpected) in 0.348 (0.351) seconds
Can anyone shed some light on why running the tests on the physical device causes these errors?
Edit:
Here's the error/log message from the "Messages" tab in Xcode found by #CPR:
Here's the error/log code in text:
example.app encountered an error (Failed to establish communication with the test runner. (Underlying error: Unable to connect to test manager on 8b441d96d063b3b6abf55b06115441d160e85e67. (Underlying error: kAMDMuxConnectError: Could not connect to the device.)))
-----------------------------------------------------------------------
Solution:
Thanks to #CPR, the solution is to simply restart the device. Another lesson learned is to use the "Messages" tab to see logs and errors which is the right most tab next to the "Breakpoint" tab.
His full answer here.
Problem resolved in comments. For future readers, here are some good things to check if you encounter similar issues:
Check Xcode messages tab to look for build errors.
Restart device/simulator as appropriate. This fixed the problem in this case.
Uninstall/reinstall app on device/simulator.
Are certs/provisioning profiles valid? Try generating new ones and re-running.
Does it work on the simulator? If yes, this is likely an issue with the device itself.
Are breakpoints hit when you run the code? If yes, then this is a test failure. If no, the problem is happening before the tests are even run (as was the case here).
I'm new to UI Testing and am trying to integrate the feature into an existing project. I'm trying the most basic of tests to just see the framework in action but am having some difficulty. I've added a new UI Testing Bundle target to my project and am able to run my basic test. However I keep getting a "Failure attempting to launch" error that always occurs when I'm trying to actually launch the app in my test. It also says something about a "nil token for current process reference".
Here's the code that I'm testing out:
#implementation SKYPracticeUITesting
- (void)setUp {
[super setUp];
// Put setup code here. This method is called before the invocation of each test method in the class.
// In UI tests it is usually best to stop immediately when a failure occurs.
self.continueAfterFailure = NO;
// UI tests must launch the application that they test. Doing this in setup will make sure it happens for each test method.
XCUIApplication *app = [[XCUIApplication alloc] init];
// In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this.
[app launch];
}
- (void)tearDown {
// Put teardown code here. This method is called after the invocation of each test method in the class.
[super tearDown];
}
- (void)testExample {
// Use recording to get started writing UI tests.
// Use XCTAssert and related functions to verify your tests produce the correct results.
NSLog(#"Something is happening");
}
#end
Any help would be greatly appreciated, especially considering the process in Objective-C isn't very well documented.
How can I stop Xcode iOS unit tests if a fatalerror is hit?
That is in case I have 10 unit tests, but it happens that the code it calls for unit test number 5 has a coding problem (** coding issue in this case is in the test case and setup code **) and is throwing a fatalError. So in this case the unit testing stops there and does not continue to other test cases in that test class.
(not sure if this is the intended operational / process for good unit testing or not? )
Try
override func setUp() {
super.setUp()
// Put setup code here. This method is called before the invocation of each test method in the class.
continueAfterFailure = false
}
A related problem I had was stopping the unit test with a breakpoint when it fails, so that I can see the issue, without having to click or scroll through tons of test output.
Making a unit test failure breakpoint is simple, but wasn't easy to find.
Xcode 8+ there is a new breakpoint that you can add called a "Test Failure Breakpoint".
Click on Breakpoints (Left Panel ... Command + 7)
Click on + (Bottom left corner)
Click on "Test Failure Breakpoint"
References
Debugging Tests with Xcode
There is no easy fix for this. In case of an uncaught (objc) exception or a failed assert the process that runs the unit tests did receive either a mach exception or a unix signal.
Matt Gallagher has a nice solution for this, which he presents in this blog post.
It is good practice to write safe code everywhere, including your unit tests.
Regarding your problem, as Oleg Danu replied, you should set continueAfterFailure = false.
The next step is to add following before your test can crash.
var optionalVariable: Int?
XCTAssertNotNil(optionalVariable)
I recommend to add it into setUp()
In this way your test will stop before crashing Xcode, and you don't need to set any breakpoints.
When I run a single XCTest class, all tests within succeed.
However when I run it together with other XCTest, some tests in the class fail.
setUp and tearDown method are implemented correctly as following:
- (void)setUp {
[super setUp];
...
}
- (void)tearDown {
...
[super tearDown];
}
I set a breakpoint in the code that should be executed in the test. When I was running the tests in isolation, the breakpoint was reached; when I was running it with other tests, the breakpoint was not reached. I'm thinking that maybe XCTest has some caching behaviors? If so, how to turn it off?
Does anyone know why this might happen?
Thanks a bunch!
I was seeing the same behavior, and my issue was because I had a static variable in my method I was testing that was holding onto its value across tests.
I faced the similar issue when I run all tests together. I could able to fix it by putting Assertion in Dispatch Delay,
DispatchQueue.main.asyncAfter(deadline: .now() + 2.0) {
XCTAssert(self.response != nil)
}
I hope the above solution works for you as well.