When I run a single XCTest class, all tests within succeed.
However when I run it together with other XCTest, some tests in the class fail.
setUp and tearDown method are implemented correctly as following:
- (void)setUp {
[super setUp];
...
}
- (void)tearDown {
...
[super tearDown];
}
I set a breakpoint in the code that should be executed in the test. When I was running the tests in isolation, the breakpoint was reached; when I was running it with other tests, the breakpoint was not reached. I'm thinking that maybe XCTest has some caching behaviors? If so, how to turn it off?
Does anyone know why this might happen?
Thanks a bunch!
I was seeing the same behavior, and my issue was because I had a static variable in my method I was testing that was holding onto its value across tests.
I faced the similar issue when I run all tests together. I could able to fix it by putting Assertion in Dispatch Delay,
DispatchQueue.main.asyncAfter(deadline: .now() + 2.0) {
XCTAssert(self.response != nil)
}
I hope the above solution works for you as well.
Related
I have a number of asynchronous unit tests which work correctly on their own using expectations. However when I run all the tests in the suit, they do not wait for each other to complete - the asynchronous callbacks are still pending when the next tests start running. What I want is for each test to wait for the expectations in the previous test before running. These tests use a shared database, so having them overlap leads to annoying additional complexity, and failing tests when run as a suite.
- (void)testSignIn {
// This is an example of a functional test case.
// Use XCTAssert and related functions to verify your tests produce the correct results.
XCTestExpectation *expectation =
[self expectationWithDescription:#"Expectations"];
[_userManager signInWithUsername:kUserEmail andPassword:kUserPassword
success:^{
XCTAssertNotNil([_userManager getCurrentUser]);
XCTAssertNotNil([_userManager getCurrentUser].plan);
XCTAssertTrue([_userManager getCurrentUser].plan.liveStream == TRUE);
[expectation fulfill];
} failure:^(EDApiError *apiError) {
XCTAssertTrue(FALSE); // Should not fail
[expectation fulfill];
}];
[self waitForExpectationsWithTimeout:5.0 handler:^(NSError *error) {
if (error) {
NSLog(#"Timeout Error: %#", error);
}
}];
}
Use XCTWaiter or waitForExpectations(timeout:handler:) to stall the completion of each test until the expectations have been fulfilled.
This blog post explains some of the more complex pitfalls you might come across when writing asynchronous tests and how to prevent them: https://jeremywsherman.com/blog/2016/03/19/xctestexpectation-gotchas/
I've found you need to fulfil expectations on the main thread - important if your async completion block is maybe running on another thread. I've also found that the problem can be triggered by a faulty test that ran earlier, so it isn't always caused by the test the failure manifests in.
Also, watch if you're spawning lots of async blocks (I do this for crazy thread safety tests, or tests checking exclusive access to a resource is working as expected, and in the order expected). Once you fulfil the expectation, the testing moves on to the next test, but if you've fired off a lot of async blocks after that point, they're maybe still running.
I'm implementing the performance test for iOS app, and a strange result is derived from it and don't know why.
As Xcode performance test runs given code 10 times, test run on my machine always results such that the first attempt runs in a huge amount of time compared to other 9 attempts.
Jobs that I want to test performance is very simple, it iterates through given array and investigates with a certain conditions. There's no network connection, array manipulation, or reusing of previous variables whatsoever.
Test codes that I wrote is like the following (all of these result in the same behavior).
a. Initialize variables outside the measure block
- (void)testSomething {
// do some initialization
[self measureBlock:^{
// run code to test performance
}];
}
b. Initialize variables inside the measure block
- (void)testSomething {
[self measureBlock:^{
// do some initialization
// run code to test performance
}];
}
c. Initialize variables in -setUp, and use them in test method
- (void)setUp {
// do some initialization here, and capture the pointers as properties
}
- (void)testSomething {
[self measureBlock:^{
// run code to test performance, using properties initialized in setUp
}];
}
- (void)tearDown {
// clean up properties
}
The results are very strange, and the other 9 attempts seem not to be reliable because of the number of first attempt. Why does the first iteration take so much longer?
I'm new to UI Testing and am trying to integrate the feature into an existing project. I'm trying the most basic of tests to just see the framework in action but am having some difficulty. I've added a new UI Testing Bundle target to my project and am able to run my basic test. However I keep getting a "Failure attempting to launch" error that always occurs when I'm trying to actually launch the app in my test. It also says something about a "nil token for current process reference".
Here's the code that I'm testing out:
#implementation SKYPracticeUITesting
- (void)setUp {
[super setUp];
// Put setup code here. This method is called before the invocation of each test method in the class.
// In UI tests it is usually best to stop immediately when a failure occurs.
self.continueAfterFailure = NO;
// UI tests must launch the application that they test. Doing this in setup will make sure it happens for each test method.
XCUIApplication *app = [[XCUIApplication alloc] init];
// In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this.
[app launch];
}
- (void)tearDown {
// Put teardown code here. This method is called after the invocation of each test method in the class.
[super tearDown];
}
- (void)testExample {
// Use recording to get started writing UI tests.
// Use XCTAssert and related functions to verify your tests produce the correct results.
NSLog(#"Something is happening");
}
#end
Any help would be greatly appreciated, especially considering the process in Objective-C isn't very well documented.
In Xcode, is there a way for me run a single test case n times automatically?
Reason for doing this is that some of my beta testers are encountering random crashes in my app. I see the crash logs in TestFlight, along with the stack trace, but I can't reproduce the crash.
The crash happens infrequently but when it does, it always happens when users are trying to create a DB record, which then gets uploaded to a server. The problem with the crash logs is that my code does not make an appearance in their stack traces (all UIKit & CoreFoundation stuff - and different each time).
My solution is to run the test for that part of the app 100s of times, with the exception breakpoint set, to try to trigger the bug in my dev environment. But I don't know how to do this automatically.
It took 7 years, but as of Xcode 13, support for test repetition is now built-in.
From the Xcode 13 release notes:
Enable test repetition in your test plan, xcodebuild, or by running your test from the test diamond by Control-clicking and selecting Run Repeatedly to bring up the test repetition dialog.
You can read my answer here.
Basically you need to override invokeTest method
override func invokeTest() {
for time in 0...15 {
print("this test is being invoked: \(time) times")
super.invokeTest()
}
}
In Xcode as such, no.
You can create an XCTestCase class that hooks into the test-running methods it inherits to return multiple runs, but that tends to be annoying and mostly undocumented.
It's probably easier to instead have a "meta-test" that calls out to your other test method repeatedly:
func testOnce() {}
func testManyTimes() {
for _ in 0..<1000 { testOnce() }
}
You might need to call out to some per-test setup/teardown methods. You could perhaps work around that by instead making the loop body be something like:
let test = XCTestCase(selector: #selector(testOnce))
test.invokeTest()
This would lean on the XCTest machinery that your standard tests use, but it might gripe about not being wired into an XCTestCaseRun (or not).
I have a test suite in which there are many test classes each one having many tests.
Now if I define
+ (void)tearDown
in any class, then it will be run after all the tests of that class only.
and
- (void)tearDown
will be run after each test.
I want to define a tearDown method and want it to be run after all the tests of all the classes.
Fox eg, if I have 5 classes each having 7 tests each. So I want to run this tearDown function after all the 35 tests.
Since you're looking for a way to call __gcov_flush() after all tests have finished, see https://qualitycoding.org/ios-7-code-coverage/. The latest code from Google Toolbox for Mac shows how to do this using test observers, with versions for both SenTestingKit and XCTest.
If I understand correctly your question, you can take note of completion of each method using a global variable/flag using completion block methods like this:
+ (void)myMethod:(UIView *)exampleView completion:(void (^)(BOOL finished))completion {
if (completion) {
completion(finished);
}
}
Look at this for a better explanation.
Then create a method that checks if all taskes are executed and that runs final test when needed like this:
- (void)finalTest {
if (CHECK GLOBAL FLAG FOR COMPLETION OF OTHER TASKES) {
// do final test
} else {
// repeat check after 0.1 seconds
[self performSelector:#selector(finalTest)
withObject:nil
afterDelay:0.1];
}
}