I'm testing real web service calls with OCMock.
Right now I'm doing something like:
- (void)testWebservice
{
id mydelegatemock = [OCMockObject mockForProtocol:#protocol(MySUTDelegate)];
[[mydelegatemock expect] someMethod:[OCMArg any]];
[SUT sutWithDelegate:mydelegatemock];
// we need to wait for real result
[[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:2.0]];
[(OCMockObject*)mydelegatemock verify];
}
It works fine, but it implies that every such test will take 2 seconds.
Is there a way I can set a timeout of e.g. 2 seconds, and let a call to someMethod of mydelegatemock immediately verify and complete the test case?
I do this using a handy utility function I found at this link:
#import <Foundation/Foundation.h>
#import <OCMock/OCMock.h>
#interface TestUtils : NSObject
+ (void)waitForVerifiedMock:(OCMockObject *)mock delay:(NSTimeInterval)delay;
#end
And the implementation:
#import "TestUtils.h"
#implementation TestUtils
+ (void)waitForVerifiedMock:(OCMockObject *)inMock delay:(NSTimeInterval)inDelay
{
NSTimeInterval i = 0;
while (i < inDelay)
{
#try
{
[inMock verify];
return;
}
#catch (NSException *e) {}
[[NSRunLoop currentRunLoop] runUntilDate:[NSDate dateWithTimeIntervalSinceNow:0.5]];
i+=0.5;
}
[inMock verify];
}
#end
This allows me to to wait up to a maximum delay (in seconds) without waiting the full amount each time.
I would separate the functional testing of your web services (if you need to do that at all) from the unit testing of your class that processes the web service result.
To unit test, you should mock the web service call, providing a mock result. Then your test would verify that, for that well-defined result, your class behaves accordingly.
If you also want to do functional testing of your web service (say that it returns a specific response given some request), you don't need to mock anything--just call the service and make assertions on the result.
By separating out your tests, you have finer control over the test runs. For example, you could run your fast-running unit tests every time you change code, but run your slow-running functional tests nightly, on a dedicated server, or as needed. And when a test breaks, you'll know whether it's your code or something wrong with the web service.
You could also switch to GHUnit, like a fellow member suggests in the answer to this related question:
SenTestingKit in Xcode 4: Asynchronous testing?
You can find GHUnit here
https://github.com/gabriel/gh-unit
Related
I have a number of asynchronous unit tests which work correctly on their own using expectations. However when I run all the tests in the suit, they do not wait for each other to complete - the asynchronous callbacks are still pending when the next tests start running. What I want is for each test to wait for the expectations in the previous test before running. These tests use a shared database, so having them overlap leads to annoying additional complexity, and failing tests when run as a suite.
- (void)testSignIn {
// This is an example of a functional test case.
// Use XCTAssert and related functions to verify your tests produce the correct results.
XCTestExpectation *expectation =
[self expectationWithDescription:#"Expectations"];
[_userManager signInWithUsername:kUserEmail andPassword:kUserPassword
success:^{
XCTAssertNotNil([_userManager getCurrentUser]);
XCTAssertNotNil([_userManager getCurrentUser].plan);
XCTAssertTrue([_userManager getCurrentUser].plan.liveStream == TRUE);
[expectation fulfill];
} failure:^(EDApiError *apiError) {
XCTAssertTrue(FALSE); // Should not fail
[expectation fulfill];
}];
[self waitForExpectationsWithTimeout:5.0 handler:^(NSError *error) {
if (error) {
NSLog(#"Timeout Error: %#", error);
}
}];
}
Use XCTWaiter or waitForExpectations(timeout:handler:) to stall the completion of each test until the expectations have been fulfilled.
This blog post explains some of the more complex pitfalls you might come across when writing asynchronous tests and how to prevent them: https://jeremywsherman.com/blog/2016/03/19/xctestexpectation-gotchas/
I've found you need to fulfil expectations on the main thread - important if your async completion block is maybe running on another thread. I've also found that the problem can be triggered by a faulty test that ran earlier, so it isn't always caused by the test the failure manifests in.
Also, watch if you're spawning lots of async blocks (I do this for crazy thread safety tests, or tests checking exclusive access to a resource is working as expected, and in the order expected). Once you fulfil the expectation, the testing moves on to the next test, but if you've fired off a lot of async blocks after that point, they're maybe still running.
I'm new to UI Testing and am trying to integrate the feature into an existing project. I'm trying the most basic of tests to just see the framework in action but am having some difficulty. I've added a new UI Testing Bundle target to my project and am able to run my basic test. However I keep getting a "Failure attempting to launch" error that always occurs when I'm trying to actually launch the app in my test. It also says something about a "nil token for current process reference".
Here's the code that I'm testing out:
#implementation SKYPracticeUITesting
- (void)setUp {
[super setUp];
// Put setup code here. This method is called before the invocation of each test method in the class.
// In UI tests it is usually best to stop immediately when a failure occurs.
self.continueAfterFailure = NO;
// UI tests must launch the application that they test. Doing this in setup will make sure it happens for each test method.
XCUIApplication *app = [[XCUIApplication alloc] init];
// In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this.
[app launch];
}
- (void)tearDown {
// Put teardown code here. This method is called after the invocation of each test method in the class.
[super tearDown];
}
- (void)testExample {
// Use recording to get started writing UI tests.
// Use XCTAssert and related functions to verify your tests produce the correct results.
NSLog(#"Something is happening");
}
#end
Any help would be greatly appreciated, especially considering the process in Objective-C isn't very well documented.
I have a test suite in which there are many test classes each one having many tests.
Now if I define
+ (void)tearDown
in any class, then it will be run after all the tests of that class only.
and
- (void)tearDown
will be run after each test.
I want to define a tearDown method and want it to be run after all the tests of all the classes.
Fox eg, if I have 5 classes each having 7 tests each. So I want to run this tearDown function after all the 35 tests.
Since you're looking for a way to call __gcov_flush() after all tests have finished, see https://qualitycoding.org/ios-7-code-coverage/. The latest code from Google Toolbox for Mac shows how to do this using test observers, with versions for both SenTestingKit and XCTest.
If I understand correctly your question, you can take note of completion of each method using a global variable/flag using completion block methods like this:
+ (void)myMethod:(UIView *)exampleView completion:(void (^)(BOOL finished))completion {
if (completion) {
completion(finished);
}
}
Look at this for a better explanation.
Then create a method that checks if all taskes are executed and that runs final test when needed like this:
- (void)finalTest {
if (CHECK GLOBAL FLAG FOR COMPLETION OF OTHER TASKES) {
// do final test
} else {
// repeat check after 0.1 seconds
[self performSelector:#selector(finalTest)
withObject:nil
afterDelay:0.1];
}
}
I use a subclass of NSOperation to upload large files to AWS S3 using Amazon's iOS SDK (v1.3.2). This all works fine, but some beta testers experience deadlocks (iOS 5.1.1). The result is that the NSOperationQueue in which the operations are scheduled is blocked as only one operation is allowed to run at one time. The problem is that I cannot reproduce the issue whereas the beta testers experience this problem every single time.
The operation is quite complex due to how the AWS iOS SDK works. However, the problem is not related to the AWS iOS SDK as far as I know based on my testing. The operation's main method is pasted below. The idea of the operation's main method is based on this Stack Overflow question.
- (void)main {
// Operation Should Terminate
_operationShouldTerminate = NO;
// Notify Delegate
dispatch_async(dispatch_get_main_queue(), ^{
[self.delegate operation:self isPreparingUploadWithUuid:self.uuid];
});
// Increment Network Activity Count
[self incrementNetworkActivityCount];
// Verify S3 Credentials
[self verifyS3Credentials];
while (!_operationShouldTerminate) {
if ([self isCancelled]) {
_operationShouldTerminate = YES;
} else {
// Create Run Loop
[[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:[NSDate distantFuture]];
}
}
// Decrement Network Activity Count
[self decrementNetworkActivityCount];
NSLog(#"Operation Will Terminate");
}
The method that finalizes the multipart upload sets the boolean _operationShouldTerminate to YES to terminate the operation. That method looks like this.
- (void)finalizeMultipartUpload {
// Notify Delegate
dispatch_async(dispatch_get_main_queue(), ^{
[self.delegate operation:self didFinishUploadingUploadWithUuid:self.uuid];
});
// Operation Should Terminate
_operationShouldTerminate = YES;
NSLog(#"Finalize Multipart Upload");
}
The final log statement is printed to the console, but the while loop in the main method does not seem to exit as the final log statement in the operation's main method is not printed to the console. As a result, the operation queue in which the operation is scheduled, is blocked and any scheduled operations are not executed as a result.
The operation's isFinished method simply returns _operationShouldTerminate as seen below.
- (BOOL)isFinished {
return _operationShouldTerminate;
}
It is odd that the while loop is not exited and it is even more odd that it does not happen on any of my own test devices (iPhone 3GS, iPad 1, and iPad 3). Any help or pointers are much appreciated.
The solution to the problem is both complex and simple as it turns out. What I wrongly assumed was that the methods and delegate callbacks of the operation were executed on the same thread, that is, the thread on which the operation's main method was called. This is not always the case.
Even though this was true in my test and on my devices (iPhone 3GS), which is why I did not experience the problem myself. My beta testers, however, used devices with multicore processors (iPhone 4/4S), which caused some of the code to be executed on a thread different from the thread on which the operation's main method was invoked.
The result of this is that _operationShouldTerminate was modified in the finalizeMultipartUpload method on the wrong thread. This in turn means that the while loop of the main method was not exited properly resulting in the operation deadlocking.
In short, the solution is to update _operationShouldTerminate on the same thread as the main method was invoked on. This will properly exit the while loop and exit the operation.
There are a number of problems with your code, and I can offer two solutions:
1) read up on Concurrent NSOperations in Apple's Concurrency Programming Guide. To keep the runLoop "alive" you have to add either a port or schedule a timer. The main loop should contain a autorelease pool as you may not get one (see Memory Management in that same memo). You need to implement KVO to let the operationQueue know when your operation is finished.
2) Or, you can adopt a small amount of field tested hardened code and reuse it. That Xcode project contains three classes of interest to you: a ConcurrentOperation file that does well what you are trying to accomplish above. The Webfetcher.m class shows how to subclass the concurrent operation to perform an asynchronous URL fetch from the web. And the OperationsRunner is a small helper file you can add to any kind of class to manage the operations queue (run, cancel, query, etc). All of the above are less than 100 lines of code, and provide a base for you to get your code working. The OperationsRunner.h file provide a "how to do" too.
How can I unit test my NSURLConnection delegate?
I made a ConnectionDelegate class which conforms to different protocols to serve data from the web to different ViewControllers. Before I get too far I want to start writing my unit tests.
But I don't know how to test them as a unit without the internet connection. I would like also what I should do to treat the asynchronous callbacks.
This is similar to Jon's response, couldn't fit it into a comment, though. The first step is to make sure you are not creating a real connection. The easiest way to achieve this is to pull the creation of the connection into a factory method and then substitute the factory method in your test. With OCMock's partial mock support this could look like this.
In your real class:
- (NSURLConnection *)newAsynchronousRequest:(NSURLRequest *)request
{
return [[NSURLConnection alloc] initWithRequest:request delegate:self];
}
In your test:
id objectUnderTest = /* create your object */
id partialMock = [OCMockObject partialMockForObject:objectUnderTest];
NSURLConnection *dummyUrlConnection = [[NSURLConnection alloc]
initWithRequest:[NSURLRequest requestWithURL:[NSURL URLWithString:#"file:foo"]]
delegate:nil startImmediately:NO];
[[[partialMock stub] andReturn:dummyUrlConnection] newAsynchronousRequest:[OCMArg any]];
Now, when your object under test tries to create the URL connection it actually gets the dummy connection created in the test. The dummy connection doesn't have to be valid, because we're not starting it and it never gets used. If your code does use the connection you could return another mock, one that mocks NSURLConnection.
The second step is to invoke the method on your object that triggers the creation of the NSURLConnection:
[objectUnderTest doRequest];
Because the object under test is not using the real connection we can now call the delegate methods from the test. For the NSURLResponse we're using another mock, the response data is created from a string that's defined somewhere else in the test:
int statusCode = 200;
id responseMock = [OCMockObject mockForClass:[NSHTTPURLResponse class]];
[[[responseMock stub] andReturnValue:OCMOCK_VALUE(statusCode)] statusCode];
[objectUnderTest connection:dummyUrlConnection didReceiveResponse:responseMock];
NSData *responseData = [RESPONSE_TEXT dataUsingEncoding:NSASCIIStringEncoding];
[objectUnderTest connection:dummyUrlConnection didReceiveData:responseData];
[objectUnderTest connectionDidFinishLoading:dummyUrlConnection];
That's it. You've effectively faked all the interactions the object under test has with the connection, and now you can check whether it is in the state it should be in.
If you want to see some "real" code, have a look at the tests for a class from the CCMenu project that uses NSURLConnections. This is a little bit confusing because the class that's tested is named connection, too.
http://ccmenu.svn.sourceforge.net/viewvc/ccmenu/trunk/CCMenuTests/Classes/CCMConnectionTest.m?revision=129&view=markup
EDIT (2-18-2014): I just stumbled across this article with a more elegant solution.
http://www.infinite-loop.dk/blog/2011/04/unittesting-asynchronous-network-access/
Essentially, you have the following method:
- (BOOL)waitForCompletion:(NSTimeInterval)timeoutSecs {
NSDate *timeoutDate = [NSDate dateWithTimeIntervalSinceNow:timeoutSecs];
do {
[[NSRunLoop currentRunLoop] runMode:NSDefaultRunLoopMode beforeDate:timeoutDate];
if([timeoutDate timeIntervalSinceNow] < 0.0)
break;
} while (!done);
return done;
}
At the end of your test method, you make sure things haven't timed out:
STAssertTrue([self waitForCompletion:5.0], #"Timeout");
Basic format:
- (void)testAsync
{
// 1. Call method which executes something asynchronously
[obj doAsyncOnSuccess:^(id result) {
STAssertNotNil(result);
done = YES;
}
onError:^(NSError *error) [
STFail();
done = YES;
}
// 2. Determine timeout
STAssertTrue([self waitForCompletion:5.0], #"Timeout");
}
==============
I'm late to the party, but I came across a very simple solution. (Many thanks to http://www.cocoabuilder.com/archive/xcode/247124-asynchronous-unit-testing.html)
.h file:
#property (nonatomic) BOOL isDone;
.m file:
- (void)testAsynchronousMethod
{
// 1. call method which executes something asynchronously.
// 2. let the run loop do its thing and wait until self.isDone == YES
self.isDone = NO;
NSDate *untilDate;
while (!self.isDone)
{
untilDate = [NSDate dateWithTimeIntervalSinceNow:1.0]
[[NSRunLoop currentRunLoop] runUntilDate:untilDate];
NSLog(#"Polling...");
}
// 3. test what you want to test
}
isDone is set to YES in the thread that the asynchronous method is executing.
So in this case, I created and started the NSURLConnection at step 1 and made the delegate of it this test class. In
- (void)connection:(NSURLConnection *)connection didReceiveResponse:(NSURLResponse *)response
I set self.isDone = YES;. We break out of the while loop and the test is executed. Done.
I avoid networking in unit tests. Instead:
I isolate NSURLConnection within a method.
I create a testing subclass, overriding that method to remove all traces of NSURLConnection.
I write one test to ensure that the method in question will get invoked when I want. Then I know it'll fire off an NSURLConnection in real life.
Then I concentrate on the more interesting part: Synthesize mock NSURLResponses with various characteristics, and pass them to the NSURLConnectionDelegate methods.
My favorite way of doing this is to subclass NSURLProtocol and have it respond to all http requests - or other protocols for that matter. You then register the test protocol in your -setup method and unregisters it in your -tearDown method.
You can then have this test protocol serve some well known data back to your code so you can validate it in your unit tests.
I have written a few blog articles about this subject. The most relevant for your problem would probably be Using NSURLProtocol for Injecting Test Data and Unit Testing Asynchronous Network Access.
You may also want to take a look my ILCannedURLProtocol which is described in the previous articles. The source is available at Github.
you might want to give this a chance:
https://github.com/marianoabdala/ZRYAsyncTestCase
Here's some sample code of a NSURLConnection being unit tested:
https://github.com/marianoabdala/ZRYAsyncTestCase/blob/12a84c7f1af1a861f76c7825aef6d9d6c53fd1ca/SampleProject/SampleProjectTests/SampleProjectTests.m#L33-L56
Here's what I do:
Get XAMPP Control Panel
http://www.apachefriends.org/en/xampp.html
Start the apache server
In your ~/Sites folder, put a test file (whatever data you want, we'll call it my file.test).
Start your delegate using the URL http://localhost/~username/myfile.test
Stop the apache server when not using it.