So, i have already existing project at my hands and i’m trying to create some UI tests by using this new fancy UI Testing Bundle provided by apple. The problem is that test target doesn't have access to any external framework (and i need to do some setup with one of them). Adding framework in build phases and coping framework search path from main target doesn't do anything.
After day of browsing i found out only one thing, that ”makes things kinda different”. By setting up Bundle Loader and Test Host to $(BUILT_PRODUCTS_DIR)/App.app/App , i still couldn't import external frameworks to test.m, but i could import classes that do that for them self. And it all would be fine and dandy unless it didn't break some stuff. By setting Bundle and Host now my UI test is unable to execute launch method:
[[[XCUIApplication alloc] init] launch];
It crashes with error: Assertion Failure: UI Testing Failure - App state is still not terminated.
In the end i could remove launch method from setup and trigger every single test manually, so it restarts application every time before executing, but this solution seems so wrong (especially for some bigger projects). Does anyone know proper way to handle this problem?
What I've done for this is add an environment variable to XCUIApplication to specify UI tests are being run. I then have a pre-processor check for #DEBUG in a main part of the application, and then check if the test environment variable has been set; if it has, do the necessary steps for the UI tests.
Essentially, this will allow you to configure your app to how you need it for the UI tests to run. It also means the pre-processor check will strip out that setup code entirely for the release build.
Related
I have an iOS app in Xcode 8.2. It has a test target / scheme, for which “Gather coverage data” is checked in the scheme’s Test / Info settings. Coverage data is not gathered. I see how many times a line was iterated over in the gutter as usual, but the Report navigator’s test runs don’t indicate any coverage at all.
I’m wondering if this is because I’ve set the tests to run hostless, i.e. without needing to actually fire up my app – they’re pure logic tests.
Is this possible?
Yes a hostless XCTest target should gather code coverage data.
'iOS Unit Testing' (aka XCTest) bundles, which test a dynamic framework or something else which doesn't require the application environment to exist to run, should happily collect code coverage data and display it in Xcode. Even of the Host Application is set to None. This works either when running Xcode > Product > Test on the Scheme for the framework under test or on the Scheme for the unit tests themselves (if the test bundle is listed in the Test pane of the Scheme editor).
Your problem must be somewhere else, sorry. Its hard for me to guess what the problem is, I suggest you try making a fresh project and see if you can reproduce the problem.
I once asked a question related to XCTests. And in one of the answers I was told that it is a common practice to use a separate test target (other than the main app) when running unit tests (at least, in iOS development). I tried to find some sources about it, but I couldn't
I understand, that it is probably a best practice, so I would really like to understand it. Could someone explain to me why is it important, what benefits do I get from it and how should I go about doing it? Links to some articles explaining the issue will be much appreciated.
P.S. I understand that I need special environment for tests (fake in-memory database, mocked networking layer, etc.), but up until now I managed to achieve it without a separate test host. But I believe that there might be a better way.
To answer your points:
Why is using a separate test target important?
Separation of concerns. A unit test target does not have the same kind of structure as a regular app target - it contains a test bundle, which references the app target under test, as well as the test classes and any helper classes required. It is not a duplicate of the app target with test code added to it - in fact it should not even contain the code under test. This means that you don't have to make any special effort to keep the test target in sync with the main app target - the fact that the test bundle loads the main app handles all that for you. (You made a comment on Richard Ross's answer to your previous q suggesting you've run into the difficulties duplication causes already).
How should I go about doing this? (checked on Xcode 7).
Assuming you are wanting to add a target to an existing project that doesn't have any tests, select the main project, and then click on the '+' beneath the list of targets in the project. You can also use the File->New->Target menu option.
You'll see a menu asking you to choose a template for your new target. Select 'Test' and within test select 'iOS Unit Testing Bundle'.
Check the fields in the next screen - they should be correct by default - but you might want to double check the 'Target to be Tested' field value is correct if you have a lot of targets in the project/workspace. Click 'OK' and you should have a functioning unit test bundle, with an example test that you should be able to run using Apple-U or Product->Test You'll still need to #import the app classes if you're using ObjC.
If you are creating a new project, all you need to do is tick the 'Include Unit Test' box when creating the project - no other steps are required.
Apple Docs (with links to relevant WWDC presentations)
Tutorials. Most of the tutorials around are a bit out of date. But not that much has changed, so just look at the docs and figure it out. The two below might be useful, otherwise just google. From a quick glance, most of them seem to assume that the project had unit tests set up at the beginning
http://www.raywenderlich.com/22590/beginning-automated-testing-with-xcode-part-12 (2012/iOS 6, but the process is still broadly the same. Also deals with Jenkins, Git and running tests from the CLI).
Unit testing in OSX - most recent post - not iOS, I know, but more up to date than most of the iOS tutorials (Oct 2015), and gives step by step instructions (including setting the unit test target in the build schemes, which probably won't be necessary in your case). Probably worth a look anyway.
I am trying to use UIAutomation for testing an iOS 8 Today extension widget. I can change the target to the extension, launch it, but then unable to do anything after that.
Has anyone had any success in using UIAutomation with the extensions?
I hope I have the right end of the stick on this one.
I think you want to create a set of UI Tests which run like you can do with a normal application.
I found that I was not able to create a specific UI Tests target to then set the target application as WidgetExtension, as you would normally.
However I was able to set the WidgetExtension target to build and run onto a device (>= iOS 14) then within my UI Tests, I was able to record some steps which would allow me to write a clearer UI Test (place the cursor inside a test function to use record, you might already know this).
The tricky bit would be keeping the WidgetExtension target up-to-date onto the testing simulator to run your automated tests.
I have a set of automated test cases set up in instruments using tuneup.js to test an app. I decided to use tuneup.js as it allowed me to separate my tests into individual test cases and run the whole set from one individual script, this works fine if all the tests run ok, however if one fails, all the tests fail as the simulator is left in an unknown state (I have written my tests so they all start and end on the same login screen) Is there a way to reset the simulator, or restart the app between test cases?
Try to launch tests from the command line. UI Automation allows to execute only one test in one run. After the test will be completed (does not matter if it was failed or passed) - application will be kicked by the system (UIAutomation). At least it works with real devices.
Your command line launch script will work in the following manner:
1. Reads configuration file (can be any file txt or xml) with path to your tests. At this point you will have an array with path to your tests and total tests count.
2. Then using simple 'for' loop (from 1 to 'testcount') it will launch UIAutomation with required parameters. One of the parameters will be the path to your test script that was read from the configuration file.
You can also put the path to the 'configuration file' as a parameter for your command line launch script. This will allow you to run any test set simply calling the launch script with required configuration file.
I wrote a script that will reset the contents & settings of all versions and devices for the iOS Simulator. It grabs the device names and version numbers from the menu, so it will include any new devices or iOS versions that Apple releases simulators for.
It's easy to run manually or use in a build-script. I would suggest adding it as a Pre-Action Run Script before the build.
https://github.com/michaelpatzer/ResetAllSimulators
Having failed tests leave your app in an unknown state is one of the main problems with using Apple's instruments tool as-is. We solved this in a framework called Illuminator (on GitHub, and inspired by tuneup.js) in two ways.
First, we wrote an automation bridge -- a channel for RPC with the app being tested, which allows us to reset our app before each test.
In cases where that's not sufficient, the Illuminator test runner has an option to re-run each failed test in its own pristine launch of the simulator (e.g. with --retest 1x,solo).
I've found plenty of people using Xcode 3 to do this, but it no longer seems to work in Xcode 4. Or, it works partially…
I've added a preprocessor macro "TEST_TARGET" for Debug and Release under my Test target, and if I use #ifdef TEST_TARGET in the actual unit tests, that works as expected.
However, I really want to log some extra information from a source file that's just part of the main app when it's being run under unit tests (i.e. just a standard source file, not a unit test file). That source file doesn't appear to "see" the define. I've stepped through with the debugger, and the code within #ifdef is never executed.
Is there a way to tell my app is being run under a unit test target?
Here's how you can do it: you can test for something that's loaded when the tests are loaded, but not when they aren't.
For instance:
if (NSClassFromString(#"SenTest")) {
NSLog(#"Extra info when running tests");
}
You could also add categories to your classes that are only present in the test target, which might also be helpful.