Don't launch simulator when running unittests - ios

Some background:
I have iOS application with a target configured to run unitTests. And I am running build automation tool jenkins on my MacBook which automatically builds this application and run all tests (using command line xcodebuild tool).
Everything worked fine with Xcode 4. This build automation tool was running under different user and was running all these tests.
I switched to Xcode 5 recently and it started to fail, because it can't launch Simulator.
The problem
I have a scheme UnitTests which is configured to run tests (logic tests). A I run these test using one of two methods:
Command U in Xcode
Or command line "/usr/bin/xcodebuild -scheme UnitTests -sdk iphonesimulator -configuration Release clean build TEST_AFTER_BUILD=YES "
In both cases, it tries to start simulator. However, per my understand it doesn't need it. Anyway it runs on top x86 and it doesn't look like any apps are installed on Simulator.
Is there a way to get rid of this pesky simulator start (because it breaks my build automation)?
Update 1
Seems to find very similar question, but can't get it working:
Run logic tests in Xcode 4 without launching the simulator
Update 2
I found VERY relevant and interesting question/answer: Apple CI / Xcode Service and Jenkins

Using xCode 7 and xCtool.
xctool is capable of executing unit tests without the simulator.
To get this working,
1 . Update target settings to run without a host app.
Select your project --> then test target --> Set the host application to none.
2. Install xctool , if you don't have it.
brew install xctool
3. Run the tests using terminal with xctool.
xctool -workspace yourWorkspace.xcworkspace -scheme yourScheme run-tests -sdk iphonesimulator

You can create a Mac OSX Unit Test instead of an iOS unit test. This requires that you not include any iOS specific libraries in the unit tests though. You can do this via the following:
Select the project -> the target drop down -> "Add Target..."
Select "Mac OSX" -> "Other" -> "Cocoa Unit Testing Bundle"
Create the testing bundle as you would a normal project
You can now add sources to the unit test and run it like an iOS test without launching the simulator.

I've asked the same question to apple engineers. Unfortunately it doesn't seem you can accomplish this and stay with iOS at the same time. There are some tricks you can do to check if testing. You could put this code snippet in your AppDelegate.h or some other global class to say not load a root viewcontroller and prevent any wierdo ui stuff from corrupting your unit tests:
static BOOL isTesting() {
BOOL isTesting = !isEmpty([[[NSProcessInfo processInfo] environment] objectForKey:#"XCInjectBundle"]);
return isTesting;
}
I've also had an apple engineer verify this is a legitimate check. And to give credit where credit is due, this is from: Programmatically determine current target (run or test) in iOS project
EDIT:
I've also had success with this and it's a little more straight forward:
static BOOL isTesting() {
return [[[NSProcessInfo processInfo] processName] isEqualToString:#"xctest"];
}

A osx test target can become a huge hassle because you have to manage yourself which source file to include. Putting #testable import YourAppName on top of your XCTest files is way more convenient. So just prevent your app from launching in case of a XCTest run.
In your AppDelegate put: (Swift 3 solution)
func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool
{
if ProcessInfo.processInfo.environment["XCInjectBundleInto"] != nil {
return false
}
...
This wont prevent the simulator from launching, but will save you a lot of time.

Workaround:
App will still launch but you can #if to define what you don't want to run.
Approach:
Create a custom build configuration called Test by duplicating Debug (Project > Info > create new configuration)
In Build Settings > Active Compilation Conditions for Test add TESTING
Edit Scheme > Info > Build Configuration, set build configuration as Test
Use #if !TESTING #endif around the code you don't want to execute when testing.
Frameworks:
If you have embedded frameworks, create the same build configuration in the framework, so that the framework binary is properly linked.

You can run unit and ui tests in headless mode with Xcode 9 and a command from your terminal. For reference sample commands:
Building and testing a workspace
xcrun xcodebuild -workspace "YOUR_WORKSPACE_NAME.xcworkspace" -scheme "YOUR_SCHEME" -sdk "iphonesimulator12.0" -destination "OS=12.0,name=iPhone X" -configuration Debug -enableCodeCoverage YES clean build test
For project
xcrun xcodebuild -project "YOUR_PROJECT_NAME.xcodeproj" -scheme "YOUR_SCHEME" -sdk "iphonesimulator12.0" -destination "OS=12.0,name=iPhone X" -configuration Debug -enableCodeCoverage YES clean build test

Related

run test classes using xctest framework from a class

i am facing a problem. I want to do a ui test for my application.
i have written six test class where my test cases are included.
I can run my test cases using
"command U" or the clicking in the run button.
But i don't want to do like this.
I want to run my test classes using code i mean using
xctest because after complete my execution, i have to do another task.
can anyone please give an example how i can run my test classes using xctest codding from a class file.
i dont want to use shell script or jenkins, i want it by using xctest framework.
Running tests with xcodebuild
The xcodebuild command-line tool drives tests just like the Xcode IDE. Run xcodebuild with the action test and specify different destinations with the -destination argument. For example, to test MyApp on the local OS X “My Mac 64 Bit,” specify that destination and architecture with this command:
> xcodebuild test -project MyAppProject.xcodeproj -scheme MyApp -destination 'platform=OS X,arch=x86_64'
If you have development-enabled devices plugged in, you can call them out by name or id. For example, if you have an iPod touch named “Development iPod touch” connected that you want to test your code on, you use the following command:
> xcodebuild test -project MyAppProject.xcodeproj -scheme MyApp -destination 'platform=iOS,name=Development iPod touch'
Tests can run in Simulator, too. Use the simulator to target different form factors, operating systems, and OS versions easily. Simulator destinations can be specified by name or id. For example:
> xcodebuild test -project MyAppProject.xcodeproj -scheme MyApp -destination 'platform=Simulator,name=iPhone,OS=8.1'
Source
Edit your Xcode scheme.
Click the disclosure triangle next to "Test" to reveal more options.
Select "Post-actions"
Click '+' to add any actions to run after tests are complete.
As you can see, your choices are to run a script, or send a simple email. For more information on configuring actions, see https://michele.io/the-project-file-part-2/

Is there a way to run XCTest(UI) against an archived build(.ipa)?

I am investigating how to use XCTest to close our automation test gap.
From the developer document, I only see something like this:
xcodebuild test -project MyAppProject.xcodeproj -scheme MyApp
https://developer.apple.com/library/tvos/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/08-automation.html#//apple_ref/doc/uid/TP40014132-CH7-SW1
I would like to know if there is a way to run XCTest(UI test) against an archived build (.ipa)?
or
Can we just seperate the build and test so that we can build first and test against that build later?
Thanks
If you want to separate the build and test stages, you can use the xcodebuild flags build-for-testing (which builds a xctestrun file) and test-without-building (which runs it). You may want to build-without-testing to a generic device if you want the build-for-testing to be used on multiple devices.
Check out https://medium.com/xcblog/speed-up-ios-ci-using-test-without-building-xctestrun-and-fastlane-a982b0060676 for more details

How to dynamically change target for unit tests in Xcode 7?

I have a project that has multiple different targets/schemes (~38 of them as of writing this question) and I am trying to get unit testing working properly across all of the different targets. I got things working with one target, and tried adding my testing target to all of the different schemes, but it looks like when tests run for each scheme they actually run on the same, original target.
Looking in the project file I see that there's a specific Host Application associated with my testing target, and in the build settings the Bundle Loader and Test Host point to that same Host Application.
Is there any way to override those values for each scheme to run the tests against the current scheme's build target? Or some other way to set up a single test target to run across multiple build targets?
If you run the tests from the command line, or from an CI tool, like Jenkins, you can instruct xcodebuild to use the build settings that you provide. A simple usage example would be:
xcodebuild -scheme SomeScheme test TEST_HOST=target
You can control almost (if not any) build setting from Xcode, like code coverage generation, build directory, derived data directory, code sign identity, etc.
You can select the scheme when you run tests with Xcode server.
Look at WWDC 2014 continues integration talk for a walk through on how to set it up
https://developer.apple.com/videos/play/wwdc2014-415/
It's using Xcode 6 but it's very similar process to Xcode 7
Also check this CI(continues integration) guideline from apple
https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/adopt_continuous_integration.html
If anyone is wondering how to do that with UI tests (maybe it is working with unit tests, too), this is what I came up with:
First, we need to build the application which we are going to use to host our UI tests:
xcodebuild -scheme "<appSchemeName>" build -destination "<yourDestination>"
More info about destination parameter: https://mokacoding.com/blog/xcodebuild-destination-options/
Then, we need to run tests on our newly built application:
xcodebuild -scheme "<uiTestsSchemeName>" -destination "<yourDestination>" test TEST_TARGET_NAME="<yourNewlyBuiltAppTargetName>"
Destinations should match, since after the build .app is generated in your DerivedData folder, which will be used for UI tests hosting application

Xcode Build for Profiling via Terminal

I'm trying to build my XCode project via the terminal. I'd like use Xcode's Product > Build for > Profiling option. So far I have:
xcodebuild -project "MyGame.xcodeproj" -target "MyGameEditor - Mac" -destination 'name=My Mac 64-bit' -configuration Profile
The project builds but not for profiling. I get an error that configuration 'Profile' does not exist.
Any help is appreciated.
What you are trying to sepcify with -configuration is the build configuration, not the build action. Unless you have added extra configurations to your project, you only have "Debug" and "Release" configurations.
What Xcode does when you tell it to profile is it builds the configuration that your scheme specifies to use when profiling, launches the simulator, installs the app, then launches Instruments.
So you need to do a similar thing with two command line calls, one to xcodebuild, one to instruments.
Some helpful links that should help you figure out what you need to do:
http://blog.manbolo.com/2013/04/09/launching-uiautomation-tests-in-command-line
Can the UI Automation instrument be run from the command line?
It is also worth noting that rather than specifying a target and configuration, you should just specify a scheme which provides both and other optional additional features.

Xcode 5 xcodebuild unit test seems to return status successful when tests fail

Hi I'm following examples from Test Driven iOS development and I've written a few unit tests with the new Xcode 5 and XCTest. My tests fail with the Xcode GUI client, but when I use xcodebuild the status code is 0.
xcodebuild -target TemperatureConverterTests build
and
echo $?
returns 0.
The tests are suppose to fail. Is the command to run the test cases correct?
xcodebuild command has changed with xcode 5. Here is a script to run Unit tests :
Don't forget that schemes must be shared, you've got to create your Tests Scheme, Xcode 5 didn't create it for you
xcodebuild -workspace MyApp.xcworkspace -scheme myApp-Tests -destination=build -configuration Debug -sdk iphonesimulator7.0 ONLY_ACTIVE_ARCH=YES clean build test
Hope this helps :)
If you don't want (as you rarely need) a separate (and annoying) "scheme" for your testing "target"... Use a variation on #Rémy Virin's answer...
xcodebuild -scheme YourAppOrLib -target YourAppOrLibTests
As pictured below.. this allows you to run the Unit tests for the target.... without cluttering the project with non-production schemes... In this case IndexedKeyMap is my YourAppOrLib.

Resources