iOS: how to run XCTest on device? - ios

I currently learning XCTest for the purpose of unit testing. I was able to run the default template XCtest on the simulator without any problem. I could all the green ticks in the test navigation view. However, when I ran them on the device with my app as host application, nothing happens. My app was launched on the device and the XCTest did not seem to run. I even put a break point in the test and it did not break. Also, in the simulator, the app automatically stops when tests finish, however, when running on device, my app was just kept running and never put to a stop. Am I doing something incorrect?

xCode should show this message when running XCTest on a physical device.
Logic Testing on iOS devices is not supported. You can run logic tests on the Simulator.

There's a library called GHUnit we can run unit tests on iphone device. Internally it uses xctest. Here is how you setup GHUnit.
I was able to follow the instructions and setup unit tests using xcode 5. should be okay with Xcode 6?

Related

Xcode 8 app runs on device unit test crashes with __abort_with_payload

I have a project with two targets, an app, and UI Tests for that app. I can run the app on my device. When I try to run a test on my devices, it immediately crashes with __abort_with_payload on Thread 1 and absolutely no other clue as to what is going wrong. Oddly, I have been able to run this test on other devices.
How can I get my tests running on my device?

How to I disable UITesting when running tests on a iOS8 device?

I have a project that contains both Unit tests and UI tests.
The project, per requirement, has to support both iOS8 and iOS9, and is setup with a continuous integration bot that runs tests on every commit to our repos.
I would like the bot to run tests on multiple devices with multiple iOS versions, to ensure as a minimum that our unit tests are valid across multiple OS versions.
However, I am unable to find an option to disable running UITests on devices running OS versions prior to iOS9, as UI testing is not supported on iOS8. However I would still like to run my unit tests on iOS8.
If I try and run my tests directly on the project using iPhone Simulator 8.2 I get the following error message: "UI Testing is not supported on “iPhone 6 (8.2)” because it is running iOS Simulator 8.2 and UI Testing requires iOS Simulator 9.0 or later."
Any ideas?
Untick the UI Test checkbox from edit scheme can switch it off when running test scheme by Command+U.
Add a second scheme and remove the UI test from the scheme in the test menu. Then add a bot for that scheme.

KIF iOS test failures with Apple's CI Bots

I continue to have most of my KIF UI Test cases fail with Apple's CI Bots.
The errors I get are: "Failed to find accessibility element with label ...."
I can run all and pass the tests consistently on the same machine just fine with a Command U, but when running from an Xcode bot they fail (but intermittently and inconsistently.) For example, sometimes 24/25 tests will fail with the bot and the next time 20/25 tests fail with the same bot, with no changes. Furthermore, the first few tests I ran when I initially set up the bot ran and passed perfectly.
I am running the tests on the iOS 9 Simulator Xcode 7.1 and Version 10.11.1 OS X Server.
Has anyone experienced and resolved similar UI Automated Testing issues with Xcode bots themselves?
The solution is hardware.
After testing on a Mac Pro the UI tests are passing consistently. The previous test machine was a mac mini with 8 gigs ram. Thus, if you encounter something similar with your UI Tests and Xcode bots, beef up your hardware to see if that helps.

Calabash-cucumber simulator not running same version of app as the "-cal" scheme

I start out with the "-cal" scheme running just like the Calabash setup guide tells me to. Then I run cucumber for Calabash-iOS. The simulator seems to stop and restart before executing the tests in a different (possibly cached version) of my app and the target device has changed from an iPhone 6 to running the tests in an iPhone 5s.
How can I make cucumber execute the tests in the already running "-cal" scheme? Or (better yet) how can I make it relaunch using the "-cal" scheme? I can see cases where relaunching would be valuable.
Below is the only code executed before the scenarios run. I know it says .relaunch is in there, but if I take it out then the test fails on the first step.
Before do |scenario|
#calabash_launcher = Calabash::Cucumber::Launcher.new
unless #calabash_launcher.calabash_no_launch?
#calabash_launcher.relaunch
#calabash_launcher.calabash_notify(self)
end
end
So there are few questions there.
How can I make cucumber execute the tests in the already running "-cal" scheme?
Answer: When you build the cal target you define what target to build for and when you execute the tests you define what target to execute on.
Build: -sdk iphonesimulator9.0
Execute: DEVICE_TARGET='iPhone 6 Plus (9.0)'
The simulator seems to stop and restart before executing the tests in a different (possibly cached version) of my app and the target device has changed from an iPhone 6 to running the tests in an iPhone 5s.
Answer: Also related partly to question 1 (defining target to control what setup you test on). Besides that the app will relaunch between scenarios otherwise each scenario would be dependent on what ever else had been executed before it.
The one thing you can toggle is if the app should be reinstalled between scenarios. This is helpful if you have some functionality that only shows/executes on first install/launch. To add support for reinstall you can add something like this to the launch.rb file in "Before do |scenario|"
if scenario_tags.include?('#reinstall')
#calabash_launcher.reset_app_sandbox end
The simulator seems to stop and restart before executing the tests
The simulator is quit and relaunched before each Scenario for stability. If we keep the simulator open, it quickly becomes unstable. See below for more details.
(possibly cached version) of my app
This is no longer an issue starting in run-loop 1.5* which is required by Calabash 0.16.4. What version of Calabash are you running? If you are building from the command line and from Xcode, have a look at this project Calabash iOS Smoke Test and specifically this script that stages products built by Xcode to the same directory as the command line builds.
iPhone 6 to running the tests in an iPhone 5s.
Starting in Xcode 7.1, the default simulator is the iPhone 6. Pre Xcode 7.1, the default simulator has been the iPhone 5s. Lasse is correct, use the DEVICE_TARGET to control which simulator to run on.
How can I make cucumber execute the tests in the already running "-cal" scheme? Or (better yet) how can I make it relaunch using the "-cal" scheme?
You don't have to relaunch between Scenarios. Relaunching will restart the simulator. If you don't want to relaunch, you can use a custom calabash backdoor to reset your app to a good known state before each test start. I used to do this all the time; it really speeds up the testing. However, I abandoned this approach because it is hard to maintain and as I mentioned, the simulator becomes unstable.
Or (better yet) how can I make it relaunch using the "-cal" scheme?
This I actually don't understand. Have you set the APP variable? Is another app launching? Did you run calabash setup and does your app have a Watch extension? If so, you are probably running into this problem: calabash setup adds calabash.framework to Watch extension instead of the app

Errors when trying to test app on iPhone

When I try to test my app on the iPhone I get the following errors:
However I do not get these errors when running on the iOS Simulator.
Anyone know why?
You need to add the AVFoundation framework for this to work, through the target page in Xcode. The Simulator, since it's running on OS X, handles things related to frameworks differently then the device does.

Resources