I have a project that contains both Unit tests and UI tests.
The project, per requirement, has to support both iOS8 and iOS9, and is setup with a continuous integration bot that runs tests on every commit to our repos.
I would like the bot to run tests on multiple devices with multiple iOS versions, to ensure as a minimum that our unit tests are valid across multiple OS versions.
However, I am unable to find an option to disable running UITests on devices running OS versions prior to iOS9, as UI testing is not supported on iOS8. However I would still like to run my unit tests on iOS8.
If I try and run my tests directly on the project using iPhone Simulator 8.2 I get the following error message: "UI Testing is not supported on “iPhone 6 (8.2)” because it is running iOS Simulator 8.2 and UI Testing requires iOS Simulator 9.0 or later."
Any ideas?
Untick the UI Test checkbox from edit scheme can switch it off when running test scheme by Command+U.
Add a second scheme and remove the UI test from the scheme in the test menu. Then add a bot for that scheme.
Related
I Hope someone can help me here.
I'm using XCode for UI Automation testing. For some reason, I can't run my tests on any emulators anymore. If I set the device to Generic iOS Device my tests appear, but once I change this to an emulator, my tests are no longer available.
I hope someone can help me. I have also tried deleting my current schemes and creating new ones, but then again, when I try to add my test folder to it, nothing is inside of it unless I select Genetic iOS Device first, and then the same issue occurs when I try to run my tests.
Make sure your test target has the appropriate platform and architectures. Select your project, then select the test target. Under Build Settings, make sure Architectures is set to Standard and Supported Platforms to iOS (which should be iphonesimulator and iphoneos).
Has anyone faced this issue?
Been stuck trying to do a simple Xcode Bot integration with my simple project. Keeps coming back with this error that the bot wasn't able to build the project. From what I see, the scheme is just a normal scheme with the Unit Tests and UI Unit Tests. Bot is enabled Periodically and the Devices currently I've only set it to All Simulators (tried with others but it didn't work too). One thing I notice was when remove the Tests from the Scheme the Integration worked fine.
Thanks!
Was finally able to find the issue for this! Removing UI Tests initially showed that the integration was working fine. It seems that as Xcode UI Testing is only available on iOS9, the automated building would fail (without telling you the true reason) when testing on older simulators/devices. To overcome this I configured two separate bots:
Test for Unit Test Cases
Test for UI Test Cases (specifically on iOS 9 simulators / devices)
Hope this solves the issue for anyone else facing it!
It turns out, that in automatically updating Xcode it uninstalls the iOS 9.0 simulator and replaces it with the iOS 9.1 simulator. It doesn't think to update the build settings on the Xcode server.
So to fix the problem, you have to manually open Xcode on your server, it then prompts you to install several items which you need to do. Then go to xcode preferences > Downloads and you can reinstall the iOS 9.0 simulator.
I continue to have most of my KIF UI Test cases fail with Apple's CI Bots.
The errors I get are: "Failed to find accessibility element with label ...."
I can run all and pass the tests consistently on the same machine just fine with a Command U, but when running from an Xcode bot they fail (but intermittently and inconsistently.) For example, sometimes 24/25 tests will fail with the bot and the next time 20/25 tests fail with the same bot, with no changes. Furthermore, the first few tests I ran when I initially set up the bot ran and passed perfectly.
I am running the tests on the iOS 9 Simulator Xcode 7.1 and Version 10.11.1 OS X Server.
Has anyone experienced and resolved similar UI Automated Testing issues with Xcode bots themselves?
The solution is hardware.
After testing on a Mac Pro the UI tests are passing consistently. The previous test machine was a mac mini with 8 gigs ram. Thus, if you encounter something similar with your UI Tests and Xcode bots, beef up your hardware to see if that helps.
I start out with the "-cal" scheme running just like the Calabash setup guide tells me to. Then I run cucumber for Calabash-iOS. The simulator seems to stop and restart before executing the tests in a different (possibly cached version) of my app and the target device has changed from an iPhone 6 to running the tests in an iPhone 5s.
How can I make cucumber execute the tests in the already running "-cal" scheme? Or (better yet) how can I make it relaunch using the "-cal" scheme? I can see cases where relaunching would be valuable.
Below is the only code executed before the scenarios run. I know it says .relaunch is in there, but if I take it out then the test fails on the first step.
Before do |scenario|
#calabash_launcher = Calabash::Cucumber::Launcher.new
unless #calabash_launcher.calabash_no_launch?
#calabash_launcher.relaunch
#calabash_launcher.calabash_notify(self)
end
end
So there are few questions there.
How can I make cucumber execute the tests in the already running "-cal" scheme?
Answer: When you build the cal target you define what target to build for and when you execute the tests you define what target to execute on.
Build: -sdk iphonesimulator9.0
Execute: DEVICE_TARGET='iPhone 6 Plus (9.0)'
The simulator seems to stop and restart before executing the tests in a different (possibly cached version) of my app and the target device has changed from an iPhone 6 to running the tests in an iPhone 5s.
Answer: Also related partly to question 1 (defining target to control what setup you test on). Besides that the app will relaunch between scenarios otherwise each scenario would be dependent on what ever else had been executed before it.
The one thing you can toggle is if the app should be reinstalled between scenarios. This is helpful if you have some functionality that only shows/executes on first install/launch. To add support for reinstall you can add something like this to the launch.rb file in "Before do |scenario|"
if scenario_tags.include?('#reinstall')
#calabash_launcher.reset_app_sandbox end
The simulator seems to stop and restart before executing the tests
The simulator is quit and relaunched before each Scenario for stability. If we keep the simulator open, it quickly becomes unstable. See below for more details.
(possibly cached version) of my app
This is no longer an issue starting in run-loop 1.5* which is required by Calabash 0.16.4. What version of Calabash are you running? If you are building from the command line and from Xcode, have a look at this project Calabash iOS Smoke Test and specifically this script that stages products built by Xcode to the same directory as the command line builds.
iPhone 6 to running the tests in an iPhone 5s.
Starting in Xcode 7.1, the default simulator is the iPhone 6. Pre Xcode 7.1, the default simulator has been the iPhone 5s. Lasse is correct, use the DEVICE_TARGET to control which simulator to run on.
How can I make cucumber execute the tests in the already running "-cal" scheme? Or (better yet) how can I make it relaunch using the "-cal" scheme?
You don't have to relaunch between Scenarios. Relaunching will restart the simulator. If you don't want to relaunch, you can use a custom calabash backdoor to reset your app to a good known state before each test start. I used to do this all the time; it really speeds up the testing. However, I abandoned this approach because it is hard to maintain and as I mentioned, the simulator becomes unstable.
Or (better yet) how can I make it relaunch using the "-cal" scheme?
This I actually don't understand. Have you set the APP variable? Is another app launching? Did you run calabash setup and does your app have a Watch extension? If so, you are probably running into this problem: calabash setup adds calabash.framework to Watch extension instead of the app
I currently learning XCTest for the purpose of unit testing. I was able to run the default template XCtest on the simulator without any problem. I could all the green ticks in the test navigation view. However, when I ran them on the device with my app as host application, nothing happens. My app was launched on the device and the XCTest did not seem to run. I even put a break point in the test and it did not break. Also, in the simulator, the app automatically stops when tests finish, however, when running on device, my app was just kept running and never put to a stop. Am I doing something incorrect?
xCode should show this message when running XCTest on a physical device.
Logic Testing on iOS devices is not supported. You can run logic tests on the Simulator.
There's a library called GHUnit we can run unit tests on iphone device. Internally it uses xctest. Here is how you setup GHUnit.
I was able to follow the instructions and setup unit tests using xcode 5. should be okay with Xcode 6?