I continue to have most of my KIF UI Test cases fail with Apple's CI Bots.
The errors I get are: "Failed to find accessibility element with label ...."
I can run all and pass the tests consistently on the same machine just fine with a Command U, but when running from an Xcode bot they fail (but intermittently and inconsistently.) For example, sometimes 24/25 tests will fail with the bot and the next time 20/25 tests fail with the same bot, with no changes. Furthermore, the first few tests I ran when I initially set up the bot ran and passed perfectly.
I am running the tests on the iOS 9 Simulator Xcode 7.1 and Version 10.11.1 OS X Server.
Has anyone experienced and resolved similar UI Automated Testing issues with Xcode bots themselves?
The solution is hardware.
After testing on a Mac Pro the UI tests are passing consistently. The previous test machine was a mac mini with 8 gigs ram. Thus, if you encounter something similar with your UI Tests and Xcode bots, beef up your hardware to see if that helps.
Related
I have a project that contains both Unit tests and UI tests.
The project, per requirement, has to support both iOS8 and iOS9, and is setup with a continuous integration bot that runs tests on every commit to our repos.
I would like the bot to run tests on multiple devices with multiple iOS versions, to ensure as a minimum that our unit tests are valid across multiple OS versions.
However, I am unable to find an option to disable running UITests on devices running OS versions prior to iOS9, as UI testing is not supported on iOS8. However I would still like to run my unit tests on iOS8.
If I try and run my tests directly on the project using iPhone Simulator 8.2 I get the following error message: "UI Testing is not supported on “iPhone 6 (8.2)” because it is running iOS Simulator 8.2 and UI Testing requires iOS Simulator 9.0 or later."
Any ideas?
Untick the UI Test checkbox from edit scheme can switch it off when running test scheme by Command+U.
Add a second scheme and remove the UI test from the scheme in the test menu. Then add a bot for that scheme.
Has anyone faced this issue?
Been stuck trying to do a simple Xcode Bot integration with my simple project. Keeps coming back with this error that the bot wasn't able to build the project. From what I see, the scheme is just a normal scheme with the Unit Tests and UI Unit Tests. Bot is enabled Periodically and the Devices currently I've only set it to All Simulators (tried with others but it didn't work too). One thing I notice was when remove the Tests from the Scheme the Integration worked fine.
Thanks!
Was finally able to find the issue for this! Removing UI Tests initially showed that the integration was working fine. It seems that as Xcode UI Testing is only available on iOS9, the automated building would fail (without telling you the true reason) when testing on older simulators/devices. To overcome this I configured two separate bots:
Test for Unit Test Cases
Test for UI Test Cases (specifically on iOS 9 simulators / devices)
Hope this solves the issue for anyone else facing it!
It turns out, that in automatically updating Xcode it uninstalls the iOS 9.0 simulator and replaces it with the iOS 9.1 simulator. It doesn't think to update the build settings on the Xcode server.
So to fix the problem, you have to manually open Xcode on your server, it then prompts you to install several items which you need to do. Then go to xcode preferences > Downloads and you can reinstall the iOS 9.0 simulator.
I start out with the "-cal" scheme running just like the Calabash setup guide tells me to. Then I run cucumber for Calabash-iOS. The simulator seems to stop and restart before executing the tests in a different (possibly cached version) of my app and the target device has changed from an iPhone 6 to running the tests in an iPhone 5s.
How can I make cucumber execute the tests in the already running "-cal" scheme? Or (better yet) how can I make it relaunch using the "-cal" scheme? I can see cases where relaunching would be valuable.
Below is the only code executed before the scenarios run. I know it says .relaunch is in there, but if I take it out then the test fails on the first step.
Before do |scenario|
#calabash_launcher = Calabash::Cucumber::Launcher.new
unless #calabash_launcher.calabash_no_launch?
#calabash_launcher.relaunch
#calabash_launcher.calabash_notify(self)
end
end
So there are few questions there.
How can I make cucumber execute the tests in the already running "-cal" scheme?
Answer: When you build the cal target you define what target to build for and when you execute the tests you define what target to execute on.
Build: -sdk iphonesimulator9.0
Execute: DEVICE_TARGET='iPhone 6 Plus (9.0)'
The simulator seems to stop and restart before executing the tests in a different (possibly cached version) of my app and the target device has changed from an iPhone 6 to running the tests in an iPhone 5s.
Answer: Also related partly to question 1 (defining target to control what setup you test on). Besides that the app will relaunch between scenarios otherwise each scenario would be dependent on what ever else had been executed before it.
The one thing you can toggle is if the app should be reinstalled between scenarios. This is helpful if you have some functionality that only shows/executes on first install/launch. To add support for reinstall you can add something like this to the launch.rb file in "Before do |scenario|"
if scenario_tags.include?('#reinstall')
#calabash_launcher.reset_app_sandbox end
The simulator seems to stop and restart before executing the tests
The simulator is quit and relaunched before each Scenario for stability. If we keep the simulator open, it quickly becomes unstable. See below for more details.
(possibly cached version) of my app
This is no longer an issue starting in run-loop 1.5* which is required by Calabash 0.16.4. What version of Calabash are you running? If you are building from the command line and from Xcode, have a look at this project Calabash iOS Smoke Test and specifically this script that stages products built by Xcode to the same directory as the command line builds.
iPhone 6 to running the tests in an iPhone 5s.
Starting in Xcode 7.1, the default simulator is the iPhone 6. Pre Xcode 7.1, the default simulator has been the iPhone 5s. Lasse is correct, use the DEVICE_TARGET to control which simulator to run on.
How can I make cucumber execute the tests in the already running "-cal" scheme? Or (better yet) how can I make it relaunch using the "-cal" scheme?
You don't have to relaunch between Scenarios. Relaunching will restart the simulator. If you don't want to relaunch, you can use a custom calabash backdoor to reset your app to a good known state before each test start. I used to do this all the time; it really speeds up the testing. However, I abandoned this approach because it is hard to maintain and as I mentioned, the simulator becomes unstable.
Or (better yet) how can I make it relaunch using the "-cal" scheme?
This I actually don't understand. Have you set the APP variable? Is another app launching? Did you run calabash setup and does your app have a Watch extension? If so, you are probably running into this problem: calabash setup adds calabash.framework to Watch extension instead of the app
I currently learning XCTest for the purpose of unit testing. I was able to run the default template XCtest on the simulator without any problem. I could all the green ticks in the test navigation view. However, when I ran them on the device with my app as host application, nothing happens. My app was launched on the device and the XCTest did not seem to run. I even put a break point in the test and it did not break. Also, in the simulator, the app automatically stops when tests finish, however, when running on device, my app was just kept running and never put to a stop. Am I doing something incorrect?
xCode should show this message when running XCTest on a physical device.
Logic Testing on iOS devices is not supported. You can run logic tests on the Simulator.
There's a library called GHUnit we can run unit tests on iphone device. Internally it uses xctest. Here is how you setup GHUnit.
I was able to follow the instructions and setup unit tests using xcode 5. should be okay with Xcode 6?
I use new XCode Continuous Integration service running on Mac OS X 10.9. with using Mac OS X Server.
So I created new bot with testing target: Run tests on All Simulators (Without any devices)
After integration I receive such an error:
Integration failed. The bot was unable to find or use one of the
devices on which it is configured to integrate.
How can this problem be solved? I do not want to connect any devices to server for integration testing.
Thanks in advance,
Does the app you're testing support all devices? Try setting up the bot for "Specific Devices" and choose a simulator you have built to. Also make sure current code changes are pushed to the repository the Xcode server pulls from.
EDIT 11/19/13
On your server machine, make sure you have the latest Xcode release installed (currently 5.0.2).
Open Xcode and configure the Preferences, in Downloads, to install the command line tools, and be sure you have downloaded/installed a Simulator you want to test against. If you're building for iOS 6.1, for example, you'll need to install the simulator.
In your Mac OS X Server Xcode admin panel, in the "Builds" section, make sure you choose that Xcode app version. The server will use that Xcode app to do the builds.
When you edit the bot settings, where you configure which devices to test with, choose "Specific Devices" and choose a simulator you have successfully built to on your development machine, for example, choose the iPad Retina (6.1), or whatever your app supports.
If a specific simulator device does not appear in the list of Selected Devices, the simulator is probably not installed on your server machine. See step 2 above.
(Hope this helps)
This is similar in description, upgrading to Xcode 7.3 and the latest OS X Server, I was getting this error.
Bot Issue for MB Bot (build service error) Integration #69 of MB Bot
Open in Xcode:
xcbot://P3P4/botID/e6fd861f04bdb3ae4b3d731a1f0c4f9f/integrationID/cf7e287759d2045e134163c25b01518a
Assertion: Could not test because no devices or simulators were
available. File: (null):(null)
Introduced 1 integration ago
After digging into this a bit, and trying the "Fix..." button and installing all the different simulators for older OS versions to no avail on the server. Finally, I found that it was doing the checkout but immediately failing before it started building. Disabling the tests allowed it to compile. Then I noticed that the Build Configuration in the Bot configuration Perform Test Action, was set to Use Schema Settings, and that the scheme was older. After changing this in the schema, the diff looked like this in the Schema file.
- version = "1.3">
+ version = "1.7">
This seems to have fixed the problem. This is probably a rarer situation, but thought I would post this answer if someone ran into a similar problem as it took the span of several days to figure out.