I've been trying to find a way to run a single iOS unit test among multiple tests with the command line on an iOS device.
Here is what I've found thus far...
I can use xcodebuild test [options] to build & run every tests.
Found a tool called xctest in /Applications/Xcode.app/Contents/Developer/usr/bin. I've looked up in the man page and it is possible to run a single test. When I try using it, I get an error stating that I'm using x86_64 architecture and then a posix error pops after. This seems to indicate that it might be the xctest for Mac OS X apps.
Found another xctest in for the iPhone Simulator.. but I want to run it on a iOS device
I know that it is possible to select the tests I wish to run in a scheme, but this is problematic since I would need a very skillful & patient monkey to manually do this in a build system.
By the way, I am not using Apple's own Continuous Integration system with bots and using it is out of question because another build system is used for every other platform.
My question is: How can I run a single unit test (XCTest) from the command line with build automation in mind?
P.S: I don't have any code signing issues.
Try using xctool which can run tests on devices https://github.com/facebook/xctool
Related
I have a project that has multiple different targets/schemes (~38 of them as of writing this question) and I am trying to get unit testing working properly across all of the different targets. I got things working with one target, and tried adding my testing target to all of the different schemes, but it looks like when tests run for each scheme they actually run on the same, original target.
Looking in the project file I see that there's a specific Host Application associated with my testing target, and in the build settings the Bundle Loader and Test Host point to that same Host Application.
Is there any way to override those values for each scheme to run the tests against the current scheme's build target? Or some other way to set up a single test target to run across multiple build targets?
If you run the tests from the command line, or from an CI tool, like Jenkins, you can instruct xcodebuild to use the build settings that you provide. A simple usage example would be:
xcodebuild -scheme SomeScheme test TEST_HOST=target
You can control almost (if not any) build setting from Xcode, like code coverage generation, build directory, derived data directory, code sign identity, etc.
You can select the scheme when you run tests with Xcode server.
Look at WWDC 2014 continues integration talk for a walk through on how to set it up
https://developer.apple.com/videos/play/wwdc2014-415/
It's using Xcode 6 but it's very similar process to Xcode 7
Also check this CI(continues integration) guideline from apple
https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/adopt_continuous_integration.html
If anyone is wondering how to do that with UI tests (maybe it is working with unit tests, too), this is what I came up with:
First, we need to build the application which we are going to use to host our UI tests:
xcodebuild -scheme "<appSchemeName>" build -destination "<yourDestination>"
More info about destination parameter: https://mokacoding.com/blog/xcodebuild-destination-options/
Then, we need to run tests on our newly built application:
xcodebuild -scheme "<uiTestsSchemeName>" -destination "<yourDestination>" test TEST_TARGET_NAME="<yourNewlyBuiltAppTargetName>"
Destinations should match, since after the build .app is generated in your DerivedData folder, which will be used for UI tests hosting application
I am new with calabash-ios and have a dummy question is:
How possibly could I run a calabash test with real device which have installed ipa file (included calabash framework) and without xcodeproj?
Problem is I am doing QA and Dev only gives me ipa file to test without source code.
UPDATE:
I am able to run the ipa file in console and cucumber by adding BUNDLE_ID. And make sure the CFNetwork.framework is including in app framework to establish the connection.
But I got another problem to run the scenario:
1. When I run the command below:
DEVICE_ENDPOINT=http://192.168.1.9:37265 NO_LAUNCH=1 BUNDLE_ID=com.example.appname DEVICE_TARGET=UDID cucumber
Then I got this error:
A JSON text must at least contain two octets! (JSON::ParserError)
features/my_first.feature:8:in `Then I touch "LOG-IN/ CREATE ACCOUNT"'
all other steps was skipped
Then I run the console code
DEVICE_ENDPOINT=http://192.168.1.9:37265 NO_LAUNCH=1 BUNDLE_ID=com.example.appname DEVICE_TARGET=UDID calabash-ios console
Then I run server_version
I got reply back that said connection SUCCEED
But when I run start_test_server_in_background
It took me too long to wait and I see the app my physical device kept crashed and launch again and again until it stop as:
Unable to start. Make sure you've set APP_BUNDLE_PATH to a build supported by this simulator version
Calabash::Cucumber::Launcher::StartError: Calabash::Cucumber::Launcher::StartError:
"Timed out waiting for UIAutomation run-loop Error while writing to fifo. RunLoop::Fifo::NoReaderConfiguredError.
2015-10-01 10:49:41.634 instruments[15640:10558880] WebKit Threading Violation - initial use of WebKit from a secondary thread.
2015-10-01 10:49:42.574 instruments[15640:10558978] Attempting to change event horizon while disengage
2015-10-01 10:49:42.577 instruments[15640:10558877] Attempting to change event horizon while disengage
Instruments Trace Error : Target failed to run: Permission to debug app
Anyone can help please! I tried to search some solution but they almost issue on simulator and they contain xcodeproj. As a QA, I dont have the source code, only .ipa file. :(
Thanks
You should set 3 variables:
$ DEVICE_TARGET=<UDID> \
DEVICE_ENDPOINT=http://<ip>:37265 \
BUNDLE_ID=com.example.YourApp cucumber
The device and the host computer should be on the same network.
The .ipa must be installed on your device.
You can do this in Xcode by dragging-and-dropping in the Device Manager window or with a third-party tool like ideviceinstaller (available with homebrew).
You should never set NO_LAUNCH.
http://calabashapi.xamarin.com/ios/file.ENVIRONMENT_VARIABLES.html#label-NO_LAUNCH
You also need to enable you device for develop and turn on UIAutomation in the Settings.app > Developer.
Enabling device for development
RunLoop::Fifo::NoReaderConfiguredError
UPDATE In response to this question, I rewrote the Testing on Physical Devices Calabash iOS wiki page. Wow, that content was ancient!
As long as the file is build with calabash then that is not a problem. You just call cucumber with APP_BUNDLE_PATH to where ever your .ipa file is and should be all set.
But you should be aware that .ipa needs to be build to match the target you run the tests on.
I've set up jenkins on my local machine. Than I created a sample app with one failing and one succeeding unit test. When I issue the following command in the terminal
xcodebuild -scheme 'SampleWithTest' -sdk iphonesimulator7.0
-destination platform='iOS Simulator',OS=7.0,name='iPhone Retina (3.5-inch)' clean test
than it clean build the project, starts the simulator and makes the tests. This is how it should work.
When I run the same command via jenkins, it clean builds the project and ends with ** TEST SUCCEEDED ** without starting the simulator or print out the test results. Jenkins just fakes, that the tests was successful.
I test it on my local machine. Nothing headless, remote, source control, slave, code signing or whatever could make any trouble.
The Jenkins Xcode Plugin has not been updated to work with Xcode 5 and the XCTest framework. See this issue. Jenkins and the Xcode Plugin do not understand XCTest output.
Additionally, Xcode 5 changed how Xcode interacts with the simulator and devices when running tests. While there is something like a "headless mode" now for the simulator, access to that functionality isn't available to the Xcode plugin. This means that in some cases, it looks to the Xcode plugin as if the tests run and exit immediately, which is not accurate.
Xcode is known to be pretty tricky to setup with Jenkins, Travis or other non-Apple CI solutions.
Unit Tests support is even worse so you should try using xctool instead.
Related blog entry here.
So we're building this iOS application. We have performance tests that are automatically run by Jenkins. Of course these tests must be run on the device.
Right now we have a lot of trouble with some tests, and I suspect that they don't clean up after themselves, leaving the app's Documents folder in an uncertain state. Obviously I can fix the cleanup, but this will take a huge time.
I'd love to be able to just delete the app from the device before running the tests. Is there any way we can do this via xcodebuild or any other automatable tool. Could be even applescript or a non-official tool. The only rule is that we may not jailbreak the device.
One multi-platform option is libimobiledevice and the ideviceinstaller utility. It runs on Linux, Mac and Windows and even supports iOS 9.
As noted in the library webpage, it does not depend on jailbreak.
To delete an app do:
ideviceinstaller -uninstall com.somecompany.someapp
In Xcode 5, I have a workspace with multiple projects and a scheme that should run all the test targets of the various projects. When I kick off the tests about 3 test project targets pass. So far, so good. But at the fourth test target I get an error message from Xcode telling me that the simulator can't be launched because it is already in use.
Is there any way to kill the simulator between each test target(!) in a single(!) scheme?
Or are there other solutions?
Edit: I have opened a rdar for this: http://openradar.appspot.com/15153136
Thanks!
I have come into this error whenever i try to run my functional tests using calabash. One way to do this maybe be to run a script in the after each build process to kill the simulator.
this guy has wrote a bash script that does this
http://cdrussell.blogspot.co.uk/2012/04/script-to-kill-iphone-simulator-if.html
Might be worth running this as a build phase for each test target