I would like to run my tests using xctool without having to build the tests on the machine prior. Basically, I want to build the tests on one machine, transfer them to several other machines and run them in parallel from there.
So I need to know what files the command
xctool run-tests requires in order to run the tests, so I can transfer those files to each computer in order to parallelize my runs.
You probably want the file produced by xctool build-tests, which builds the tests but does not run them.
The last file output by xctool on the command-line is the path the final .xctest – or at least it is for me. Either way, that's what you need.
https://github.com/facebook/xctool#building-tests
If you just want to execute that test bundle, you can do something like this:
xcrun -sdk iphonesimulator simctl spawn booted $(xcrun --show-sdk-platform-path -sdk iphonesimulator)/Developer/Library/Xcode/Agents/xctest /path/to/tests.xctest
Related
Here is how we are setup while facing the issue.
We have a project which has both UI and Unit tests.
Uses fastlane.
Has CI/CD implementation using Jenkins which runs on an aws ec2 instance.
Quite recently we moved to Xcode12 (I know!!! However, better late than never)
Problem statement:
After moving to xcode12, the jenkins pipeline is unable to generate the Coverage.profdata file which in turn will be used by Slather to derive code coverage metrics. Yes, did not happen when running xcode11.
We are using the xcodebuild command after pointing it to the appropriate command line tool version (in this case it is 12.4).
First we build the app using (after getting the simulator ID)
xcodebuild build-for-testing -workspace Example.xcworkspace -scheme ExampleUITests -derivedDataPath ./derivedData -destination 'platform=iOS Simulator,id=<simulator_id>'
Then we run the test using
xcodebuild test-without-building -workspace Example.xcworkspace -scheme ExampleUITests -enableCodeCoverage YES -destination 'platform=iOS Simulator,id=<simulator_id>' -derivedDataPath ./derivedData
All the UI tests run successfully but end up with the following statements when the test suite completes.
warning: <Path_to_derived_data>/Build/ProfileData/<Some_UUID>/<Filename>.profraw: Invalid instrumentation profile data (file header is corrupt)
error: No profiles could be merged.
P.S: This is not happening on my local machine. Only on the mac in the Jenkins ec2 instance.
We faced a similar issue. We were also using fastlane.
And the fix which worked for us was to ensure we delete any artifacts from the previous build. The situation you've described is quite similar to situation we faced. The first run for unit tests generates a few coverage artifacts, after which executing UI tests also generates coverage artifacts and then xcov/slather is unable to parse these artifacts.
Therefore, can you check if deleting the Logs/Test folder in derived data helps? If it doesn't try clearing derived data entirely and then check if that helps resolve the issue on CI.
See related threads on xcov
I am investigating how to use XCTest to close our automation test gap.
From the developer document, I only see something like this:
xcodebuild test -project MyAppProject.xcodeproj -scheme MyApp
https://developer.apple.com/library/tvos/documentation/DeveloperTools/Conceptual/testing_with_xcode/chapters/08-automation.html#//apple_ref/doc/uid/TP40014132-CH7-SW1
I would like to know if there is a way to run XCTest(UI test) against an archived build (.ipa)?
or
Can we just seperate the build and test so that we can build first and test against that build later?
Thanks
If you want to separate the build and test stages, you can use the xcodebuild flags build-for-testing (which builds a xctestrun file) and test-without-building (which runs it). You may want to build-without-testing to a generic device if you want the build-for-testing to be used on multiple devices.
Check out https://medium.com/xcblog/speed-up-ios-ci-using-test-without-building-xctestrun-and-fastlane-a982b0060676 for more details
Playing with iOS 9.0, Xcode GM and the new Xcuitesting framework. How can I run a specific test from the command line instead of having to click play in the ide? I know there's a shortcut to run all the tests but I want to run a single test.
Thanks!
You should be able to do that with xctool command line tool for running specific test class,
xctool -workspace YourWorkspace.xcworkspace -scheme YourScheme test -only myUITestTarget:UITestClass
You can also use xcodebuild to run the entire test suite as explained here,
https://krausefx.com/blog/run-xcode-7-ui-tests-from-the-command-line
Is it possible to configure Xcode in such a way so when I run all of my test each one will be performed on a "new iPhone Simulator". I'm testing UI related stuff and I want each test to start on a newly opened application.
Currently, application is starting only once and all of the tests are starting one after another and sometimes the ending state after one test is not good for the next test to start.
Solved
I used XCTool (https://github.com/facebook/xctool)
I was able to run one particulatr test by invoking command:
xctool -workspace WorkspaceName.xcworkspace -scheme Workspace -sdk iphonesimulator run-tests -only TestTarget:TestClass/test_Some_Method
Then I simply wrote a script to run all of the tests I had. This way every test triggers simulator to run application once again from scratch.
You could, before the next test, delete the app as you would do it in a real iPhone.
When I run
xcodebuild -workspace ~/Documents/JudgecardXSwift/JudgecardXSwift.xcworkspace -scheme JudgecardXSwift -destination 'platform=iOS Simulator,name=iPhone 6' clean test
from command line, I can successfully run all of my tests.
Now I want to add a run script phase to my Xcode project so it will always run my tests each time I run the app. However, when I added the run script phase with the above xcodebuild command, my build always hangs:
and as you can see, it must be hanging on the run script phase I added, because it has just finished the build phase before it titled Run custom shell script 'Copy Pods Resources':
Is my method of adding a run script phase with the xcodebuild command the proper way to automatically run tests with every build? Why does it cause my build process to hang? Is it causing some kind of infinite loop with each xcodebuild command kicking off another through its run script phase?
Bit late here, so thought I would add my personal experience, but xcodebuild will kick off another build for you, importantly xcodebuild via command line as you have may not run your Build Phases as expected. I have only experienced silent failing copy commands, with those I could view the command in log files, but it did not actually copy the file.
So it may hit an infinite loop, more likely though is it's running through that build, hitting xcodebuild command, and then building as normal without some Build Phases executing properly and thus without running your tests.
In Xcode 9 this works. I've added a run script with:
echo "warning: 🚕🚕🚕 Started running myTests 🚕🚕🚕"
xcodebuild -sdk iphonesimulator -workspace myWorkspace.xcworkspace -scheme "myWorkspaceTests" test -destination 'platform=iOS Simulator,name=iPhone X,OS=11.2' CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO
And if one of the tests fail, you're gonna get a build error.