I've set up jenkins on my local machine. Than I created a sample app with one failing and one succeeding unit test. When I issue the following command in the terminal
xcodebuild -scheme 'SampleWithTest' -sdk iphonesimulator7.0
-destination platform='iOS Simulator',OS=7.0,name='iPhone Retina (3.5-inch)' clean test
than it clean build the project, starts the simulator and makes the tests. This is how it should work.
When I run the same command via jenkins, it clean builds the project and ends with ** TEST SUCCEEDED ** without starting the simulator or print out the test results. Jenkins just fakes, that the tests was successful.
I test it on my local machine. Nothing headless, remote, source control, slave, code signing or whatever could make any trouble.
The Jenkins Xcode Plugin has not been updated to work with Xcode 5 and the XCTest framework. See this issue. Jenkins and the Xcode Plugin do not understand XCTest output.
Additionally, Xcode 5 changed how Xcode interacts with the simulator and devices when running tests. While there is something like a "headless mode" now for the simulator, access to that functionality isn't available to the Xcode plugin. This means that in some cases, it looks to the Xcode plugin as if the tests run and exit immediately, which is not accurate.
Xcode is known to be pretty tricky to setup with Jenkins, Travis or other non-Apple CI solutions.
Unit Tests support is even worse so you should try using xctool instead.
Related blog entry here.
Related
Here is how we are setup while facing the issue.
We have a project which has both UI and Unit tests.
Uses fastlane.
Has CI/CD implementation using Jenkins which runs on an aws ec2 instance.
Quite recently we moved to Xcode12 (I know!!! However, better late than never)
Problem statement:
After moving to xcode12, the jenkins pipeline is unable to generate the Coverage.profdata file which in turn will be used by Slather to derive code coverage metrics. Yes, did not happen when running xcode11.
We are using the xcodebuild command after pointing it to the appropriate command line tool version (in this case it is 12.4).
First we build the app using (after getting the simulator ID)
xcodebuild build-for-testing -workspace Example.xcworkspace -scheme ExampleUITests -derivedDataPath ./derivedData -destination 'platform=iOS Simulator,id=<simulator_id>'
Then we run the test using
xcodebuild test-without-building -workspace Example.xcworkspace -scheme ExampleUITests -enableCodeCoverage YES -destination 'platform=iOS Simulator,id=<simulator_id>' -derivedDataPath ./derivedData
All the UI tests run successfully but end up with the following statements when the test suite completes.
warning: <Path_to_derived_data>/Build/ProfileData/<Some_UUID>/<Filename>.profraw: Invalid instrumentation profile data (file header is corrupt)
error: No profiles could be merged.
P.S: This is not happening on my local machine. Only on the mac in the Jenkins ec2 instance.
We faced a similar issue. We were also using fastlane.
And the fix which worked for us was to ensure we delete any artifacts from the previous build. The situation you've described is quite similar to situation we faced. The first run for unit tests generates a few coverage artifacts, after which executing UI tests also generates coverage artifacts and then xcov/slather is unable to parse these artifacts.
Therefore, can you check if deleting the Logs/Test folder in derived data helps? If it doesn't try clearing derived data entirely and then check if that helps resolve the issue on CI.
See related threads on xcov
I have a project that has multiple different targets/schemes (~38 of them as of writing this question) and I am trying to get unit testing working properly across all of the different targets. I got things working with one target, and tried adding my testing target to all of the different schemes, but it looks like when tests run for each scheme they actually run on the same, original target.
Looking in the project file I see that there's a specific Host Application associated with my testing target, and in the build settings the Bundle Loader and Test Host point to that same Host Application.
Is there any way to override those values for each scheme to run the tests against the current scheme's build target? Or some other way to set up a single test target to run across multiple build targets?
If you run the tests from the command line, or from an CI tool, like Jenkins, you can instruct xcodebuild to use the build settings that you provide. A simple usage example would be:
xcodebuild -scheme SomeScheme test TEST_HOST=target
You can control almost (if not any) build setting from Xcode, like code coverage generation, build directory, derived data directory, code sign identity, etc.
You can select the scheme when you run tests with Xcode server.
Look at WWDC 2014 continues integration talk for a walk through on how to set it up
https://developer.apple.com/videos/play/wwdc2014-415/
It's using Xcode 6 but it's very similar process to Xcode 7
Also check this CI(continues integration) guideline from apple
https://developer.apple.com/library/ios/documentation/IDEs/Conceptual/xcode_guide-continuous_integration/adopt_continuous_integration.html
If anyone is wondering how to do that with UI tests (maybe it is working with unit tests, too), this is what I came up with:
First, we need to build the application which we are going to use to host our UI tests:
xcodebuild -scheme "<appSchemeName>" build -destination "<yourDestination>"
More info about destination parameter: https://mokacoding.com/blog/xcodebuild-destination-options/
Then, we need to run tests on our newly built application:
xcodebuild -scheme "<uiTestsSchemeName>" -destination "<yourDestination>" test TEST_TARGET_NAME="<yourNewlyBuiltAppTargetName>"
Destinations should match, since after the build .app is generated in your DerivedData folder, which will be used for UI tests hosting application
Is it possible to configure Xcode in such a way so when I run all of my test each one will be performed on a "new iPhone Simulator". I'm testing UI related stuff and I want each test to start on a newly opened application.
Currently, application is starting only once and all of the tests are starting one after another and sometimes the ending state after one test is not good for the next test to start.
Solved
I used XCTool (https://github.com/facebook/xctool)
I was able to run one particulatr test by invoking command:
xctool -workspace WorkspaceName.xcworkspace -scheme Workspace -sdk iphonesimulator run-tests -only TestTarget:TestClass/test_Some_Method
Then I simply wrote a script to run all of the tests I had. This way every test triggers simulator to run application once again from scratch.
You could, before the next test, delete the app as you would do it in a real iPhone.
In Xcode 5, I have a workspace with multiple projects and a scheme that should run all the test targets of the various projects. When I kick off the tests about 3 test project targets pass. So far, so good. But at the fourth test target I get an error message from Xcode telling me that the simulator can't be launched because it is already in use.
Is there any way to kill the simulator between each test target(!) in a single(!) scheme?
Or are there other solutions?
Edit: I have opened a rdar for this: http://openradar.appspot.com/15153136
Thanks!
I have come into this error whenever i try to run my functional tests using calabash. One way to do this maybe be to run a script in the after each build process to kill the simulator.
this guy has wrote a bash script that does this
http://cdrussell.blogspot.co.uk/2012/04/script-to-kill-iphone-simulator-if.html
Might be worth running this as a build phase for each test target
I am upgrading a Jenkins CI server from XCode 4.6 to XCode 5 and iOS SDK 7.
To run unit tests, I must now use xcodebuild test. I am piping the output of this command to a Ruby script, that generates test reports.
This does not work any longer after the upgrade. I think the problem lies in the fact that xcodebuild test launches a new process to actually run the tests. The last statement in my log is
/bin/sh -c /Volumes/Data/Jenkins/jobs/MyProject-Unit/workspace/Build/Intermediates/Redacted.build/Debug-iphonesimulator/UnitTests.build/Script-261C7EA912FD9A4B00F7A53E.sh
The output after that point can not be found in the Jenkins log, but when I run it on my own machine, I get the test results printed in the console.
What is happening here, and how can I get the test reports back ?
I'm on the same trail. I believe if you look into that script, it will turn around and call this script:
${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests
This is script that used to run unit tests. Now, in Xcode 5, it just tells you to remove the Run Script build phase that Xcode 4 automatically creates. If you run RunUnitTests from Xcode 5, it just says:
RunUnitTests is obsolete. To run unit tests for your target, use the
Test scheme action in the Xcode IDE and the test action in xcodebuild.
You can remove the Run Script build phase that invokes RunUnitTests
from your unit test bundle target.
I've gone ahead and done what it says (removed the Run Script phase) and I still get no output under Jenkins, perhaps because, as you surmise, it's a new process, and Jenkins sees the first process complete, and jumps out.
Update: More info here: How can I get Xcode 5 to run iOS unit tests on a Jenkins slave?