I am working with a CI system, Jenkins, that calls a script to launch automated KIF tests on our iOS application.
While running, the application crashes because of an uncaught exception in our code in -[KIFTestActor runBlock:complete:timeout].
Unfortunately, no error is reported by the automated KIF tests, and so Jenkins incorrectly reports that the automated testing was a success. I had thought that perhaps abort(), which is in the stackcrawl, was causing the application to exit without an error code, so I registered a handler to catch uncaught exceptions and called _exit(-1) to make sure we were returning an error code. That was able to catch the exception and call _exit(-1), but the Jenkins job still thinks that everything passed.
Has anyone dealt with this before and figured how to deal with this?
I see this post to which no one has not yet responded.
This is how we launch the tests:
xcodebuild clean test -workspace MyApp.xcworkspace -scheme MyApp -destination OS=latest,name="iPhone Retina (4-inch)"
The only way we solved this was by parsing the output of the xcodebuild test command.
Thanks to the post by Leonardo Borges.
Related
Context -
We have E2E Testing running on the CI on the iPhone devices.
The script performs the following steps -
Build the Test runner app from code.
Extracts the app file from the IPA.
Re-codesign the app with the required Provision Profiles.
Creates a .xctestrun file
Uses the xcodebuild command like the following to run the test -
xcodebuild -sdk iphoneos -destination id=<device_id> -xctestrun <path_to_xctestrun_file> -resultBundlePath <path_to_test_bundle_file> test-without-building -only-testing:E2ETestsBundle.xctest/<test_filter>
The above command installs the runner and target app into the device and starts executing the test.
The apps get installed and uninstalled at each run during a single execution.
Problem -
During the execution, the xcodebuild CLI tool, suddenly loses connection to the device and is not able to install the apps into the device and then results in the following error -
Testing failed:
E2ETestsBundle.xctest:
E2ETestsBundle-Runner encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. (Underlying Error: Test runner exited before starting test execution. If you believe this error represents a bug, please attach the log file at /var/folders/1r/0xlnbmzd0yj3m_cc7p95yb4r0000gn/T/tmpl7d9p1s6/testResults.bundle/Staging/1_Test/Diagnostics/E2ETestsBundle-Configuration-Test Scheme Action-Iteration-1/E2ETestsBundle/Session-E2ETestsBundle-2022-04-13_205520-muaMCx.log))
** TEST EXECUTE FAILED **
Points to note -
The xcodebuild CLI tool will keep on failing with the error once it occurred until recovered.
Even while the xcodebuild is in a failed state, the devices are still accessible via Xcode IDE and other CLI tools like libimobiledevice and are able to install both the test runner and target applications.
This is occurring intermittently across all devices models and across subversions of iOS 14 and 15.
Recovery -
The only way to recover is to reboot the device.
Even after a reboot, the devices start facing this exception again after a few days of execution.
As we can’t keep on rebooting the device every time before execution, we need a solution to fix this problem.
Any help would be appreciated!!
Here is how we are setup while facing the issue.
We have a project which has both UI and Unit tests.
Uses fastlane.
Has CI/CD implementation using Jenkins which runs on an aws ec2 instance.
Quite recently we moved to Xcode12 (I know!!! However, better late than never)
Problem statement:
After moving to xcode12, the jenkins pipeline is unable to generate the Coverage.profdata file which in turn will be used by Slather to derive code coverage metrics. Yes, did not happen when running xcode11.
We are using the xcodebuild command after pointing it to the appropriate command line tool version (in this case it is 12.4).
First we build the app using (after getting the simulator ID)
xcodebuild build-for-testing -workspace Example.xcworkspace -scheme ExampleUITests -derivedDataPath ./derivedData -destination 'platform=iOS Simulator,id=<simulator_id>'
Then we run the test using
xcodebuild test-without-building -workspace Example.xcworkspace -scheme ExampleUITests -enableCodeCoverage YES -destination 'platform=iOS Simulator,id=<simulator_id>' -derivedDataPath ./derivedData
All the UI tests run successfully but end up with the following statements when the test suite completes.
warning: <Path_to_derived_data>/Build/ProfileData/<Some_UUID>/<Filename>.profraw: Invalid instrumentation profile data (file header is corrupt)
error: No profiles could be merged.
P.S: This is not happening on my local machine. Only on the mac in the Jenkins ec2 instance.
We faced a similar issue. We were also using fastlane.
And the fix which worked for us was to ensure we delete any artifacts from the previous build. The situation you've described is quite similar to situation we faced. The first run for unit tests generates a few coverage artifacts, after which executing UI tests also generates coverage artifacts and then xcov/slather is unable to parse these artifacts.
Therefore, can you check if deleting the Logs/Test folder in derived data helps? If it doesn't try clearing derived data entirely and then check if that helps resolve the issue on CI.
See related threads on xcov
We have an iOS unit test (XCTest) suite we run on Jenkins using this command:
xcodebuild -workspace MyApp.xcworkspace -configuration Debug -scheme MyScheme -destination platform='iOS Simulator',name='iPhone 6' clean test RUN_UNIT_TESTS=YES
Every once in a while (less than once a day) we get the suite failing with this error:
Testing failed:
MyTest encountered an error (Test session exited(-1) without completing. If you believe this error represents a bug, please attach the log file at /var/folders/bp/z5grkxn50z1ffym24hsk17fc00007q/T/com.apple.dt.XCTest-status/Session-2015-08-14_14:28:55-cPXXXX.log)
** TEST FAILED **
...
Build step 'Execute shell' marked build as failure
My guess is that the app is crashing during the test run. Every once in a while I'll see the test suite crash when running the app on my machine in Xcode. I'm usually busy in the middle of trying to get my work done and don't want to investigate so I'll re-run the tests and they'll pass. When this happens on my machine I've noticed the crash happening in a background thread. It's probably some dependency we're not stubbing in our test suite that's not configured correctly under test and it's trying to make network requests.
My question is, is there a way to get a backtrace from such crashes? I looked in the log file mentioned in the error message but that file doesn't have any more information on the crash.
It might also be worth mentioning that our test suite is configured to run within the iOS simulator, if that makes a difference.
Running XCTest-based tests from the cmd-line causes the tests to silently pass, even when a test fails when run via the IDE.
Added a new XCTest-based test target to an iOS App.
Ran tests with the following command:
xcodebuild clean test -workspace VoucherBlaster.xcworkspace -scheme VoucherBlaster -configuration Debug -destination OS=7.1,name=iPad
xcodebuild reported tests as passing, even though the example test failed.
Used SenTest instead and the build reported failed tests, as expected.
Questions:
Is this a common/known issue? (If it is I'd normally be surprised that the problem made it all the way into Xcode5.1 GA, but it wouldn't be the first time :-/ )
Is there a work-around?
I've set up jenkins on my local machine. Than I created a sample app with one failing and one succeeding unit test. When I issue the following command in the terminal
xcodebuild -scheme 'SampleWithTest' -sdk iphonesimulator7.0
-destination platform='iOS Simulator',OS=7.0,name='iPhone Retina (3.5-inch)' clean test
than it clean build the project, starts the simulator and makes the tests. This is how it should work.
When I run the same command via jenkins, it clean builds the project and ends with ** TEST SUCCEEDED ** without starting the simulator or print out the test results. Jenkins just fakes, that the tests was successful.
I test it on my local machine. Nothing headless, remote, source control, slave, code signing or whatever could make any trouble.
The Jenkins Xcode Plugin has not been updated to work with Xcode 5 and the XCTest framework. See this issue. Jenkins and the Xcode Plugin do not understand XCTest output.
Additionally, Xcode 5 changed how Xcode interacts with the simulator and devices when running tests. While there is something like a "headless mode" now for the simulator, access to that functionality isn't available to the Xcode plugin. This means that in some cases, it looks to the Xcode plugin as if the tests run and exit immediately, which is not accurate.
Xcode is known to be pretty tricky to setup with Jenkins, Travis or other non-Apple CI solutions.
Unit Tests support is even worse so you should try using xctool instead.
Related blog entry here.