Running XCTest-based tests from the cmd-line causes the tests to silently pass, even when a test fails when run via the IDE.
Added a new XCTest-based test target to an iOS App.
Ran tests with the following command:
xcodebuild clean test -workspace VoucherBlaster.xcworkspace -scheme VoucherBlaster -configuration Debug -destination OS=7.1,name=iPad
xcodebuild reported tests as passing, even though the example test failed.
Used SenTest instead and the build reported failed tests, as expected.
Questions:
Is this a common/known issue? (If it is I'd normally be surprised that the problem made it all the way into Xcode5.1 GA, but it wouldn't be the first time :-/ )
Is there a work-around?
Related
Here is how we are setup while facing the issue.
We have a project which has both UI and Unit tests.
Uses fastlane.
Has CI/CD implementation using Jenkins which runs on an aws ec2 instance.
Quite recently we moved to Xcode12 (I know!!! However, better late than never)
Problem statement:
After moving to xcode12, the jenkins pipeline is unable to generate the Coverage.profdata file which in turn will be used by Slather to derive code coverage metrics. Yes, did not happen when running xcode11.
We are using the xcodebuild command after pointing it to the appropriate command line tool version (in this case it is 12.4).
First we build the app using (after getting the simulator ID)
xcodebuild build-for-testing -workspace Example.xcworkspace -scheme ExampleUITests -derivedDataPath ./derivedData -destination 'platform=iOS Simulator,id=<simulator_id>'
Then we run the test using
xcodebuild test-without-building -workspace Example.xcworkspace -scheme ExampleUITests -enableCodeCoverage YES -destination 'platform=iOS Simulator,id=<simulator_id>' -derivedDataPath ./derivedData
All the UI tests run successfully but end up with the following statements when the test suite completes.
warning: <Path_to_derived_data>/Build/ProfileData/<Some_UUID>/<Filename>.profraw: Invalid instrumentation profile data (file header is corrupt)
error: No profiles could be merged.
P.S: This is not happening on my local machine. Only on the mac in the Jenkins ec2 instance.
We faced a similar issue. We were also using fastlane.
And the fix which worked for us was to ensure we delete any artifacts from the previous build. The situation you've described is quite similar to situation we faced. The first run for unit tests generates a few coverage artifacts, after which executing UI tests also generates coverage artifacts and then xcov/slather is unable to parse these artifacts.
Therefore, can you check if deleting the Logs/Test folder in derived data helps? If it doesn't try clearing derived data entirely and then check if that helps resolve the issue on CI.
See related threads on xcov
I am running the following command from command line to build/test my workspace. The idea is to port it eventually to a CI system. But what happens is that one of the runs succeed but all of the following ones fail. They fail in an odd manner in that the build completes as part of the command but then, I see the message which kicks off the simulator and I see the simulator up but the tests are never run on it. The command is either hung on it forever or it fails with no error message. I tried using xcrun simctl erase all before the command to make sure the simulator is clean. I also tried avoiding specifying derivedDatapath and OBJROOT hoping that would make the builds clean but so far, I am stuck here.
xcodebuild test -workspace <workspace-name> -scheme <scheme-name> -configuration Coverage -sdk iphonesimulator -destination platform='iOS Simulator',OS=9.0,name='iPhone 6'
The final error I see is /usr/bin/touch -c /home-folder/Library/Developer/Xcode/DerivedData/.../Build/Products/Coverage-iphonesimulator/Tests.xctest ** TEST FAILED **
The code runs smoothly from within XCode and the tests always pass. I believe this is some issue with the terminal app connecting to the simulator or the simulator being in a weird state after an earlier run but I could not figure out what could be causing the problem. Any suggestions would be great.
XCode Version: 7.0.1
I don't know if that is the issue, but shouldn't test be at the end of the line? Like this:
xcodebuild -workspace <workspace-name> -scheme <scheme-name> -configuration Coverage -sdk iphonesimulator -destination platform='iOS Simulator',OS=9.0,name='iPhone 6' test
We have an iOS unit test (XCTest) suite we run on Jenkins using this command:
xcodebuild -workspace MyApp.xcworkspace -configuration Debug -scheme MyScheme -destination platform='iOS Simulator',name='iPhone 6' clean test RUN_UNIT_TESTS=YES
Every once in a while (less than once a day) we get the suite failing with this error:
Testing failed:
MyTest encountered an error (Test session exited(-1) without completing. If you believe this error represents a bug, please attach the log file at /var/folders/bp/z5grkxn50z1ffym24hsk17fc00007q/T/com.apple.dt.XCTest-status/Session-2015-08-14_14:28:55-cPXXXX.log)
** TEST FAILED **
...
Build step 'Execute shell' marked build as failure
My guess is that the app is crashing during the test run. Every once in a while I'll see the test suite crash when running the app on my machine in Xcode. I'm usually busy in the middle of trying to get my work done and don't want to investigate so I'll re-run the tests and they'll pass. When this happens on my machine I've noticed the crash happening in a background thread. It's probably some dependency we're not stubbing in our test suite that's not configured correctly under test and it's trying to make network requests.
My question is, is there a way to get a backtrace from such crashes? I looked in the log file mentioned in the error message but that file doesn't have any more information on the crash.
It might also be worth mentioning that our test suite is configured to run within the iOS simulator, if that makes a difference.
Is it possible to configure Xcode in such a way so when I run all of my test each one will be performed on a "new iPhone Simulator". I'm testing UI related stuff and I want each test to start on a newly opened application.
Currently, application is starting only once and all of the tests are starting one after another and sometimes the ending state after one test is not good for the next test to start.
Solved
I used XCTool (https://github.com/facebook/xctool)
I was able to run one particulatr test by invoking command:
xctool -workspace WorkspaceName.xcworkspace -scheme Workspace -sdk iphonesimulator run-tests -only TestTarget:TestClass/test_Some_Method
Then I simply wrote a script to run all of the tests I had. This way every test triggers simulator to run application once again from scratch.
You could, before the next test, delete the app as you would do it in a real iPhone.
Has anyone been able to successfully pass the KW_SPEC variable to xctool. I'm trying to run a single KIWI spec by using https://github.com/kiwi-bdd/Kiwi/wiki/Kiwi-FAQ#q-how-do-i-run-a-single-spec-describecontextit.
I can run all the tests successfully with xctool but it doesn't seem to pick up the KW_SPEC value. I've tried it in a lot of different places with the command line but no luck.
e.g.:
xctool -destination 'platform=iOS Simulator,name=iPad Retina,OS=latest'
-sdk iphonesimulator -workspace SampleProject.xcworkspace
-scheme SampleProject KW_SPEC=NewAssessmentTests.m:12 test
-only SampleProject_Acceptance_Tests
Using Kiwi v2.3.1 and xctool 0.2.3
Cheers,
Mo
Update 11/03/2015
#OhadSchneider's comment made me realise that KW_SPEC worked for me because I had set it in the test configuration for my scheme (Edit scheme->Test->Arguments->Environment variables). Setting the variable from shell doesn't work as that variable applies only to the actual build, and not when executing the unit test target.
But there's a workaround to this, by modifying the test phase of your scheme and adding a KW_SPEC environment variable with the value $KW_SPEC, this will expand when running xcodebuild to the value passed to the xcodebuild command (as in my original answer). After this, xcode will will gracefully use the passed KW_SPEC variable, xctool still has the skipped teste marked as failure issue.
Original answer
If you want KW_SPEC as an environment variable to xctool (or to any *nix tool), then you have to place it before the command, otherwise it will be considered a build setting:
KW_SPEC=NewAssessmentTests.m:12 xctool
-destination 'platform=iOS Simulator,name=iPad Retina,OS=latest'
-sdk iphonesimulator -workspace SampleProject.xcworkspace
-scheme SampleProject test
-only SampleProject_Acceptance_Tests
This will however lead to another problem: xctool will report as errored the tests that don't run, and will report the test as failed, even if no tests have failed. xcodebuild doesn't have this problem as it either doesn't do unit tests discovery, or ignore tests that didn't run, a thing that xctool fails to do.