Code Coverage is commonly used with tests in Xcode. I would like to use it for manually executed app. Can I do it, possibly with third-party tools?
For example: I build and launch the app on device, perform some actions with it and then look at code coverage results.
The solution was suggested by someone here: https://github.com/SlatherOrg/slather/issues/209
You could try having a XCUITest that sleeps forever, then manually use the app, and see if on termination the coverage files are generated.
I simply tried the solution and it worked perfectly for me:
class FooTests: XCTestCase {
override func setUp() {
super.setUp()
let app = XCUIApplication()
app.launch()
}
func testBalances() {
sleep(30)
XCTAssert(true)
}
}
After the test succeeded on XCTAssert(true), I could see the code coverage for the manual use cases. You can play around with the sleep(30) to match your requirement.
Probably you might have already figured it out, but it was possible before Xcode7. The way we achieved this was, to set the "Instrument Program Flow" and "Generate Test Coverage files" flags to Yes, on your project, and later add a "flush" code somewhere inside you application, to write the coverage data files. This "flushing" part actually writes the coverage data files, which can be used later by other tools such as gcovr or lcov, to get your coverage data. Once you interact with the app, either manually or through automated tests, the coverage data gets written.
However, with Xcode7, looks like the coverage data is limited to only Xcode unit tests. I am still trying to figure out, if there is any way to gather the coverage data, by interacting with the application manually, or through automated tests.
Code coverage will record which parts of your code your test have run. But you could Build some UITests which would preform some actions as you said. UI can record UI tests to repeat the actions you perform on the simulator then when you run the test it will repeat what you did. The coverage will then show which parts of the code where excited during the UITests.
Look for some info on UITesting in Xcode 7. There is a good demo in one of the developer sessions from wwdc15
https://developer.apple.com/videos/wwdc/2015/?id=406
Related
I have a few UIView class which is coming under coverage even though. I haven't added them under unit test target. Neither in unit test nor UI unit test. but still those file showing.
I do not know whether they are included in test coverage.
Q: How we can remove them from unit test coverage?
Q: Are they involve in unit test coverage score?
Class included in unit test coverage:
Classes not added in test targets:
I think that your problem is related with gather code coverage for all targets.
create new target - You can duplicate existing one.
Change Host Application to brand new target.
Change your test scheme settings to new target as source of code coverage. Please have a look at screenshots:
Remember add to your test file #testable import UnitTestExampleCodeCoverage
Add to new target files which you want.
In my example proj i have: AppDelegate, ViewController, ViewController2, ViewController3, ViewController4, View, View2, View3, View4. I added to new target: AppDelegate, ViewController3, ViewController4, View3, View4 so only this five classes should be base to calculate code coverage.
Add tests to your classes.
Code coverage report contain only classes added to new target. Please remember that old target can be your main target - the new one is only kind of abstract layer to achieve your goal: "remove some classes from code coverage report".
Here is whole project: https://github.com/mStaruchowicz/SO55723759
Happy testing ;)
Hi #Sandy, to answer your questions:
How we can remove them from unit test coverage?
As of Xcode 10.2 there is no way to exclude files from the test coverage report.
Are they involve in unit test coverage score?
Yes they are. All the files contribute to the total coverage %.
If you want to compute a test coverage score that doesn't take into account certain files, your best chance is to write a script to parse the xccov output.
I can't find any official documentation of how to interact with it but this post is a good place to start.
When you active coverage in your project, all files from the target will compute the coverage.
I recommend for you to use XCOV:
https://github.com/nakiostudio/xcov
You can run the coverage and put your ignored files to improve a correct coverage.
If you want customizable UT coverage report where you can ignore the files you wish, Slather can be a best tool.
you can ignore the files from coverage calculation by configuring the files in .slather.yml file under ignore option.
Please find the details here how to ignore.
When unit tests run,
Your app is launched
Tests are injected and run
The problem is the app launch fires up your normal sequence, creating views and maybe initiating some network calls. This not only interferes with code coverage reporting — it potentially poisons your unit tests. For example, if you test notifications by having a unit test fire off a notification, you need to know that the only objects listening for that notification were created by the test, not lying around from app launch.
Another problem with ignoring coverage of certain files is: How are you going to see what your unit test coverage is of those files?
The solution is to use a different app delegate for testing that does almost nothing.
I have an iOS app in Xcode 8.2. It has a test target / scheme, for which “Gather coverage data” is checked in the scheme’s Test / Info settings. Coverage data is not gathered. I see how many times a line was iterated over in the gutter as usual, but the Report navigator’s test runs don’t indicate any coverage at all.
I’m wondering if this is because I’ve set the tests to run hostless, i.e. without needing to actually fire up my app – they’re pure logic tests.
Is this possible?
Yes a hostless XCTest target should gather code coverage data.
'iOS Unit Testing' (aka XCTest) bundles, which test a dynamic framework or something else which doesn't require the application environment to exist to run, should happily collect code coverage data and display it in Xcode. Even of the Host Application is set to None. This works either when running Xcode > Product > Test on the Scheme for the framework under test or on the Scheme for the unit tests themselves (if the test bundle is listed in the Test pane of the Scheme editor).
Your problem must be somewhere else, sorry. Its hard for me to guess what the problem is, I suggest you try making a fresh project and see if you can reproduce the problem.
I am running test cases in application with enabled code coverage data Xcode 7 Beta 2. But I am able to get only few files coverage data while my all test cases are running successfully.
Some files has covered all codes by unit test cases but still showing 3% code coverage.
For example:
This is the result of code coverage, as you can see on the right side, there is an info how many times these lines of code was called during tests. In this case - 0.
But...
here is a place in tests where we can see that this function was called indeed. How many times? oh... at least once. This number is delivered by info on the right side.
So the code above should be marked as called, and not be grayed out:-)
Can anyone explain this? Why does this happen?
IT WORKS.
Since Apple released #testable keyword to import your project into test target, you don't have to add your files to both target anymore:
So just remove every file from your test target:
Wherever you need access to your file from your test target just import your target using: #testable import MyApp
Do this for every file in your project.
Then code coverage will be working fine.
Read more from Swift 2 + Xcode 7: Unit Testing Access Made Easy!!!!
If you need to know how to work with code coverage read How to use code coverage in Xcode 7?
As #Gerd Castan mentioned earlier is: "So it appears to me that a tested method shows a coverage of 0 when there exists at least one target where this method is not tested."
Solution is simple. Do not let compiler think that this file is included in more that one target, instead import your module using #testable keyword.
I think I found out what XCTest coverage ist doing and it makes some sense:
My setup:
class1 compiled into target1
class2 compiled into target1 and into target2
Test setup:
import XCTest
#testable import target1
class MyTests: XCTestCase {
func testSomething() {
someMethodFromClass1()
someMethodFromClass2()
}
}
What I find is that class1 (compiled into target1) shows test coverage and class2 (compiled into target1 and into target2) shows no test coverage.
So it appears to me that a tested method shows a coverage of 0 when there exists at least one target where this method is not tested.
And this makes a lot of sense, because testing a method in a target doesn't say anything about how it behaves in a different target.
Apple wants us to test all targets.
Update
One more hint to back this theory:
go to the report navigator
and click on coverage.
If you have more than one target, you see your files grouped by target.
And if you have one file in two targets, you see your file twice.
If you have one file in both targets, the code coverage of this one file is shown for both targets. And (at least in my projects) one file has different blue lines in each target:
coverage in target 1:
coverage of same file in the same project in the same test run in target 2:
If you look at your test coverage in the source editor, apple has to decide which coverage it shows to you.
I think showing the target with the lowest coverage is the best apple can do in the source editor.
simple fix for a special case:
If your only second target is your test target: don't compile into your test target and use #testable import.
For all other cases you have to test each target.
I checked at Apple developers forums for the subject and after reading through various posts I guess I came across the solution.
In order for the thing to work it is necessary to:
Remove all your application source files from the test target
In your unit-test sources put #testable import <NameOfYourModule>
Re-build and re-run tests
I tested this out with my current project, and the results are much better.
Original solution recipe can be found at: http://natashatherobot.com/swift-2-xcode-7-unit-testing-access/
Also it seems that the functionality is a bit raw-ish, hence bugs possible, and Apple suggests submitting bug reports when things do not work as expected:
I've personally seen code coverage results for some very large projects. Right now the support works best for applications and frameworks. If that's what you're testing, then it would be best if you could file a bug report at https://bugreport.apple.com so that we can investigate your particular circumstances. For that matter, a bug report would be a good thing no matter what type of project you have. If possible, it's best to actually in the project with the report. If you can't do that, describe its setup in as much detail as possible. Pictures are good.
Original thread: https://forums.developer.apple.com/message/9733#9733
Bear in mind that there are multiple ways to cover code with tests, you may test all functions, or you may be covering all instructions in the functions, but you may not be covering all the execution paths that are possible.
Or Xcode coverage stuff may be broken, but it's hard to tell if you don't give details on what kind of coverage are you expecting it to check.
This happens because .swift file of your project selected for both targets by default.
Manually select and remove test target for files works for me.
I am using UIAutomation to test an app, and I would like to find out my code coverage. But since javascript has no preprocessor, that means that gcov and similar are not an option. Has anyone come up with a solution for this?
For Xcode version 4.5 and newer:
Set the “Generate Test Coverage Files” build setting to Yes.
Set the “Instrument Program Flow” build setting to Yes.
This will generate code coverage files every time you run your application in the simulator and exit the application. A detailed explanation of these two steps can be found at the beginning of http://qualitycoding.org/xcode-code-coverage/.
For any unit tests, code coverage files will be generated every time you hit the test button and the tests complete. For UIAutomation, it is a little bit more tricky. You have to ensure the application exits at the conclusion of your tests. The easiest way I found to do this is to turn off multitasking. Add UIApplicationExitsOnSuspend in your MyAppName-Info.plist file and set this option to 'YES'. Run your UI automation test and at the end of it you can exit the app either by manually pressing the home button in the simulator or using the UIATarget.localTarget().deactivateAppForDuration() method.
Note: if your app has any UI Automation tests that rely on the deactivateAppForDuration() method, the tests will terminate upon running the command.
Code Coverage is only used for Unit Testing, there is no Code Coverage for UIAutomation because there is no way to tell how many elements on screen has been "touched" by UIAutoamtion
I have enabled test coverage with no problem using the Generate Test Coverage Files and Instrument Program Flow with the fopen$UNIX2003 and fwrite$UNIX2003 hack.
But the problem is, when you use XCode to run tests it ends up launching the simulator which launches your app. When that happens the output for the test coverage is not truly correct because it thinks certain parts of the code are touched because they are executed when the app launches, not because a test touched them.
Is there a better way to see what code was actually touched by a test?
EDIT:
So not really an answer to my question, but a somewhat "solution" that will at least get better coverage numbers can be found here: Run logic tests in Xcode 4 without launching the simulator