We are setting up a Swift iOS Project, build with fastlane, to be analysed by SonarQube with the SonarSwift Plugin.
Everything works so far except for information about the Tests.
We archived to add Code coverage by generating a report with slather (llvm-cov) and renaming it Coverage.report and filling sonar.swift.coverage.reportPath with the path.
We also generate *.junit files and feed them into sonar.junit.reportsPath. But those seem to never be included.
We tried only the path ./reports and naming the files with TEST- prefix and .xml ending. We also tried a direct Path ./reports/TEST-report.xml.
Did anybody got this to work? If yes, How? Is this even supported by SonarQube?
According to https://community.sonarsource.com/t/number-of-unit-tests-is-zero/26104/6
JUnit reports only support Java.
For Swift, we need to convert the report to a generic type
Firstly, For Coverage (https://gist.github.com/lucascorrea/80f3d57a7f97b2365ef39839eefe68a1)
run test to generate test data
convert to coverage data with xcrun llvm-cov show that sonar can recognize
configure sonar.swift.coverage.reportPath to the report path generated by xcrun
you can run sonar-scanner to submit coverage data
sh("xcodebuild -workspace ../brewdog.xcworkspace -scheme brewdog -sdk iphonesimulator -enableCodeCoverage YES -destination 'platform=iOS Simulator,name=iPhone 7,OS=11.3' GCC_GENERATE_TEST_COVERAGE_FILES=YES build test -derivedDataPath ../DerivedData")
sh("xcrun llvm-cov show -instr-profile=$(find ../DerivedData -iname 'Coverage.profdata') ../DerivedData/Build/Products/Debug-iphonesimulator/brewdog.app/brewdog > ../DerivedData/Coverage.report")
Secondly, For UnitTest (https://github.com/azohra/forsis)
After you done this, you can check information like count of unit tests, success, fail, duration (There are some pictures in the above link)
This one is the same, you need to convert the test execution report to generic type with this tool
set up the convert
**[IMPORTANT]**configure sonar.tests AND sonar.testExecutionReportPaths in sonar-project.properties
run sonar-scanner
After done the two things
You can check the coverage AND unit test in the column in sonarqube website!!
Related
I have a set of tests for an iOS app written using XCTest, and would like to include a code coverage report in the pipeline.
At the moment, I am using scan followed by xcov with fastlane, but every time I run the tests a different subset of targets are included in the report - effectively making any code coverage summary meaningless.
I believe this is an issue with the scan step running the tests and collecting the coverage information rather than xcov, as if I repeatedly run the coverage report step by itself it behaves consistently.
I have tried with both "Gather coverage for all targets" enabled and a subset of targets selected and see the same issues.
Any help hugely appreciated!
In summary:
Tried:
- Running XCTest tests with `scan` in fastlane
- Creating a coverage report with `xcov` in fastlane
(and also the same thing but running the tests through the XCode GUI)
Expected:
- A consistent code coverage report
Recieved:
- A well-formatted coverage report every time,
but with an inconsistent subset of targets included in it
Edit: extra details as requested
Fastlane
The fastlane tasks (with names anonymised) are as follows. These are run one after the other in the order listed below in my pipeline.
desc "GH clean"
lane :gh_clean do
clear_derived_data
clean_build_artifacts
end
desc "GH unit test"
lane :gh_unit_test do
scan(scheme: "ClientTestScheme", device: "iPhone 14 Pro")
end
desc "GH test coverage report"
lane :gh_coverage_report do
xcov(scheme: "ClientTestScheme", output_directory: "./coverage_reports")
end
The above produces a nice-looking report of the test coverage, but the included targets are inconsistent.
Non-fastlane
I've also tried a non-fastlane approach with the following bash script and I see the same inconsistent behaviour in terms of which targets are included:
rm -rf Build/Logs/Test/*.xcresult
rm data.json
xcodebuild -project ClientProject.xcodeproj/ -scheme ClientTestScheme -derivedDataPath Build/ -destination 'platform=iOS Simulator,OS=16.2,name=iPhone 14 Pro' -enableCodeCoverage YES clean build test CODE_SIGN_IDENTITY="" CODE_SIGNING_REQUIRED=NO
xcrun xccov view --report Build/Logs/Test/*.xcresult --json > data.json
I want to have a run script that looks at the number of previous warnings (probably during the compile phase), and generates an error which stops the run/buid if the number of warnings is too large.
I am working on an app that has a large amount of warnings that happen during compile. They do not stop the app from compiling, but I want to stop devs from adding more warnings as they add features.
I have looked everywhere, but so far have not found a way to use previous build output in the run script phase of the build. Is there any way of doing this? Some kind of env variable, or a way of monitoring the build output in a custom script as it happens?
I see a similar question here:
Is there a way to access Xcodes current build output from a build phase run script?
But it seems that the output is only available after the build is complete in that answer. Is there any other option that would allow the build to be failed before it finishes?
Any insight into the build system in Xcode would be appreciated! Cheers
Additional context:
All I need is for Xcode's default warnings to have a feature similar to SwiftLint's warning_threshold
https://stackoverflow.com/a/52256459/7623867
I have looked into that problem as well for quite some time (I wanted to use warnings for quality measurement). The only way I have found is to compile the whole project with xcodebuild, with a command similar to:
xcrun xcodebuild -project MyProject.xcodeproj/ -scheme MyScheme -destination "platform=iOS Simulator" build -quiet | tee xcodebuild.log
and then counting the warnings using AWK or some other tool.
Here is an example.
Using xcodebuild I can run a specific test or group of tests. But is there a way to list all tests without running them, using xcodebuild or any other tool that comes with xcode? Facebook's xctool can do this but I hope to not take dependency on third party tools.
Better late than never - I also needed to do this and couldn't find a way using xcode, so I wrote a python script that parses the test files and gets the names of XCTest test functions:
import re
import os
all_tests = []
TEST_FILES = os.path.expanduser("~/git/ios-proj/Test")
test_case_pattern = re.compile(r"^\s*func\s(test\w*)\(\)[\s\w]*\{")
for test_file in os.listdir(TEST_FILES):
all_tests += [test_case_pattern.match(line)[1] for line in
open(os.path.join(TEST_FILES, test_file), "r").readlines() if test_case_pattern.match(line)]
print(all_tests)
You can get the test names by using "strings". it's not an xcode command, but is normally available on the mac. Run "strings" on the following file and grep whatever marks your tests. in my case, all tests start with "test_":
strings <Tests-Target>-Runner.app/PlugIns/<Tests-Target>.xctest/<Tests-Target>|grep "test_"
an easy way to find this file is by running:
grep -r <single test name> <derived data folder for your project>
I cannot think of any way to sort automatically... but
If you want to test one of the classes by one command So you can run your test one by one through this command :
xcodebuild test -destination 'platform=iOS Simulator,name=iPhone 7,OS=11.1' '-only-testing:APPNAME-TESTTARGET/NAMEOFTHECLASS'
also you can see your list of targets, builds configurations, and schemes by using this command :
xcodebuild -list
Summary
I'm trying to get a code coverage report from my project. However when I try to build and run tests, there is no CodeCoverage folder created in the derived data folder.
How am I trying to do this?
I am running xcodebuild as follows:
xcodebuild \
-workspace <some_workspace> \
-scheme <some_scheme> \
-destination <some_destination> \
-derivedDataPath <some_path> \
-enableCodeCoverage YES \
build test
What is the problem?
For my workspace/project it fails at the very end with a line:
xcodebuild: error: Failed to build workspace <some_workspace> with scheme <some_scheme>.
Reason: Could not determine bundle identifier for <some_test_target>'s TEST_HOST: "<some_path>/Build/Intermediates/CodeCoverage/Products/Debug-iphonesimulator/<some_product>.app"
At first it seemed directly linked to the TEST_HOST issue, but that's not the case here.
There is simply no CodeCoverage folder:
<some_path>/Build/Intermediates/CodeCoverage
What have I tried?
I tried the same with a clean new project, same running the same xcodebuild command, which succeeds. Within the Build/Intermediates/ folder exists the CodeCoverage folder.
However, in my own project, which is more complex, the Build/Intermediates/ folder contains a bunch of *.build folders (related to the app and the various frameworks, HockeySDK.build for example) and PrecompiledHeaders but no CodeCoverage folder.
When looking through the output of xcodebuild I never see a reference to the CodeCoverage folder for my project. However, for the test project, the first mention is at:
Write auxiliary files
/bin/mkdir -p /<some_path>/Build/Intermediates/CodeCoverage/Intermediates/<some_project>.build/Debug-iphonesimulator/<some_project>.build/Objects-normal/x86_64
For my project I see:
Write auxiliary files
/bin/mkdir -p /<some_path>/Build/Intermediates/<some_project>.build/Debug-iphonesimulator/<some_project>.build/Objects-normal/x86_64
Workaround
Something else that seemed to 'trick' it into working is to set the Host Application to None. After doing this it builds and tests, but tests fail due to the lack of host. However, this causes the CodeCoverage folder to be created, and if the Host Application is correctly set again, running the build and tests works fine, with a code coverage report produced.
You have to assure the package name is equal in all configurations. Xcode modifies it in case you are using unsupported characters. For instance Xcode replaces "-" with "_".
In all configurations, go to:
Project -> Build Settings -> Product Module
and set the exact same name without spaces.
I'm experiencing failures while running xctool build. My command is as follows:
xctool clean build run-tests -only 'MyApp KIF Tests' -workspace MyApp.xcworkspace/ -scheme 'Debug' -destination "platform=iOS Simulator,name=iPhone 6,OS=latest" -sdk iphonesimulator -reporter junit -reporter plain
The build fails with the following error:
Failed to query the list of test cases in the test bundle: Test bundle not found at: /Users/myusername/Library/Developer/Xcode/DerivedData/MyApp-blbvnisblktossbxwcmnxafkngts/Build/Products/Debug-iphonesimulator/MyApp KIF Tests.xctest
I looked up the MyApp KIF Tests.xctest in the path specified in the error message, and indeed I couldn't find that file. However, this file does exist under a different path:
/Users/myusername/Library/Developer/Xcode/DerivedData/MyApp-blbvnisblktossbxwcmnxafkngts/Build/Products/Debug-iphoneos/MyApp KIF Tests.xctest
The only difference between these two paths is that one is under Debug-iphonesimulator, and the other is under Debug-iphoneos.
I also noticed that under target -> Build Settings -> Build Locations -> Per-configuration Build Products Path -> Debug, the value is $(BUILD_DIR)/$(CONFIGURATION)$(EFFECTIVE_PLATFORM_NAME), which was evaluating to build/OktaVerify.build/Debug-iphoneos. So the first question is: why does $(EFFECTIVE_PLATFORM_NAME) evaluate to Debug-iphoneos? What config do I need to change to make it evaluate to Debug-iphonesimulator so that xctool can find it?
Next, I hardcoded the value of $(CONFIGURATION)$(EFFECTIVE_PLATFORM_NAME) to Debug-iphonesimulator, wishing to see it working. However, I got the same error. My second question is, why does the xctest file appear under Debug-iphoneos folder even the build products path specifies Debug-iphonesimulator? What setting is specifying Debug-iphoneos?
Environment is Xcode 7.2 and xctool version 0.2.8.
I solved the issue by removing "-sdk iphonesimulator" from the xctool command. This way it'll automatically choose the appropriate sdk depending on the target type.