I am trying to instrument the code coverage using bazel and testwell ctc++.
As per testwell we need to just pre-append the ctcwrap utility and it will create MON.sym and MON.dat file respectively.
But the command is not working. There are no files created.
My question is more about knowing the best way to instrument third party coverage tools using bazel.
Related
I followed The Clang manual and used __llvm_profile_write_buffer to collect coverage cprofile data inside the instrumented program.
This works well, but to actually generate a coverage report the recommended way is to use the llvm-cov tool like this:
llvm-cov show ./foo -instr-profile=foo.profdata
This tool needs access to the binary which does not play well with __llvm_profile_write_buffer .
Is there a way to generate a coverage report similar to what llvm-cov does, but inside the process, from the buffer updated by __llvm_profile_write_buffer ?
I guess this would involve accessing the symbol table from within the process, which I think is doable?
Use case : I would like to upload the coverage report from within the process to a remote server without having to execute an external tool.
Thanks for your help,
Antoine
I have gone through the Opencover wiki documentation and tried a lot to figure out what would be the filter criteria for not to include test assembly as part of code coverage. Here is the problem
for eg I have many assemblies starts with sample name like sample.submodule.assembly1.dll, sample.submodule.assembly2.dll, my tests assembly also starts with sample like sample.submodule.tests.dll, here I applied the filter criteria for openCover
1.-filter: "+[sample*]* -[*tests]*"
it didn't work, not generating report file.
-filter: "+[sample*]* -[sample.submodule.tests]*" didn't work, not generating report file,
-filter: "+[sample*]* -[*]*tests*" didn't work, not generating report file too,
can somebody please advise what can be the filter criteria here to exclude all the test files from code coverage
First run OpenCover without any filters.
Now you can look at the XML report produced (or you can use ReportGenerator to turn it into HTML) and identify assemblies/modules that you wish to exclude.
now you can apply filters using the filter switch e.g.
-filter:"+[*]* -[*.tests]* -[*.Tests]*"
NOTE: no space between : and first "
or
"-filter:+[*]* -[*.tests]* -[*.Tests]*"
if you are talking about writing unit test using visual studio nunit adaptor then you have open cover UI visual studio extension available for all such purpose. it is wonderful.
Step 1) Install open cover from latest relese build https://github.com/opencover/opencover/releases
Step 2) Usuall it will install C:\Users\goma1940\AppData\Local\Apps\OpenCover (%localappdata%\Apps\OpenCover)
Steo 3) Install VS Extn from gallery https://visualstudiogallery.msdn.microsoft.com/6950a046-8919-4935-8542-c6f37956f688
Step 4) you have open cover test explorer and open cover coverage result pane as below…
Recently I started using BullseyeCoverage.
I'm going through the steps: compiling with BullseyeCoverage, running some test cases on the binaries created, generating a coverage report.
In the coverage report there are: function coverage, and condition/decision coverage. However, there is no line coverage. I tried to find a way of generating line coverage statistics, unsuccessfully. I thought of using covbr to this end, but, I need something that will cover all of my sources altogether.
Thanks for your help!
Bullseye does not support line coverage (which is also called statement coverage).For reasons, see http://www.bullseye.com/statementCoverage.html
I am working on automating my iOS project's build process using Jenkins and xcodebuild command line tool. I am also able to test the application on simulator using UIAutomation testing scripts (instruments command line utility). But I would like to use other instruments like Time Profiler, Leaks, etc.. along with the automation to automatically test for irregular behavior.
Although I can make a custom template and run it with automation, but the problem is I can not parse the results of these other Instruments. Is there a way, for example, to know that there is a leak in my code, and trigger an email (from jenkins)?
It's possible (I've done it), but it is terribly complicated to do well. You need something to translate the Instruments results plist to something Jenkins can understand. You can use XSLT (I wouldn't recommend it), or write a script in perl/python/etc.
The Jenkins XCode plugin is an excellent tool for automating Xcode from Jenkins.
https://wiki.jenkins-ci.org/display/JENKINS/Xcode+Plugin
That aside, building a CI environment around UIAutomation is doable, but there are a lot of gotchas - the Jenkins user has to have the correct set of permissions to use the dev tools, has to be in the right group, and has to be logged into the UI for the simulator to run correctly from Instruments. Make sure you can get that far before you invest too much time into parsing the output of the Instruments run.
What tools are out there that can perform code coverage analysis at the machine code level rather than the source code level? I'm looking for a possible solution to perform fuzz testing on software that I do not have source code access.
I think the IBM Rational test coverage tools instrument object code.
Assuming you had such a tool, but no access to the source, what exactly
would code coverage mean, other than 100%?
If you didn't have 100% coverage, you'd know you hadn't exercised something.
But you would have no way of knowing what.
For compiled code (not Java), try Valgrind.
Old post... but my two cents.
If you have a bunch of jars and if you know what classes/methods you are using, you can instrument the jars with Emma and run your sample application against those jars.
In my case, I have jars which are actually proprietary components (to generate html code) which our company uses to build it's web-pages. We have a sample application that utilizes these components and a bunch of tests that are run against the sample app. I wrote an ant task to copy the maven dependencies to a directory, instrument them and run the tests against these instrumented jars. This task is invoked from the maven POM and is hence part of the build process.
Also, as part of the build process, we process the emma coverage data to produce a report. This report shows the classes and methods in the jar for which we do not have the source code! Hope this helps.
If you have the number of entry points (public methods), you can test the coverage for that. I don't know any tool for that though.
Otherwise you would have to test the assembly code coverage, and I don't know if it is possible.