I have gone through the Opencover wiki documentation and tried a lot to figure out what would be the filter criteria for not to include test assembly as part of code coverage. Here is the problem
for eg I have many assemblies starts with sample name like sample.submodule.assembly1.dll, sample.submodule.assembly2.dll, my tests assembly also starts with sample like sample.submodule.tests.dll, here I applied the filter criteria for openCover
1.-filter: "+[sample*]* -[*tests]*"
it didn't work, not generating report file.
-filter: "+[sample*]* -[sample.submodule.tests]*" didn't work, not generating report file,
-filter: "+[sample*]* -[*]*tests*" didn't work, not generating report file too,
can somebody please advise what can be the filter criteria here to exclude all the test files from code coverage
First run OpenCover without any filters.
Now you can look at the XML report produced (or you can use ReportGenerator to turn it into HTML) and identify assemblies/modules that you wish to exclude.
now you can apply filters using the filter switch e.g.
-filter:"+[*]* -[*.tests]* -[*.Tests]*"
NOTE: no space between : and first "
or
"-filter:+[*]* -[*.tests]* -[*.Tests]*"
if you are talking about writing unit test using visual studio nunit adaptor then you have open cover UI visual studio extension available for all such purpose. it is wonderful.
Step 1) Install open cover from latest relese build https://github.com/opencover/opencover/releases
Step 2) Usuall it will install C:\Users\goma1940\AppData\Local\Apps\OpenCover (%localappdata%\Apps\OpenCover)
Steo 3) Install VS Extn from gallery https://visualstudiogallery.msdn.microsoft.com/6950a046-8919-4935-8542-c6f37956f688
Step 4) you have open cover test explorer and open cover coverage result pane as below…
Related
I've installed the extension Fine Code Coverage (Version 1.1.191) in Visual Studio 2022 Community Edition. I got a few xUnit test-projects in my solution. I'm getting the coverage statistics and the report by using the following settings:
run the tests from the visual studio Test Explorer (not the xunit testrunner)
Test => Options => Fine Code Coverage => RunMsCodeCoverage:
selection of "No" or "IfInRunSettings" produces the mentioned output
selection of "Yes" and not providing runsettings fails to produce any output and the Output window for FCC gives the message "No cobertura files for ms code coverage"
The README of the project states for the usage of MS Code Coverage with FCC:
Firstly you need to change the RunMsCodeCoverage option from No.
Ms code coverage requires a runsettings file that is configured appropriately for code coverage. This requires that you have the ms code coverage package and have pointed to it with the TestAdaptersPaths element as well as specifying the ms data collector.
...
FCC does not require you to do this. If you do not provide a runsettings and RunMsCodeCoverage is Yes then FCC will generate one. If RunMsCodeCoverage is IfInRunSettings then if the project has runsettings that includes the ms data collector element configured correctly then FCC will process the collected results.
As pointed out in my settings above I do not get a result if "RunMsCodeCoverage" is set to "Yes" without providing runsettings. On the other hand setting "IfInRunSettings" without providing runsettings I'm getting the desired output (are the runsettings generated automatically in this case?). So I find the description in the README in contrast to my results. Could it be that the documentation in this point is referring to the enterprise edition of VS? I'd like to be able to understand the different setup options/requirements so anyone who can shed a light on these is welcome.
I'm setting up automated regression testing for an FPGA project, almost exactly as described here:
Continuous integration of complex reconfigurable systems
Now I want to get test results (from VHDL REPORT statements in ModelSim simulation) to appear in Jenkins testing reports. My understanding is that Jenkins only natively supports jUnit format, and I looked for plugins supporting non-XML formats but didn't see any.
Generating valid XML from VHDL REPORT statements would be very difficult, since the simulation may immediately terminate depending on the severity. Which means that the closing tags would have to be duplicated in every single possible exit path for every single test -- not the most maintainable approach.
So, do you know of any straightforward way to convert plain text into jUnit (or another format, if supported by Jenkins)? If something doesn't already exist, is there an advantage to writing a Jenkins plugin vs just throwing together a perl script? Any other suggestions?
You should take a look at the XUnit Plugin. The Plugin reads test results from a number of tools, and seems adaptable to custom formats. From the documentation the plugin is able to read not only xml, but also csv and txt. For custom format you need to specify some style sheet for the transformation, I am not quite sure if this will go all the way for you. But even if it does not, I suppose the plugin should be easy to extend for your own format.
Old post but today there is a unit testing framework for VHDL that we've developed. It solves the problem by generating a report on the JUnit format. It also handles the case when the simulation stops due to a severe error. The tool is free and open source and can be found at https://github.com/LarsAsplund/vunit
I found that this custom tool is used for generation of .cs files from SpecFlow's .feature file.
Is there a way to use this tool outside of VS?
I would like to call this tool from console for specific .feature file when building my project with NAnt.
Regards,
Vajda Vladimir
You sure can! That is if you mean: "Can I generate unit test code from my .feature-files without using Visual Studio"
I've written about that here: http://www.marcusoft.net/2010/12/specflowexe-and-mstest.html but in short you can use the SpecFlow.exe with the "generateAll" switch, and it will inspect your project settings and generate the appropiate unit tests for you (in my example it's MsTest but it can be any of the supported testing frameworks).
The SpecFlow.exe's help we get the following concise help:
Generate tests from all feature files in a project
usage: specflow generateall projectFile [/force] [/verbose]
projectFile Visual Studio Project File containing features
So for a project called Specs.csproj it would be:
"%ProgramFiles(x86)%\TechTalk\SpecFlow\SpecFlow.exe" generateAll Specs\Specs.csproj /force /verbose
You can read more in my blog post - but this is basically it.
What tools are out there that can perform code coverage analysis at the machine code level rather than the source code level? I'm looking for a possible solution to perform fuzz testing on software that I do not have source code access.
I think the IBM Rational test coverage tools instrument object code.
Assuming you had such a tool, but no access to the source, what exactly
would code coverage mean, other than 100%?
If you didn't have 100% coverage, you'd know you hadn't exercised something.
But you would have no way of knowing what.
For compiled code (not Java), try Valgrind.
Old post... but my two cents.
If you have a bunch of jars and if you know what classes/methods you are using, you can instrument the jars with Emma and run your sample application against those jars.
In my case, I have jars which are actually proprietary components (to generate html code) which our company uses to build it's web-pages. We have a sample application that utilizes these components and a bunch of tests that are run against the sample app. I wrote an ant task to copy the maven dependencies to a directory, instrument them and run the tests against these instrumented jars. This task is invoked from the maven POM and is hence part of the build process.
Also, as part of the build process, we process the emma coverage data to produce a report. This report shows the classes and methods in the jar for which we do not have the source code! Hope this helps.
If you have the number of entry points (public methods), you can test the coverage for that. I don't know any tool for that though.
Otherwise you would have to test the assembly code coverage, and I don't know if it is possible.
I haven't yet installed my license of NCover 3, and am still running 1.5.8 on my build server. I am trying to exclude full assemblies and specific classes that I don't want included in the report, because they are artificially lowering the coverage results.
In NCoverExplorer, I was playing around with the options because there is a coverage exclusions section in the Options tab where you can specify full namespaces. I've entered the fully qualified classes, and for some reason, only a handful of them get excluded, and I cannot figure out why. For example, when I add System.ComponentModel.Composition to the list, it never gets excluded!
Is this just a bug in 1.5.8 that I have to live with for now, since it is a beta and also no longer supported? Although I do have a new license for the server, I'd like to be able to do some coverage at home on my personal computer.
I found a really great article on using a CoverageExcludeAttribute to make NCover automatically skip those classes / methods that are marked with this attribute. Is this the best option?
Did you try using regular expressions to include/exclude the assemblies -
//ias AuctionSniper([.\w]*?)(?<!Tests)
includes all assemblies that begin with AuctionSniper but don't end with Tests e.g. AuctionSniper.Main.exe
You can specify multiple patterns separated by semicolons.
or //ias .*vendorsupplied.*;.*tests
This works with NCover 3 - you can give it a try if it works for the free/community edition.
One way to get part way there is to also use the "assemblies to include in coverage" option. This allowed me to ignore System.ComponentModel.Composition. However, exlcuding most of the other classes and namespaces still doesn't work. It's a little odd how when I exclude an entire class, it only excludes the methods, but not any of the contained classes.