Is there any way to measure code coverage with DUnit? Or are there any free tools accomplishing that? What do you use for that? What code coverage do you usually go for?
Jim McKeeth: Thanks for the detailed answer. I am talking about unit testing in the sense of a TDD approach, not only about unit tests after a failure occured. I'm interested in the code coverage I can achieve with some basic prewritten unit tests.
I have just created a new open source project on Google Code with a basic code coverage tool for Delphi 2010. https://sourceforge.net/projects/delphicodecoverage/
Right now it can measure line coverage but I'm planning to add class and method coverage too.
It generates html reports with a summary as well as marked up source showing you what lines are covered (green), which were not (red) and the rest of the lines that didn't have any code generated for them.
Update:
As of version 0.3 of Delphi Code Coverage you can generate XML reports compatible with the Hudson EMMA plugin to display code coverage trends within Hudson.
Update:
Version 0.5 brings bug fixes, increased configurability and cleaned up reports
Update:
Version 1.0 brings support for emma output, coverage of classes and methods and coverage of DLLs and BPLs
I don't know of any free tools. AQTime is almost the defacto standard for profiling Delphi. I haven't used it, but a quick search found Discover for Delphi, which is now open source, but just does code coverage.
Either of these tools should give you an idea of how much code coverage your unit tests are getting.
Are you referring to code coverage from unit tests or stale code? Generally I think only testable code that has a failure should be covered with a unit test (yes I realize that may be starting a holy war, but that is where I stand). So that would be a pretty low percentage.
Now stale code on the other hand is a different story. Stale code is code that doesn't get used. You most likely don't need a tool to tell you this for a lot of your code, just look for the little Blue Dots after you compile in Delphi. Anything without a blue dot is stale. Generally if code is not being used then it should be removed. So that would be 100% code coverage.
There are other scenarios for stale code, like if you have special code to handle if the date ever lands on the 31st of February. The compiler doesn't know it can't happen, so it compiles it in and gives it a blue dot. Now you can write a unit test for that, and test it and it might work, but then you just wasted your time a second time (first for writing the code, second for testing it).
There are tools to track what code paths get used when the program runs, but that is only simi-reliable since not all code paths will get used every time. Like that special code you have to handle leap year, it will only run every four years. So if you take it out then your program will be broken every four years.
I guess I didn't really answer your question about DUnit and Code Coverage, but I think I may have left you with more questions then you started with. What kind of code coverage are you looking for?
UPDATE: If you are taking a TDD approach then no code is written until you write a test for it, so by nature you have 100 test coverage. Of course just because each method is exercised by a test does not mean that its entire range of behaviors is exercised. SmartInspect provides a really easy method to measure which methods are called along with timing, etc. It is a little less then AQTime, but not free. With some more work on your part you can add instrumentation to measure every code path (branches of "if" statements, etc.) Of course you can also just add your own logging to your methods to achieve a coverage report, and that is free (well, expect for your time, which is probably worth more then the tools). If you use JEDI Debug then you can get a call stack too.
TDD really cannot easily be applied retroactively to existing code without a lot of refactoring. Although the newer Delphi IDEs have the ability to generate unit test stubs for each public method, which then gives you 100% coverage of your public methods. What you put in those stubs determines how effective that coverage is.
I use Discover for Delphi and it does the work, for unit testing with DUnit and Functional testing with TestComplete.
Discover can be configured to run from the command line for automation.
As in:
Discover.exe Project.dpr -s -c -m
Discover works great for me. It hardly slows down your application, unlike AQTime. This may not be a problem for you anyway, of course. I think the recent versions of AQTime perform better in this respect.
I've been using Discover" for years, worked excellently up to and including BDS2006 (which was the last pre-XE* version of Delphi i used and still use), but its current opensourced state, it's unclear how to make it work with XE* versions of Delphi. A shame really, because I loved this tool, fast and convenient in almost every way.
So now I'm moving to delphi-code-coverage...
Related
The Delphi Linker strips out any functions that aren't actually used, thus reducing the executable size.
Is there any way to stop the Delphi Linker doing this? e.g. a compiler switch?
To those wondering "why?"...
I am trying to use the delphi-code-coverage tool, but it only reports on code that is actually compiled into the executable. Which makes it not very useful. If I could get Delphi to include all code, I'm hoping I could then get some useful code coverage statistics.
I should mention that I have DUnit tests in a separate project to my application. So even though the code is "unused" in the DUnit project, it is used in the actual application.
See here for more details.
Your code-coverage tool is measuring the wrong thing. It works off the map file instead of the source code, so it will only report on live code instead of on all code in a project. The linker already filters out the dead code, and in a blank unit-test project, all code is dead code. There is no way to tell Delphi to include dead code in an EXE.
Run the code-coverage tool on your application to get a list of functions that needs testing. Then, write code in your unit-test project that mentions all those functions. (It doesn't have to call everything yet, and it certainly doesn't have to test it all. We're just making sure it's linked to the unit-test project.) Now the coverage tool can get an accurate measurement of what's been executed and what hasn't.
I'm trying a long-time to find algorithm PSOLA (Pitch Synchronous Overlap Add) or WSOLA (Waveform Similarity Overlap Add) which are Acoustic or Signal Processing Algorithms.
I found it in c++, but I have no experience in c++ and is difficult to pass it to Pascal. Anyone have this code in Pascal or know where to find it to copy?
Something like this example that is in c++
http://sourceforge.net/projects/mffmtimescale/files/v3%20stable/v3.9/WSOLA.v.3.9.zip/download?use_mirror=ufpr
Try the SoundTouch DLL, it comes with a Delphi import unit, so you can use the DLL directly. It should not be too hard to compile it. Just download the free VC++ Express from the MS download site and compile it with that or ask someone with VC++ to compile it for you.
FWIW, who knows, with a few modifications, it might also compile with C++Builder.
About one decade ago, I've used praat in FSeqEdit (Delphi program) to do these type of calculations, but I think the same approach would still work fine today.
I wrote some praat-scripts and execute them via praatcons.exe (console version of praat). You can download the console version from this page:
http://www.fon.hum.uva.nl/praat/download_win.html
That works pretty good.
I usually take this approach:
I manually check what type of conversions and calculations need to be done via the GUI version of praat. Once I find what I need, I create a script for it, and run that with the console version.
Praat is very powerful, so if you didn't know about it yet, make sure to check it out.
There's a page that shows how to work with PSOLA resynthesis here:
http://www.ling.ohio-state.edu/~kyoon/praat-tut/praat-tut2.html
Let me know if you want to see some example code on how to integrate it into your Delphi application (it's pretty straightforward actually).
I want to enforce coding standards for our Delphi codebase.
A few colleagues have suggested Code Healer and Pascal Analyzer. I've had a look at these tools and they aren't suitable.
I was hoping to be able to do the same thing that CheckStyle for Java or StyleCop for C# can do
Some newer editions of Delphi offer Audits and Metrics in the Model view, which can also be configured to set allowed limits. They do not run from command line for build integration afaik so I found them not very helpful.
I know the highly customizable Java (and .Net tools) like PMD, FindBugs and CheckStyle which generate XML or HTML statistic reports, and also integrate very well with build tools (Ant, Maven, Hudosn) - but for Delphi nothing comparable has crossed my road so far.
It seems those 2 are the most used. You can also try:
http://jedicodeformat.sourceforge.net/
The best one is Pascal Analyzer (PAL) by Peganza, which you said you tried and found unsuitable, but did not say why. I will say a bit in its favor: It's Commercial, inexpensive, and so worth it. They recently released version 5, and if version 5 doesn't do what you want, you should tell them what you want, because they have always answered my requests whenever I have mentioned a feature I wish the product would add.
We use it instead of the high-end SKUs of Delphi's metrics because it costs less and does more than the built-in $3000 stuff. I think it costs about $160 us.
I am a happy customer. Here is a sample of some of the metric areas that I like:
convention compliance - class names that don't start with T, exception types don't start with E, class fields not in private, identifiers with goofy names, class visibility confusion or bad order, local identifier/unit outer scope identifier clashes. Inconsistent case, Many many many more!
The weakness is that the output is plain text in a "TMemo" control. Of course, I have found a lot of ways to take that output and write my own small sort/filter utilities to mine even more useful stuff from the reports. A powerful tool that you won't be able to live without once you try it.
I realize you said in your answer that you tried that already, but if it's not what you want, it's already the best LINT like tool for Pascal that currently exists.
If you're into writing your own style checking, you can write a .exe in Delphi to look for bad things being committed. Call that in a pre-commit hook into your repository.
You can examine the differences of a checkin by using SVNLOOK.
ex:
excerpt from pre-commit.bat
SVNLOOK diff -t "%2" "%1" | MyCustomFilter.exe
IF %ERRORLEVEL% == 0 GOTO EOJ
EXIT 1
:EOJ
EXIT 0
We're trying to untangle a hairball of 100's of units, removing some.
It would be helpful if there was tool that would show us what units were explicitly using unit X.
Penganza doesn't seem to have a report that does that. (Although it has lots of other useful reports.)
Can anyone suggest a tool or strategy for doing this, other than just hiding unit x and then hitting F9 ... repeatedly?
MMX (Model Maker Code Explorer) has a nice unit dependency analyzer (it is especially good at detecting cycles).
For more details, see this answer.
--jeroen
From a similar question here
You might want to take a look at at
CnPack.
CnPack includes a Uses cleaner
wizard wich hasn't failed me yet.
GExperts can show Project Dependencies.
Peganza Pascal Analyzer can do the work. I haven't worked with it much, but a former dev here wrote a system that uses PAL to do the analysis, then dumped the results into a database, and then there's a browser app that lets you enter a unit name and it returns the list of units affected, whether they would need to be rebuilt if the unit changed, or if the interface changed. We use lots of BPLs so you can sometimes change a unit and you don't have to re-build other binaries that use your unit, unless the interface changed. This saves us lots of work (hundreds of BPLs and EXEs).
Chris
Headway Software's Structure 101g (and Restructure 101g) can do that really well, with the Delphi plugin.
Disclaimer: I wrote the flavors to analyze Delphi. I use them professionally, helping clients.
We've just released a freeware utility that does exactly what you need plus quite a bit more. It's called the Delphi Unit Dependency Scanner (DUDs) and you can download it here: http://www.easy-ip.net/delphi-unit-dependency-scanner.html
Sorry it's a bit late!
I was going to mention Icarus, but when I googled them I got this stack overflow answer, which you might want to check out.
Then again, sometimes I just like to delete my whole Unit Output Directory, then count my new DCU's, and that works too.
The reason you may like Icarus and not GExperts is that it doesn't rely on you to have properly maintained the uses statements in your project file.
A newcomer in this field is the Delphi Plugin for Sonar. It does not list unit dependencies but can find unused files and "dead" code (and more).
Implemented features:
Counting lines of code, statements, number of files
Counting number of classes, number of packages, methods, accessors
Counting number of public API (methods, classes and fields)
Counting comments ratio, comment lines (including blank lines)
CPD (code duplication, how many lines, block and in how many files)
Code Complexity (per method, class, file; complexity distribution
over methods, classes and files)
LCOM4 and RFC
Code colorization
Unit tests reports
Assembler syntax in grammar
Include statement
Parsing preprocessor statements
Rules
Code coverage reports
Source code highlight for unit tests
“Dead” code recognition
Unused files recognition
Our team had been using Delphi 6 for many years, then switched to Delphi 2006 years ago. With both versions we have the following problem: frequently the compiler complains about a unit which is supposedly used recursively. This unit is a 40k LOC unit which is at the core of a project with almost 1 million LOC (third party included).
The error message is incorrect: a full build on the project always works. Unfortunately, the error message does not tell us where the supposed circular reference is, just the name of that unit. Sometimes it even happens that valid error messages are listed 2-4 times until that circular reference problem is "found". Clearly the compiler is running in a circle here. Because of the size of that project it is hard to find the problem manually. Therefore I made a tool which proves that there really is no circular reference (the tool creates a directed dependency graph of the units and determines the coherence components in that graph - there are none except if I deliberately put some in).
This is not only affecting F9 compilation but also code completion / insight which is not working most of the time. Sometimes it works when I press ctrl-space a second time...
Any ideas how we can isolate or even fix the problem? Note that it will be very hard to split the 40k LOC unit into smaller ones because it contains about 15 large classes which depend on each other in the interface section (I know it's bad but should work anyway).
Update
We are constantly refactoring but this is one tough unit to refactor because everything depends on everything, almost. Have been trying to get around it via interfaces but we are talking about some classes with 100s of methods and properties. And it would be slower.
Upgrading to D2009 may be an option down the road but right now we're stuck with D2006 (the unicode stuff and the price tag are two of the stoppers here). Question is anyway if it would help since the problem is in there since D6 at least.
About trimming the uses clauses, we have been doing this frequently with Icarus. But that did not help so far. We are down to 90 custom units in the interface section now. However, with a true circular reference the problem could be in any unit. Also tried to add all units to the dpr.
The project shares a lot of code with other projects, and there are some IFDEFs. However, the defines are not set up in project options but via a common include file. Therefore all modules should see the same defines. Also, the problem reoccurs shortly after a full rebuild without switching to another project.
I will probably be downvoted for this. In D2005 I had a 10k loc unit (datamodule) that flat out stopped compiling. Had to separate out some datasets/code to another datamodule. That 10k unit was and is a mess. You really should consider refactoring out some code to other units. My module has since D2005 / separation grown even worse, but it still compiles in D2007. So my answer is a) refactor and b) upgrade to D2009.
It seems clear that this is due to a slight difference between the background compiler and the real thing. You could look around (QualityCentral) what's known on that topic.
Also, since you didn't explicitly state this, you should remove unnecessary units and move uses statements down to implementation if possible. Maybe your tool could help with this.
And just to be sure you should check the unit aliases and Path settings.
You write that a full build does always succeed, but shortly after the incremental build fails with this error. Assuming that you experience this in the IDE, have you tried to use the command line compiler dcc32 to do incremental builds?
If you don't feed it the "-Q" switch (which probably most Makefiles or scripts for command line builds do) it will output a lot of information what files it compiles in what order. You could either try to do an incremental build after the error appeared in the IDE, or you could keep a command line open next to the IDE and Alt+Tab to it for compilation, skipping compilation in the IDE completely.
I simply assume you have a way to build using dcc32, one way or another - with the size of your project I can't imagine otherwise.
We regularly fall in similar problems, and we never managed (or bothered long enough) to find the precise cause. There seems to be a problem in the order which Delphi chooses to compile the units when hitting Ctrl-F9, which is incompatible with the actual dependency order of the units.
Have you tried deleting "MyBigFatUnit.dcu" before hitting Ctrl-F9?
Have you tried to re-order the declaration of your units in your dpr/dpk files, so that units appear in a correct compilation order? (i.e.: if unit B depends on unit A, unit A should appear first in the dpr/dpk)
Do you have any other projects that use part of the same codebase? If you compile one of them under different compiler settings or IFDEFs, it might change certain things in some of the DCUs which would lead to a circular dependency. A full build rebuilds all DCUs and then the problem goes away.
Try Icarus (free) from Peganza. If that does not tell you what the problem is, try their Pascal Analyzer.
We have this problem as well, also with a fairly large codebase.
We are currently using D2009, but have had this problem with all previous versions of Delphi.
It most frequently happens immediately after doing an update from source control, so I suspect there is some timestamp issue within the Delphi build process.
In our case, if Ctrl-F9 fails and reports the circular reference, a second Ctrl-F9 will generally work
A way I have been told to deal with this is to open another arbitrary file in the project, change that file, save it, and then try running the incremental compile again. Surprisingly enough, this usually works.
We have a 4 MLOC project where this comes up from time to time and this "solution" works for me.
I've fought this before, in my experience the error is quasi-legitimate. It's been a quite a while (the error has nothing to do with the version) but my memory of the situation is that it involves a loop in which part of the loop is in the implementation.
Unit A uses B in the implementation. Unit B uses A in the interface. If you compile B first it calls for A but since the call for B is in the implementation it succeeds. If you compile A first it calls for B, B turns around and calls for A in the interface, boom. Such loops are only safe if both cross references are in the implementation.
The solution is to design things so there is a minimum of stuff used in the interface and to make certain there's nothing resembling a loop in those units. So long as you keep your type definitions separate from units with code this is pretty easy to do.
The error coming and going depending on what you are doing is a hallmark of this issue as it comes down to how you enter the loop. When you do a full build the order is consistent and you either get it 100% or 0%, it's not random.