Using a gem like SimpleCov I can see my test coverage on a line-by-line basis for all the files I specify. Is there a way to see test coverage on a method-by-method basis? For example, if my tests engaged a method at all, that method would be considered 'covered.'
if my tests engaged a method at all, that method would be considered
'covered.'
That's a very arbitrary definition of covered methods. What if it has 100 lines but it returns on first guard clause?
Because of this (I imagine) this is not feasible to implement - there would be a problem to even agree what is a covered method.
Such metric would lie (reporting bigger coverage). If you assumed that method is covered if all lines are covered - it would lie too, reporting smaller coverage.
Note: line coverage is also lying a bit (check ternary operators for example), but it's the smallest liar...
Related
I am new to using gcov and gcovr and I wanted to get the statement coverage of a given function. It is coded in C, compiled with minGW and called from Matlab (which I use to later process the coverage information given by gcov).
I am executing the code in two different ways: for the first one I am using Simulink, in which the function inputs are given by the outputs of other functions that encompass the dynamic process I modelled on Simulink. For the second one, I am using the editor on Matlab and defining directly the inputs to the function.
Because the Simulink - executed code depends on secondary functions whose output I cannot control (contrary to the second way), I expected the statement coverage of the first execution to be worse than the second but to have the same number of statement lines (since it is exactly the same code). However, I found that:
For some function callers inside the function, the second method counts the few lines of the caller (like the first line and the following lines when the input and output variables are too long to fit in a single line), adding up statements that in reality don't exist.
The first method doesn't take into account some variable definitions at the beginning of the code, not counting them as line statements (for instance, setting input variables to 0).
Has anybody also encountered this discrepancy when getting the statement coverage of the same function? Do you know why this may be?
Thank you very much in advance!
What is the benefit of using paragraphs and sections for executing pieces of code, instead of using a subprogram instead? As far as I can see paragraphs and sections are dangerous because they have an non intuitive control flow, its easy to fall through and execute stuff you never meant to execute, and there is no variable (item) scoping, therefore it encourages a style of programming where everything is visible to everything else. Its a slippery soup.
I read a lot, but I could not find anything related to the comparative benefit of paragraphs/sections vs a subprogram. I also asked online some people in some COBOL forum, but their answers were along the lines of "is this a joke" or "go learn programming"(!!!).
I do not wish to engage in a discussion of stylistic preferences, everyone writes the way that their brain works, I only want to know, is there any benefit to using paragraphs/sections for flow control? As in, are there any COBOL operations that can be done only by using paragraphs/sections? Or is it just a remnant of an early way of thinking about code?
Because no other language I know of has mimicked that, so it either has some mechanical concrete essential reason to exist in COBOL, or it is a stylistic preference of the COBOL people. Can someone illuminate me on what is happening?
These are multiple questions... the two most important ones:
Are there any COBOL operations that can be done only by using paragraphs/sections?
Yes. A likely not complete list:
USE statements in DECLARATIVES can only apply to a paragraph or a section. These are used for handling file errors and exceptions. Not all compilers support this COBOL standard feature in full.
Segmentation (primary: a program that is only partially loaded in memory) is only possible with sections; but that is to be considered a "legacy feature" (at least I don't know of people actually using it this way explicitly); see the comment of Gilbert Le Blanc for more details on this
fall-through, many other languages have this feature with a kind of a switch statement (COBOL's EVALUATE, which is not the same as a common switch but can be used similar has an explicit break and no fall-through)
GO TO DEPENDING ON (could be recoded to achieve something similar with EVALUATE and then either PERFORM, if the paragraphs are expected to fall-through, which is not uncommon, then that creates a lot of extra code)
GO TO in general and especially nice - the old obsolete ALTER statement
PERFORM statement, format 1 "out-of-line"
file state is only shared between programs when you define it as EXTERNAL, and you often want to have a file state being limited to a single program
up to COBOL85: EXIT statement (plain without anything else, actually doing nothing else then a CONTINUE would)
What is the benefit of using paragraphs and sections for executing pieces of code, instead of using a subprogram instead?
shared data (I guess you know of programs with static data or otherwise (module)global data that is shared between functions/methods and also different source code files)
much less overhead than a CALL is
consistency:
you know what's in your code, you don't know what another program does (or at least: you cannot guarantee that it will do the same some years later exactly the same)
easier to extend/change: adding another variable (and also removing part of it, change its size) to a CALL USING means that you also have to adjust the called program - and all programs that call this, even when you place the complete definition in a copybook, which is very reasonable, this means you have to recompile all programs that use this
a section/paragraph is always available (it is already loaded when the program runs), a CALLed program may not be available or lead to an exception, for example because it cannot be loaded as its parameters have changed
less stuff to code
Note: While not all compilers support this you can work around nearly all of the runtime overhead and consistency issue when you use one source files with multiple program definitions in (possibly nested) and using a static call-convention. This likely gives you the "modern" view you aim for with scope-limitation of variables, within the programs either persistent (like local-static) when defined in WORKING-STORAGE or always passed when in LINKAGE or "local-temporary" when in LOCAL-STORAGE.
Should all code of an application be in one program?
[I've added this one to not lead to bad assumptions] Of course not!
Using sub-programs and also user-defined functions (possibly even nested providing the option for "scoped" and "shared" data) is a good thing where you have a "feature boundary" (for example: access to data, user-interface, ...) or with "modern" COBOL where you have a "language boundary" (for example: direct CALLs of C/Java/whatever), but it isn't "jut for limiting a counter to a section" - in this case: either define a variable which state is not guaranteed to be available after any PERFORM or define one for the section/paragraph; in both cases it would be reasonable to use a prefix telling you this.
Using that "separate by boundary" approach also takes care of the "bad habit of everything being seen by everyone" issue (which is in any case only true for "all sections/paragraphs in the same program).
Personal side note: I would only use paragraphs where it is a "shop/team rule" (it is better to stay consistent then to do things different "just because they are better" [still providing an option to possibly change the common rule]) or for GO TO, which I normally not use.
SECTIONs and EXIT SECTION + EXIT PERFORM [CYCLE] (and very rarely GOBACK/EXIT PROGRAM) make paragraphs nearly unnecessary.
very short answer. subroutines!!
Subroutines execute in the context of the calling routine. Two virtues: no parameter passing, easy to create. In some languages, subroutines are private to (and are part of) the calling (invoking) routine (see various dialects of BASIC).
direct answer: Section and Paragraph support a different way of thinking about programming. Higher performance than call subprogram. Supports overlays. The "fall thru" aspect can be quite useful, a feature rather than a vice. They may be necessary depending on what you are doing with a specific COBOL compiler.
See also PL/1, BAL/360, architecture 360/370/...
As a veteran Cobol dinosaur, I would say asking about the benefit is not the right question. I used paragraph (or section) differently than a subprogram. The right question in my opinion is when to use them logically. If I can make an analogy, if you have a Dog java class, you will write Dog-appropriate methods within it. If there's a cat involved, you may need a helper class. In this case the helper class is the subprogram. Though, you can instead code the helper class methods inside the Dog class, but that will be bad coding.
In any other language I would recommend putting self contained functions into subroutines.
However in COBOL not so much. If the code is very likely to be used in other programs then a subroutine is a good idea. Otherwise not!
The reason being the total lack of any checks on the number type or existence of passed parameters at compile time. Small errors in call statements lead to program crashes at run time. Limiting the use of sub-routines and carefully checking the calling code for errors makes for a more reliable program.
Using paragraphs any type mismatch will be flagged at compile time, or, an automatic conversion will occur.
Background
I have a test suite and I need to know the coverage of the project.
I have played around with mix test --cover but I find the native erlang's coverage analysis tool to be insufficient at best.
The native coverage tool doesn't tell you about branch coverage nor function coverage. It's only metric seems to be relevant lines which I have no idea how they calculate. For all I know, this is just the most basic form of test coverage: see if a given text line was executed.
What have you tried?
I have tried Coverex but the result was disastrous. Not only does it suffer from the same issues that the native tool does, it also seems not produce correct results as it counts imported modules as untested.
Or maybe it is doing a great job and my code is poorly tested, but I can't know for sure because it doesn't tell me how it is evaluating my code. Have 40% coverage in a file? What am I missing? I can't know, the tool wont tell me.
I am now using ExCoveralls. It is considerably better than the previous options, it allows me to easily configure which folders I want to ignore, but it uses the native coverage tool, so it suffers pretty much from the same issues.
What do you want?
I was hoping to find something among the lines of Istanbul, or in this case nyc:
https://github.com/istanbuljs/nyc
It's test coverage analysis tells me everything I need to know, metrics and all:
Branches, Functions, Lines, Statements, everything you need to know is there.
Questions
Is there any tool that uses Istanbul for code coverage metrics with Elixir instead of the native erlang one?
If not, is there a way to configure the native coverage tool to give me more information?
Which metrics does the native coverage tool uses ?
The native coverage tool inserts "bump" calls on every line of the source code, recording module, function, arity, clause number and line number:
bump_call(Vars, Line) ->
A = erl_anno:new(0),
{call,A,{remote,A,{atom,A,ets},{atom,A,update_counter}},
[{atom,A,?COVER_TABLE},
{tuple,A,[{atom,A,?BUMP_REC_NAME},
{atom,A,Vars#vars.module},
{atom,A,Vars#vars.function},
{integer,A,Vars#vars.arity},
{integer,A,Vars#vars.clause},
{integer,A,Line}]},
{integer,A,1}]}.
(from cover.erl)
The code inserted by the function above is:
ets:update_counter(?COVER_TABLE,
{?BUMP_REC_NAME, Module, Function, Arity, Clause, Line}, 1)
That is, increment the entry for the given module / function / line in question by 1. After all tests have finished, cover will use the data in this table and show how many times a given line was executed.
As mentioned in the cover documentation, you can get coverage for modules, functions, function clauses and lines. It looks like ExCoveralls only uses line coverage in its reports, but there is no reason it couldn't do all four types of coverage.
Branch coverage is not supported. Seems like supporting branch coverage would require expanding the "bump" record and updating cover.erl to record that information. Until someone does that, coverage information is only accurate when branches appear on different lines. For example:
case always_false() of
true ->
%% this line shows up as not covered
do_something();
false ->
ok
end.
%% this line shows up as covered, even though do_something is never called
always_false() andalso do_something()
To add to #legoscia excellent response, I also want to clarify why cover does not do statements evaluation. According to this discussion in the official forum:
https://elixirforum.com/t/code-coverage-tools-for-elixir/18102/10
The code is first compiled into erlang and then from erlang into a modified binary file (but no .beam file is created) that is automatically loaded into memory and executed.
Because of the way erlang code works, a single statement can have several instructions:
and single line can result in multiple VM “statements”, for example:
Integer.to_string(a + 1)
Will result with 2 instructions:
{line,[{location,"lib/tasks.ex",6}]}.
{gc_bif,'+',{f,0},1,[{x,0},{integer,1}],{x,0}}.
{line,[{location,"lib/tasks.ex",6}]}.
{call_ext_only,1,{extfunc,erlang,integer_to_binary,1}}.
Therefore it is rather tricky for an automatic analysis tool to provide statement coverage because it is hard to match statements to instructions, especially as in theory a compiler is free to reorder commands as it pleases as long as the result is the same.
I am using the python API of the Z3 solver to search for optimized schedules. It works pretty well apart from that it sometimes is very slow even for small graphs (sometimes its very quick though). The reason for that is probably that the constraints of my scheduling problem are quite complex.
I am trying to speed things up and stumbled on some articles about incremental solving.
As far I understood, you can use incremental solving to prune some of the search space by only applying parts of the constraints.
So my original code was looking like that:
for constraint in constraint_set:
self._opt_solver.add(constraint)
self._opt_solver.minimize(some_objective)
self._opt_solver.check()
model = self._opt_solver.mode()
I changed it now to the following:
for constraint in constraint_set:
self._opt_solver.push(constraint)
self._opt_solver.check()
self._opt_solver.minimize(some_objective)
self._opt_solver.check()
model = self._opt_solver.mode()
I basically substituted the "add" command by the "push" command and added a check() after each push.
So first of all: is my general approach correct?
Furthermore, I get an exception which I can't get rid of:
self._opt_solver.push(constraint) TypeError: push() takes 1 positional argument but 2 were given
Can anyone give me a hint, what I am doing wrong. Also is there maybe a z3py tutorial that explains (with some examples maybe) how to use incremental solving with the python api.
My last question is: Is that at all the right way of minimizing the execution time of the solver or is there a different/better way?
The function push doesn't take an argument. It creates a "backtracking" point that you can pop to later on. See here: http://z3prover.github.io/api/html/classz3py_1_1_solver.html#abc4ae989afee7ad164844640537107d9
So, it seems push isn't really what you want/need here at all. You should simply add your constraints one-by-one and call check. However, I very much doubt checking after each addition is going to speed anything up significantly. The optimizing solver (as opposed to the regular one), in particular, usually solves everything from scratch. (See the relevant discussion here: https://github.com/Z3Prover/z3/issues/1577)
Regarding incremental: The python API is automatically "incremental." Incremental simply means the ability to call the command check() multiple times, without the solver forgetting what it has seen before. (i.e., call check, assert more facts, call check again; the second check will take into account all the assertions from the very beginning.) You shouldn't make any assumptions regarding this will give you speed over calling check just once at the very end: It entirely depends on the heuristics and the decision procedures involved, which is dependent on the problem at hand.
I'd like to improve specs for a convenience method that returns an array of users whose purchases are due tomorrow and will be paid using a certain payment method (ignoring other payment methods).
The specs are something along the lines of:
it "doesn't return users without purchases being due tomorrow" do
# create user and associated records that don't fit the time condition
expect(subject).to eq([])
end
context "with users who have purchases due tomorrow" do
it "returns users with $CERTAIN_PAYMENT_METHOD" do
# create different users with different payment methods,
# all matching the time condition.
expect(subject).to eq([user_1, user_2])
end
it "doesn't return users without $CERTAIN_PAYMENT_METHOD" do
# create user with credit card,
# matching the time condition.
expect(subject).to eq([])
end
end
I see three possible approaches here:
Approach used above: Set up arbitrary combinations of records, and expect things specific to a single condition, always asserting the whole array. The setup can be long and repetitive.
Set up arbitrary combinations, and expect things to be included/excluded in the returned array. These "soft expectations" are great to read but can be very error-prone.
Set up (nearly) all combinations of records, and have a single expectation. This creates no overlap but also leaves developers in the dark as to why this method returns this specific array.
All approaches have their drawbacks, and I'm wondering if there's a best practice how to test methods like these?
Deciding between Option 1 and 2 can be tricky, but Option 3 is without a doubt the worst option. Unit Tests serve two purposes* - they drive clean code design and they document what the code does for the developers of the future. As you have correctly pointed out, having "one huge test" leaves developers in the dark as to why the method behaves as it does, and therefore destroys half of the benefit of the tests. Furthermore, these kinds of tests are not the sort of tests that drive out clean code design, so you are probably loosing that benefit too.
Whether Option 1 or 2 is better depends on your situation. Repetitive set up code is not necessarily a problem (research DAMP vs DRY with respect to TDD) and if it is a problem it could be improved by extracting common set up code so it can be shared between test cases (although this can be a smell if taken too far). In any case, I don't see how Option 2 solves the issue of repetitive set up code - the only difference between the two options is how you make the assertions.
The issue of "stock" vs "weak" assertions is a common theme (research strict mocks vs loose mocks) and is the main difference between Option 1 and 2. The problem with Option 1 is that making 1 change to production code can result in having to change the assertions for every single test case. For this reason I would advise having a couple of tests that assert on the whole array (to test ordering and completeness etc) but for most tests to be as specific as possible and only test include/excludes. This allows the test to clearly document which types of set ups result in which output values, rather than every test having a long list of assertions, only one of which is interesting to the current test case.
The guiding principal is, don't test the same thing twice. If, by asserting every element in every test, you feel like you are testing the same thing twice then it's probably not the right way to go.
* OK, sometimes they catch bugs too, but on their own they don't tell you that your system works and is bug free. That is the purpose of other forms of testing that test your entire system. Green unit tests do not give you enough confidence to release to production.