I'm just starting out with Coveralls and testing coverage and I see info such as this. What do the arrays mean? How should I interpret these? I can't seem to find anywhere in the docs or online regarding the meaning of these.
First index is block number and the second is branch number assigned by GCC
Basing my theory on lcov documentation for branch coverage information
BRDA: line number,block number,branch number,taken
Block number and branch number are gcc internal IDs for the branch.
Taken is either '-' if the basic block containing the branch was never
executed or a number indicating how often that branch was taken.
Background
I have a test suite and I need to know the coverage of the project.
I have played around with mix test --cover but I find the native erlang's coverage analysis tool to be insufficient at best.
The native coverage tool doesn't tell you about branch coverage nor function coverage. It's only metric seems to be relevant lines which I have no idea how they calculate. For all I know, this is just the most basic form of test coverage: see if a given text line was executed.
What have you tried?
I have tried Coverex but the result was disastrous. Not only does it suffer from the same issues that the native tool does, it also seems not produce correct results as it counts imported modules as untested.
Or maybe it is doing a great job and my code is poorly tested, but I can't know for sure because it doesn't tell me how it is evaluating my code. Have 40% coverage in a file? What am I missing? I can't know, the tool wont tell me.
I am now using ExCoveralls. It is considerably better than the previous options, it allows me to easily configure which folders I want to ignore, but it uses the native coverage tool, so it suffers pretty much from the same issues.
What do you want?
I was hoping to find something among the lines of Istanbul, or in this case nyc:
https://github.com/istanbuljs/nyc
It's test coverage analysis tells me everything I need to know, metrics and all:
Branches, Functions, Lines, Statements, everything you need to know is there.
Questions
Is there any tool that uses Istanbul for code coverage metrics with Elixir instead of the native erlang one?
If not, is there a way to configure the native coverage tool to give me more information?
Which metrics does the native coverage tool uses ?
The native coverage tool inserts "bump" calls on every line of the source code, recording module, function, arity, clause number and line number:
bump_call(Vars, Line) ->
A = erl_anno:new(0),
{call,A,{remote,A,{atom,A,ets},{atom,A,update_counter}},
[{atom,A,?COVER_TABLE},
{tuple,A,[{atom,A,?BUMP_REC_NAME},
{atom,A,Vars#vars.module},
{atom,A,Vars#vars.function},
{integer,A,Vars#vars.arity},
{integer,A,Vars#vars.clause},
{integer,A,Line}]},
{integer,A,1}]}.
(from cover.erl)
The code inserted by the function above is:
ets:update_counter(?COVER_TABLE,
{?BUMP_REC_NAME, Module, Function, Arity, Clause, Line}, 1)
That is, increment the entry for the given module / function / line in question by 1. After all tests have finished, cover will use the data in this table and show how many times a given line was executed.
As mentioned in the cover documentation, you can get coverage for modules, functions, function clauses and lines. It looks like ExCoveralls only uses line coverage in its reports, but there is no reason it couldn't do all four types of coverage.
Branch coverage is not supported. Seems like supporting branch coverage would require expanding the "bump" record and updating cover.erl to record that information. Until someone does that, coverage information is only accurate when branches appear on different lines. For example:
case always_false() of
true ->
%% this line shows up as not covered
do_something();
false ->
ok
end.
%% this line shows up as covered, even though do_something is never called
always_false() andalso do_something()
To add to #legoscia excellent response, I also want to clarify why cover does not do statements evaluation. According to this discussion in the official forum:
https://elixirforum.com/t/code-coverage-tools-for-elixir/18102/10
The code is first compiled into erlang and then from erlang into a modified binary file (but no .beam file is created) that is automatically loaded into memory and executed.
Because of the way erlang code works, a single statement can have several instructions:
and single line can result in multiple VM “statements”, for example:
Integer.to_string(a + 1)
Will result with 2 instructions:
{line,[{location,"lib/tasks.ex",6}]}.
{gc_bif,'+',{f,0},1,[{x,0},{integer,1}],{x,0}}.
{line,[{location,"lib/tasks.ex",6}]}.
{call_ext_only,1,{extfunc,erlang,integer_to_binary,1}}.
Therefore it is rather tricky for an automatic analysis tool to provide statement coverage because it is hard to match statements to instructions, especially as in theory a compiler is free to reorder commands as it pleases as long as the result is the same.
I'd like to compare two rules to see whether or not they are identical (in particular, I'd like to be able to test a bazel target before and after a commit to see if it has changed)
Is there a way to accomplish this, perhaps with bazel query?
You can try bazel query with --output=build to have bazel print out the rule with everything expanded (e.g. macros evaluated, globs expanded, expressions evaluated, etc) before and after the change, and compare the results. See https://docs.bazel.build/versions/master/query.html#output-build for more information.
I am really confuse with the statement coverage. I did some search on the Internet. Some say the statement coverage only go through the true condition, which in this case is 1-2-3-4-12. However, others say the statement coverage should cover as many statements as possible, which I believe in this case will be 1-2-3-5-6-7-8-9-10-11-12. Which one is correct?
For the branch coverage, I believe I should test both the true/false conditions without considering the loop, which I will use the value 1)x=-1 and 2)x=0
For the path coverage, I think I should test all the paths so compared to branch coverage I will need to test the loop also. So I am going to use the value 1) x= -1, 2)x = 0 and 3) x = 10.
Is my answers correct?
Thank you in advance
I think "statement coverage" refers to "meaningful/useful/normal".
The nubmer of statement may vary by breaking one statement into two.
The "true" condition may not be useful for us just like the example you gave. Normally we want a positive number to get its factorial. You can switch the "true" and "false" conditions whatever you like.
To calculate Statement Coverage, find out the shortest number of paths following which all the nodes will be covered.
So in your case :
1-2-3-5-6-7-8-9-10-11-12
This path is the shortest and covering maximum number of nodes but not all
so we have to take one more path , which is :
1-2-3-4-12
So in this exercise , the value of SC=2
I don't know how it happens that the option named "Restrict where this project can be run" from Jenkins seems to allow only a single value inside "Label Expression" field.
I tried lots of combinations in order to add more than one label and I wasn't able to find any way to put two.
I need to mention that I need AND between these labels.
The irony is that this option even has an Info button which loads some documentations, which is missing to say how an expression is supposed to look like. Another small nail in the Jenkins UX coffin. On this one neither Google helped.
There is a Jenkins bug causing the help text not to be shown. It is present since 1.585 and fixed since 1.621 (or 1.609.3 respectively).
Here is the help text:
If you want to always run this project on a specific node/slave, just specify its name. This works well when you have a small number of nodes.
As the size of the cluster grows, it becomes useful not to tie projects to specific slaves, as it hurts resource utilization when slaves may come and go. For such situation, assign labels to slaves to classify their capabilities and characteristics, and specify a boolean expression over those labels to decide where to run.
Valid Operators
The following operators are supported, in the order of precedence.
(expr) parenthesis
!expr negation
expr&&expr and
expr||expr or
a -> b "implies" operator. Equivalent to !a|b. For example, windows->x64 could be thought of as "if run on a Windows slave, that slave must be 64bit." It still allows Jenkins to run this build on linux.
a <-> b "if and only if" operator. Equivalent to a&&b || !a&&!b. For example, windows<->sfbay could be thought of as "if run on a Windows slave, that slave must be in the SF bay area, but if not on Windows, it must not be in the bay area."
All operators are left-associative (i.e., a->b->c <-> (a->b)->c ) An expression can contain whitespace for better readability, and it'll be ignored.
Label names or slave names can be quoted if they contain unsafe characters. For example, "jenkins-solaris (Solaris)" || "Windows 2008"