Is there a way to print the test description of an erlang test generator that uses fixtures? Using a generator makes it tricky to tell what test is actually failing and printing the description would help out.
Example:
-module(math_test).
-include_lib("eunit/include/eunit.hrl").
-define(test(Desc, F), {Desc, {setup, fun setup/0, fun cleanup/1, F}}).
setup() ->
ok.
cleanup(_) ->
ok.
math_test_ () ->
[
?test("adds two numbers", fun add_two_numbers/0),
?test("subtract two numbers", fun subtract_two_numbers/0),
?test("undefined method called", fun undefined_error/0)
].
add_two_numbers () ->
?assertEqual(2, 1 + 3).
subtract_two_numbers () ->
?assertEqual(1, 2 - 2).
undefined_error () ->
undefined_module:uh_oh().
And then running it
[root#a7c901c022bb src]# rebar3 eunit --module=math_test
===> Verifying dependencies...
===> Compiling math
===> Performing EUnit tests...
FFF
Failures:
1) math_test:math_test_/0
Failure/Error: ?assertEqual(2, 1 + 3)
expected: 2
got: 4
%% /src/_build/test/lib/math/src/math_test.erl:20:in `math_test:-add_two_numbers/0-fun-0-/1`
Output:
Output:
2) math_test:math_test_/0
Failure/Error: ?assertEqual(1, 2 - 2)
expected: 1
got: 0
%% /src/_build/test/lib/math/src/math_test.erl:23:in `math_test:-subtract_two_numbers/0-fun-0-/1`
Output:
Output:
3) math_test:math_test_/0
Failure/Error: {error,undef,[{undefined_module,uh_oh,[],[]}]}
Output:
The first 2 errors are ok, but not great -- you can at least see in the assertion where things actually went wrong.
However the 3rd error (calling undefined module/method) is where things go horribly wrong -- there's no real way to tell where it came from!
Is there a way to improve things, like printing the test description with the failure log?
One thing you can do is putting the test description on the test itself, not the entire setup tuple. That is, change this line:
-define(test(Desc, F), {Desc, {setup, fun setup/0, fun cleanup/1, F}}).
to:
-define(test(Desc, F), {setup, fun setup/0, fun cleanup/1, {Desc, F}}).
With that change, the test descriptions are printed:
Failures:
1) math_test:math_test_/0: adds two numbers
Failure/Error: ?assertEqual(2, 1 + 3)
expected: 2
got: 4
%% /tmp/math_test/mylib/_build/test/lib/mylib/test/math_test.erl:20:in `math_test:-add_two_numbers/0-fun-0-/0`
Output:
Output:
2) math_test:math_test_/0: subtract two numbers
Failure/Error: ?assertEqual(1, 2 - 2)
expected: 1
got: 0
%% /tmp/math_test/mylib/_build/test/lib/mylib/test/math_test.erl:23:in `math_test:-subtract_two_numbers/0-fun-0-/0`
Output:
Output:
3) math_test:math_test_/0: undefined method called
Failure/Error: {error,undef,[{undefined_module,uh_oh,[],[]}]}
Output:
Another thing to try is specifying the test functions with the ?_test macro instead of as plain fun terms:
math_test_ () ->
[
?test("adds two numbers", ?_test(add_two_numbers())),
?test("subtract two numbers", ?_test(subtract_two_numbers())),
?test("undefined method called", ?_test(undefined_error()))
].
The ?_test macro remembers the line number it appeared on, and includes it in the output on test failure:
1) math_test:math_test_/0:14: adds two numbers
[...]
2) math_test:math_test_/0:15: subtract two numbers
[...]
3) math_test:math_test_/0:16: undefined method called
[...]
Now you can tell which line those tests were invoked from.
Yet another way to do it is to have the individual functions return eunit "test objects" instead of just running the tests. That would involve using ?_assertEqual instead of ?assertEqual, or wrapping the entire thing in ?_test:
math_test_ () ->
[
?test("adds two numbers", add_two_numbers()),
?test("subtract two numbers", subtract_two_numbers()),
?test("undefined method called", undefined_error())
].
add_two_numbers () ->
?_assertEqual(2, 1 + 3).
subtract_two_numbers () ->
?_assertEqual(1, 2 - 2).
undefined_error () ->
?_test(undefined_module:uh_oh())
Then the output contains both the line numbers and the names of the individual test functions:
Failures:
1) math_test:add_two_numbers/0:20: adds two numbers
Failure/Error: ?assertEqual(2, 1 + 3)
expected: 2
got: 4
%% /tmp/math_test/mylib/_build/test/lib/mylib/test/math_test.erl:20:in `math_test:-add_two_numbers/0-fun-0-/0`
Output:
Output:
2) math_test:subtract_two_numbers/0:23: subtract two numbers
Failure/Error: ?assertEqual(1, 2 - 2)
expected: 1
got: 0
%% /tmp/math_test/mylib/_build/test/lib/mylib/test/math_test.erl:23:in `math_test:-subtract_two_numbers/0-fun-0-/0`
Output:
Output:
3) math_test:undefined_error/0:26: undefined method called
Failure/Error: {error,undef,[{undefined_module,uh_oh,[],[]}]}
Output:
The answers by #legoscia are good, but I also suspect that it's the error reporting implemented by rebar3 that's suboptimal for that kind of error. Running the tests directly from eunit with its default output, you get this:
2> eunit:test(math_test).
math_test: math_test_...*failed*
in function math_test:'-add_two_numbers/0-fun-0-'/0 (math_test.erl, line 22)
**error:{assertEqual,[{module,math_test},
{line,22},
{expression,"1 + 3"},
{expected,2},
{value,4}]}
output:<<"">>
math_test: math_test_...*failed*
in function math_test:'-subtract_two_numbers/0-fun-0-'/0 (math_test.erl, line 25)
**error:{assertEqual,[{module,math_test},
{line,25},
{expression,"2 - 2"},
{expected,1},
{value,0}]}
output:<<"">>
math_test: math_test_...*failed*
in function undefined_module:uh_oh/0
called as uh_oh()
**error:undef
output:<<"">>
=======================================================
Failed: 3. Skipped: 0. Passed: 0.
With the 'verbose' option, it also prints the descriptions before each setup. Furthermore, moving the description to the test fun, and using the ?_test(...) macro to create test funs with more positional information than a plain fun, as #legoscia suggested, you get this output:
math_test:18: math_test_ (undefined method called)...*failed*
in function undefined_module:uh_oh/0
called as uh_oh()
**error:undef
output:<<"">>
You could report this to the rebar3 maintainers.
Related
I am currently writing a test suite using busted/luassert and since I have put some assertions in a separate function I am getting inaccurate stack traces. For example, consider the following test suite (a_spec.lua):
local function my_custom_assertion(x) -- 1
assert.is_true(x > 0) -- 2 <-
end -- 3
-- 4
describe("My test suite", function() -- 5
it("they are positive", function() -- 6
my_custom_assertion(-10) -- 7 <-
my_custom_assertion(-20) -- 8 <-
end) -- 9
end) -- 10
When I run it, my test case fails but the stack trace points to line 2 so I can't tell which of the two assertions was the one that failed.
$busted spec/a_spec.lua
◼
0 successes / 1 failure / 0 errors / 0 pending : 0.002242 seconds
Failure → spec/a_spec.lua # 6
My test suite they are positive
spec/a_spec.lua:2: Expected objects to be the same.
Passed in:
(boolean) false
Expected:
(boolean) true
Is there a way I could have it point to line 7 or 8 instead? One way this would be possible is if luassert's assert.is_true function had a level parameter similar to the builtin error function.
Looking at luassert's source code it seems that it does care about the stack level but I haven't been able to figure out if this feature is internal or if it is exposed to the user somehow.
Instead of creating a custom assertion by creating a function that calls assert.xyzz(), create a function that returns true or false and register it with assert:register.
See the second example in the README.
Turns out that there is a way to solve my actual problem of finding out which of the assertions was the one that fired without needing to change how I write my tests. By invoking busted with the -v (--verbose) option it prints a full stack trace when an assertion fails instead of just providing a single line number.
$ busted -v spec/a_spec.lua
0 successes / 1 failure / 0 errors / 0 pending : 0.003241 seconds
Failure → spec/a_spec.lua # 6
My test suite they are positive
spec/a_spec.lua:2: Expected objects to be the same.
Passed in:
(boolean) false
Expected:
(boolean) true
stack traceback:
spec/a_spec.lua:2: in upvalue 'my_custom_assertion'
spec/a_spec.lua:7: in function <spec/a_spec.lua:6>
That mention of line number 7 lets me know what assertion was the one that failed.
If I declare a function
test(A) -> 3.
Erlang generates a warning about variable A not being used. However the definition
isEqual(X,X) -> 1.
Doesn't produce any warning but
isEqual(X,X) -> 1;
isEqual(X,Y) -> 0.
again produces a warning but only for the second line.
The reason why that doesn't generate a warning is because in the second case you are asserting (through pattern matching), by using the same variable name, that the first and second arguments to isEqual/2 have the same value. So you are actually using the value of the argument.
It might help to understand better if we look at the Core Erlang code produced from is_equal/2. You can get .core source files by compiling your .erl file in the following way: erlc +to_core pattern.erl (see here for pattern.erl).
This will produce a pattern.core file that will look something like this (module_info/[0,1] functions removed):
module 'pattern' ['is_equal'/2]
attributes []
'is_equal'/2 = fun (_cor1,_cor0) ->
case <_cor1,_cor0> of
%% Line 5
<X,_cor4> when call 'erlang':'=:=' (_cor4, X) ->
1
%% Line 6
<X,Y> when 'true' ->
0
end
As you can see, each function clause from is_equal/2 in the .erl source code gets translated to a case clause in Core Erlang. X does get used in the first clause since it needs to be compared to the other argument. On the other hand neither X or Y are used in the second clause.
I'm trying to log all calls to a function with dbg for debugging (thanks to this answer). Here's the code:
-module(a).
-export([main/0]).
trace_me(_, _, _) ->
ok.
main() ->
dbg:start(),
dbg:tracer(),
dbg:tpl(a, trace_me, 3, []),
dbg:p(all, c),
LargeBinary = binary:copy(<<"foo">>, 10000),
trace_me(foo, bar, LargeBinary).
The problem is that one of the argument is a really large binary, and the code above prints the complete binary with every call:
1> c(a).
{ok,a}
2> a:main().
(<0.57.0>) call a:trace_me(foo,bar,<<"foofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoofoo...lots of foos omitted...">>)
ok
Is it possible to (without modifying trace_me/3):
Only print the the first 2 arguments for each call?
Print the first 2 arguments + first few bytes of the 3rd argument or just pass the 3rd argument through a custom function before printing?
I don't know with dbg, but you can use the very good and small redbug:
trace_me(_, _, _) ->
ok.
do_redbug() ->
% 1. print arguments as dbg:
% redbug:start("a:trace_me"),
% 2. print arity only:
% redbug:start("a:trace_me", [arity]),
% 3. Specify the formatting depth:
redbug:start("a:trace_me", [{print_depth, 10}]),
LargeBinary = binary:copy(<<"foo">>, 100000),
trace_me(foo, bar, LargeBinary).
Have a look at the (quite terse) redbug documentation in the above link for more options and for how to pass your own printer/formatter function.
Note also that redbug, as opposed to dbg, is safe to use on a production system. Check out this presentation by the redbug author: Taking the printf out of printf Debugging.
I am looking to just run a few unit tests, I have added them them in an .fs file. I want to call them from the applications entry point as and when when I want to run them.
This the test I have written
namespace InvoiceApp
open System
open NUnit.Framework
open FsUnit
module Test =
let testPassed testName = printfn "Test Passed!! - %s" testName
let testFailed testName = printfn "Test Failed!! - %s" testName
[<TestFixture>]
type Test() =
[<TestCase(200,10,20)>]
let ``When profitmarging is 10% and total is 200 expect 20 `` x y z () =
try
Math.percentage x y |> should equal z
testPassed "When profitmarging is 10% and total is 200 expect 20"
with
| :? NUnit.Framework.AssertionException as ex ->
testFailed "When profitmarging is 10% and total is 200 expect 20"
printfn "%s" ex.Message
How can I call these tests from the entry point in a different .fs file?
A little while back I built an NUnit test runner as an F# module which should support your simple execution in a command line app scenario, see Running TAP.
To set it up, simply include this F# snippet and invoke the tests with:
Tap.Run typeof<InvoiceApp.Test>
Note: you will also need to change the test case to be a member function and use a non-generic tupled argument for either NUnit's built-in runner or the TAP runner to see it, i.e.
member test.``When profitmarging is 10% and total is 200 expect 20 `` (x:int,y:int,z:int) =
Take a look at SimpleTestRunner and RemoteTestRunner, ala this question.
So I've been using Erlang for the last eight hours, and I've spent two of those banging my head against the keyboard trying to figure out the exception error my console keeps returning.
I'm writing a dice program to learn erlang. I want it to be able to call from the console through the erlang interpreter. The program accepts a number of dice, and is supposed to generate a list of values. Each value is supposed to be between one and six.
I won't bore you with the dozens of individual micro-changes I made to try and fix the problem (random engineering) but I'll post my code and the error.
The Source:
-module(dice2).
-export([d6/1]).
d6(1) ->
random:uniform(6);
d6(Numdice) ->
Result = [],
d6(Numdice, [Result]).
d6(0, [Finalresult]) ->
{ok, [Finalresult]};
d6(Numdice, [Result]) ->
d6(Numdice - 1, [random:uniform(6) | Result]).
When I run the program from my console like so...
dice2:d6(1).
...I get a random number between one and six like expected.
However when I run the same function with any number higher than one as an argument I get the following exception...
**exception error: no function clause matching dice2:d6(1, [4|3])
... I know I I don't have a function with matching arguments but I don't know how to write a function with variable arguments, and a variable number of arguments.
I tried modifying the function in question like so....
d6(Numdice, [Result]) ->
Newresult = [random:uniform(6) | Result],
d6(Numdice - 1, Newresult).
... but I got essentially the same error. Anyone know what is going on here?
This is basically a type error. When Result is a list, [Result] is a list with one element. E.g., if your function worked, it would always return a list with one element: Finalresult.
This is what happens (using ==> for "reduces to"):
d6(2) ==> %% Result == []
d6(2, [[]]) ==> %% Result == [], let's say random:uniform(6) gives us 3
d6(1, [3]) ==> %% Result == 3, let's say random:uniform(6) gives us 4
d6(0, [4|3]) ==> %% fails, since [Result] can only match one-element lists
Presumably, you don't want [[]] in the first call, and you don't want Result to be 3 in the third call. So this should fix it:
d6(Numdice) -> Result = [], d6(Numdice, Result). %% or just d6(Numdice, []).
d6(0, Finalresult) -> {ok, Finalresult};
d6(Numdice, Result) -> d6(Numdice - 1, [random:uniform(6) | Result]).
Lesson: if a language is dynamically typed, this doesn't mean you can avoid getting the types correct. On the contrary, it means that the compiler won't help you in doing this as much as it could.