Does anyone know if this is possible with SpecFlow? In the spirit of not having more than one assertion per test, I was hoping SpecFlow would treat each "Then" as a separate test.
I have a Scenario with multiple "Then" steps that looks something like this (snippet):
When a summary for day <day number> is requested
Then the summary well id should be "134134"
And the summary well name should be "My Well --oops!"
And the summary violated rules should be <violations>
And the summary variance should be <variance>
Examples:
| day number | violations | variance |
| 0 | BadTotal,HpOnLp | -33 |
| 3 | BadTotal | -133.33 |
| 5 | | 0 |
The second assertion "My Well --oops!" should fail. What I want is for SpecFlow to test the assertions that follow
I get:
When a summary for day 0 is requested
-> done: DprSummarySteps.WhenASummaryForDayIsRequested(0) (0.0s)
Then the summary well id should be "134134"
-> done: DprSummarySteps.ThenTheSummaryWellIdShouldBe("134134") (0.0s)
And the summary well name should be "My Well --oops!"
-> error: Assert.AreEqual failed. Expected:<My Well --oops!>. Actual:<My Well>.
And the summary violated rules should be BadTotal,HpOnLp
-> skipped because of previous errors
And the summary variance should be -33
-> skipped because of previous errors
Assert.AreEqual failed. Expected:<My Well --oops!>. Actual:<My Well>.
What I want:
When a summary for day 0 is requested
-> done: DprSummarySteps.WhenASummaryForDayIsRequested(0) (0.0s)
Then the summary well id should be "134134"
-> done: DprSummarySteps.ThenTheSummaryWellIdShouldBe("134134") (0.0s)
And the summary well name should be "My Well --oops!"
-> error: Assert.AreEqual failed. Expected:<My Well --oops!>. Actual:<My Well>.
And the summary violated rules should be BadTotal,HpOnLp
-> done: DprSummarySteps.ThenTheViolatedRulesShouldBe("BadTotal,HpOnLp") (0.0s)
And the summary variance should be -33
-> done: DprSummarySteps.ThenTheVarianceShouldBe(-33) (0.0s)
Assert.AreEqual failed. Expected:<My Well --oops!>. Actual:<My Well>.
I don't believe that you can do this but my question to you would be why do you want that? With any unit test it will fail at the first assertion. you wouldn't expect a test to continue execution after an assertion failed and this is no different. Surely knowing that the test failed for some reason is enough. In this specific case you might be able to make separate independent assertions which provide useful info, but in the general case assertions that come after one that has failed might be completely meaningless.
If you want to have each assertion independent of the others then you need to break it into several scenarios with each of you current And steps as its own Then step. You might be able to use a Background step to do the common setup.
I'm not sure that this will help you though as it seems like your assertions are all related to the examples so you end up needing to repeat the examples.
You can do this now if you upgrade to Specflow 2.4 and are using Nunit, see something like https://github.com/nunit/docs/wiki/Multiple-Asserts
Assert.Multiple(() =>
{
Assert.AreEqual(5.2, result.RealPart, "Real part");
Assert.AreEqual(3.9, result.ImaginaryPart, "Imaginary part");
});
I have identified one way to do this. So what happens internally in specflow, before running any step, the TestExecutionEngine checks the LastExecutionStatus of Scenario, and it does not proceed if it is not OK.. So what we can do is, in [AfterStep] hook, add the following line :
typeof(ScenarioContext).GetProperty("ScenarioExecutionStatus").SetValue(this.ScenarioContext, ScenarioExecutionStatus.OK);
Replace this.ScenarioContext with whatever object with type ScenarioContext exist. This will make the current status as OK and will allow you to proceed to next step.
Keep in mind this won't allow you to catch all assert failures in a single step.
Related
On the Wikipedia page of Finite State Machines it shows a graphic of the automata types:
I've never heard of combinational logic being included in the automata theory, normally just the Chomsky hierarchy, which stars with FSM. How then would combinational logic be written using a state machine?
For example, if we have an AND gate, I'd see it in a circuit diagram as something like:
______
A ------- | |
| AND |------- C
B ------- |______|
And the states would be: 1(A) & 1(B) --> 1(C), 1&0->0, 0&1->0, 0&0->0. But this involves two initial states rather than one, and also the input to a 'gate' is the combination of two inputs rather than one, so how would this be shown using a FSM? I suppose it could be possible doing something like the following -- with the input symbols being {0,1} and the output {0,1} like a Moore machine.
1 1
s0 ----> s2 -----> s3:1
| | 0
------> s3:0 --0,1--|
0 ^----------|
But this seems a bit useless to me so maybe I'm getting it wrong, what then would be a proper way to model Combinational logic in a state diagram?
Here would be a simpler way to diagram the above, where the Input and Output states are either ON (1) or OFF (0) to make it more intuitive.
I have some eVars that are registering a huge number of Nones.
These variables are set at the same time a corresponding traffic variable is set, for instance:
s.prop20 = someValue;
s.eVar20 = someValue;
When I give a check in Omniture, I see something like:
prop20
someValue1 | 12
someValue2 | 9
someValue3 | 5
.......
eVar20
None | 1987
someValue1 | 12
someValue2 | 9
someValue3 | 5
.......
I'm very confuse about that. One hypotheses of mine is that the variable eVar20 is created even when it is not set, is there a way to avoid this Nones?
Thanks
It has everything to do with how the attribution and expiration works differently between eVars and props.
Props do not have any attribution/expiration options. They are designed as 'traffic variables' and therefore only relate to the hit of data they were defined on.
eVars have flexible options for attribution (most recent, linear, original) and expiration (visit, 30 days etc).
As such what you're seeing is a default reporting option whereby when the metric you're looking at was recorded the 'none' value just states that there was no active eVar at that time.
Let me know if any of this is unclear.
Say I have a function,foo/1, whose spec is -spec foo(atom()) -> #r{}., where #r{} is a record defined as -record(r, {a :: 1..789})., however, I have foo(a) -> 800. in my code, when I run dialyzer against it, it didn't warn me about this, (800 is not a "valid" return value for function foo/1), can I make dialyzer warn me about this?
Edit
Learn You Some Erlang says:
Dialyzer reserves the right to expand this range into a bigger one.
But I couldn't find how to disable this.
As of Erlang 18, the handling of integer ranges is done by erl_types:t_from_range/2. As you can see, there are a lot of generalizations happening to get a "safe" overapproximation of a range.
If you tried to ?USE_UNSAFE_RANGES (see the code) it is likely that your particular error would be caught, but at a terrible cost: native compilation and dialyzing of recursive integer functions would not ever finish!
The reason is that the type analysis for recursive functions uses a simple fixpoint approach, where the initial types accept the base cases and are repeatedly expanded using the recursive cases to include more values. At some point overapproximations must happen if the process is to terminate. Here is a concrete example:
fact(1) -> 1;
fact(N) -> N * fact(N - 1).
Initially fact/1 is assumed to have type fun(none()) -> none(). Using that to analyse the code, the second clause is 'failing' and only the first one is ok. Therefore after the first iteration the new type is fun(1) -> 1. Using the new type the second clause can succeed, expanding the type to fun(1|2) -> 1|2. Then fun(1|2|3) -> 1|2|6 this continues until the ?SET_LIMIT is reached in which case t_from_range stops using the individual values and type becomes fun(1..255) -> pos_integer(). The next iteration expands 1..255 to pos_integer() and then fun(pos_integer()) -> pos_integer() is a fixpoint!
Incorrect answer follows (explains the first comment below):
You should get a warning for this code if you use the -Woverspecs option. This option is not enabled by default, since Dialyzer operates under the assumption that it is 'ok' to over-approximate the return values of a function. In your particular case, however, you actually want any extra values to produce warnings.
I've created the snippet below based on this tutorial. The last two lines (feed_squid(FeederRP) and feed_red_panda(FeederSquid)) are obviously violating the defined constraints, yet Dialyzer finds them okay. This is quite disappointing, because this is exactly the type of error I want to catch with a tool performing static analysis.
There is an explanation provided in the tutorial:
Before the functions are called with the wrong kind of feeder, they're
first called with the right kind. As of R15B01, Dialyzer would not
find an error with this code. The observed behaviour is that as soon
as a call to a given function succeeds within the function's body,
Dialyzer will ignore later errors within the same unit of code.
What is the rationale for this behavior? I understand that the philosophy behind success typing is "to never cry wolf", but in the current scenario Dialyzer plainly ignores the intentionally defined function specifications (after it sees that the functions have been called correctly earlier). I understand that the code does not result in a runtime crash. Can I somehow force Dialyzer to always take my function specifications seriously? If not, is there a tool that can do it?
-module(zoo).
-export([main/0]).
-type red_panda() :: bamboo | birds | eggs | berries.
-type squid() :: sperm_whale.
-type food(A) :: fun(() -> A).
-spec feeder(red_panda) -> food(red_panda());
(squid) -> food(squid()).
feeder(red_panda) ->
fun() ->
element(random:uniform(4), {bamboo, birds, eggs, berries})
end;
feeder(squid) ->
fun() -> sperm_whale end.
-spec feed_red_panda(food(red_panda())) -> red_panda().
feed_red_panda(Generator) ->
Food = Generator(),
io:format("feeding ~p to the red panda~n", [Food]),
Food.
-spec feed_squid(food(squid())) -> squid().
feed_squid(Generator) ->
Food = Generator(),
io:format("throwing ~p in the squid's aquarium~n", [Food]),
Food.
main() ->
%% Random seeding
<<A:32, B:32, C:32>> = crypto:rand_bytes(12),
random:seed(A, B, C),
%% The zoo buys a feeder for both the red panda and squid
FeederRP = feeder(red_panda),
FeederSquid = feeder(squid),
%% Time to feed them!
feed_squid(FeederSquid),
feed_red_panda(FeederRP),
%% This should not be right!
feed_squid(FeederRP),
feed_red_panda(FeederSquid).
Minimizing the example quite a bit I have these two versions:
First one that Dialyzer can catch:
-module(zoo).
-export([main/0]).
-type red_panda_food() :: bamboo.
-type squid_food() :: sperm_whale.
-spec feed_squid(fun(() -> squid_food())) -> squid_food().
feed_squid(Generator) -> Generator().
main() ->
%% The zoo buys a feeder for both the red panda and squid
FeederRP = fun() -> bamboo end,
FeederSquid = fun() -> sperm_whale end,
%% CRITICAL POINT %%
%% This should not be right!
feed_squid(FeederRP),
%% Time to feed them!
feed_squid(FeederSquid)
Then the one with no warnings:
[...]
%% CRITICAL POINT %%
%% Time to feed them!
feed_squid(FeederSquid)
%% This should not be right!
feed_squid(FeederRP).
Dialyzer's warnings for the version it can catch are:
zoo.erl:7: The contract zoo:feed_squid(fun(() -> squid_food())) -> squid_food() cannot be right because the inferred return for feed_squid(FeederRP::fun(() -> 'bamboo')) on line 15 is 'bamboo'
zoo.erl:10: Function main/0 has no local return
... and is a case of preferring to trust its own judgement against a user's tighter spec.
For the version it doesn't catch, Dialyzer assumes that the feed_squid/1 argument's type fun() -> bamboo is a supertype of fun() -> none() (a closure that will crash, which, if not called within feed_squid/1, is still a valid argument). After the types have been inferred, Dialyzer cannot know if a passed closure is actually called within a function or not.
Dialyzer still gives a warning if the option -Woverspecs is used:
zoo.erl:7: Type specification zoo:feed_squid(fun(() -> squid_food())) -> squid_food() is a subtype of the success typing: zoo:feed_squid(fun(() -> 'bamboo' | 'sperm_whale')) -> 'bamboo' | 'sperm_whale'
... warning that nothing prevents this function to handle the other feeder or any given feeder! If that code did check for the closure's expected input/output, instead of being generic, then I am pretty sure that Dialyzer would catch the abuse. From my point of view, it is much better if your actual code checks for erroneous input instead of you relying on type specs and Dialyzer (which never promised to find all the errors anyway).
WARNING: DEEP ESOTERIC PART FOLLOWS!
The reason why the error is reported in the first case but not the second has to do with the progress of module-local refinement. Initially the function feed_squid/1 has success typing (fun() -> any()) -> any(). In the first case the function feed_squid/1 will first be refined with just the FeederRP and will definitely return bamboo, immediately falsifying the spec and stopping further analysis of main/0. In the second, the function feed_squid/1 will first be refined with just the FeederSquid and will definitely return sperm_whale, then refined with both FeederSquid and FeederRP and return sperm_whale OR bamboo. When then called with FeederRP the expected return value success-typing-wise is sperm_whale OR bamboo. The spec then promises that it will be sperm_whale and Dialyzer accepts it. On the other hand, the argument should be fun() -> bamboo | sperm_whale success-typing-wise, it is fun() -> bamboo so that leaves it with just fun() -> bamboo. When that is checked against the spec (fun() -> sperm_whale), Dialyzer assumes that the argument could be fun() -> none(). If you never call the passed function within feed_squid/1 (something that Dialyzer's type system doesn't keep as information), and you promise in the spec that you will always return sperm_whale, everything is fine!
What can be 'fixed' is for the type system to be extended to note when a closure that is passed as an argument is actually used in a call and warn in cases where the only way to 'survive' some part of the type inference is to be fun(...) -> none().
(Note, I am speculating a bit here. I have not read the dialyzer code in detail).
A "Normal" full-fledged type checker has the advantage that type checking is decidable. We can ask "Is this program well-typed" and get either a Yes or a No back when the type checker terminates. Not so for the dialyzer. It is essentially in the business of solving the halting problem. The consequence is that there will be programs which are blatantly wrong, but still slips through the grips of the dialyzer.
However, this is not one of those cases :)
The problem is two-fold. A success type says "If this function terminates normally, what is its type?". In the above, our feed_red_panda/1 function terminates for any argument matching fun (() -> A) for an arbitrary type A. We could call feed_red_panda(fun erlang:now/0) and it should also work. Thus our two calls to the function in main/0 does not give rise to a problem. They both terminate.
The second part of the problem is "Did you violate the spec?". Note that often, specs are not used in the dialyzer as a fact. It infers the types itself and uses the inference patterns instead of your spec. Whenever a function is called, it is annotated with the parameters. In our case, it will be annotated with the two generator types: food(red_panda()), food(squid()). Then a function local analysis is made based on these annotations in order to figure out if you violated the spec. Since the correct parameters are present in the annotations, we must assume the function is used correctly in some part of the code. That it is also called with squids could be an artifact of code which are never called due to other circumstances. Since we are function-local we don't know and give the benefit of doubt to the programmer.
If you change the code to only make the wrong call with a squid-generator, then we find the spec-discrepancy. Because we know the only possible call site violates the spec. If you move the wrong call to another function, it is not found either. Because the annotation is still on the function and not on the call site.
One could imagine a future variant of the dialyzer which accounted for the fact that each call-site can be handled individually. Since the dialyzer is changing as well over time, it may be that it will be able to handle this situation in the future. But currently, it is one of the errors that will slip through.
The key is to notice that the dialyzer cannot be used as a "Checker of well-typedness". You can't use it to enforce structure on your programs. You need to do that yourself. If you would like more static checking, it would probably be possible to write a type checker for Erlang and run it on parts of your code base. But you will run into trouble with code upgrades and distribution, which are not easy to handle.
I'm trying to learn a little of the mindset of functional programming in F#, so any tips are appreciated. Right now I'm making a simple recursive function which takes a list and returns the i:th element.
let rec nth(list, i) =
match (list, i) with
| (x::xs, 0) -> x
| (x::xs, i) -> nth(xs, i-1)
The function itself seems to work, but it warns me about an incomplete pattern. I'm not sure what to return when I match the empty list in this case, since if I for example do the following:
| ([], _) -> ()
The whole function is treated like a function that takes a unit as argument. I want it to treat is as a polymorphic function.
While I'm at it, I may as well ask how far is reasonable to go to check for valid arguments when designing a function when developing seriously. Should I check for everything, so "misuse" of the function is prevented? In the above example I could for example specify the function to try to access an element in the list that is larger than its size. I hope my question isn't too confusing :)
You can learn a lot about the "usual" library design by looking at the standard F# libraries. There is already a function that does what you want called List.nth, but even if you're implementing this as an exercise, you can check how the function behaves:
> List.nth [ 1 .. 3 ] 10;;
System.ArgumentException: The index was outside the range
of elements in the list. Parameter name: index
The function throws System.ArgumentException with some additional information about the exception, so that users can easily find out what went wrong. To implement the same functionality, you can use the invalidArg function:
| _ -> invalidArg "index" "Index is out of range."
This is probably better than just using failwith which throws a more general exception. When using invalidArg, users can check for a specific type of exceptions.
As kvb noted, another option is to return option 'a. Many standard library functions provide both a version that returns option and a version that throws an exception. For example List.pick and List.tryPick. So, maybe a good design in your case would be to have two functions - nth and tryNth.
If you want your function to return a meaningful result and to have the same type as it has now, then you have no alternative but to throw an exception in the remaining case. A matching failure will throw an exception, so you don't need to change it, but you may find it preferable to throw an exception with more relevant information:
| _ -> failwith "Invalid list index"
If you expect invalid list indices to be rare, then this is probably good enough. However, another alternative would be to change your function so that it returns an 'a option:
let rec nth = function
| x::xs, 0 -> Some(x)
| [],_ -> None
| _::xs, i -> nth(xs, i-1)
This places an additional burden on the caller, who must now explicitly deal with the possibility of failure.
Presumably, if taking an empty list is invalid, you're best off just throwing an exception?
Generally the rules for how defensive you should be don't really change from language to language - I always go by the guideline that if it's public be paranoid about validating input, but if it's private code, you can be less strict. (Actually if it's a large project, and it's private code, be a little strict... basically strictness is proportional to the number of developers who might call your code.)