I've created the snippet below based on this tutorial. The last two lines (feed_squid(FeederRP) and feed_red_panda(FeederSquid)) are obviously violating the defined constraints, yet Dialyzer finds them okay. This is quite disappointing, because this is exactly the type of error I want to catch with a tool performing static analysis.
There is an explanation provided in the tutorial:
Before the functions are called with the wrong kind of feeder, they're
first called with the right kind. As of R15B01, Dialyzer would not
find an error with this code. The observed behaviour is that as soon
as a call to a given function succeeds within the function's body,
Dialyzer will ignore later errors within the same unit of code.
What is the rationale for this behavior? I understand that the philosophy behind success typing is "to never cry wolf", but in the current scenario Dialyzer plainly ignores the intentionally defined function specifications (after it sees that the functions have been called correctly earlier). I understand that the code does not result in a runtime crash. Can I somehow force Dialyzer to always take my function specifications seriously? If not, is there a tool that can do it?
-module(zoo).
-export([main/0]).
-type red_panda() :: bamboo | birds | eggs | berries.
-type squid() :: sperm_whale.
-type food(A) :: fun(() -> A).
-spec feeder(red_panda) -> food(red_panda());
(squid) -> food(squid()).
feeder(red_panda) ->
fun() ->
element(random:uniform(4), {bamboo, birds, eggs, berries})
end;
feeder(squid) ->
fun() -> sperm_whale end.
-spec feed_red_panda(food(red_panda())) -> red_panda().
feed_red_panda(Generator) ->
Food = Generator(),
io:format("feeding ~p to the red panda~n", [Food]),
Food.
-spec feed_squid(food(squid())) -> squid().
feed_squid(Generator) ->
Food = Generator(),
io:format("throwing ~p in the squid's aquarium~n", [Food]),
Food.
main() ->
%% Random seeding
<<A:32, B:32, C:32>> = crypto:rand_bytes(12),
random:seed(A, B, C),
%% The zoo buys a feeder for both the red panda and squid
FeederRP = feeder(red_panda),
FeederSquid = feeder(squid),
%% Time to feed them!
feed_squid(FeederSquid),
feed_red_panda(FeederRP),
%% This should not be right!
feed_squid(FeederRP),
feed_red_panda(FeederSquid).
Minimizing the example quite a bit I have these two versions:
First one that Dialyzer can catch:
-module(zoo).
-export([main/0]).
-type red_panda_food() :: bamboo.
-type squid_food() :: sperm_whale.
-spec feed_squid(fun(() -> squid_food())) -> squid_food().
feed_squid(Generator) -> Generator().
main() ->
%% The zoo buys a feeder for both the red panda and squid
FeederRP = fun() -> bamboo end,
FeederSquid = fun() -> sperm_whale end,
%% CRITICAL POINT %%
%% This should not be right!
feed_squid(FeederRP),
%% Time to feed them!
feed_squid(FeederSquid)
Then the one with no warnings:
[...]
%% CRITICAL POINT %%
%% Time to feed them!
feed_squid(FeederSquid)
%% This should not be right!
feed_squid(FeederRP).
Dialyzer's warnings for the version it can catch are:
zoo.erl:7: The contract zoo:feed_squid(fun(() -> squid_food())) -> squid_food() cannot be right because the inferred return for feed_squid(FeederRP::fun(() -> 'bamboo')) on line 15 is 'bamboo'
zoo.erl:10: Function main/0 has no local return
... and is a case of preferring to trust its own judgement against a user's tighter spec.
For the version it doesn't catch, Dialyzer assumes that the feed_squid/1 argument's type fun() -> bamboo is a supertype of fun() -> none() (a closure that will crash, which, if not called within feed_squid/1, is still a valid argument). After the types have been inferred, Dialyzer cannot know if a passed closure is actually called within a function or not.
Dialyzer still gives a warning if the option -Woverspecs is used:
zoo.erl:7: Type specification zoo:feed_squid(fun(() -> squid_food())) -> squid_food() is a subtype of the success typing: zoo:feed_squid(fun(() -> 'bamboo' | 'sperm_whale')) -> 'bamboo' | 'sperm_whale'
... warning that nothing prevents this function to handle the other feeder or any given feeder! If that code did check for the closure's expected input/output, instead of being generic, then I am pretty sure that Dialyzer would catch the abuse. From my point of view, it is much better if your actual code checks for erroneous input instead of you relying on type specs and Dialyzer (which never promised to find all the errors anyway).
WARNING: DEEP ESOTERIC PART FOLLOWS!
The reason why the error is reported in the first case but not the second has to do with the progress of module-local refinement. Initially the function feed_squid/1 has success typing (fun() -> any()) -> any(). In the first case the function feed_squid/1 will first be refined with just the FeederRP and will definitely return bamboo, immediately falsifying the spec and stopping further analysis of main/0. In the second, the function feed_squid/1 will first be refined with just the FeederSquid and will definitely return sperm_whale, then refined with both FeederSquid and FeederRP and return sperm_whale OR bamboo. When then called with FeederRP the expected return value success-typing-wise is sperm_whale OR bamboo. The spec then promises that it will be sperm_whale and Dialyzer accepts it. On the other hand, the argument should be fun() -> bamboo | sperm_whale success-typing-wise, it is fun() -> bamboo so that leaves it with just fun() -> bamboo. When that is checked against the spec (fun() -> sperm_whale), Dialyzer assumes that the argument could be fun() -> none(). If you never call the passed function within feed_squid/1 (something that Dialyzer's type system doesn't keep as information), and you promise in the spec that you will always return sperm_whale, everything is fine!
What can be 'fixed' is for the type system to be extended to note when a closure that is passed as an argument is actually used in a call and warn in cases where the only way to 'survive' some part of the type inference is to be fun(...) -> none().
(Note, I am speculating a bit here. I have not read the dialyzer code in detail).
A "Normal" full-fledged type checker has the advantage that type checking is decidable. We can ask "Is this program well-typed" and get either a Yes or a No back when the type checker terminates. Not so for the dialyzer. It is essentially in the business of solving the halting problem. The consequence is that there will be programs which are blatantly wrong, but still slips through the grips of the dialyzer.
However, this is not one of those cases :)
The problem is two-fold. A success type says "If this function terminates normally, what is its type?". In the above, our feed_red_panda/1 function terminates for any argument matching fun (() -> A) for an arbitrary type A. We could call feed_red_panda(fun erlang:now/0) and it should also work. Thus our two calls to the function in main/0 does not give rise to a problem. They both terminate.
The second part of the problem is "Did you violate the spec?". Note that often, specs are not used in the dialyzer as a fact. It infers the types itself and uses the inference patterns instead of your spec. Whenever a function is called, it is annotated with the parameters. In our case, it will be annotated with the two generator types: food(red_panda()), food(squid()). Then a function local analysis is made based on these annotations in order to figure out if you violated the spec. Since the correct parameters are present in the annotations, we must assume the function is used correctly in some part of the code. That it is also called with squids could be an artifact of code which are never called due to other circumstances. Since we are function-local we don't know and give the benefit of doubt to the programmer.
If you change the code to only make the wrong call with a squid-generator, then we find the spec-discrepancy. Because we know the only possible call site violates the spec. If you move the wrong call to another function, it is not found either. Because the annotation is still on the function and not on the call site.
One could imagine a future variant of the dialyzer which accounted for the fact that each call-site can be handled individually. Since the dialyzer is changing as well over time, it may be that it will be able to handle this situation in the future. But currently, it is one of the errors that will slip through.
The key is to notice that the dialyzer cannot be used as a "Checker of well-typedness". You can't use it to enforce structure on your programs. You need to do that yourself. If you would like more static checking, it would probably be possible to write a type checker for Erlang and run it on parts of your code base. But you will run into trouble with code upgrades and distribution, which are not easy to handle.
Related
When I compile the following module:
-module(x).
-export([inp/0]).
f(X) ->
g(X).
g(X) ->
error(X).
inp() ->
f(123).
And evaluate x:inp() I get the following output:
[{x,g,1,[{file,"x.erl"},{line,8}]},
{erl_eval,do_apply,6,[{file,"erl_eval.erl"},{line,689}]},
{erl_eval,try_clauses,8,[{file,"erl_eval.erl"},{line,919}]},
{shell,exprs,7,[{file,"shell.erl"},{line,686}]},
{shell,eval_exprs,7,[{file,"shell.erl"},{line,642}]},
{shell,eval_loop,3,[{file,"shell.erl"},{line,627}]}]
Where did the calls to f and inp go? This behavior makes it significantly harder to track the causes of errors in my case, how can I get the full stacktrace?
I am using OTP24
This is because of Erlang's compiler optimization. The compiler deduces that, in this specific case, functions f() and inp() are only used to pass a number to function g() and they cannot be used for anything else, not even theoretically. So the compiler "optimizes them away" and de facto only compiles function g().
When should I use "ignore" instead of "()"?
I attempted to write the following:
let log = fun data medium -> ()
I then received the following message:
Lint: 'fun _ -> ()' might be able to be refactored into 'ignore'.
So I updated the declaration to the following:
let log = fun data medium -> ignore
Is there any guidance on why I might use one over the other?
My gut tells me that I should use ignore when executing an actual expression.
In this case though, I'm declaring a high-order function.
Are my assumptions accurate?
The linter message that you got here is a bit confusing. The ignore function is just a function that takes anything and returns unit:
let ignore = fun x -> ()
Your log function is a bit similar to ignore, but it takes two parameters:
let log = fun data medium -> ()
In F#, this is actually a function that returns another function (currying). You can write this more explicitly by saying:
let log = fun data -> fun medium -> ()
Now, you can see that a part of your function is actually the same thing as ignore. You can write:
let log = fun data -> ignore
This means the same thing as your original function and this is what the linter is suggesting. I would not write the code in this way, because it is less obvious what the code does (it actually takes two arguments) - I guess the linter is looking just for the simple pattern, ignoring the fact that sometimes the refactoring is not all that useful.
Never, at least not in the way shown in the question.
Substituting between ignore and () is not meaningful, as they are different concepts:
ignore is a generic function with one argument and unit return. Its type is 'T -> unit.
() is the only valid value of type unit. It is not a function at all.
Therefore, it's not valid to do the refactor shown in the question. The first version of log takes two curried arguments, while the second version takes three.
What Lint is trying to suggest isn't quite clear. ignore is a function with one argument; it's not obvious how (or why) it should be used to refactor a method that takes two curried arguments. fun _ _ -> () would be an okay and quite readable way to ignore two arguments.
Say I have a function,foo/1, whose spec is -spec foo(atom()) -> #r{}., where #r{} is a record defined as -record(r, {a :: 1..789})., however, I have foo(a) -> 800. in my code, when I run dialyzer against it, it didn't warn me about this, (800 is not a "valid" return value for function foo/1), can I make dialyzer warn me about this?
Edit
Learn You Some Erlang says:
Dialyzer reserves the right to expand this range into a bigger one.
But I couldn't find how to disable this.
As of Erlang 18, the handling of integer ranges is done by erl_types:t_from_range/2. As you can see, there are a lot of generalizations happening to get a "safe" overapproximation of a range.
If you tried to ?USE_UNSAFE_RANGES (see the code) it is likely that your particular error would be caught, but at a terrible cost: native compilation and dialyzing of recursive integer functions would not ever finish!
The reason is that the type analysis for recursive functions uses a simple fixpoint approach, where the initial types accept the base cases and are repeatedly expanded using the recursive cases to include more values. At some point overapproximations must happen if the process is to terminate. Here is a concrete example:
fact(1) -> 1;
fact(N) -> N * fact(N - 1).
Initially fact/1 is assumed to have type fun(none()) -> none(). Using that to analyse the code, the second clause is 'failing' and only the first one is ok. Therefore after the first iteration the new type is fun(1) -> 1. Using the new type the second clause can succeed, expanding the type to fun(1|2) -> 1|2. Then fun(1|2|3) -> 1|2|6 this continues until the ?SET_LIMIT is reached in which case t_from_range stops using the individual values and type becomes fun(1..255) -> pos_integer(). The next iteration expands 1..255 to pos_integer() and then fun(pos_integer()) -> pos_integer() is a fixpoint!
Incorrect answer follows (explains the first comment below):
You should get a warning for this code if you use the -Woverspecs option. This option is not enabled by default, since Dialyzer operates under the assumption that it is 'ok' to over-approximate the return values of a function. In your particular case, however, you actually want any extra values to produce warnings.
What is the difference in ending a function with end and ok in Erlang? I've been trying to grasp the meaning in the following code:
-module(esOne).
-export([start/1, func/1]).
start(Par) ->
io:format("Client: I am ~p, spawned by the server: ~p~n",[self(),Par]),
spawn(esOne, func, [self()]),
Par ! {onPid, self()},
serverEsOne ! {onName, self()},
receiveMessage(),
ok.
receiveMessage() ->
receive
{reply, N} ->
io:format("Client: I received a message: ~p~n",[N])
after
5000->
io:format("Client: I received no message, i quit~n",[])
end.
func(Parent)->
io:format("Child: I am ~p, spawned from ~p~n",[self(),Parent]).
This code works in conjunction with another .erl file that acts as server. I managed to write this only through analyzing the given server file and copying it's behavior. First I thought ok was used to end every function, but that is not the case as I can't end receiveMessage() with ok. Then I thought I could maybe end every function with end, but start(Par) will give an error if I replace ok by end. Not only that, but in the server file I see that ok and end are used within functions to end loops. The way they're used look the same to me, yet they clearly fulfill a separate function as one cannot be replaced by another. Some clarification would be much appreciated.
Two points to understand:
Some code block types in Erlang are closed with an "end". So if ... end, case ... end, receive ... [after N] ... end and so on. It is certainly possible to use "end" as its own atom in place of OK, but that is not what is happening above.
Every function in Erlang returns some value. If you aren't explicit about it, it returns the value of the last expression. The "=" operator isn't assignment to a variable like in other languages, it is assignment to a symbol as in math, meaning that reassigning is effectively a logical assertion. If the assertion fails, the process throws an exception (meaning it crashes, usually).
When you end something with "ok" or any other atom you are providing a known final value that will be returned. You don't have to do anything with it, but if you want the calling process to assert that the function completed or crash if anything unusual happened then you can:
do_stuff() ->
ok = some_func().
instead of
do_stuff() ->
some_func().
If some_func() may have had a side effect that can fail, it will usually return either ok or {error, Reason} (or something similar). By checking that the return was ok we prevent the calling process from continuing execution if something bad happened. That is central to the Erlang concept of "let it crash". The basic idea is that if you call a function that has a side-effect and it does anything unexpected at all, you should crash immediately, because proceeding with bad data is worse than not proceeding at all. The crash will be cleaned up by the supervisor, and the system will be restored to a known state instead of being in whatever random condition was left after the failure of the side-effect.
A variation on the bit above is to have the "ok" part appear in a tuple if the purpose of a function is to return a value. You can see this in any dict-type handling library, for example. The reason some data returning functions have a return type of {ok, Value} | {error, Reason} instead of just Value | {error, Reason} is to make pattern matching more natural.
Consider the following case clauses:
case dict:find(Key, Dict) of
{ok, Value} ->
Value;
{error, Reason} ->
log(error, Reason),
error
end.
And:
case finder(Key, Struct) of
Value ->
Value;
{error, Reason}
log(error, Reason),
error
end.
In the first example we match the success condition first. In the second version, though, this is impossible because the error clause could never match; any return at all would always be represented by Value. Oops.
Most of the time (but not quite always) functions that return a value or crash will return just the value. This is especially true of pure functions that carry no state but what you pass in and have no side effects (for example, dict:fetch/2 gives the value directly, or crashes the calling process, giving you an easy choice which way you want to do things). Functions that return a value or signal an error usually wrap a valid response in {ok, Value} so it is easy to match.
I'm trying to learn a little of the mindset of functional programming in F#, so any tips are appreciated. Right now I'm making a simple recursive function which takes a list and returns the i:th element.
let rec nth(list, i) =
match (list, i) with
| (x::xs, 0) -> x
| (x::xs, i) -> nth(xs, i-1)
The function itself seems to work, but it warns me about an incomplete pattern. I'm not sure what to return when I match the empty list in this case, since if I for example do the following:
| ([], _) -> ()
The whole function is treated like a function that takes a unit as argument. I want it to treat is as a polymorphic function.
While I'm at it, I may as well ask how far is reasonable to go to check for valid arguments when designing a function when developing seriously. Should I check for everything, so "misuse" of the function is prevented? In the above example I could for example specify the function to try to access an element in the list that is larger than its size. I hope my question isn't too confusing :)
You can learn a lot about the "usual" library design by looking at the standard F# libraries. There is already a function that does what you want called List.nth, but even if you're implementing this as an exercise, you can check how the function behaves:
> List.nth [ 1 .. 3 ] 10;;
System.ArgumentException: The index was outside the range
of elements in the list. Parameter name: index
The function throws System.ArgumentException with some additional information about the exception, so that users can easily find out what went wrong. To implement the same functionality, you can use the invalidArg function:
| _ -> invalidArg "index" "Index is out of range."
This is probably better than just using failwith which throws a more general exception. When using invalidArg, users can check for a specific type of exceptions.
As kvb noted, another option is to return option 'a. Many standard library functions provide both a version that returns option and a version that throws an exception. For example List.pick and List.tryPick. So, maybe a good design in your case would be to have two functions - nth and tryNth.
If you want your function to return a meaningful result and to have the same type as it has now, then you have no alternative but to throw an exception in the remaining case. A matching failure will throw an exception, so you don't need to change it, but you may find it preferable to throw an exception with more relevant information:
| _ -> failwith "Invalid list index"
If you expect invalid list indices to be rare, then this is probably good enough. However, another alternative would be to change your function so that it returns an 'a option:
let rec nth = function
| x::xs, 0 -> Some(x)
| [],_ -> None
| _::xs, i -> nth(xs, i-1)
This places an additional burden on the caller, who must now explicitly deal with the possibility of failure.
Presumably, if taking an empty list is invalid, you're best off just throwing an exception?
Generally the rules for how defensive you should be don't really change from language to language - I always go by the guideline that if it's public be paranoid about validating input, but if it's private code, you can be less strict. (Actually if it's a large project, and it's private code, be a little strict... basically strictness is proportional to the number of developers who might call your code.)