I am wondering if there is a reason why dafny is unable to verify my program?
https://rise4fun.com/Dafny/Ip1s
Am I missing some additional invariant?
The problem is that your definition of s and your construction of o go in "different directions". The recursive case of s defines s(i) in terms of
i[0] and what is "previously" defined by s(i[1..]). In contrast, the loop iteration defines the new o in terms of i[n] and the previous value of o. It would take an inductively proven lemma to establish the proof obligations in your current program, and Dafny does not invent such lemmas by itself.
For the record in this answer, here is what you started with:
function s(i: seq<int>): seq<int> {
if |i| == 0 then [] else
if i[0] == 42 then [i[0]] + s(i[1..])
else s(i[1..])
}
method q (i: seq<int>) returns (o: seq<int>)
ensures o == s(i)
{
var n := 0;
o := [];
while n < |i|
invariant n <= |i| && o == s(i[..n])
{
if i[n] == 42 {
o := o + [i[n]];
}
n := n + 1;
}
}
There are four ways out.
One way out is to define a different version of s, call it s', that recurses from the other end of the given sequence. Then, replace s by s' in your method specification and loop invariant. This is a fine solution, unless for some reason you really prefer s, not s', in your method specification.
A second way out is to define such an s' and to prove a lemma that s(i) and s'(i) return the same value. This will let you keep s in your method specification, at the cost of having two function definitions and having to write (and prove and use) a lemma.
A third way out is to change the loop to iterate "downward" instead of "upward". That is, start n at |i| and decrement n in the loop body. (As usual, an increment of n is typically best done at the end of the loop body (post-increment), whereas a decrement of n is typically best done at the beginning of the loop body (pre-decrement).)
A fourth way out is to change the way you write the loop invariant about o. Currently, the invariant speaks about what you already have computed, that is, o == s(i[..n]). You can instead write the invariant in terms of what is yet to be computed, as in o + s(i[n..]) == s(i), which you can read as "once I have added s(i[n..]) to o, I will have s(i)". Here is that version of q:
method q(i: seq<int>) returns (o: seq<int>)
ensures o == s(i)
{
var n := 0;
o := [];
while n < |i|
invariant n <= |i| && o + s(i[n..]) == s(i)
{
if i[n] == 42 {
o := o + [i[n]];
}
n := n + 1;
}
}
You may also be interested in watching this episode of Verification Corner on this subject.
Rustan
Related
I have never felt so woefully inadequate as I am when trying to prove to Dafny that my program is correct, so I need your help: The given program looks as follows:
method doingMath(N: int, M: int) returns (s: int)
requires N <= M //given precondition
ensures 2*s == M*(M+1)-N*(N+1) //given postcondition
{
var a: int := N;
var r: int := 0;
while a < M
invariant FIND ME!
{
a := a+1;
r := r+a;
}
return r;
}
As first step I wanted to figure out what the loop does, so I made a table:
With that I worked out a loop invariant for r:
invariant r == (a-N)*N + (a-N)*((a-N)+1)/2
which holds before (0==0), and after each iteration of the loop (following the formula). Clearly it does not satisfy the Termination criteria
When the loop terminates, the loop invariant along with the reason that the loop terminated gives us a useful property.
Since the loop guard is simple enough I figured the complete invariant should be
invariant a<=M && r == (a-N)*N + (a-N)*((a-N)+1)/2
And thus my invariant satisfies Initialization, Maintenance and Termination. However Dafny complains that
Error: This loop invariant might not be maintained by the loop.
How do I make Dafny happy?
I managed to stay clear of any non-linear arithmetic hick-ups. Here's how I think of the problem:
You're trying to establish a postcondition that, for the sake of clarity, I will write as P(s, N, M), that is, some function of s, N, and M. One technique for coming up with a loop that does this is "replace a constant by a variable". What this means is that you pick one of the constants of the desired postcondition (here, you can choose either N or M, since s is not a constant) and replace it by a variable that is going to change in each loop iteration. Let's pick M as the constant and let's introduce (as you had already done in your program) a as the variable. Since we picked M as the constant, we'll want the final value of a to be M, so we'll start a at N. We then have:
method doingMath(N: int, M: int) returns (s: int)
requires N <= M
ensures P(s, N, M)
{
var a := N;
while a < M
invariant N <= a <= M
invariant P(s, N, a) // postcondition, but with variable a instead of constant M
}
If you type in this program (but expand out the P(s, N, a) to the actual condition), then you will find that Dafny proves the postcondition. In other words, the verifier is giving you the information that if you can establish and maintain this loop invariant, then the program will correctly establish the postcondition.
You can see this yourself, too. The negation of the loop guard gives you M <= a, which combined with the loop invariant a <= M gives you a == M. When you combine a == M and the loop invariant P(s, N, a), you get the postcondition P(s, N, M).
Great. But the verifier issues a complaint that the loop invariant does not hold on entry. This is because we didn't provide any initial value for s. Since a has the initial value N, we need to find a value for s that satisfies P(s, N, N). That value is 0, so we update the program to
method doingMath(N: int, M: int) returns (s: int)
requires N <= M
ensures P(s, N, M)
{
var a := N;
s := 0;
while a < M
invariant N <= a <= M
invariant P(s, N, a)
}
Next, let's write the loop body. (Notice how I have started with the loop invariant, rather than starting with loop body and then trying to figure out an invariant. For these sorts of programs, I find that's the easiest way.) We already decided that we want to vary a from the initial value N up to the final value M, so we add the assignment a := a + 1;:
method doingMath(N: int, M: int) returns (s: int)
requires N <= M
ensures 2*s == M*(M+1) - N*(N+1)
{
var a := N;
s := 0;
while a < M
invariant N <= a <= M
invariant P(s, N, a)
{
a := a + 1;
}
}
This addresses termination. The final thing we need to do is update s inside the loop so that the invariant is maintained. This is mostly easily done backward, in a goal-directed fashion. Here's how: At the end of the loop body, we want to make sure P(s, N, a) holds. That means we want the condition P(s, N, a + 1) to hold before the assignment to a. You obtain this condition (again, remember we're working backward) by replacing a in the desired condition with (the right-hand side of the assignment) a + 1.
Okay, so before the assignment to a, we want to have P(s, N, a + 1), and what we've got just inside the loop body is the invariant P(s, N, a). Now, it's time for me to expand P(...) to your actual condition. Alright, we have
2*s == a*(a+1) - N*(N+1) (*)
and we want
2*s == (a+1)*(a+2) - N*(N+1) (**)
Let's rewrite (a+1)*(a+2) in (**) as 2*(a+1) + a*(a+1). So, (**) can equivalently be written as
2*s == 2*(a+1) + a*(a+1) - N*(N+1) (***)
If you compare (***) (which is what we want) with (*) (which is what we've got), then you notice that the right-hand side of (***) is 2*(a+1) more than the right-hand side of (*). So, we must arrange to increase the left-hand side with the same amount.
If you increase s by a+1, then the left-hand side 2*s goes up by 2*(a+1), which is what we want. So, our final program is
method doingMath(N: int, M: int) returns (s: int)
requires N <= M
ensures 2*s == M*(M+1) - N*(N+1)
{
var a := N;
s := 0;
while a < M
invariant N <= a <= M
invariant 2*s == a*(a+1) - N*(N+1)
{
s := s + a + 1;
a := a + 1;
}
}
If you want, you can swap the order of the assignments to s and a. This will give you
method doingMath(N: int, M: int) returns (s: int)
requires N <= M
ensures 2*s == M*(M+1) - N*(N+1)
{
var a := N;
s := 0;
while a < M
invariant N <= a <= M
invariant 2*s == a*(a+1) - N*(N+1)
{
a := a + 1;
s := s + a;
}
}
In summary, we have built the loop body from the loop invariant, and we designed this loop invariant by "replacing a constant with a variable" in the postcondition.
Rustan
You are running afoul of the curse of nonlinear arithmetic. Any time you rely on nontrivial properties of multiplication, Dafny will have a hard time with your program.
Here is one way to fix your specific proof. Sorry that it is so messy. I'm sure it can be cleaned up, but I just hacked something together to show you the idea.
function {:opaque} mul(a: int, b: int): int
{
a * b
}
lemma MulZero1(a: int)
ensures mul(0, a) == 0
{
reveal mul();
}
lemma MulNeg1(a: int, b: int)
ensures mul(-a, b) == -mul(a, b)
{
reveal mul();
}
lemma MulNeg2(a: int, b: int)
ensures mul(a, -b) == -mul(a, b)
{
reveal mul();
}
lemma MulInd(a: nat, b: int)
ensures mul(a, b) == if a == 0 then 0 else mul(a-1, b) + b
{
reveal mul();
}
lemma MulEven(a: int, b: int)
requires b % 2 == 0
decreases if a < 0 then -a + 1 else a
ensures mul(a, b) % 2 == 0
{
if a < 0 {
MulNeg1(a, b);
MulEven(-a, b);
} else if a == 0 {
MulZero1(b);
} else {
calc {
mul(a, b) % 2;
{ MulInd(a, b); }
(mul(a-1, b) + b) % 2;
mul(a-1, b) % 2;
{ MulEven(a-1, b); }
0;
}
}
}
lemma MulComm(a: int, b: int)
ensures mul(a, b) == mul(b, a)
{
reveal mul();
}
lemma MulAdjEven(a: int)
ensures mul(a, a + 1) % 2 == 0
{
var m := a % 2;
if m == 0 {
MulComm(a, a+1);
MulEven(a+1, a);
} else {
assert m == 1;
assert (a + 1) % 2 == 0;
MulEven(a, a+1);
}
}
method doingMath(N: int, M: int) returns (s: int)
requires N <= M //given precondition
ensures 2*s == mul(M,M+1) - mul(N,N+1) //given postcondition
{
var a: int := N;
var r: int := 0;
assert mul(a-N, N) + mul(a-N, (a-N)+1)/2 == 0 by {
reveal mul();
}
while a < M
invariant a <= M
invariant r == mul(a-N, N) + mul(a-N, (a-N)+1)/2
{
a := a+1;
r := r+a;
assert r == mul(a-N, N) + mul(a-N, (a-N)+1)/2 by {
reveal mul();
}
}
calc {
2*r;
2* (mul(M-N, N) + mul(M-N, (M-N)+1)/2);
{ MulAdjEven(M-N); }
2*mul(M-N, N) + mul(M-N, (M-N)+1);
{ reveal mul(); }
mul(M,M+1) - mul(N,N+1);
}
return r;
}
Multiplication is hard for Dafny, so we manually wrap it in an opaque function. This gives us fine-grained control of when Dafny is allowed to "know" that the function is really multiplication.
Then we can replace all the occurrences of multiplication in your method by calls to mul. This makes Dafny fail quickly. (That's a big improvement over timing out!) Then we can selectively reveal the definition of mul where we need it, or we can prove lemmas about mul.
The hardest lemma is MulEven. Try replacing its body/proof by reveal mul(); and you will see that Dafny times out. Instead, I had to prove it by induction. This proof itself necessitated several other lemmas about multiplication. Fortunately, all of them were easy.
You may also want to take a look at the math library developed as part of the IronFleet project here. (Start by reading the files whose names contain the word "nonlinear"; those are the lowest-level proofs that are closest to the axioms.) They use a similar approach to build up a large body of facts about multiplication (and division and modulo), so that those functions can remain opaque everywhere else in the codebase, improving Dafny's performance.
The overarching problem is that when I add modifies to a method, suddenly some of my loop invariants no longer check correctly.
I have worked around this by extracting that loop out into its own method, however that feels very hacky.
method Merge (arr : array<int>, l : int, m : int, r : int) returns (res : array<int>)
requires 0 <= l < m < r <= arr.Length
requires sorted_slice(arr, l, m);
requires sorted_slice(arr, m, r);
ensures sorted_slice(res, l, r)
{
var ia := l;
var ib := m;
res := new int[r - l];
var ri := 0;
while ri < res.Length
decreases res.Length - ri
invariant ri == (ia - l) + (ib - m)
//Ensure that the ia/ib is within the sorted slice at all times
invariant l <= ia <= m
invariant m <= ib <= r
// r[:ri] is sorted
invariant forall j, k :: (0 <= j <= k < ri) && (0 <= j <= k < res.Length) ==> res[j] <= res[k]
invariant forall ja, jr :: (ia <= ja < m) && (0 <= jr < ri < res.Length) ==> res[jr] <= arr[ja]
invariant forall jb, jr :: (ib <= jb < r) && (0 <= jr < ri < res.Length) ==> res[jr] <= arr[jb]
{
if ia >= m {
res[ri] := arr[ib];
ib := ib + 1;
ri := ri + 1;
} else if ib >= r {
res[ri] := arr[ia];
ia := ia + 1;
ri := ri + 1;
} else {
if arr[ia] < arr[ib]
{
res[ri] := arr[ia];
ia := ia + 1;
ri := ri + 1;
} else {
res[ri] := arr[ib];
ib := ib + 1;
ri := ri + 1;
}
}
}
...
}
Specifically the 4th and 5th loop invariants fail when I add modifies arr to Merge's signature.
Why might this be occurring? I can understand that I might need to add an invariant to the loop saying that it doesn't edit arr, however I can't find how to do that?
A loop inherits any modifies clause of the enclosing method [0]. So, if your method says modifies arr, then, in effect, so does your loop. This means that the verifier will treat the loop as if it may modify the elements of arr, whether or not the loop body actually does [1]. Therefore, you are indeed correct that you need to add something to the loop specification that says that the loop doesn't actually modify arr.
Your method is also allowed to modify the elements of res, because array res is "newly allocated" inside the method. This means that your loop is allowed to modify both arr and res, if your method says modifies arr.
So, you want to override the inherited modifies clause, so that you can restrict the effective modifies clause for the loop to be only res. To do that, write
modifies res
among the decreases and invariant clauses on the loop.
Fine points, just for reference:
[0] For a nested loop, the inner loop inherits the effective modifies clause of the enclosing loop.
[1] If the verifier can determine by a simple syntactic scan that the loop body couldn't possibly modify anything in the heap, then the verifier uses this fact, regardless of modifies clauses.
Btw, for your program, you can omit the explicit decreases clause, because Dafny will infer it for you.
A method that copies the negative elements of an array of integers into another array has the property that the set of elements in the result is a subset of the elements in the original array, which stays the same during the copy.
The problem in the code below is that, as soon as we write something in the result array, Dafny somehow forgets that the original set is unchanged.
How to fix this?
method copy_neg (a: array<int>, b: array<int>)
requires a != null && b != null && a != b
requires a.Length == b.Length
modifies b
{
var i := 0;
var r := 0;
ghost var sa := set j | 0 <= j < a.Length :: a[j];
while i < a.Length
invariant 0 <= r <= i <= a.Length
invariant sa == set j | 0 <= j < a.Length :: a[j]
{
if a[i] < 0 {
assert sa == set j | 0 <= j < a.Length :: a[j]; // OK
b[r] := a[i];
assert sa == set j | 0 <= j < a.Length :: a[j]; // KO!
r := r + 1;
}
i := i + 1;
}
}
Edit
Following James Wilcox's answer, replacing inclusions of sets with predicates on sequences is what works the best.
Here is the complete specification (for an array with distinct elements). The post-condition has to be detailed a bit in the loop invariant and a dumb assert remains in the middle of the loop, but all ghost variables are gone, which is great.
method copy_neg (a: array<int>, b: array<int>)
returns (r: nat)
requires a != null && b != null && a != b
requires a.Length <= b.Length
modifies b
ensures 0 <= r <= a.Length
ensures forall x | x in a[..] :: x < 0 <==> x in b[..r]
{
r := 0;
var i := 0;
while i < a.Length
invariant 0 <= r <= i <= a.Length
invariant forall x | x in b[..r] :: x < 0
invariant forall x | x in a[..i] && x < 0 :: x in b[..r]
{
if a[i] < 0 {
b[r] := a[i];
assert forall x | x in b[..r] :: x < 0;
r := r + 1;
}
i := i + 1;
}
}
This is indeed confusing. I will explain why Dafny has trouble proving this below, but first let me give a few ways to make it go through.
First workaround
One way to make the proof go through is to insert the following forall statement after the line b[r] := a[i];.
forall x | x in sa
ensures x in set j | 0 <= j < a.Length :: a[j]
{
var j :| 0 <= j < a.Length && x == old(a[j]);
assert x == a[j];
}
The forall statement is a proof that sa <= set j | 0 <= j < a.Length :: a[j]. I will come back to why this works below.
Second workaround
In general, when reasoning about arrays in Dafny, it is best to use the a[..] syntax to convert the array to a mathematical sequence, and then work with that sequence. If you really need to work with the set of elements, you can use set x | x in a[..], and you will have a better time than if you use set j | 0 <= j < a.Length :: a[j].
Systematically replacing set j | 0 <= j < a.Length :: a[j] with set x | x in a[..] causes your program to verify.
Third solution
Popping up a level to specifying your method, it seems like you don't actually need to mention the set of all elements. Instead, you can get away with saying something like "every element of b is an element of a". Or, more formally forall x | x in b[..] :: x in a[..]. This is not quite a valid postcondition for your method, because your method may not fill out all of b. Since I'm not sure what your other constraints are, I'll leave that to you.
Explanations
Dafny's sets with elements of type A are translated to Boogie maps [A]Bool, where an element maps to true iff it is in the set. Comprehensions such as set j | 0 <= j < a.Length :: a[j] are translated to Boogie maps whose definition involves an existential quantifier. This particular comprehension translates to a map that maps x to
exists j | 0 <= j < a.Length :: x == read($Heap, a, IndexField(j))
where the read expression is the Boogie translation of a[j], which, in particular, makes the heap explicit.
So, to prove that an element is in the set defined by the comprehension, Z3 needs to prove an existential quantifier, which is hard. Z3 uses triggers to prove such quantifiers, and Dafny tells Z3 to use the trigger read($Heap, a, IndexField(j)) when trying to prove this quantifier. This turns out to not be a great trigger choice, because it mentions the current value of the heap. Thus, when the heap changes (ie, after updating b[r]), the trigger may not fire, and you get a failing proof.
Dafny lets you customize the trigger it uses for set comprehensions using a {:trigger} attribute. Unfortunately, there is no great choice of trigger at the Dafny level. However, a reasonable trigger for this program at the Boogie/Z3 level would be just IndexField(j) (though this is likely a bad trigger for such expressions in general, since it is overly general). Z3 itself will infer this trigger if Dafny doesn't tell it otherwise. You can Dafny to get out of the way by saying {:autotriggers false}, like this
invariant sa == set j {:autotriggers false} | 0 <= j < a.Length :: a[j]
This solution is unsatisfying and requires detailed knowledge of Dafny's internals. But now that we've understood it, we can also understand the other workarounds I proposed.
For the first workaround, the proof goes through because the forall statement mentions a[j], which is the trigger. This causes Z3 to successfully prove the existential.
For the second workaround, we have simplified the set comprehension expression so that it no longer introduces an existential quantifier. Instead the comprehension set x | x in a[..], is translated to a map that maps x to
x in a[..]
(ignoring details of how a[..] is translated). This means that Z3 never has to prove an existential, so the otherwise very similar proof goes through.
The third solution works for similar reasons, since it uses no comprehensions and thus no problematic existential quantifiers/
I am building a function that counts of many times a character appears in a string after the nth position.
countCh ("aaabbbccc", 3, 'b')
val it: int = 2
In C, I would use an accumulator with a while loop. But I am trying to learn the F# functional face, where this approach is discouraged.
So I used guards to test few conditions and build the function:
let rec countCh (s:string, n:int, ch:char) =
match s, n, ch with
| (s, n, ch) when n > s.Length -> 0 //p1
| (s, n, ch) when n < 0 -> 0 //p2
| (s, n, ch) when s.[n] <> ch -> countCh(s, n + 1, ch) //p3
| (s, n, ch) when s.[n] = ch -> 1 + countCh(s, n + 1, ch) //p4
The coexistence of patterns 3 and 4 is problematic (impossible, I am afraid). Even if it compiles, I have not been able to make it work. How can this task functionally be handled?
First, the coexistence of these branches is not problematic. They don't conflict with each other. Why do you think that it's problematic? Is it because you get an "Incomplete pattern match" compiler warning? That warning does not tell you that the branches conflict, it tells you that the compiler can't prove that the four branches cover all possibilities. Or do you think that for some other reason? If you want your questions to be answered accurately, you'll have to ask them more clearly.
Second, you're abusing the pattern matching. Look: there are no patterns! The patterns in every branch are exactly the same, and trivial. Only guards are different. This looks very counterintuitively within a match, but would be plainly expressed with if..elif:
let rec countCh (s:string) n ch =
if n >= s.Length || n < 0 then 0
elif s.[n] = ch then 1 + countCh s (n + 1) ch
else countCh s (n + 1) ch
NOTE 1: see how I made the parameters curried? Always use curried form, unless there is a very strong reason to use tupled. Curried parameters are much more convenient to use on the caller side.
NOTE 2: your condition n > s.Length was incorrect: string indices go from 0 to s.Length-1, so the bail condition should be n >= s.Length. It is corrected in my code.
Finally, since this is an exercise, I must point out that the recursion is not tail recursion. Look at the second branch (in my code): it calls the function recursively and then adds one to the result. Since you have to do something with the result of the recursive call, the recursion can't be "tail". This means you risk stack overflow on very long inputs.
To make this into tail recursion, you need to turn the function "inside out", so to say. Instead of returning the result from every call, you need to pass it into every call (aka "accumulator"), and only return from the terminal case:
let rec countCh (s:string) n ch countSoFar =
if n >= s.Length || n < 0 then countSoFar
elif s.[n] = ch then countCh s (n+1) ch (countSoFar+1)
else countCh s (n+1) ch countSoFar
// Usage:
countCh "aaaabbbccc" 5 'b' 0
This way, every recursive call is the "last" call (i.e. the function doesn't do anything with the result, but passes it straight out to its own caller). This is called "tail recursion" and can be compiled to work in constant stack space (as opposed to linear).
I agree with the other answers, but I'd like to help you with your original question. You need to indent the function, and you have an off by one bug:
let rec countCh (s:string, n:int, ch:char) =
match s, n, ch with
| s, n, _ when n >= s.Length-1 -> 0 //p1
| s, _, _ when n < 0 -> 0 //p2
| s, n, ch when s.[n+1] <> ch -> countCh(s, n+2, ch) //p3
| s, n, ch when s.[n+1] = ch -> 1 + countCh(s, n+2, ch) //p4
I'd suggest to not write it yourself, but ask the library functions for help:
let countCh (s: string, n, c) =
s.Substring(n+1).ToCharArray()
|> Seq.filter ((=) c)
|> Seq.length
Or use Seq.skip, along with the fact that you can drop the conversion to character array:
let countCh (s: string, n, c) =
s
|> Seq.skip (n + 1)
|> Seq.filter ((=) c)
|> Seq.length
I am completely new to F# (started using it today) and relatively new to functional programming (I have minor experience with Lisp). I want to exit a function by returning a value when a certain condition is met so that the rest of the loop is not executed. Here is a C# illustration of what I want to do:
bool CheckRow (int n, int i)
{
for(int j = 0; j < 9; j++)
if (n == sudoku[i][j])
return false;
return true;
}
I tried implementing the same function in F# like this (sudoku is an array2D):
let CheckRow (n : int) (i : int) : bool =
for j = 0 to 8 do
if (n = sudoku.[i, j]) then
false
true
However, I get the following error at false within the if: "This expression was expected to have type unit but here has type bool". What is the proper way to "return" from within a F# function?
Higher-order functions are nice of course, but at some point someone has to write a loop (e.g. to implement the higher-order function), and that'll eventually be you, so it's nice to know how to write loops in F#. There is no early return from a for loop in F#, but other types of loops do allow this:
// While loop, imperative style
let checkRow n i =
let mutable clear = true
let mutable j = 0
while clear && j < 9 do
clear <- n <> sudoku.[i, j]
j <- j + 1
clear
// Tail-recursive style - more idiomatic F#
let checkRow n i =
let rec loop j =
if j = 9 then true
elif sudoku.[i, j] = n then false
else loop (j + 1)
loop 0
Normally you shouldn't need to break function earlier but rather end recursion on some case, otherwise call function recursively. Here recursion might be hidden because you are operating on lists or matrixes.
List.forall is one of those functions that implement recursion over the list and returns the result on the first occasion. You could write you function this way:
let CheckRow (expectedValue : int) (rowIndex : int) =
[0..8] |> List.forall (fun colIndex ->
sudoku.[rowIndex, colIndex] <> expectedValue)