Gave VisualRust another try and to see how far they got, I wrote a few lines of code. And as usual, the code causes me to write a question on stackoverflow...
See first, read my question later:
fn make_counter( state : &mut u32 ) -> Box<Fn()->u32>
{
Box::new(move || {let ret = *state; *state = *state + 1; ret })
}
fn test_make_counter() {
let mut cnt : u32 = 0;
{
let counter = make_counter( & mut cnt );
let x1 = counter();
let x2 = counter();
println!("x1 = {} x2 = {}",x1,x2);
}
}
fn alt_make_counter ( init : u32 ) -> Box<Fn()->u32> {
let mut state = init;
Box::new(move || {let ret = state; state = state + 1; ret })
}
fn test_alt_make_counter() {
let counter = alt_make_counter( 0u32 );
let x1 = counter();
let x2 = counter();
println!("x1 = {} x2 = {}",x1,x2);
}
fn main() {
test_make_counter();
test_alt_make_counter();
}
The difference between make_counter() and alt_make_counter() is, that in one case, the state is a pointer to a mutable u32 passed to the function and in the other case, it is a mutable u32 defined inside the function. As the test_make_counter() function shows clearly, there is no way, that the closure lives longer than the variable cnt. Even if I removed the block inside test_make_counter() they would still have the identical lifetime. With the block, the counter will die before cnt. And yet, Rust complains:
src\main.rs(4,2): error : captured variable state does not outlive the enclosing closure
src\main.rs(3,1): warning : note: captured variable is valid for the anonymous lifetime #1 defined on the block at 3:0
If you look at the alt_make_counter() function now, the lifetime of state should basically cause the same error message, right? If the code captures the state for the closure, it should not matter if the pointer is passed in or if the variable is bound inside the function, right? But obviously, those 2 cases are magically different.
Who can explain, why they are different (bug, feature, deep insight, ...?) and if there is a simple rule one can adopt which prevents wasting time over such issues now and then?
The difference is not in using a local variable vs. using a parameter. Parameters are perfectly ordinary locals. In fact, this version of alt_make_counter works1:
fn alt_make_counter (mut state: u32) -> Box<FnMut() -> u32> {
Box::new(move || {let ret = state; state = state + 1; ret })
}
The problem is that the closure in make_counter closes over a &mut u32 instead of u32. It doesn't have its own state, it uses an integer somewhere else as its scratch space. And thus it needs to worry about the lifetime of that location. The function signature needs to communicate that the closure can only work while it can still use the reference that was passed in. This can be expressed with a lifetime parameter:
fn make_counter<'a>(state: &'a mut u32) -> Box<FnMut() -> u32 + 'a> {
Box::new(move || {let ret = *state; *state = *state + 1; ret })
}
Note that 'a is also attached to the FnMut() -> u32 (though with a different syntax because it's a trait).
The simplest rule to avoid such trouble is to not use references when they cause problems. There is no good reason for this closure to borrow its state, so don't do it. I don't know whether you fall under this, but I've seen a bunch of people that were under the impression that &mut is the primary or only way to mutate something. That is wrong. You can just store it by value and then just mutate that directly by storing it, or the larger structure in which it is contained, in a local variable that is tagged as mut. A mutable reference is only useful if the results of the mutation needs to be shared with some other code and you can't just pass the new value to that code.
Of course, sometimes juggling references in complicated ways is necessary. Unfortunately there doesn't seem to be a quick and easy way to learn to deal with those confidently. It's a big pedagogic challenge, but so far it appears everyone just struggled for a while and then progressively had fewer problems as they get more experienced. No, there is no single simple rule that solves all lifetime woes.
1 The return type has to be FnMut in all cases. You just didn't get an error about that yet because your current error happens at an earlier stage in the compilation.
Related
I'm struggling to express my code in a way to please the borrow-checker.
I have a function create_task which creates a future of some database operations. There's a stream of values where each element needs to be inserted to a database within a transaction. The problem is sharing the transaction between multiple closures as it has also mutably borrowed the connection object.
#![feature(conservative_impl_trait)]
extern crate futures;
extern crate rusqlite;
use futures::prelude::*;
use futures::{future, stream};
use rusqlite::Connection;
fn main() {
let task = create_task();
task.wait().unwrap();
}
fn create_task() -> impl Future<Item = (), Error = ()> {
let mut conn = Connection::open("temp.db").unwrap();
conn.execute("CREATE TABLE IF NOT EXISTS temp (val INTEGER)", &[]).unwrap();
// tx takes a mut ref to conn!
let tx = conn.transaction().unwrap();
stream::iter_ok::<_, ()>(vec![1, 2, 3])
.for_each(|val| {
// tx borrowed here!
tx.execute("INSERT INTO temp (val) VALUES (?1)", &[&val]).unwrap();
future::ok(())
})
.map(|_| {
// tx moved/consumed here!
tx.commit().unwrap();
})
}
There are multiple issues with the code:
conn does not live long enough. It needs to be also moved to the closures. Perhaps as an Rc<Connection> because of two closures?
conn can't be simply shared as an Rc because of mutability requirements. Perhaps Rc<RefCell<Connection>> is a more suitable type?
the borrow-checker does not know that the borrow to tx ends after the first for_each closure, therefore it cannot be moved to the second map closure. Once again, moving it as Rc<Transaction> to both closures might be reasonable?
I've been fiddling around with those ideas and know that the desired lifetimes are possible and make sense, but have not been able to express my code to the compiler in a correct way.
I believe your first problem is that you haven't fully grasped how lazy futures are. You are creating a Connection inside of create_task, taking a reference to it, putting that reference into a stream/future, then attempting to return that future. None of the closures have even executed at this point.
You cannot return a reference to a value created in a function. Don't try to store the transaction and the connection in the same struct, either.
Instead, accept a reference to a Connection and return a Future containing that lifetime.
The next problem is that the compiler doesn't know how the closures will be called or in what order. Instead of trying to close over the transaction, let it "flow" from one to the other, letting the ownership system ensure that it's always in the right place.
#![feature(conservative_impl_trait)]
extern crate futures;
extern crate rusqlite;
use futures::prelude::*;
use futures::{future, stream};
use rusqlite::Connection;
fn main() {
let mut conn = Connection::open("temp.db").unwrap();
conn.execute("CREATE TABLE IF NOT EXISTS temp (val INTEGER)", &[]).unwrap();
let task = create_task(&mut conn);
task.wait().unwrap();
}
fn create_task<'a>(conn: &'a mut rusqlite::Connection) -> impl Future<Item = (), Error = ()> + 'a {
let tx = conn.transaction().unwrap();
stream::iter_ok::<_, ()>(vec![1, 2, 3])
.fold(tx, |tx, val| {
tx.execute("INSERT INTO temp (val) VALUES (?1)", &[&val]).unwrap();
future::ok(tx)
})
.map(move |tx| {
tx.commit().unwrap();
})
}
A giant word of warning: If execute isn't asynchronous, you really shouldn't be using it inside of a future like that. Any blocking operation will cause all of your futures to stall out. You probably should be running the synchronous workload on a separate thread / threadpool.
See also:
Is there any way to return a reference to a variable created in a function?
How to store a SqliteConnection and SqliteStatement objects in the same struct in Rust?
Why can't I store a value and a reference to that value in the same struct?
I'm trying to write a function which pushes an element onto the end of a sorted vector only if the element is larger than the last element already in the vector, otherwise returns an error with a ref to the largest element. This doesn't seem to violate any borrowing rules as far as I cant tell, but the borrow checker doesn't like it. I don't understand why.
struct MyArray<K, V>(Vec<(K, V)>);
impl<K: Ord, V> MyArray<K, V> {
pub fn insert_largest(&mut self, k: K, v: V) -> Result<(), &K> {
{
match self.0.iter().next_back() {
None => (),
Some(&(ref lk, _)) => {
if lk > &k {
return Err(lk);
}
}
};
}
self.0.push((k, v));
Ok(())
}
}
error[E0502]: cannot borrow `self.0` as mutable because it is also borrowed as immutable
--> src/main.rs:15:9
|
6 | match self.0.iter().next_back() {
| ------ immutable borrow occurs here
...
15 | self.0.push((k, v));
| ^^^^^^ mutable borrow occurs here
16 | Ok(())
17 | }
| - immutable borrow ends here
Why doesn't this work?
In response to Paolo Falabella's answer.
We can translate any function with a return statement into one without a return statement as follows:
fn my_func() -> &MyType {
'inner: {
// Do some stuff
return &x;
}
// And some more stuff
}
Into
fn my_func() -> &MyType {
let res;
'outer: {
'inner: {
// Do some stuff
res = &x;
break 'outer;
}
// And some more stuff
}
res
}
From this, it becomes clear that the borrow outlives the scope of 'inner.
Is there any problem with instead using the following rewrite for the purpose of borrow-checking?
fn my_func() -> &MyType {
'outer: {
'inner: {
// Do some stuff
break 'outer;
}
// And some more stuff
}
panic!()
}
Considering that return statements preclude anything from happening afterwards which might otherwise violate the borrowing rules.
If we name lifetimes explicitly, the signature of insert_largest becomes fn insert_largest<'a>(&'a mut self, k: K, v: V) -> Result<(), &'a K>. So, when you create your return type &K, its lifetime will be the same as the &mut self.
And, in fact, you are taking and returning lk from inside self.
The compiler is seeing that the reference to lk escapes the scope of the match (as it is assigned to the return value of the function, so it must outlive the function itself) and it can't let the borrow end when the match is over.
I think you're saying that the compiler should be smarter and realize that the self.0.push can only ever be reached if lk was not returned. But it is not. And I'm not even sure how hard it would be to teach it that sort of analysis, as it's a bit more sophisticated than the way I understand the borrow checker reasons today.
Today, the compiler sees a reference and basically tries to answer one question ("how long does this live?"). When it sees that your return value is lk, it assigns lk the lifetime it expects for the return value from the fn's signature ('a with the explicit name we gave it above) and calls it a day.
So, in short:
should an early return end the mutable borrow on self? No. As said the borrow should extend outside of the function and follow its return value
is the borrow checker a bit too strict in the code that goes from the early return to the end of the function? Yes, I think so. The part after the early return and before the end of the function is only reachable if the function has NOT returned early, so I think you have a point that the borrow checked might be less strict with borrows in that specific area of code
do I think it's feasible/desirable to change the compiler to enable that pattern? I have no clue. The borrow checker is one of the most complex pieces of the Rust compiler and I'm not qualified to give you an answer on that. This seems related to (and might even be a subset of) the discussion on non-lexical borrow scopes, so I encourage you to look into it and possibly contribute if you're interested in this topic.
For the time being I'd suggest just returning a clone instead of a reference, if possible. I assume returning an Err is not the typical case, so performance should not be a particular worry, but I'm not sure how the K:Clone bound might work with the types you're using.
impl <K, V> MyArray<K, V> where K:Clone + Ord { // 1. now K is also Clone
pub fn insert_largest(&mut self, k: K, v: V) ->
Result<(), K> { // 2. returning K (not &K)
match self.0.iter().next_back() {
None => (),
Some(&(ref lk, _)) => {
if lk > &k {
return Err(lk.clone()); // 3. returning a clone
}
}
};
self.0.push((k, v));
Ok(())
}
}
Why does returning early not finish outstanding borrows?
Because the current implementation of the borrow checker is overly conservative.
Your code works as-is once non-lexical lifetimes are enabled, but only with the experimental "Polonius" implementation. Polonius is what enables conditional tracking of borrows.
I've also simplified your code a bit:
#![feature(nll)]
struct MyArray<K, V>(Vec<(K, V)>);
impl<K: Ord, V> MyArray<K, V> {
pub fn insert_largest(&mut self, k: K, v: V) -> Result<(), &K> {
if let Some((lk, _)) = self.0.iter().next_back() {
if lk > &k {
return Err(lk);
}
}
self.0.push((k, v));
Ok(())
}
}
I am working through this Rust tutorial, and I'm trying to solve this problem:
Implement a function, incrementMut that takes as input a vector of integers and modifies the values of the original list by incrementing each value by one.
This seems like a fairly simple problem, yes?
I have been trying to get a solution to compile for a while now, and I'm beginning to lose hope. This is what I have so far:
fn main() {
let mut p = vec![1i, 2i, 3i];
increment_mut(p);
for &x in p.iter() {
print!("{} ", x);
}
println!("");
}
fn increment_mut(mut x: Vec<int>) {
for &mut i in x.iter() {
i += 1;
}
}
This is what the compiler says when I try to compile:
Compiling tut2 v0.0.1 (file:///home/nate/git/rust/tut2)
/home/nate/git/rust/tut2/src/main.rs:5:12: 5:13 error: use of moved value: `p`
/home/nate/git/rust/tut2/src/main.rs:5 for &x in p.iter() {
^
/home/nate/git/rust/tut2/src/main.rs:3:16: 3:17 note: `p` moved here because it has type `collections::vec::Vec<int>`, which is non-copyable
/home/nate/git/rust/tut2/src/main.rs:3 increment_mut(p);
^
error: aborting due to previous error
Could not compile `tut2`.
To learn more, run the command again with --verbose.
I also tried a version with references:
fn main() {
let mut p = vec![1i, 2i, 3i];
increment_mut(&p);
for &x in p.iter() {
print!("{} ", x);
}
println!("");
}
fn increment_mut(x: &mut Vec<int>) {
for &mut i in x.iter() {
i += 1i;
}
}
And the error:
Compiling tut2 v0.0.1 (file:///home/nate/git/rust/tut2)
/home/nate/git/rust/tut2/src/main.rs:3:16: 3:18 error: cannot borrow immutable dereference of `&`-pointer as mutable
/home/nate/git/rust/tut2/src/main.rs:3 increment_mut(&p);
^~
error: aborting due to previous error
Could not compile `tut2`.
To learn more, run the command again with --verbose.
I feel like I'm missing some core idea about memory ownership in Rust, and it's making solving trivial problems like this very difficult, could someone shed some light on this?
There are a few mistakes in your code.
increment_mut(&p), given a p that is Vec<int>, would require the function increment_mut(&Vec<int>); &-references and &mut-references are completely distinct things syntactically, and if you want a &mut-reference you must write &mut p, not &p.
You need to understand patterns and how they operate; for &mut i in x.iter() will not do what you intend it to: what it will do is take the &int that each iteration of x.iter() produces, dereference it (the &), copying the value (because int satisfies Copy, if you tried it with a non-Copy type like String it would not compile), and place it in the mutable variable i (mut i). That is, it is equivalent to for i in x.iter() { let mut i = *i; … }. The effect of this is that i += 1 is actually just incrementing a local variable and has no effect on the vector. You can fix this by using iter_mut, which produces &mut int rather than &int, and changing the &mut i pattern to just i and the i += 1 to *i += 1, meaning “change the int inside the &mut int.
You can also switch from using &mut Vec<int> to using &mut [int] by calling .as_mut_slice() on your vector. This is a better practice; you should practically never need a reference to a vector as that is taking two levels of indirection where only one is needed. Ditto for &String—it’s exceedingly rare, you should in such cases work with &str.
So then:
fn main() {
let mut p = vec![1i, 2i, 3i];
increment_mut(p.as_mut_slice());
for &x in p.iter() {
print!("{} ", x);
}
println!("");
}
fn increment_mut(x: &mut [int]) {
for i in x.iter_mut() {
*i += 1;
}
}
In the following code, does the "box 5i" get properly freed when exiting the "main" scope? The wording on their pointer guide seems to indicate that variables with box types act as if there's an automatic "free()" call when the variable goes out of scope. However, if you "free()" on "a" in this code, it would only end up freeing the "box 8i" that is on the heap. What happens to the "box 5i" that "a" was originally pointing to?
fn foo(a: &mut Box<int>) {
*a = box 8i;
}
fn main() {
let mut a = box 5i;
println!("{}", a); // -> "5"
foo(&mut a);
println!("{}", a); // -> "8"
}
By default, overwriting a memory location will run the destructor of the old value. For Box<...> this involves running the destructor of the contents (which is nothing for an int) and freeing the allocation, so if a has type &mut Box<T>, *a = box value is equivalent to (in C):
T_destroy(**a);
free(*a);
*a = malloc(sizeof T);
**a = value;
In some sense, the answer to your question is yes, because the type system guarantees that *a = box ... can only work if a is the only reference to the old Box, but unlike most garbage collected/managed languages this is all determined statically, not dynamically (it is a direct consequence of ownership and linear/affine types).
I am writing a compiler of mini-pascal in Ocaml. I would like my compiler to accept the following code for instance:
program test;
var
a,b : boolean;
n : integer;
begin
...
end.
I have difficulties in dealing with the declaration of variables (the part following var). At the moment, the type of variables is defined like this in sib_syntax.ml:
type s_var =
{ s_var_name: string;
s_var_type: s_type;
s_var_uniqueId: s_uniqueId (* key *) }
Where s_var_uniqueId (instead of s_var_name) is the unique key of the variables. My first question is, where and how I could implement the mechanism of generating a new id (actually by increasing the biggest id by 1) every time I have got a new variable. I am wondering if I should implement it in sib_parser.mly, which probably involves a static variable cur_id and the modification of the part of binding, again don't know how to realize them in .mly. Or should I implement the mechanism at the next stage - the interpreter.ml? but in this case, the question is how to make the .mly consistent with the type s_var, what s_var_uniqueId should I provide in the part of binding?
Another question is about this part of statement in .mly:
id = IDENT COLONEQ e = expression
{ Sc_assign (Sle_var {s_var_name = id; s_var_type = St_void}, e) }
Here, I also need to provide the next level (the interpreter.ml) a variable of which I only know the s_var_name, so what could I do regarding its s_var_type and s_var_uniqueId here?
Could anyone help? Thank you very much!
The first question to ask yourself is whether you actually need an unique id. From my experience, they're almost never necessary or even useful. If what you're trying to do is making variables unique through alpha-equivalence, then this should happen after parsing is complete, and will probably involve some form of DeBruijn indices instead of unique identifiers.
Either way, a function which returns a new integer identifier every time it is called is:
let unique =
let last = ref 0 in
fun () -> incr last ; !last
let one = unique () (* 1 *)
let two = unique () (* 2 *)
So, you can simply assign { ... ; s_var_uniqueId = unique () } in your Menhir rules.
The more important problem you're trying to solve here is that of variable binding. Variable x is defined in one location and used in another, and you need to determine that it happens to be the same variable in both places. There are many ways of doing this, one of them being to delay the binding until the interpreter. I'm going to show you how to deal with this during parsing.
First, I'm going to define a context: it's a set of variables that allows you to easily retrieve a variable based on its name. You might want to create it with hash tables or maps, but to keep things simple I will be using List.assoc here.
type s_context = {
s_ctx_parent : s_context option ;
s_ctx_bindings : (string * (int * s_type)) list ;
s_ctx_size : int ;
}
let empty_context parent = {
s_ctx_parent = parent ;
s_ctx_bindings = [] ;
s_ctx_size = 0
}
let bind v_name v_type ctx =
try let _ = List.assoc ctx.s_ctx_bindings v_name in
failwith "Variable is already defined"
with Not_found ->
{ ctx with
s_ctx_bindings = (v_name, (ctx.s_ctx_size, v_type))
:: ctx.s_ctx_bindings ;
s_ctx_size = ctx.s_ctx_size + 1 }
let rec find v_name ctx =
try 0, List.assoc ctx.s_ctx_bindings v_name
with Not_found ->
match ctx.s_ctx_parent with
| Some parent -> let depth, found = find v_name parent in
depth + 1, found
| None -> failwith "Variable is not defined"
So, bind adds a new variable to the current context, find looks for a variable in the current context and its parents, and returns both the bound data and the depth at which it was found. So, you could have all global variables in one context, then all parameters of a function in another context that has the global context as its parent, then all local variables in a function (when you'll have them) in a third context that has the function's main context as the parent, and so on.
So, for instance, find 'x' ctx will return something like 0, (3, St_int) where 0 is the DeBruijn index of the variable, 3 is the position of the variable in the context identified by the DeBruijn index, and St_int is the type.
type s_var = {
s_var_deBruijn: int;
s_var_type: s_type;
s_var_pos: int
}
let find v_name ctx =
let deBruijn, (pos, typ) = find v_name ctx in
{ s_var_deBruijn = deBruijn ;
s_var_type = typ ;
s_var_pos = pos }
Of course, you need your functions to store their context, and make sure that the first argument is the variable at position 0 within the context:
type s_fun =
{ s_fun_name: string;
s_fun_type: s_type;
s_fun_params: context;
s_fun_body: s_block; }
let context_of_paramlist parent paramlist =
List.fold_left
(fun ctx (v_name,v_type) -> bind v_name v_type ctx)
(empty_context parent)
paramlist
Then, you can change your parser to take into account the context. The trick is that instead of returning an object representing part of your AST, most of your rules will return a function that takes a context as an argument and returns an AST node.
For instance:
int_expression:
(* Constant : ignore the context *)
| c = INT { fun _ -> Se_const (Sc_int c) }
(* Variable : look for the variable inside the contex *)
| id = IDENT { fun ctx -> Se_var (find id ctx) }
(* Subexpressions : pass the context to both *)
| e1 = int_expression o = operator e2 = int_expression
{ fun ctx -> Se_binary (o, e1 ctx, e2 ctx) }
;
So, you simply propagate the context "down" recursively through the expressions. The only clever parts are those when new contexts are created (you don't have this syntax yet, so I'm just adding a placeholder):
| function_definition_expression (args, body)
{ fun ctx -> let ctx = context_of_paramlist (Some ctx) args in
{ s_fun_params = ctx ;
s_fun_body = body ctx } }
As well as the global context (the program rule itself does not return a function, but the block rule does, and so a context is created from the globals and provided).
prog:
PROGRAM IDENT SEMICOLON
globals = variables
main = block
DOT
{ let ctx = context_of_paramlist None globals in
{ globals = ctx;
main = main ctx } }
All of this makes the implementation of your interpreter much easier due to the DeBruijn indices: you can have a "stack" which holds your values (of type value) defined as:
type stack = value array list
Then, reading and writing variable x is as simple as:
let read stack x =
(List.nth stack x.s_var_deBruijn).(x.s_var_pos)
let write stack x value =
(List.nth stack x.s_var_deBruijn).(x.s_var_pos) <- value
Also, since we made sure that function parameters are in the same order as their position in the function context, if you want to call function f and its arguments are stored in the array args, then constructing the stack is as simple as:
let inner_stack = args :: stack in
(* Evaluate f.s_fun_body with inner_stack here *)
But I'm sure you'll have a lot more questions to ask when you start working on your interpeter ;)
How to create a global id generator:
let unique =
let counter = ref (-1) in
fun () -> incr counter; !counter
Test:
# unique ();;
- : int = 0
# unique ();;
- : int = 1
Regarding your more general design question: it seems that your data representation does not faithfully represent the compiler phases. If you must return a type-aware data-type (with this field s_var_type) after the parsing phase, something is wrong. You have two choices:
devise a more precise data representation for the post-parsing AST, that would be different from the post-typing AST, and not have those s_var_type fields. Typing would then be a conversion from the untyped to the typed AST. This is a clean solution that I would recommend.
admit that you must break the data representation semantics because you don't have enough information at this stage, and try to be at peace with the idea of returning garbage such as St_void after the parsing phase, to reconstruct the correct information later. This is less typed (as you have an implicit assumption on your data which is not apparent in the type), more pragmatic, ugly but sometimes necessary. I don't think it's the right decision in this case, but you will encounter situation where it's better to be a bit less typed.
I think the specific choice of unique id handling design depends on your position on this more general question, and your concrete decisions about types. If you choose a finer-typed representation of post-parsing AST, it's your choice to decide whether to include unique ids or not (I would, because generating a unique ID is dead simple and doesn't need a separate pass, and I would rather slightly complexify the grammar productions than the typing phase). If you choose to hack the type field with a dummy value, it's also reasonable to do that for variable ids if you wish to, putting 0 as a dummy value and defining it later; but still I personally would do that in the parsing phase.