Passing a closure that modifies its environment to a function in Rust - closures

I have a closure that captures and modifies its environment. I want to pass this closure to a function that accepts closures:
fn main() {
let mut integer = 5;
let mut closure_variable = || -> i32 {
integer += 1;
integer
};
execute_closure(&mut closure_variable);
}
fn execute_closure(closure_argument: &mut Fn() -> i32) {
let result = closure_argument();
println!("Result of closure: {}", result);
}
Because the closure modifies its environment, this fails:
error[E0525]: expected a closure that implements the `Fn` trait, but this closure only implements `FnMut`
--> src/main.rs:3:32
|
3 | let mut closure_variable = || -> i32 {
| ________________________________^
4 | | integer += 1;
5 | | integer
6 | | };
| |_____^
7 | execute_closure(&mut closure_variable);
| --------------------- the requirement to implement `Fn` derives from here
|
note: closure is `FnMut` because it mutates the variable `integer` here
--> src/main.rs:4:9
|
4 | integer += 1;
| ^^^^^^^
As I understand from When does a closure implement Fn, FnMut and FnOnce?, this means that my closure actually is expanded to a struct that implements the trait FnMut. This trait is mutable, meaning calling the function changes the (implicit) object. I think this correct, because the variable integer should be modified after calling execute_closure().
How do I convince the compiler this is okay and that I actually want to call a FnMut function? Or is there something fundamentally wrong with how I use Rust in this example?

If you can change the function that accepts the closure...
Accept a FnMut instead of a Fn:
fn main() {
let mut integer = 5;
execute_closure(|| {
integer += 1;
integer
});
}
fn execute_closure<F>(mut closure_argument: F)
where
F: FnMut() -> i32,
{
let result = closure_argument();
println!("Result of closure: {}", result);
}
If you can not change the function that accepts the closure...
Use interior mutability provided by types like Cell or RefCell:
use std::cell::Cell;
fn main() {
let integer = Cell::new(5);
execute_closure(|| {
integer.set(integer.get() + 1);
integer.get()
});
}
fn execute_closure<F>(closure_argument: F)
where
F: Fn() -> i32,
{
let result = closure_argument();
println!("Result of closure: {}", result);
}
Or is there something fundamentally wrong with how I use Rust in this example?
Perhaps. An argument of type &mut Fn() -> i32 cannot mutate the variables it has closed over, so the error message makes sense to me.
It's kind of similar to the type &mut &u8 — you could alter the outer reference to point to another immutable reference, but you cannot "ignore" the inner immutability and change the numeric value.
Aside:
The original code uses dynamic dispatch because there is a trait object that provides indirection. In many cases you'd see this version that I posted above, which uses static dispatch and can be monomorphized. I've also inlined the closure as that's the normal syntax.
Here's the original version with just enough changes to work:
fn main() {
let mut integer = 5;
let mut closure_variable = || -> i32 {
integer += 1;
integer
};
execute_closure(&mut closure_variable);
}
fn execute_closure(closure_argument: &mut FnMut() -> i32) {
let result = closure_argument();
println!("Result of closure: {}", result);
}

Related

Returning closures which capture outer variables in Rust

As the title states I am looking to return a closure from a function which has some initial, mutable state. In the following examples CowRow is a struct with a time field. It also has a String field, so is thus not copyable. Concretely, I would like a function that looks something like:
pub fn agg1() -> Box<Fn(&CowRow)> {
let res = 0;
Box::new(move |r| { res += r.time; })
}
Of course, this produces the error:
src/queries.rs:9:25: 9:38 error: cannot assign to captured outer variable in an `Fn` closure
src/queries.rs:9 Box::new(move |r| { res += r.time; })
^~~~~~~~~~~~~
src/queries.rs:9:14: 9:41 help: consider changing this closure to take self by mutable reference
src/queries.rs:9 Box::new(move |r| { res += r.time; })
^~~~~~~~
It is my understanding that Rust needs to know about the size of returned values and because closures borrow their stack frame from their environment we need to introduce the Box and move to get a size for the return and put the closure on the heap.
Is there some way to also put res on the heap in this closures environment? Or otherwise allow for this behaviour? Of course I have looked at: Cannot borrow captured outer variable in an `Fn` closure as mutable but this seems overly complicated and it's not clear to me how this would perform in the case of multiple threads running this function simultaneously.
Another technique I tried was to change the closure to take a mutable reference to an i32 which I can initialise outside of the agg function. Example:
pub fn agg0() -> Box<Fn(&CowRow, &mut i32)> {
Box::new(move |r, &mut acc| { acc += r.time; })
}
However, this produces the error:
src/queries.rs:4:35: 4:48 error: re-assignment of immutable variable `acc` [E0384]
src/queries.rs:4 Box::new(move |r, &mut acc| { acc += r.time; })
^~~~~~~~~~~~~
src/queries.rs:4:35: 4:48 help: run `rustc --explain E0384` to see a detailed explanation
src/queries.rs:4:28: 4:31 note: prior assignment occurs here
src/queries.rs:4 Box::new(move |r, &mut acc| { acc += r.time; })
This one is a total mystery to me.
You need to do two things here: make res mutable, and return an FnMut closure, not an Fn one:
pub struct CowRow {
time: u64,
}
pub fn agg1() -> Box<FnMut(&CowRow) -> u64> {
let mut res = 0;
Box::new(move |r| { res += r.time; res })
}
fn main() {
let mut c = agg1();
let moo = CowRow { time: 2 };
println!("{:?}", c(&moo));
println!("{:?}", c(&moo));
println!("{:?}", c(&moo));
}
The Fn trait forbids the implementor from changing itself when invoked. Since this closure is modifying its own state, this means it cannot be Fn [1]. Instead, you need to use FnMut which does allow mutation of the closure's captured environment.
[1]: Unless you involve interior mutability, of course.
DK. already said how to fix agg1, but I wanted to explain what's wrong with agg0, for completeness.
pub fn agg0() -> Box<Fn(&CowRow, &mut i32)> {
Box::new(move |r, &mut acc| { acc += r.time; })
}
We can infer from agg0's return type that the type of the closure's second parameter is &mut i32. &mut acc is a pattern that deconstructs the mutable reference, defining acc as an i32, initialized to a copy of the referenced value. You can't mutate it, because you didn't define acc to be mutable (you'd need to write &mut mut acc instead of &mut acc), but that's not what you want anyway, because then you'd be mutating a copy. What you want is to mutate the pointed-to integer, so you need to define your closure like this:
pub fn agg0() -> Box<Fn(&CowRow, &mut i32)> {
Box::new(move |r, acc| { *acc += r.time; })
}
Here, the type of acc is &mut i32, so in order to mutate the i32, we need to dereference the pointer first (this yields an lvalue that refers to the i32 behind the pointer; it's not a copy!).

How do I return a reference to a closure's argument from the closure?

I am a Rust newbie, and don't understand all the rules for lifetime elision and inference. I can't seem to get returning a reference into an argument from a closure to work, and the errors don't help much for someone with my amount of knowledge.
I can use a proper function with lifetime annotations in place of a closure, but can't figure out a way to annotate these lifetimes on the closure.
struct Person<'a> {
name: &'a str,
}
impl<'a> Person<'a> {
fn map<F, T>(&'a self, closure: F) -> T
where F: Fn(&'a Person) -> T
{
closure(self)
}
}
fn get_name<'a>(person: &'a Person) -> &'a str {
person.name
}
fn main() {
let p = Person { name: "hello" };
let s: &str = p.map(|person| person.name); // Does not work
// let s: &str = p.map(get_name); // Works
println!("{:?}", s);
}
And here is the compiler error:
error[E0495]: cannot infer an appropriate lifetime due to conflicting requirements
--> src/main.rs:19:34
|
19 | let s: &str = p.map(|person| person.name); // Does not work
| ^^^^^^^^^^^
|
note: first, the lifetime cannot outlive the anonymous lifetime #1 defined on the body at 19:33...
--> src/main.rs:19:34
|
19 | let s: &str = p.map(|person| person.name); // Does not work
| ^^^^^^^^^^^
note: ...so that expression is assignable (expected &str, found &str)
--> src/main.rs:19:34
|
19 | let s: &str = p.map(|person| person.name); // Does not work
| ^^^^^^^^^^^
note: but, the lifetime must be valid for the method call at 19:18...
--> src/main.rs:19:19
|
19 | let s: &str = p.map(|person| person.name); // Does not work
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
note: ...so that pointer is not dereferenced outside its lifetime
--> src/main.rs:19:19
|
19 | let s: &str = p.map(|person| person.name); // Does not work
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^
What is happening here, and how to change the closure for it to work?
I think I was confusing the lifetime of reference to the Person with the lifetime parameter of the struct (of the &str reference). This compiles:
struct Person<'a> {
name: &'a str,
}
impl<'a> Person<'a> {
fn map<F, T>(&self, closure: F) -> T
where F: Fn(&Person<'a>) -> T
{
closure(self)
}
}
fn get_name<'a>(person: &Person<'a>) -> &'a str {
person.name
}
fn main() {
let p = Person { name: "hello" };
println!("{:?}", p.map(|person| person.name));
println!("{:?}", p.map(get_name));
}
I'm not sure if my understanding of the lifetime annotations is correct, and perhaps someone else can pick apart the compiler error above. The language is extremely confusing.

How can I move a captured variable into a closure within a closure?

This code is an inefficient way of producing a unique set of items from an iterator. To accomplish this, I am attempting to use a Vec to keep track of values I've seen. I believe that this Vec needs to be owned by the innermost closure:
fn main() {
let mut seen = vec![];
let items = vec![vec![1i32, 2], vec![3], vec![1]];
let a: Vec<_> = items
.iter()
.flat_map(move |inner_numbers| {
inner_numbers.iter().filter_map(move |&number| {
if !seen.contains(&number) {
seen.push(number);
Some(number)
} else {
None
}
})
})
.collect();
println!("{:?}", a);
}
However, compilation fails with:
error[E0507]: cannot move out of captured outer variable in an `FnMut` closure
--> src/main.rs:8:45
|
2 | let mut seen = vec![];
| -------- captured outer variable
...
8 | inner_numbers.iter().filter_map(move |&number| {
| ^^^^^^^^^^^^^^ cannot move out of captured outer variable in an `FnMut` closure
This is a little surprising, but isn't a bug.
flat_map takes a FnMut as it needs to call the closure multiple times. The code with move on the inner closure fails because that closure is created multiple times, once for each inner_numbers. If I write the closures in explicit form (i.e. a struct that stores the captures and an implementation of one of the closure traits) your code looks (a bit) like
struct OuterClosure {
seen: Vec<i32>
}
struct InnerClosure {
seen: Vec<i32>
}
impl FnMut(&Vec<i32>) -> iter::FilterMap<..., InnerClosure> for OuterClosure {
fn call_mut(&mut self, (inner_numbers,): &Vec<i32>) -> iter::FilterMap<..., InnerClosure> {
let inner = InnerClosure {
seen: self.seen // uh oh! a move out of a &mut pointer
};
inner_numbers.iter().filter_map(inner)
}
}
impl FnMut(&i32) -> Option<i32> for InnerClosure { ... }
Which makes the illegality clearer: attempting to move out of the &mut OuterClosure variable.
Theoretically, just capturing a mutable reference is sufficient, since the seen is only being modified (not moved) inside the closure. However things are too lazy for this to work...
error: lifetime of `seen` is too short to guarantee its contents can be safely reborrowed
--> src/main.rs:9:45
|
9 | inner_numbers.iter().filter_map(|&number| {
| ^^^^^^^^^
|
note: `seen` would have to be valid for the method call at 7:20...
--> src/main.rs:7:21
|
7 | let a: Vec<_> = items.iter()
| _____________________^
8 | | .flat_map(|inner_numbers| {
9 | | inner_numbers.iter().filter_map(|&number| {
10| | if !seen.contains(&number) {
... |
17| | })
18| | .collect();
| |__________________^
note: ...but `seen` is only valid for the lifetime as defined on the body at 8:34
--> src/main.rs:8:35
|
8 | .flat_map(|inner_numbers| {
| ___________________________________^
9 | | inner_numbers.iter().filter_map(|&number| {
10| | if !seen.contains(&number) {
11| | seen.push(number);
... |
16| | })
17| | })
| |_________^
Removing the moves makes the closure captures work like
struct OuterClosure<'a> {
seen: &'a mut Vec<i32>
}
struct InnerClosure<'a> {
seen: &'a mut Vec<i32>
}
impl<'a> FnMut(&Vec<i32>) -> iter::FilterMap<..., InnerClosure<??>> for OuterClosure<'a> {
fn call_mut<'b>(&'b mut self, inner_numbers: &Vec<i32>) -> iter::FilterMap<..., InnerClosure<??>> {
let inner = InnerClosure {
seen: &mut *self.seen // can't move out, so must be a reborrow
};
inner_numbers.iter().filter_map(inner)
}
}
impl<'a> FnMut(&i32) -> Option<i32> for InnerClosure<'a> { ... }
(I've named the &mut self lifetime in this one, for pedagogical purposes.)
This case is definitely more subtle. The FilterMap iterator stores the closure internally, meaning any references in the closure value (that is, any references it captures) have to be valid as long as the FilterMap values are being thrown around, and, for &mut references, any references have to be careful to be non-aliased.
The compiler can't be sure flat_map won't, e.g. store all the returned iterators in a Vec<FilterMap<...>> which would result in a pile of aliased &muts... very bad! I think this specific use of flat_map happens to be safe, but I'm not sure it is in general, and there's certainly functions with the same style of signature as flat_map (e.g. map) would definitely be unsafe. (In fact, replacing flat_map with map in the code gives the Vec situation I just described.)
For the error message: self is effectively (ignoring the struct wrapper) &'b mut (&'a mut Vec<i32>) where 'b is the lifetime of &mut self reference and 'a is the lifetime of the reference in the struct. Moving the inner &mut out is illegal: can't move an affine type like &mut out of a reference (it would work with &Vec<i32>, though), so the only choice is to reborrow. A reborrow is going through the outer reference and so cannot outlive it, that is, the &mut *self.seen reborrow is a &'b mut Vec<i32>, not a &'a mut Vec<i32>.
This makes the inner closure have type InnerClosure<'b>, and hence the call_mut method is trying to return a FilterMap<..., InnerClosure<'b>>. Unfortunately, the FnMut trait defines call_mut as just
pub trait FnMut<Args>: FnOnce<Args> {
extern "rust-call" fn call_mut(&mut self, args: Args) -> Self::Output;
}
In particular, there's no connection between the lifetime of the self reference itself and the returned value, and so it is illegal to try to return InnerClosure<'b> which has that link. This is why the compiler is complaining that the lifetime is too short to be able to reborrow.
This is extremely similar to the Iterator::next method, and the code here is failing for basically the same reason that one cannot have an iterator over references into memory that the iterator itself owns. (I imagine a "streaming iterator" (iterators with a link between &mut self and the return value in next) library would be able to provide a flat_map that works with the code nearly written: would need "closure" traits with a similar link.)
Work-arounds include:
the RefCell suggested by Renato Zannon, which allows seen to be borrowed as a shared &. The desugared closure code is basically the same other than changing the &mut Vec<i32> to &Vec<i32>. This change means "reborrow" of the &'b mut &'a RefCell<Vec<i32>> can just be a copy of the &'a ... out of the &mut. It's a literal copy, so the lifetime is retained.
avoiding the laziness of iterators, to avoid returning the inner closure, specifically.collect::<Vec<_>>()ing inside the loop to run through the whole filter_map before returning.
fn main() {
let mut seen = vec![];
let items = vec![vec![1i32, 2], vec![3], vec![1]];
let a: Vec<_> = items
.iter()
.flat_map(|inner_numbers| {
inner_numbers
.iter()
.filter_map(|&number| if !seen.contains(&number) {
seen.push(number);
Some(number)
} else {
None
})
.collect::<Vec<_>>()
.into_iter()
})
.collect();
println!("{:?}", a);
}
I imagine the RefCell version is more efficient.
It seems that the borrow checker is getting confused at the nested closures + mutable borrow. It might be worth filing an issue. Edit: See huon's answer for why this isn't a bug.
As a workaround, it's possible to resort to RefCell here:
use std::cell::RefCell;
fn main() {
let seen = vec![];
let items = vec![vec![1i32, 2], vec![3], vec![1]];
let seen_cell = RefCell::new(seen);
let a: Vec<_> = items
.iter()
.flat_map(|inner_numbers| {
inner_numbers.iter().filter_map(|&number| {
let mut borrowed = seen_cell.borrow_mut();
if !borrowed.contains(&number) {
borrowed.push(number);
Some(number)
} else {
None
}
})
})
.collect();
println!("{:?}", a);
}
I came across this question after I ran into a similar issue, using flat_map and filter_map. I solved it by moving the filter_map outside the flat_map closure.
Using your example:
let a: Vec<_> = items
.iter()
.flat_map(|inner_numbers| inner_numbers.iter())
.filter_map(move |&number| {
if !seen.contains(&number) {
seen.push(number);
Some(number)
} else {
None
}
})
.collect();

Conflicting lifetime requirements when storing closure capturing returned value

EDIT:
I'm trying to create a vector of closures inside a function, add a standard closure to the vector, and then return the vector from the function. I'm getting an error about conflicting lifetimes.
Code can be executed here.
fn vec_with_closure<'a, T>(f: Box<FnMut(T) + 'a>) -> Vec<Box<FnMut(T) + 'a>>
{
let mut v = Vec::<Box<FnMut(T)>>::new();
v.push(Box::new(|&mut: t: T| {
f(t);
}));
v
}
fn main() {
let v = vec_with_closure(Box::new(|t: usize| {
println!("{}", t);
}));
for c in v.iter_mut() {
c(10);
}
}
EDIT 2:
Using Rc<RefCell<...>> together with move || and the Fn() trait as opposed to the FnMut()m as suggested by Shepmaster, helped me produce a working version of the above code. Rust playpen version here.
Here's my understanding of the problem, slightly slimmed down:
fn filter<F>(&mut self, f: F) -> Keeper
where F: Fn() -> bool + 'static //'
{
let mut k = Keeper::new();
self.subscribe(|| {
if f() { k.publish() }
});
k
}
In this method, f is a value that has been passed in by-value, which means that filter owns it. Then, we create another closure that captures f by-reference. We are then trying to save that closure somewhere, so all the references in the closure need to outlive the lifetime of our struct (I picked 'static for convenience).
However, f only lives until the end of the method, so it definitely won't live long enough. We need to make the closure own f. It would be ideal if we could use the move keyword, but that causes the closure to also move in k, so we wouldn't be able to return it from the function.
Trying to solve that led to this version:
fn filter<F>(&mut self, f: F) -> Keeper
where F: Fn() -> bool + 'static //'
{
let mut k = Keeper::new();
let k2 = &mut k;
self.subscribe(move || {
if f() { k2.publish() }
});
k
}
which has a useful error message:
error: `k` does not live long enough
let k2 = &mut k;
^
note: reference must be valid for the static lifetime...
...but borrowed value is only valid for the block
Which leads to another problem: you are trying to keep a reference to k in the closure, but that reference will become invalid as soon as k is returned from the function. When items are moved by-value, their address will change, so references are no longer valid.
One potential solution is to use Rc and RefCell:
fn filter<F>(&mut self, f: F) -> Rc<RefCell<Keeper>>
where F: Fn() -> bool + 'static //'
{
let mut k = Rc::new(RefCell::new(Keeper::new()));
let k2 = k.clone();
self.subscribe(move || {
if f() { k2.borrow_mut().publish() }
});
k
}

Type aliases for closures

I would've thought the following code would work:
use std::num::{Num};
use std::fmt::{Show};
pub type GradFn<T : Num> = for<'a> fn(&'a [T]) -> (T, Vec<T>);
fn minimize<T : Show, F>(f : GradFn<T>, x0 : &[T]) {
// some no-op to test types
print!("{}",f(x0))
}
fn main() {
let xSquared : GradFn<f64> = |x : &[f64]| -> (f64, Vec<f64>) {
return (x[0] * x[0], vec![2.0 * x[0]]);
};
let (fx, grad) = xSquared(vec![2.0f64].as_slice());
print!("{}", fx);
}
But I'm getting a compiler error (see here):
<anon>:12:32: 14:4 error: mismatched types: expected `fn(&'a [f64]) -> (f64, collections::vec::Vec<f64>)`, found `|&[f64]| -> (f64, collections::vec::Vec<f64>)` (expected extern fn, found fn)
<anon>:12 let xSquared : GradFn<f64> = |x : &[f64]| -> (f64, Vec<f64>) {
<anon>:13 return (x[0] * x[0], vec![2.0 * x[0]]);
<anon>:14 };
fn does not define a closure type; it defines bare function pointers (i.e. pointers to functions defined with the fn keyword). That's why you can't assign a closure to a GradFn. Instead, you want to use either Fn, FnMut or FnOnce.
There are a few more changes I needed to make to get this code to compile:
On minimize, the f parameter as you wrote it receives an unsized type by value, which is forbidden. You also put an F type parameter that you don't use. You probably meant to constrain F and to use F as the type of f.
The compiler won't let us use a type alias in a type parameter constraint; we need to spell out the trait fully. This means the type alias is basically useless.
I removed the type annotation on xSquared, which is unnecessary. This let me remove the type alias completely.
Here's the final code:
#![feature(unboxed_closures)]
use std::num::{Num};
use std::fmt::{Show};
fn minimize<T: Show, F: FnMut(&[T]) -> (T, Vec<T>)>(mut f: F, x0: &[T]) {
// some no-op to test types
print!("{}", f(x0))
}
fn main() {
let xSquared = |x: &[f64]| -> (f64, Vec<f64>) {
return (x[0] * x[0], vec![2.0 * x[0]]);
};
let (fx, grad) = xSquared(vec![2.0f64].as_slice());
print!("{}", fx);
}
If your GradFn is actually going to be a bare function pointer (as opposed to a closure) you can keep the type alias, like this:
use std::num::Num;
use std::fmt::Show;
// this type has to point to a bare function, not a closure
pub type GradFn<T> = for<'a> fn(&'a [T]) -> (T, Vec<T>);
fn minimize<T>(f : GradFn<T>, x0 : &[T])
where T: Show + Num {
// some no-op to test types
println!("{}",f(x0))
}
fn main() {
// this is now a bare function
fn x_squared(x : &[f64]) -> (f64, Vec<f64>) {
return (x[0] * x[0], vec![2.0 * x[0]]);
}
// and this is a pointer to it, that uses your type alias
let x_sq : GradFn<f64> = x_squared;
let (fx, grad) = x_sq(&[2f64]);
println!("fx: {} - grad: {}", fx, grad);
minimize(x_sq, &[3f64]);; // works with minimize as well
}

Resources