How can I check if std::io::Cursor has unconsumed data? - stream

I am writing a low-level network app that deals with TCP sockets where I often need to process binary data streams. When some data is available, I read it into u8 array, then wrap into std::io::Cursor<&[u8]> and then pass it to handlers. In a handler, I often need to know if there is some more data in the Cursor or not.
Imagine that the handle function receives data and then processes it in chunks using the handle_chunk function. For simplicity, assume that chunk size is fixed at 10 bytes; if the data size is not divisible by 10, it's an error. This simple logic can be implemented in the following way:
fn handle(mut data: Cursor<&[u8]>) {
while !data.empty() {
if let Err(err) = handle_chunk(&mut data) {
eprintln!("Error while handling data: {}", err);
}
}
}
fn handle_chunk(data: &mut Cursor<&[u8]>) -> Result<(), String> {
// Returns Err("unexpected EOF".to_string()) if chunk is incomplete
// ...
}
However, Cursor does not have an empty() method or any other method capable of telling if there is more data to process. The working solution that I could come up with is:
fn handle(data: Cursor<&[u8]>) {
let data = data.into_inner();
let len = data.len();
let mut data = Cursor::new(data);
while (data.position() as usize) < len - 1 {
if let Err(err) = handle_chunk(&mut data) {
eprintln!("Error while handling data: {}", err);
}
}
}
This looks hacky and inelegant though. Is there a better solution? Maybe there is a different tool in the Rust standard library that fits here better than Cursor?

Your code can be simplified by using Cursor::get_ref to avoid breaking up the input and putting it back together:
fn handle(mut data: Cursor<&[u8]>) {
let len = data.get_ref().len();
while (data.position() as usize) < len - 1 {
if let Err(err) = handle_chunk(&mut data) {
eprintln!("Error while handling data: {}", err);
}
}
}
Now, you haven't shown any code that requires a Cursor. Many times, people think it's needed to convert a &[u8] to something that implements Read, but it's not. Read is implemented for &'a [u8]:
use std::io::Read;
fn handle(mut data: &[u8]) {
while !data.is_empty() {
if let Err(err) = handle_chunk(&mut data) {
eprintln!("Error while handling data: {}", err);
}
}
}
fn handle_chunk<R: Read>(mut data: R) -> Result<(), String> {
let mut b = [0; 10];
data.read_exact(&mut b).unwrap();
println!("Chunk: {:?}", b);
Ok(())
}
fn main() {
let d: Vec<u8> = (0..20).collect();
handle(&d)
}
By having mut data: &[u8] and using &mut data, the code will update the slice variable in place to advance it forward. We can't easily go backward though.
an empty() method
Rust style indicates that an empty method would be a verb — this would remove data (if it were possible). The method you want should be called is_empty, as seen on slices.

Related

Why this Rust code allocates buffers on same memory region?

I don't understand the behaviour of this piece of code... I'm writing an RTOS an this issue is halting me. I really don't get why the code acts this way.
Here is some code I tested on the playground that shows the issue.
https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=cc6cc0ec8bfe76f65e1baaa67caaf9e6
use core::fmt;
use core::fmt::Display;
struct StackPointer(*const usize);
impl Display for StackPointer {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0 as usize)
}
}
struct Stack<const WORDS: usize> {
pub sp: StackPointer,
pub mem: [usize; WORDS],
}
impl<const WORDS: usize> Stack<WORDS> {
pub fn new() -> Self {
let mem = [0; WORDS];
let sp = StackPointer(mem.as_ptr() as *const usize);
Self {
mem,
sp,
}
}
}
struct PCB<const WORDS: usize> {
pub stack: Stack<WORDS>,
}
impl<const WORDS: usize> PCB<WORDS> {
pub fn new() -> Self {
Self {
stack: Stack::new(),
}
}
}
fn main() {
let pcb1 = PCB::<128>::new();
let pcb2 = PCB::<128>::new();
let pcb3 = PCB::<128>::new();
println!("sp1: {}, sp2: {}, sp3: {}", pcb1.stack.sp, pcb2.stack.sp, pcb3.stack.sp);
}
I don't understand the behaviour of this piece of code... I'm writing an RTOS an this issue is halting me. I really don't get why the code acts this way.
Because you're writing broken code.
let mem = [0; WORDS];
this reserves WORDS words on the stack (incidentally why is it usize?)
let sp = StackPointer(mem.as_ptr() as *const usize);
this takes a pointer to a location in the current stackframe, where you've put your array.
Self {
mem,
sp,
}
this then blissfully copies the data out of the current stackframe and into the parent stackframe, while keeping a pointer to the now-popped stackframe.
So on each call to PCB::<128>::new(); you're going to create a stackframe, allocate an array into that stackframe, take a pointer to that array (in the stackframe), then pop the stackframe.
All the stackframes being in the same location (on top of main's stackframe) they're at roughly the same offset, hence the array is at the same offset in all calls, and all your nonsensical StackPointer store data to the same location, which will be filled with nonsense as soon as you call an other function.

How can I efficiently extract the first element of a futures::Stream in a blocking manner?

I've got the following method:
pub fn load_names(&self, req: &super::MagicQueryType) -> ::grpcio::Result<::grpcio::ClientSStreamReceiver<String>> {
My goal is to get the very first element of grpcio::ClientSStreamReceiver; I don't care about the other names:
let name: String = load_names(query)?.wait().nth(0)?;
It seems inefficient to call wait() before nth(0) as I believe wait() blocks the stream until it receives all the elements.
How can I write a more efficient solution (i.e., nth(0).wait()) without triggering build errors? Rust's build errors for futures::stream::Stream look extremely confusing to me.
The Rust playground doesn't support grpcio = "0.4.4" so I cannot provide a link.
To extract the first element of a futures::Stream in a blocking manner, you should convert the Stream to an iterator by calling executor::block_on_stream and then call Iterator::next.
use futures::{executor, stream, Stream}; // 0.3.4
use std::iter;
fn example() -> impl Stream<Item = i32> {
stream::iter(iter::repeat(42))
}
fn main() {
let v = executor::block_on_stream(example()).next();
println!("{:?}", v);
}
If you are using Tokio, you can convert the Stream into a Future with StreamExt::into_future and annotate a function with #[tokio::main]:
use futures::{stream, Stream, StreamExt}; // 0.3.4
use std::iter;
use tokio; // 0.2.13
fn example() -> impl Stream<Item = i32> {
stream::iter(iter::repeat(42))
}
#[tokio::main]
async fn just_one() -> Option<i32> {
let (i, _stream) = example().into_future().await;
i
}
fn main() {
println!("{:?}", just_one());
}
See also:
How do I synchronously return a value calculated in an asynchronous Future in stable Rust?
How to select between a future and stream in Rust?

How do I extract messages from an unbounded queue every N seconds and spawn them on to a Tokio handler?

I am trying to extract messages (which are futures themselves) from an unbounded queue every N seconds and spawn them into the Tokio handler.
I’ve tried dozens of variations but I cannot seem to find the right approach. It looks like it should be possible, but I always hit a future type mismatch or end up with borrow issues.
This is the code that shows more or less what I want:
let fut = Interval::new_interval(Duration::from_secs(1))
.for_each(|num| vantage_dequeuer.into_future() )
.for_each(|message:VantageMessage |{
handle.spawn(message);
return Ok(());
})
.map_err(|e| panic!("delay errored; err={:?}", e));
core.run(fut);
Complete code:
extern crate futures; // 0.1.24
extern crate tokio; // 0.1.8
extern crate tokio_core; // 0.1.17
use futures::future::ok;
use futures::sync::mpsc;
use futures::{Future, Stream};
use std::thread;
use std::time::Duration;
use tokio::timer::Interval;
use tokio_core::reactor::Core;
type VantageMessage = Box<Future<Item = (), Error = ()> + Send>;
fn main() {
let (enqueuer, dequeuer) = mpsc::unbounded();
let new_fut: VantageMessage = Box::new(ok(()).and_then(|_| {
println!("Message!");
return Ok(());
}));
enqueuer.unbounded_send(new_fut);
let joinHandle = worker(Some(dequeuer));
joinHandle.join();
}
/*
Every second extract one message from dequeuer (or wait if not available)
and spawn it in the core
*/
fn worker(
mut vantage_dequeuer: Option<mpsc::UnboundedReceiver<VantageMessage>>,
) -> thread::JoinHandle<()> {
let dequeuer = dequeuer.take().unwrap();
let joinHandle = thread::spawn(|| {
let mut core = Core::new().unwrap();
let handle = core.handle();
let fut = Interval::new_interval(Duration::from_secs(1))
.for_each(|num| vantage_dequeuer.into_future())
.for_each(|message: VantageMessage| {
handle.spawn(message);
return Ok(());
})
.map_err(|e| panic!("delay errored; err={:?}", e));
core.run(fut);
println!("Returned!");
});
return joinHandle;
}
Playground
error[E0425]: cannot find value `dequeuer` in this scope
--> src/main.rs:33:20
|
33 | let dequeuer = dequeuer.take().unwrap();
| ^^^^^^^^ not found in this scope
error[E0599]: no method named `into_future` found for type `std::option::Option<futures::sync::mpsc::UnboundedReceiver<std::boxed::Box<(dyn futures::Future<Item=(), Error=()> + std::marker::Send + 'static)>>>` in the current scope
--> src/main.rs:38:46
|
38 | .for_each(|num| vantage_dequeuer.into_future())
| ^^^^^^^^^^^
|
= note: the method `into_future` exists but the following trait bounds were not satisfied:
`&mut std::option::Option<futures::sync::mpsc::UnboundedReceiver<std::boxed::Box<(dyn futures::Future<Item=(), Error=()> + std::marker::Send + 'static)>>> : futures::Stream`
Interval and UnboundedReceiver are both streams, so I'd use Stream::zip to combine them:
The zipped stream waits for both streams to produce an item, and then returns that pair. If an error happens, then that error will be returned immediately. If either stream ends then the zipped stream will also end.
extern crate futures; // 0.1.24
extern crate tokio; // 0.1.8
extern crate tokio_core; // 0.1.17
use futures::{
future::ok,
sync::mpsc,
{Future, Stream},
};
use std::{thread, time::Duration};
use tokio::timer::Interval;
use tokio_core::reactor::Core;
type VantageMessage = Box<Future<Item = (), Error = ()> + Send>;
pub fn main() {
let (tx, rx) = mpsc::unbounded();
let new_fut: VantageMessage = Box::new(ok(()).and_then(|_| {
println!("Message!");
Ok(())
}));
tx.unbounded_send(new_fut).expect("Unable to send");
drop(tx); // Close the sending side
worker(rx).join().expect("Thread had a panic");
}
fn worker(queue: mpsc::UnboundedReceiver<VantageMessage>) -> thread::JoinHandle<()> {
thread::spawn(|| {
let mut core = Core::new().unwrap();
let handle = core.handle();
core.run({
Interval::new_interval(Duration::from_secs(1))
.map_err(|e| panic!("delay errored; err={}", e))
.zip(queue)
.for_each(|(_, message)| {
handle.spawn(message);
Ok(())
})
})
.expect("Unable to run reactor");
println!("Returned!");
})
}
Note that this doesn't actually wait for any of the spawned futures to complete before the reactor shuts down. If you want that, I'd switch to tokio::run and tokio::spawn:
fn worker(queue: mpsc::UnboundedReceiver<VantageMessage>) -> thread::JoinHandle<()> {
thread::spawn(|| {
tokio::run({
Interval::new_interval(Duration::from_secs(1))
.map_err(|e| panic!("delay errored; err={}", e))
.zip(queue)
.for_each(|(_, message)| {
tokio::spawn(message);
Ok(())
})
});
println!("Returned!");
})
}

"error: closure may outlive the current function" but it will not outlive it

When I try to compile the following code:
fn main() {
(...)
let mut should_end = false;
let mut input = Input::new(ctx);
input.add_handler(Box::new(|evt| {
match evt {
&Event::Quit{..} => {
should_end = true;
}
_ => {}
}
}));
while !should_end {
input.handle();
}
}
pub struct Input {
handlers: Vec<Box<FnMut(i32)>>,
}
impl Input {
pub fn new() -> Self {
Input {handlers: Vec::new()}
}
pub fn handle(&mut self) {
for a in vec![21,0,3,12,1] {
for handler in &mut self.handlers {
handler(a);
}
}
}
pub fn add_handler(&mut self, handler: Box<FnMut(i32)>) {
self.handlers.push(handler);
}
}
I get this error:
error: closure may outlive the current function, but it borrows `should_end`, which is owned by the current function
I can't simply add move to the closure, because I need to use should_end later in the main loop. I mean, I can, but since bool is Copy, it will only affect the should_end inside the closure, and thus the program loops forever.
As far as I understand, since input is created in the main function, and the closure is stored in input, it couldn't possibly outlive the current function. Is there a way to express to Rust that the closure won't outlive main? Or is there a possibility that I can't see that the closure will outlive main? In the latter case, it there a way to force it to live only as long as main?
Do I need to refactor the way I'm handling input, or is there some way I can make this work. If I need to refactor, where can I look to see a good example of this in Rust?
Here's a playpen of a simplified version. It is possible I made a mistake in it that could crash your browser. I happened to me once, so, beware.
In case it is needed, the rest of my code is available. All the relevant info should be in either main.rs or input.rs.
The problem is not your closure, but the add_handler method. Fully expanded it would look like this:
fn add_handler<'a>(&'a mut self, handler: Box<FnMut(i32) + 'static>)
As you can see, there's an implicit 'static bound on the trait object. Obviously we don't want that, so we introduce a second lifetime 'b:
fn add_handler<'a, 'b: 'a>(&'a mut self, handler: Box<FnMut(i32) + 'b>)
Since you are adding the handler object to the Input::handlers field, that field cannot outlive the scope of the handler object. Thus we also need to limit its lifetime:
pub struct Input<'a> {
handlers: Vec<Box<FnMut(i32) + 'a>>,
}
This again requires the impl to have a lifetime, which we can use in the add_handler method.
impl<'a> Input<'a> {
...
pub fn add_handler(&mut self, handler: Box<FnMut(i32) + 'a>) {
self.handlers.push(handler);
}
}
Now all that's left is using a Cell to control access to your should_end flag.
Here is an example of the fixed code:
use std::cell::Cell;
fn main() {
let should_end = Cell::new(false);
let mut input = Input::new();
input.add_handler(Box::new(|a| {
match a {
1 => {
should_end.set(true);
}
_ => {
println!("{} {}", a, should_end.get())
}
}
}));
let mut fail_safe = 0;
while !should_end.get() {
if fail_safe > 20 {break;}
input.handle();
fail_safe += 1;
}
}
pub struct Input<'a> {
handlers: Vec<Box<FnMut(i32) + 'a>>,
}
impl<'a> Input<'a> {
pub fn new() -> Self {
Input {handlers: Vec::new()}
}
pub fn handle(&mut self) {
for a in vec![21,0,3,12,1,2] {// it will print the 2, but it won't loop again
for handler in &mut self.handlers {
handler(a);
}
}
}
pub fn add_handler(&mut self, handler: Box<FnMut(i32) + 'a>) {
self.handlers.push(handler);
}
}

Create a moving average (and other FIR filters) using ReactiveCocoa

I'm still getting started with ReactiveCocoa and functional reactive programming concepts, so maybe this is a dumb question.
ReactiveCocoa seem naturally designed to react to streams of live data, touch events or accelerometer sensor input etc.
Is it possible to apply finite impulse response filters in ReactiveCocoa in an easy, reactive fashion? Or if not, what would be the least-ugly hacky way of doing this? How would one go about implementing something like a simple moving average?
Ideally looking for an Swift 2 + RA4 solution but also interested in if this is possible at all in Objective C and RA2/RA3.
What you actually need is a some sort of period buffer, which will keep a period of values buffered and only start sending out when the buffer has reached capacity (the code below is heavenly inspired on takeLast operator)
extension SignalType {
func periodBuffer(period:Int) -> Signal<[Value], Error> {
return Signal { observer in
var buffer: [Value] = []
buffer.reserveCapacity(period)
return self.observe { event in
switch event {
case let .Next(value):
// To avoid exceeding the reserved capacity of the buffer, we remove then add.
// Remove elements until we have room to add one more.
while (buffer.count + 1) > period {
buffer.removeAtIndex(0)
}
buffer.append(value)
if buffer.count == period {
observer.sendNext(buffer)
}
case let .Failed(error):
observer.sendFailed(error)
case .Completed:
observer.sendCompleted()
case .Interrupted:
observer.sendInterrupted()
}
}
}
}
}
based on that you can map it to any algorithm you want
let pipe = Signal<Int,NoError>.pipe()
pipe.0
.periodBuffer(3)
.map { Double($0.reduce(0, combine: +))/Double($0.count) } // simple moving average
.observeNext { print($0) }
pipe.1.sendNext(10) // does nothing
pipe.1.sendNext(11) // does nothing
pipe.1.sendNext(15) // prints 12
pipe.1.sendNext(7) // prints 11
pipe.1.sendNext(9) // prints 10.3333
pipe.1.sendNext(6) // prints 7.3333
Probably the scan signal operator is what you're looking for. Inspired by Andy Jacobs' answer, I came up with something like this (a simple moving average implementation):
let (signal, observer) = Signal<Int,NoError>.pipe()
let maxSamples = 3
let movingAverage = signal.scan( [Int]() ) { (previousSamples, nextValue) in
let samples : [Int] = previousSamples.count < maxSamples ? previousSamples : Array(previousSamples.dropFirst())
return samples + [nextValue]
}
.filter { $0.count >= maxSamples }
.map { $0.average }
movingAverage.observeNext { (next) -> () in
print("Next: \(next)")
}
observer.sendNext(1)
observer.sendNext(2)
observer.sendNext(3)
observer.sendNext(4)
observer.sendNext(42)
Note: I had to move average method into a protocol extension, otherwise the compiler would complain that the expression was too complex. I used a nice solution from this answer:
extension Array where Element: IntegerType {
var total: Element {
guard !isEmpty else { return 0 }
return reduce(0){$0 + $1}
}
var average: Double {
guard let total = total as? Int where !isEmpty else { return 0 }
return Double(total)/Double(count)
}
}

Resources