How to read part of a chunked stream - dart

In Dart, stdin is a Stream<List<int>>. Bytes come in in chunks. What I want is a function that reads from stdin until I get some character (say '\0'), and then returns, so that future readers of stdin get the data after the '\0'.
Unfortunately because of the chunking, the '\0' byte might be in the middle of a chunk, so I kind of want to read a chunk, remove part of it, and then push it back to the start of the stream. But there isn't any way to do this.
Another option would be readByteSync() but reading bytes one at a time is going to be slow and this is in a GUI program so I can't use sync methods.

I think actually because a Stream<> can only ever be listened to once - even if a previous listener cancels its subscription - the only way is to have something permanently filtering the stream's events until the end of time. So you may as well just split the stream into two streams:
import 'dart:async';
class StdioPreambleSplitter {
StdioPreambleSplitter(this._input) {
var preambleFinished = false;
_input.listen((chunk) {
if (preambleFinished) {
_dataStream.add(chunk);
} else {
final nullByte = chunk.indexOf(0);
if (nullByte == -1) {
_dataStream.add(chunk);
} else {
preambleFinished = true;
_preambleStream.add(chunk.sublist(0, nullByte));
_dataStream.add(chunk.sublist(nullByte));
}
}
});
}
Stream<List<int>> preambleStream() {
return _preambleStream.stream;
}
Stream<List<int>> dataStream() {
return _dataStream.stream;
}
final Stream<List<int>> _input;
final StreamController<List<int>> _preambleStream = new StreamController();
final StreamController<List<int>> _dataStream = new StreamController();
}
Hopefully it doesn't add too much overhead.

Related

Dart - Get the last or the first value of a stream

I have a stream and I need to use the last value of this stream, and if there is no value emitted by this stream I need to wait for the fist value. I only want to use this value once. What is the correct way to do it?
Sounds like you want the most recent event emitted by a stream (which is presumably a broadcast stream, because otherwise there is no events until you listen), or, if there has been no events before, you want the next event instead.
For a plain Dart Stream, that's impossible. It doesn't remember previous events. You need to have listened to that stream previously in order to know what the most recent event was (but if you do that, it doesn't have to be a broadcast stream anyway).
You can build your own memorizing stream wrapper fairly easily (but as always with asynchronous programming, you need to be careful about race conditions)
// Copyright 2021 Google LLC.
// SPDX-License-Identifier: Apache-2.0
import "dart:async";
/// Listens to [source] to returned stream.
///
/// Each listener on the returned stream receives the most recent
/// event sent on [source] followed by all further events of [source]
/// until they stop listening.
/// If there has been no events on [source] yet, only the further events
/// are forwarded.
Stream<T> mostRecentStream<T>(Stream<T> source) {
var isDone = false;
var hasEvent = false;
T? mostRecentEvent;
List<MultiStreamController>? pendingListeners;
var listeners = <MultiStreamController>[];
void forEachListener(void Function(MultiStreamController) action) {
var active = 0;
var originalLength = listeners.length;
for (var i = 0; i < listeners.length; i++) {
var controller = listeners[i];
if (controller.hasListener) {
listeners[active++] = controller;
if (i < originalLength) action(controller);
}
}
listeners.length = active;
}
source.listen((event) {
mostRecentEvent = event;
hasEvent = true;
forEachListener((controller) {
controller.addSync(event);
});
}, onError: (e, s) {
forEachListener((controller) {
controller.addErrorSync(e, s);
});
}, onDone: () {
isDone = true;
for (var controller in listeners) {
controller.close();
}
listeners.clear();
});
return Stream<T>.multi((controller) {
if (hasEvent) controller.add(mostRecentEvent as T);
if (isDone) {
controller.close();
} else {
listeners.add(controller);
}
});
}
With that, you can simply do var recentStream = mostRecentStream(yourStream) and then later do recentStream.first to get either the most recent event or, if there is none, the next event (if there is one, you get an error if the stream is completely empty).

TCP input data packets are combined in Dart

I have a simple TCP client in dart:
import 'dart:io';
void main() {
const sendData = "\$I,Z,0.5,5,0*\r\n";
final socket = Socket.connect("192.168.1.100", int.parse("8008"))
.timeout(Duration(seconds: 5))
.whenComplete(() {
print("Connected!");
}).catchError((_) {
print("Error!");
});
socket.then((soc) {
soc.write(sendData);
soc.listen((data) {
print(String.fromCharCodes(data).trim);
});
});
}
This program sends a special message to server and after that, the server sends back a bunch of data every 10 ms. The output is as follows:
$I,1,250,0,206*$I,1,248,0,192*$I,1,246,0,178*$I,1,245,0,165*
$I,1,244,0,153*$I,1,244,0,141*$I,1,244,0,131*$I,1,245,0,121*
$I,1,246,0,113*$I,1,248,0,105*$I,1,250,0,98*
$I,1,253,0,92*$I,2,0,0,86*$I,2,4,0,82*$I,2,8,0,79*$I,2,12,0,76*
$I,2,18,0,74*$I,2,23,0,74*$I,2,29,0,74*$I,2,36,0,75*$I,2,42,0,77*$I,2,50,0,80*$I,2,58,0,84*$I,2,66,0,89*$I,2,74,0,94*
$I,2,83,0,101*$I,2,93,0,109*$I,2,103,0,117*$I,2,113,0,126*$I,2,124,0,136*$I,2,134,0,147*
$I,2,146,0,159*$I,2,157,0,171*$I,2,169,0,185*$I,2,182,0,199*$I,2,194,0,214*$I,2,207,0,230*$I,2,220,0,246*$I,2,233,1,8*$I,2,247,1,26*$I,3,5,1,44*$I,3,19,1,64*$I,3,33,1,84*
$I,3,48,1,105*$I,3,63,1,126*$I,3,77,1,148*$I,3,93,1,171*$I,3,108,1,194*$I,3,123,1,217*
$I,3,138,1,242*$I,3,154,2,10*$I,3,169,2,35*$I,3,185,2,61*$I,3,201,2,87*$I,3,216,2,113*$I,3,232,2,140*$I,3,248,2,167*$I,4,7,2,195*
$I,4,23,2,223*$I,4,39,2,251*$I,4,54,3,23*$I,4,70,3,51*
The server sends data in $I,1,250,0,206* format, i.e. starts with $ and ends with *. As one may note, several consecutive data packages are concatenated incorrectly.
Whenever I increase the interval, for example to 200 ms, everything is ok.
What should I do?
UPDATE
Besides the BrettSutton answer which is true, the contributers in dart github gave a more complete answer here.
I decided to parse the packet in Socket listen handler, split that and append it to a list. As I want to show the data on a chart, I reset the list after 100 data. However the data could be logged as well.
soc.listen((data) {
var av = data.length;
if (av != 0) {
var stList = String.fromCharCodes(data).trim().split("\$");
stList.forEach((str) {
if (str.isNotEmpty) {
var strS = str.split(",");
if (strS != null) y = parseData(strS);
sampleList.add(y);
}
});
print(sampleList);
if (sampleList.length > 100) {
sampleList.clear();
}
print("==========");
}
});

When do Stream start publishing values to listeners?

After reading a bunch of documentation about Streams and StreamControllers in dart I tried to build a little example and was surprised of the results. All documentation I have read states that a stream starts emiting data as soon as a listener is registered. But this doesn't show any printed data:
class Order
{
String type;
Order(this.type);
}
class Pizza
{
}
void main()
{
Order order = Order("pzza");
final StreamController sc = StreamController();
sc.sink.add(order);
sc.sink.add(order);
sc.sink.add(new Order("pizza"));
Stream st = sc.stream.map((order) {
return order.type;
})
.map((orderType) {
if(orderType == "pizza")
return Pizza();
else
throw ("dude!, I don't know how to do that");
});
var sus = st.listen((pizza)
{
print("We did a pizza");
},
onError: (error)
{
print(error);
});
sus.cancel();
sc.sink.add(new Order("pizza2"));
}
I was expecting this output:
dude!, I don't know how to do that
dude!, I don't know how to do that
We did a pizza
When creating streams and adding data is all "sinked" data scheduled to be emited on the next application step?
Cheers.
You are right in that the documentation states that you listen on a stream to make it start generating events. However, streams are asynchronous so when you call the listen() method you are registering to receive events from the stream at some point in the future. Dart will then continue to run the remainder of your main function. Immediately after calling listen() you call cancel() to cancel the subscription which is why nothing is being printed.
If you remove or comment out the cancel and run it again you will see the expected output.
A slightly modified version of your code will hopefully highlight the run of events:
class Order {
String type;
Order(this.type);
}
class Pizza {}
void main() {
print("Main starts");
Order order = Order("pzza");
final StreamController sc = StreamController();
sc.sink.add(order);
sc.sink.add(order);
sc.sink.add(new Order("pizza"));
Stream st = sc.stream.map((order) {
return order.type;
}).map((orderType) {
if (orderType == "pizza")
return Pizza();
else
throw ("dude!, I don't know how to do that");
});
var sus = st.listen((pizza) {
print("We did a pizza");
}, onError: (error) {
print(error);
});
// sus.cancel();
sc.sink.add(new Order("pizza2"));
print("Main ends");
}
Running this produces the output:
Main starts
Main ends
dude!, I don't know how to do that
dude!, I don't know how to do that
We did a pizza
dude!, I don't know how to do that

create a circular stream in Dart

I try to write a chat application to chat with a computer. The user can write a message and gets a response of the computer. A chat history might look like this:
user: Hi
computer: Hello
user: What's your name?
computer: Bot
...
My circular stream based design is inspired by the ideas of Cycle.js. I've got a stream of user messages, which get transformed to a stream of computer messages, which are in turn the input of the user message stream:
|----> user message stream ---->|
| |
transform transform
| |
|<-- computer message stream <--|
This code already works:
import 'dart:io';
import 'dart:async';
void main() {
cycle(computer, user);
}
typedef Stream<T> Transform<T>(Stream<T> input);
void cycle(Transform aToB, Transform bToA) {
var aProxy = new StreamController.broadcast();
var b = aToB(aProxy.stream);
var a = bToA(b);
aProxy.add('start'); // start with user
aProxy.addStream(a);
}
Stream<String> user(Stream<String> computerMessages) {
computerMessages = computerMessages.asBroadcastStream();
computerMessages.listen((message) => print('computer: $message'));
return computerMessages.map((message) {
stdout.write('user: ');
return stdin.readLineSync();
});
}
Stream<String> computer(Stream<String> userMessages) {
var messages = <String, String>{
"Hi": "Hello",
"What's your name?": "Bot"
};
return userMessages.map((m) => messages.containsKey(m) ? messages[m] : 'What?');
}
There is only one problem. You need a start value to get a circular stream running. Therefore, I put this line in my function cycle:
aProxy.add('start'); // start with user
Actually, this logic belongs into my function user, since cycle shouldn't know the initial value(s). Moreover, I don't like to print the initial value. It should only trigger the user input stream. Thus, I changed cycle and user:
void cycle(Transform aToB, Transform bToA) {
var aProxy = new StreamController.broadcast();
var b = aToB(aProxy.stream);
var a = bToA(b);
aProxy.addStream(a);
}
Stream<String> user(Stream<String> computerMessages) {
computerMessages = computerMessages.asBroadcastStream();
computerMessages.listen((message) => print('computer: $message'));
var requestInput = new StreamController<String>.broadcast();
requestInput.add('start'); // start with user
requestInput.addStream(computerMessages); // continue on computer response
return requestInput.stream.map((message) {
stdout.write('user: ');
return stdin.readLineSync();
});
}
But with this change my application terminates immediately with no message in stdout. What's wrong?
I found a solution. Create a normal instead of a broadcast StreamController in user:
Stream<String> user(Stream<String> computerMessages) {
computerMessages = computerMessages.asBroadcastStream();
computerMessages.listen((message) => print('computer: $message'));
var requestInput = new StreamController<String>();
requestInput.add('start'); // start with user
requestInput.addStream(computerMessages); // continue on computer response
return requestInput.stream.map((message) {
stdout.write('user: ');
return stdin.readLineSync();
});
}
Even though I found a solution, I sadly can not really explain what's the problem with a broadcast StreamController. null is successfully added by requestInput.add(null); but strangely neither requestInput.addStream(computerMessages); completes nor events of computerMessages are added to requestInput. Thus, requestInput is closed and the application terminates. I would appreciate if someone could provide further explanation.

How do I open and follow a file in Dart (like tail -f)?

Is there an easy way to open a file and continously read from it without the stream getting closed on EOF? Like the Unix command tail -f.
Just reading until EOF is described in the API docs. But I can't see an obvious/simple way to block or pause the stream for more input.
One solution would be to repeatedly reopen the file and continue reading from the last known length when I detect that the file size has changed.
Somethink like this
import 'dart:io';
void main(List<String> args ) {
var file = new File("test.txt");
print(file.absolute);
var openFuture = file.open(mode: FileMode.READ);
openFuture.then((raf) => raf.length().then((len) => raf.setPosition(len)
.then((raf) => raf.readXxx...
}
You can also use Directory.watch to get notified about changes and then reopen and read from the last known position.
(even though this questions is a bunch of years old now, I stumbled across the same issue today and couldn't find a viable solution and therefore had to roll my own and wanted to share my findings with future generations of Dart programmers. ;-))
The dart:io package in conjunction with a bit of stream and async-await magic should offer everything that is needed to achieve a "tail -f"-like functionality:
import 'dart:typed_data';
import 'dart:io';
Stream<List<int>> tail(final File file) async* {
final randomAccess = await file.open(mode: FileMode.read);
var pos = await randomAccess.position();
var len = await randomAccess.length();
// Increase/decrease buffer size as needed.
var buf = Uint8List(8192);
Stream<Uint8List> _read() async* {
while (pos < len) {
final bytesRead = await randomAccess.readInto(buf);
pos += bytesRead;
yield buf.sublist(0, bytesRead);
}
}
// Step 1: read whole file
yield* _read();
// Step 2: wait for modify events and read more bytes from file
await for (final event in file.watch(events: FileSystemEvent.modify)) {
if ((event as FileSystemModifyEvent).contentChanged) {
len = await (randomAccess.length());
yield* _read();
}
}
}
This function (tail) might then be used like that:
import 'dart:convert';
import 'dart:io';
void main() {
final file = File('/var/log/messages');
tail(file).transform(utf8.decoder).transform(LineSplitter()).forEach((line) {
// Do something with the line that has been read, e.g. print it out...
print(line);
});
}

Resources