fellow dart programmers.
I am reading in a file using Stream as below.
Stream<List<int>> stream = new File(filepath).openRead();
stream
.transform(UTF8.decoder)
.transform(const LineSpilitter())
.listen((line){
// TODO: check if this is the last line of the file
var isLastLine;
});
I want to check whether the line in listen() is the last line of the file.
I don't think you can check if the current chunk of data is the last one.
You can only pass a callback that is called when the stream is closed.
Stream<List<int>> stream = new File('main.dart').openRead();
stream.
.transform(UTF8.decoder)
.transform(const LineSpilitter())
.listen((line) {
// TODO: check if this is the last line of the file
var isLastLine;
}
,onDone: (x) => print('done')); // <= add a second callback
While the answer from #Günter Zöchbauer works, the last property of streams accomplishes exactly what you are asking for (I guess the dart team added this functionality in the last 5 years).
Stream<List<int>> stream = new File('main.dart').openRead();
List<int> last = await stream.last;
But note: It is not possible to listen to the stream and use await stream.last.
This will cause an error:
StateError (Bad state: Stream has already been listened to.)
Related
I'm having trouble understanding the flow of the following code.
The code should process MERGE_SIZE lines (3 in this run), save the lines to a 'phase' file, and then process the next 3 lines and so on.
The call to savePhase has an await so I was expecting for the savePhase to complete before additional lines are processed.
As you can see in the output below every line is process and then the savePhase calls complete.
Future _sort() async {
var completer = Completer<void>();
var instance = 0;
var lineCount = MERGE_SIZE;
var phaseDirectory = Directory.systemTemp.createTempSync();
var list = <String>[];
var sentToPhase = false;
await File(filename)
.openRead()
.map(utf8.decode)
.transform(LineSplitter())
.forEach((l) async {
list.add(l);
print('$l linecount:$lineCount');
lineCount--;
if (lineCount == 0) {
lineCount = MERGE_SIZE;
instance++;
sentToPhase = true;
await savePhase(phaseDirectory, 1, instance, list, lineDelimiter);
list.clear();
print('savePhase completed');
}
});
which outputs
9 line linecount:3
8 line linecount:2
7 line linecount:1
6 line linecount:3
5 line linecount:2
4 line linecount:1
3 line linecount:3
2 line linecount:2
1 line linecount:1
savePhase completed
savePhase completed
savePhase completed
Is this something to do with the streams that openRead uses to deliver the read lines?
I thought I had await figured out, but apparently not :)
Not tested your program but I am fairly sure that your problem is that you expect the forEach() method are waiting for each Future to be completed before the next call which are not the case.
Try take a look at the following solution which are about more or less the same problem:
Sequential processing of a variable number of async functions in Dart
Flow of program
So what happens in your code is that the file your are reading seems to be small enough that the whole content can be read in one go in one of the buffers used when reading a file. This will explain why you see multiple line linecount lines before savePhase completed.
As previous mentioned, the forEach() method on Stream does not take into account that the method given as parameter does return a Future which should be awaited. You can see that in the implementation showed here:
https://api.dart.dev/stable/2.7.0/dart-async/Stream/forEach.html
So that means that the Future returned from calling forEach() does really just complete when all lines has been processed but does not wait for each Future generated for each line (remember, a async method does always return a Future regardless of it contains an await).
Since you also use shared variables between each spawned Future you will also get some funky behavior here since you e.g. share the same list but also clearing the list afterwards. So there are a potential here for errors here.
I am using EDI.Net from indice-co and i have a EDI file that contains multiple items, when i use the EdiGrammer.NewEdiFact and read the file using stream and deserialize it I get only 1 item from the file, the top most; how do i read the file using stream and deserialize it to a list?
Code Example:
var editFactParser = EdiGrammar.NewEdiFact();
var interchange = default(EdiModel.Interchange);
using (
var stream = File.Open("E:\\SomePath\\20191121020103.00000091.EDI", FileMode.Open,
FileAccess.Read))
{
interchange = new EdiSerializer().Deserialize<EdiModel.Interchange>(new StreamReader(stream),
editFactParser );
}
EdiFact File Content
UNA:+.? 'UNB+UNOA:2+DHLEUAPGW+CENTIRO+191030:1347+203516'UNH+240179+IFTSTA:D:01B:UN'BGM+77+9690108+9'DTM+9:201910301347:203'NAD+CZ+9690108'CNI+1+1032173'LOC+5+AMS::87'LOC+8+AMS::87'STS++PU+:::SHIPMENT PICKUP'RFF+CN:1297617'DTM+11:20191030:102'DTM+7:201910301329:203'GID++1'PCI+18'GIN+BN+10321732'UNT+15+240179'UNH+240180+IFTSTA:D:01B:UN'BGM+77+9690108+9'DTM+9:201910301347:203'NAD+CZ+96901083'CNI+1+2598018'LOC+5+ORY::87'LOC+8+AMS::87'STS++PL+:::PROCESSED AT LOCATION'RFF+CN:116775116'DTM+11:20191029:102'DTM+7:201910301336:203'GID++1'PCI+18'GIN+BN+2598018043'CNI+2+4911357323'LOC+5+CDG::87'LOC+8+AMS::87'STS++PL+:::PROCESSED AT LOCATION'RFF+CN:1286700'DTM+11:20191029:102'DTM+7:201910301339:203'GID++1'PCI+18'GIN+BN+49113573'CNI+3+4911401'LOC+5+CDG::87'LOC+8+AMS::87'STS++PL+:::PROCESSED AT LOCATION'RFF+CN:129007'DTM+11:20191029:102'DTM+7:201910301337:203'GID++1'PCI+18'GIN+BN+49114019'CNI+4+6194460'LOC+5+BRU::87'LOC+8+AMS::87'STS++PL+:::PROCESSED AT LOCATION'RFF+CN:127214241'DTM+11:20191029:102'DTM+7:201910301339:203'GID++1'PCI+18'GIN+BN+6194460856'CNI+5+7525715'LOC+5+ORY::87'LOC+8+AMS::87'STS++PL+:::PROCESSED AT LOCATION'RFF+CN:ECONOCOM'DTM+11:20191029:102'DTM+7:201910301336:203'GID++1'PCI+18'GIN+BN+75257154'CNI+6+752571'LOC+5+ORY::87'LOC+8+AMS::87'STS++PL+:::PROCESSED AT LOCATION'RFF+CN:ECONOCOM'DTM+11:20191029:102'DTM+7:201910301339:203'GID++1'PCI+18'GIN+BN+7525715'UNT+65+240180'UNZ+2+203516'
sorry for the late reply, I was able to solve it, it was an issue how with how I was accessing the segments and what data I was trying to get back, after some more trial and error I was able to figure it out, I currently do not have access to the code and will try and post it back here when I get to it. Thank you.
I am implementing one tokenizer. It parses the document, tokenizes it on a set of possible delimiters and then provides me combination of 1-, 2- and 3-word tokens.
I was able to achieve my goal, but only in one specific way:
Stream<String> contentStr = file.openRead().transform(utf8.decoder);
Stream<String> tokens = contentStr.transform(charSplitter).transform(tokenizer).asBroadcastStream();
var twoWordTokens = tokens.transform(sliding(2));
var threeWordTokens = tokens.transform(sliding(3));
StreamController<String> merger = StreamController();
tokens.forEach((token) => merger.add(token));
threeWordTokens.forEach((token) => merger.add(token));
twoWordTokens.forEach((token) => merger.add(token));
merger.stream.forEach(print);
As you can see I do following:
broadcast original stream of tokens
transform it to 2 additional streams by sliding window transformation
create a StreamConsumer (StreamController to be precise) and pump every event from 3 streams to that stream consumer.
then I print every element of the stream consumer to test
It works but I don't like that I add each element from source streams via StreamConsumer.add method. I wanted to use StreamController.addStream instead but that somehow does not work.
The following code gives me a Bad state: Cannot add event while adding a stream error and I understand why:
StreamController<String> merger = StreamController();
merger.addStream(tokens);
merger.addStream(twoWordTokens);
merger.addStream(threeWordTokens);
merger.stream.forEach(print);
This is per API documentation of the StreamController.addStream.
So I need to wait for each addStream returning future completion:
StreamController<String> merger = StreamController();
await merger.addStream(tokens);
await merger.addStream(twoWordTokens);
await merger.addStream(threeWordTokens);
await merger.stream.forEach(print);
But in this case I get nothing printed in the console.
If I do this:
StreamController<String> merger = StreamController();
merger.stream.forEach(print);
await merger.addStream(tokens);
await merger.addStream(twoWordTokens);
await merger.addStream(threeWordTokens);
Then only the 1-word tokens, i.e. elements of the original broadcast stream are printed. Elements of the derived streams are not.
I kind of understand why this happens, because all my streams are derived from the original broadcast stream.
Is there a better way to implement such a pipeline?
Probably my problem can be reformulated in terms of stream duplication/forking, but I can't see a way to clone a stream in Dart. If you can advice on that - please do.
I hope to allow concurrent addStream at some point, but until then, you need to handle the events indpendently:
var allAdds = [
tokens.forEach(merger.add),
twoWordTokens.forEach(merger.add),
threeWordTokens.forEach(merger.add)];
Future.wait(allAdds).then((_) { merger.close(); });
merger.stream.forEach(print);
That's if you want to control everything yourself. You can also use the StreamGroup class from package:async. It collects a number of streams and emits their events as a single stream.
This assumes that you have no error events.
I've been struggling with this for couple of hours and can't seem tof ind a solution. I'll appreciate any help.
I building a flutter app trying to follow the BLOC pattern.
I have a widget with two text fields: Country code and PhoneNumber.
I have define a Bloc with two Sink (one for each field) and a Stream with a state. The Stream State is a merge of the two sink such as:
factory AuthenticationBloc(AuthenticationRepository authRepository) {
final onPhoneChanged = PublishSubject<String>();
final onCountryChanged = PublishSubject<String>();
//oldState would be the last entry in the stream state.
final stateChangeFromThePhone = onPhoneChanged.map((newPhone)=>oldState.copyWith(phone:newPhone));
final stateChangeFromtheCountry = onCountryChanged.map((newCountry)=>oldState.copyWith(country:newCountry));
final state = Observable.merge[stateChangeFromThePhone, stateChangeFromtheCountry];
}
This is pseudo code but the idea is there. My question is how can I get access to the latest event from the state stream represented in the code by oldState?
I could define a variable on which I store this value on each new event in the state stream but looks ugly... :(
Any advice?
If I have Stream in Dart, I can use both listen and forEach, but I don't understand the difference.
So for example consider this code:
final process = await Process.start('pub', ['serve']);
process.stdout.map((l) => UTF8.decode(l)).forEach(print);
I could also have written:
process.stdout.map((l) => UTF8.decode(l)).listen(print);
Is there any difference?
The forEach function on a Stream will stop at the first error, and it won't give you a StreamSubscription to control how you listen on a stream. Using forEach is great if that's what you want - it tells you when it's done (the returned Future) and all you have to do is handle the events. If you need more control, you can use the listen call which is how forEach is implemented.
It's like the difference between Iterable.forEach and Iterable.iterator - the former just calls a callback for each element, the other gives you a way to control the iteration.