How to store csv data in a list - dart

I want to read a csv file and store the content in a list of abjects. So I created the respective object and made an inputstream to read the file. But when I try to acces the list from outside of the function it's empty, so I thing, that the data is stored in a copy of listOfStocks and not directly in the list, like I want. So how can I store the content in the list?
import 'dart:io';
import 'dart:async';
import 'dart:convert';
main(List arguments) {
final File file = new File("../Data/In.List.toDart.csv");
Stream<List> inputStream = file.openRead();
List<Stock> listOfStocks = new List();
inputStream
.transform(utf8.decoder)
.transform(new LineSplitter())
.listen((String line) {
List row = line.split(',');
Stock stock = new Stock();
stock.setSymbol(row[0]);
stock.setDesc(row[1]);
stock.setExchange(row[2]);
listOfStocks.add(stock);
print(listOfStocks.length);
},
onDone: () { print('All read.'); },
onError: (e) { print(e.toString()); });
print(listOfStocks.length);
}
class Stock {
String symbol;
String desc;
String exchange;
void setSymbol(String symbol) {
this.symbol = symbol;
}
void setDesc(String desc) {
this.desc = desc;
}
void setExchange(String exchange) {
this.exchange = exchange;
}
String getSymbol() {
return symbol;
}
String getDesc() {
return desc;
}
String getExchange() {
return exchange;
}
}

The inputStream is asynchronous, so it will take some time before the file has been read.
The inputStream.transform(...).transform(...).listen(...) call completes immediately, then the stream will emit events when something has been read from the disk, and the listOfStocks will be gradually populated.
If you need the content of the list "immediately", that is: before you do anything else, you will have to block until the list is fully populated.
You can do this in two ways:
Read the file synchronously. This makes the operating system block until the file has been read completely into memory.
var content = file.readAsStringSync(); // Defaults to UTF-8 decoding.
for (var line in LineSplitter.split(content))
...
lineOfStocks.add(stock);
}
print(listOfStocks.length);
Read the file asynchronously, but wait for the transformation to complete:
main(List arguments) async {
...
await inputStream
.transform(utf8.decoder)
.transform(const LineSplitter())
.forEach((String line) { // Use forEach instead of listen
...
listOfStocks.add(stock);
});
print(listOfStocks.length);
}
I you use forEach instead of listen, it returns a future which completes
when the stream is done. With listen, you get a done event that you have
to handle manually.

Related

Is it possible to add an item to a List contained in a Stream in Dart?

Problem
I have a Stream<List> which is being listened to in several classes and I need all the classes to receive the updated stream once a value is added to the list in the Stream.
What I have tried
void main() {
StreamedList<String> dataStream = StreamedList();
dataStream.data.listen((list) => print(list));
dataStream.updateList(['Apple', 'Orange']);
dataStream.addToList('Mango'); // This is what I want to do
}
This is the code for StreamList class
class StreamedList<T> {
StreamController<List<T>> _controller = StreamController.broadcast();
Stream<List<T>> get data => _controller.stream;
void updateList(List<T> list) {
_controller.sink.add(list);
}
void addToList(T value) {
// Is it possible to do this?
// List<T> dataList = await _controller.data;
// dataList.add(value);
// updateList(dataList);
}
void dispose() {
_controller.close();
}
}
I have tried different APIs from the dart:async library including Stream.first, etc. which return a Future<List<T>>. But the problem is that this Future resolves only after something is added to the Stream later (eg. by calling the StreamedList.updateList function).
Question
How do I add a single value to the List inside the Stream?
You are misunderstanding what a Stream does. It doesn't "contain" data. It merely accepts data on one end (the sink) and propagates it out the other end (the stream), and any listeners gain access to the streamed object. Trying to "insert an item in a list within the stream" doesn't make any conceptual sense.
If you want to push a list with an additional item, take the old list, append the item to it, then re-add the list to the stream.
class StreamedList<T> {
StreamController<List<T>> _controller = StreamController.broadcast();
Stream<List<T>> get data => _controller.stream;
List<T> _list = [];
void updateList(List<T> list) {
_list = list;
_dispatch();
}
void addToList(T value) {
_list.add(value);
_dispatch();
}
void _dispatch() {
controller.sink.add(_list);
}
void dispose() {
_list = null;
_controller.close();
}
}
If you wanted to be doubly safe, you could recreate the list after every addToList, since if a listener captured the list elsewhere and modified its contents, that would affect _list as well.
void addToList(T value) {
_list = [..._list, value];
_dispatch();
}

how to return a string from a function which is listening some stream in dart?

i have a function called foo which is listening to the stdout, what i want is to return some string which i got from stdout. here is my function;
dynamic foo(process) {
return (
process.stdout.transform(UTF8.decoder).listen((data) {
String s = data.toString();
// print(s);
if (s.contains("received event of")) {
var s1 = s.split(":");
print("${s1[1]}");
return s1[1];
}
}));
}
I want to return s1 to the calling function
here a callback function do the trick
foo(process, callback) {
process.stdout.transform(UTF8.decoder).listen((data) {
String s = data.toString();
if (s.contains("received event of")) {
String message = s.split(":")[1];
callback(message);
}
});
}
and here i am calling the method and printing the data which i get get from stream.
foo(process,(data){print(data);})
This should do what you want
Future<String> dynamic foo(process) {
return process.stdout.transform(UTF8.decoder).map((data) {
String s = data.toString();
// print(s);
if (s.contains("received event of")) {
var s1 = s.split(":");
print("${s1[1]}");
return s1[1];
} else {
return null;
}
}).where((val) => val != null).first;
}
Your custom code either returns a valid value or null.
I changed listen to map to be able to use additional stream methods.
where filters invalid values (null) and returns the first non-null value.
The caller of the foo method needs to handle the returned Future (using for example async/await) to get the value when it becomes available.
Use it like
bar() async {
...
var input = await foo(proc);
print(input);
}
I think that everybody wants this:
import 'dart:io';
import 'dart:convert';
// Using system encoding:
var outputStr = await process.stdout.transform(systemEncoding.decoder).join();
// Using UTF-8 encoding:
var outputStr = await process.stdout.transform(utf8.decoder).join();

Prevent to read file many times

I am trying to write an i18n app. The program read a json file, that contains translation from languages and it based on json structure.
{
"EN": {
"TEXT1": "Hello",
"TEXT2": "March"
},
"DE": {
"TEXT1": "Hallo",
"TEXT2": "März"
}
}
My program read the json file in async way with the file class, the whole code
import 'dart:io';
import 'dart:async';
import 'package:json_object/json_object.dart';
abstract class I18n {
static _I18n _i18n;
factory I18n(String file, String lang) {
if(_i18n == null) {
_i18n = new _I18n(file, lang);
return _i18n;
}
return _i18n;
}
Future<String> getTextByMap(String textId);
}
class _I18n implements I18n {
File _file;
String _lang;
JsonObject _jsonContainer;
JsonObject _jsonFiltered;
Future<JsonObject> _imme;
// Parameters:
// file: The whole path and filename
// lang: Expected language
_I18n(String file, this._lang) {
this._file = new File(file);
}
// Read file and return the content of file.
Future<String> _readFileFromStream() {
var com = new Completer();
this._file.exists()
.then((fileExists) {
if(!fileExists) {
throw new StateError('File not found');
}
return this._file.readAsString()
.then((stream) => com.complete(stream));
});
return com.future;
}
void _convertContentToJson(String stream) {
this._jsonContainer = new JsonObject.fromJsonString(stream);
}
Future<JsonObject> _prepareData() {
return this._readFileFromStream().then((stream) {
_convertContentToJson(stream);
this._jsonFiltered = this._jsonContainer[this._lang];
return this._jsonFiltered;
});
}
Future<String> getTextByMap(String textId) {
return this._prepareData().then((filterd) {
return filterd[textId];
});
}
}
and the main code
import 'package:i18n/i18n.dart';
void main() {
var i18n = new I18n('../hello.json', 'EN');
i18n.getTextByMap('TEXT1').then((val) => print(val));
i18n.getTextByMap('TEXT2').then((val) => print(val));
}
Everything here, happen in dart async way, read json file etc. And everytime, when i call the method
i18n.getTextByMap('TEXT1').then((val) => print(val));
it gonna read the json file again and again. I tried to rewrite the method to prevent reading json file many times
Future<String> getTextByMap(String textId) {
if(this._jsonFiltered == null)
{
return this._prepareData().then((filterd) {
return filterd[textId];
});
}
return new Future(() => this._jsonFiltered[textId]);
}
but it doesn't work too, because dart works in async way.
My question is, how can i keep this json file content in an object? Read json file only one time and keep the contents in an object, it is better then read json file everytime, that is my opinion.
It could do everything in sync way, then i wouldn't have such as problem but this is not dart terminology.
In which order do dart execute I/O operations, like this?
Future
I/O Events
My solution would be to create a class with a factory constructor. The factory constructor always returns a object of that file.
Your problem is that futures are parallel. So both calls are executed in parallel. The solution is to let the first future complete and then do other stuff to be able to get cached results.
Then you can have a read() method that reads the value of the file if it is not present in the classes "contents" attribute for example - or if that attribute is not null, it loads the file in background.
In both cases a completer or future is returned you can listen on.
EDIT Example Code:
example_async_file_factory.dart
import 'dart:io';
import 'dart:async';
class FileHolder {
String _contents = null;
String path;
static Map<String, FileHolder> _files;
factory FileHolder(String path) {
if (_files == null) {
_files = {};
}
if (_files.containsKey(path)) {
return _files[path];
} else {
final fh = new FileHolder._internal(path);
_files[path] = fh;
return fh;
}
}
FileHolder._internal(this.path);
Future<String> getContents() {
if(_contents != null) {
print("cached");
return new Future.value(_contents);
} else {
print("read");
File f = new File(this.path);
Future<String> future = f.readAsString();
Completer completer = new Completer();
future.then((String c) {
_contents = c;
completer.complete(_contents);
});
return completer.future;
}
}
}
void main() {
FileHolder f = new FileHolder("example_async_file_factory.dart");
f.getContents().then((String contents) {
print(contents.length);
FileHolder f2 = new FileHolder("example_async_file_factory.dart");
f2.getContents().then((String contents) {
print(contents.length);
});
f2.getContents().then((String contents) {
print(contents.length);
});
f.getContents().then((String contents) {
print(contents.length);
});
});
}
Output:
read
1411
cached
cached
cached
1411
1411
1411
Regards
Robert

Saving scoped object outside scope in Dart

The following leaves s null after the file read exits:
String s;
new File('etc.stk').readAsString().then((String contents) {
s = contents;
});
// s is null here.
Is there a way to save (or clone) s, or am I compelled to use it only in the .then scope?
I have a few thousand lines of compiler/interpreter code that parses and runs the file contents, and would prefer not to have them all inside the new File scope.
EDIT
To provide more context, what I am trying to do is something like
new File('etc1.stk').readAsString()
.then((String script) {
syntaxTree1 = buildTree(script);
});
new File('etc2.stk').readAsString()
.then((String script) {
syntaxTree2 = buildTree(script);
});
and have access to both syntaxTree1 and syntaxTree2 in subsequent code. I will wrap my mind around the Dart Way if I can.
EDIT
(this code is tested)
import 'dart:async' as async;
import 'dart:io' as io;
void main(args) {
// approach1: inline
async.Future.wait([
new io.File('file1.txt').readAsString(),
new io.File('file2.txt').readAsString()
]).then((values) {
values.forEach(print);
});
// approach2: load files in another function
getFiles().then((values) {
values.forEach(print);
});
}
async.Future<List> getFiles() {
return async.Future.wait([
new io.File('file1.txt').readAsString(),
new io.File('file2.txt').readAsString()
]);
}
output:
file1
file2
file1
file2
EDIT END
hint: the code is not tested
// s is null here
is because this line is executed before
s = contents
This code
new File('etc.stk').readAsString()
returns a future that is enlisted in the event queue and is executed when the actual 'thread' of execution is finished.
If you had provided more code I would have a better context for a proposed solution.
What you could do is
String s;
new File('etc.stk').readAsString().then((String contents) {
s = contents;
}).then((_) {
// s is **NOT** null here.
});
or
//String s;
new File('etc.stk').readAsString().then((String contents) {
//s = contents;
someCallback(s)
});
// s is null here.
void someCallback(String s) {
// s is **NOT** null here
}
or
Future<String> myReadAsString() {
return new File('etc.stk').readAsString();
}
myReadAsString().then((s) {
// s is **NOT** null here
}
see also:
https://www.dartlang.org/slides/2013/06/dart-streams-are-the-future.pdf
async programming in dart
https://www.dartlang.org/docs/tutorials/#futures
https://api.dartlang.org/apidocs/channels/stable/dartdoc-viewer/dart:async
https://www.dartlang.org/articles/event-loop/
https://www.dartlang.org/articles/futures-and-error-handling/
and maybe
https://www.dartlang.org/articles/creating-streams/

What is the difference between Stream<List<int>> and Stream<int> in Dart

I am trying to wrap my head around Dart Streams. In particular this example of the command line utility cat has the following lines of code:
Stream<List<int>> stream = new File(path).openRead();
// Transform the stream using a `StreamTransformer`. The transformers
// used here convert the data to UTF8 and split string values into
// individual lines.
return stream
.transform(UTF8.decoder)
.transform(const LineSplitter())
.listen((line) {
if (showLineNumbers) {
stdout.write('${lineNumber++} ');
}
stdout.writeln(line);
}).asFuture().catchError((_) => _handleError(path));
The declaration of the Stream<T> as Stream<List<int>> has me a bit confused. Why is it not declared as a Stream<int>. How does the List<> type make this different. Are the subscriber events buffered in some way if it is a List?
What Type (as in <T>) is passed to the first transform? Is it an int or a List<int>?
What type is passed to each of the next transforms and what determines their type.
Does this example read the entire file before passing the results of the transform to the next transform? If so, is there an example somewhere of how to Stream very large files similar to this Node question Parsing huge logfiles in Node.js - read in line-by-line
Good question.
UTF8 is a Utf8Codec that extends Codec<String, List<int>>. So UTF8.decoder is a Converter<List<int>, String> that takes List<int> as parameter.
LineSplitter is a Converter<String, List<String>>. So it takes String as parameter. The resulting stream of .transform(const LineSplitter()) is a Stream<String> where each line is sent.
File.openRead doesn't read the entire file before writing the first bytes to the stream. So there's no problem to deal with large files.
Alexandre Ardhuin has the first three questions right. The 4th question however is not. After taking this apart and stubbing out my own version of the code I determined the following:
Even on a 37Mb file, the the transforms only get called once.
Here is the code I used to figure it out.
import 'dart:async';
import 'dart:convert';
import 'dart:io';
void main(List<String> arguments) {
Stream<List<int>> stream = new File('Data.txt').openRead();
stream
.transform(const Utf8InterceptDecoder())
.transform(const LineSplitterIntercept())
.listen((line) {
// stdout.writeln(line);
}).asFuture().catchError((_) => print(_));
}
int lineSplitCount = 0;
class LineSplitterIntercept extends LineSplitter {
const LineSplitterIntercept() : super();
// Never gets called
List<String> convert(String data) {
stdout.writeln("LineSplitterIntercept.convert : Data:" + data);
return super.convert(data);
}
StringConversionSink startChunkedConversion(ChunkedConversionSink<String> sink) {
stdout.writeln("LineSplitterIntercept.startChunkedConversion Count:"+lineSplitCount.toString()+ " Sink: " + sink.toString());
lineSplitCount++;
return super.startChunkedConversion(sink);
}
}
int utfCount = 0;
class Utf8InterceptDecoder extends Utf8Decoder {
const Utf8InterceptDecoder() : super();
//never gets called
String convert(List<int> codeUnits) {
stdout.writeln("Utf8InterceptDecoder.convert : codeUnits.length:" + codeUnits.length.toString());
return super.convert(codeUnits);
}
ByteConversionSink startChunkedConversion(ChunkedConversionSink<String> sink) {
stdout.writeln("Utf8InterceptDecoder.startChunkedConversion Count:"+ utfCount.toString() + " Sink: "+ sink.toString());
utfCount++;
return super.startChunkedConversion(sink);
}
}

Resources