Dart read json objects from Uneditable large file - dart

i have a large file which that uneditable and i have more jsonObject into that, for example:
{"id":"#123456","v":"1"}
this isn't jsonArray which i can't parse and read it, for example:
String file= new File('file.json').readAsStringSync();
Map<String, dynamic> data = json.decode(file);
you think how can i read this objects from file and show key and values?

Try the following solution. I ended up making a parser for your input format which are returning each parsed JSON object in a Stream.
The parser will not work if any strings in your JSON contains { or }. If that is the case, I can expand the parser so it takes this into account but I don't want to make it more advanced than necessary.
import 'dart:async';
import 'dart:convert';
import 'dart:io';
Future<void> main() async {
final ids = await File('large_file.json')
.openRead()
.transform(const Utf8Decoder())
.transform<dynamic>(JsonObjectTransformer())
.map((dynamic json) => json['id'] as String)
.toList();
print(ids); // [#123456, #123456]
}
class JsonObjectTransformer extends StreamTransformerBase<String, dynamic> {
static final _openingBracketChar = '{'.codeUnitAt(0);
static final _closingBracketChar = '}'.codeUnitAt(0);
#override
Stream<dynamic> bind(Stream<String> stream) async* {
final sb = StringBuffer();
var bracketsCount = 0;
await for (final string in stream) {
for (var i = 0; i < string.length; i++) {
final current = string.codeUnitAt(i);
sb.writeCharCode(current);
if (current == _openingBracketChar) {
bracketsCount++;
}
if (current == _closingBracketChar && --bracketsCount == 0) {
yield json.decode(sb.toString());
sb.clear();
}
}
}
}
}

Related

Dart: How to convert a List<dynamic> to a List<Player>

I am getting data from a server via a signalR hook and I need to shape it into a List<Player>
Here is the receiving function, and sample data
connection.on('ReceiveLatestList', (message) {
//TODO: Convert server data into List<Player>
try {
List<Player> players = message as List<Player>;
} catch (e) {
print(e);
}
});
The message data looks like this:
What's the best way to transform this into a List<Player>
Player class
import 'dart:core';
import 'dart:math';
class Player {
int guestId;
String name;
int roomNumber;
int vote;
Player.create(this.name, this.roomNumber) {
int min = 100000000;
int max = 999999999;
var random = new Random();
this.guestId = min + random.nextInt(max - min);
this.vote = 0;
}
String toJson(Player p) {
return '{"guestId":"$guestId", "name":"$name","roomNumber":"$roomNumber","vote":"$vote"}';
}
}
Iterate json items of list and add them to List of type Player
List<Player> players= List<Player>.from(json.map((i) => Player.fromJson(i)));
and also add fromJson method to Player's class:
Player.fromJson(Map<String, dynamic> map)
{
this.someFeild= map['someFeild'];
this.someOtherFeild= map['someOtherFeild'];
...
}

Generate one file for a list of parsed files using source_gen in dart

I have a list of models that I need to create a mini reflective system.
I analyzed the Serializable package and understood how to create one generated file per file, however, I couldn't find how can I create one file for a bulk of files.
So, how to dynamically generate one file, using source_gen, for a list of files?
Example:
Files
user.dart
category.dart
Generated:
info.dart (containg information from user.dart and category.dart)
Found out how to do it with the help of people in Gitter.
You must have one file, even if empty, to call the generator. In my example, it is lib/batch.dart.
source_gen: ^0.5.8
Here is the working code:
The tool/build.dart
import 'package:build_runner/build_runner.dart';
import 'package:raoni_global/phase.dart';
main() async {
PhaseGroup pg = new PhaseGroup()
..addPhase(batchModelablePhase(const ['lib/batch.dart']));
await build(pg,
deleteFilesByDefault: true);
}
The phase:
batchModelablePhase([Iterable<String> globs =
const ['bin/**.dart', 'web/**.dart', 'lib/**.dart']]) {
return new Phase()
..addAction(
new GeneratorBuilder(const
[const BatchGenerator()], isStandalone: true
),
new InputSet(new PackageGraph.forThisPackage().root.name, globs));
}
The generator:
import 'dart:async';
import 'package:analyzer/dart/element/element.dart';
import 'package:build/build.dart';
import 'package:source_gen/source_gen.dart';
import 'package:glob/glob.dart';
import 'package:build_runner/build_runner.dart';
class BatchGenerator extends Generator {
final String path;
const BatchGenerator({this.path: 'lib/models/*.dart'});
#override
Future<String> generate(Element element, BuildStep buildStep) async {
// this makes sure we parse one time only
if (element is! LibraryElement)
return null;
String libraryName = 'raoni_global', filePath = 'lib/src/model.dart';
String className = 'Modelable';
// find the files at the path designed
var l = buildStep.findAssets(new Glob(path));
// get the type of annotation that we will use to search classes
var resolver = await buildStep.resolver;
var assetWithAnnotationClass = new AssetId(libraryName, filePath);
var annotationLibrary = resolver.getLibrary(assetWithAnnotationClass);
var exposed = annotationLibrary.getType(className).type;
// the caller library' name
String libName = new PackageGraph.forThisPackage().root.name;
await Future.forEach(l.toList(), (AssetId aid) async {
LibraryElement lib;
try {
lib = resolver.getLibrary(aid);
} catch (e) {}
if (lib != null && Utils.isNotEmpty(lib.name)) {
// all objects within the file
lib.units.forEach((CompilationUnitElement unit) {
// only the types, not methods
unit.types.forEach((ClassElement el) {
// only the ones annotated
if (el.metadata.any((ElementAnnotation ea) =>
ea.computeConstantValue().type == exposed)) {
// use it
}
});
});
}
});
return '''
$libName
''';
}
}
It seems what you want is what this issue is about How to generate one output from many inputs (aggregate builder)?
[Günter]'s answer helped me somewhat.
Buried in that thread is another thread which links to a good example of an aggregating builder:
1https://github.com/matanlurey/build/blob/147083da9b6a6c70c46eb910a3e046239a2a0a6e/docs/writing_an_aggregate_builder.md
The gist is this:
import 'package:build/build.dart';
import 'package:glob/glob.dart';
class AggregatingBuilder implements Builder {
/// Glob of all input files
static final inputFiles = new Glob('lib/**');
#override
Map<String, List<String>> get buildExtensions {
/// '$lib$' is a synthetic input that is used to
/// force the builder to build only once.
return const {'\$lib$': const ['all_files.txt']};
}
#override
Future<void> build(BuildStep buildStep) async {
/// Do some operation on the files
final files = <String>[];
await for (final input in buildStep.findAssets(inputFiles)) {
files.add(input.path);
}
String fileContent = files.join('\n');
/// Write to the file
final outputFile = AssetId(buildStep.inputId.package,'lib/all_files.txt');
return buildStep.writeAsString(outputFile, fileContent);
}
}

Prevent to read file many times

I am trying to write an i18n app. The program read a json file, that contains translation from languages and it based on json structure.
{
"EN": {
"TEXT1": "Hello",
"TEXT2": "March"
},
"DE": {
"TEXT1": "Hallo",
"TEXT2": "März"
}
}
My program read the json file in async way with the file class, the whole code
import 'dart:io';
import 'dart:async';
import 'package:json_object/json_object.dart';
abstract class I18n {
static _I18n _i18n;
factory I18n(String file, String lang) {
if(_i18n == null) {
_i18n = new _I18n(file, lang);
return _i18n;
}
return _i18n;
}
Future<String> getTextByMap(String textId);
}
class _I18n implements I18n {
File _file;
String _lang;
JsonObject _jsonContainer;
JsonObject _jsonFiltered;
Future<JsonObject> _imme;
// Parameters:
// file: The whole path and filename
// lang: Expected language
_I18n(String file, this._lang) {
this._file = new File(file);
}
// Read file and return the content of file.
Future<String> _readFileFromStream() {
var com = new Completer();
this._file.exists()
.then((fileExists) {
if(!fileExists) {
throw new StateError('File not found');
}
return this._file.readAsString()
.then((stream) => com.complete(stream));
});
return com.future;
}
void _convertContentToJson(String stream) {
this._jsonContainer = new JsonObject.fromJsonString(stream);
}
Future<JsonObject> _prepareData() {
return this._readFileFromStream().then((stream) {
_convertContentToJson(stream);
this._jsonFiltered = this._jsonContainer[this._lang];
return this._jsonFiltered;
});
}
Future<String> getTextByMap(String textId) {
return this._prepareData().then((filterd) {
return filterd[textId];
});
}
}
and the main code
import 'package:i18n/i18n.dart';
void main() {
var i18n = new I18n('../hello.json', 'EN');
i18n.getTextByMap('TEXT1').then((val) => print(val));
i18n.getTextByMap('TEXT2').then((val) => print(val));
}
Everything here, happen in dart async way, read json file etc. And everytime, when i call the method
i18n.getTextByMap('TEXT1').then((val) => print(val));
it gonna read the json file again and again. I tried to rewrite the method to prevent reading json file many times
Future<String> getTextByMap(String textId) {
if(this._jsonFiltered == null)
{
return this._prepareData().then((filterd) {
return filterd[textId];
});
}
return new Future(() => this._jsonFiltered[textId]);
}
but it doesn't work too, because dart works in async way.
My question is, how can i keep this json file content in an object? Read json file only one time and keep the contents in an object, it is better then read json file everytime, that is my opinion.
It could do everything in sync way, then i wouldn't have such as problem but this is not dart terminology.
In which order do dart execute I/O operations, like this?
Future
I/O Events
My solution would be to create a class with a factory constructor. The factory constructor always returns a object of that file.
Your problem is that futures are parallel. So both calls are executed in parallel. The solution is to let the first future complete and then do other stuff to be able to get cached results.
Then you can have a read() method that reads the value of the file if it is not present in the classes "contents" attribute for example - or if that attribute is not null, it loads the file in background.
In both cases a completer or future is returned you can listen on.
EDIT Example Code:
example_async_file_factory.dart
import 'dart:io';
import 'dart:async';
class FileHolder {
String _contents = null;
String path;
static Map<String, FileHolder> _files;
factory FileHolder(String path) {
if (_files == null) {
_files = {};
}
if (_files.containsKey(path)) {
return _files[path];
} else {
final fh = new FileHolder._internal(path);
_files[path] = fh;
return fh;
}
}
FileHolder._internal(this.path);
Future<String> getContents() {
if(_contents != null) {
print("cached");
return new Future.value(_contents);
} else {
print("read");
File f = new File(this.path);
Future<String> future = f.readAsString();
Completer completer = new Completer();
future.then((String c) {
_contents = c;
completer.complete(_contents);
});
return completer.future;
}
}
}
void main() {
FileHolder f = new FileHolder("example_async_file_factory.dart");
f.getContents().then((String contents) {
print(contents.length);
FileHolder f2 = new FileHolder("example_async_file_factory.dart");
f2.getContents().then((String contents) {
print(contents.length);
});
f2.getContents().then((String contents) {
print(contents.length);
});
f.getContents().then((String contents) {
print(contents.length);
});
});
}
Output:
read
1411
cached
cached
cached
1411
1411
1411
Regards
Robert

Extending Future

I find quite a lot about using but not about defining futures in Dart. Lets say I have letsWait() which takes quite some time. How do I use the Future class?
import 'dart:async';
void main() {
print('Let\'s get started');
ArtificialWait waitPoint = new ArtificialWait();
Future<String> future = waitPoint.letsWait();
// and how about printing the return here?
print('something fast');
}
class ArtificialWait extends Future<String> {
String letsWait() {
for (var i = 0; i < 5000000000; i++) {
// lol
}
return 'finally';
}
}
This try gives me a:
unresolved implicit call to super constructor 'Future()' class ArtificialWait extends Future<String> {
I don't know why you want to inherit from Future.
Normally you would use this like:
import 'dart:async';
void main() {
print('Let\'s get started');
artificialWait().then((e) => print(e));
// and how about printing the return here?
print('something fast');
}
Future<String> artificialWait () {
var completer = new Completer<String>();
Timer.run(() {
for (var i = 0; i < 5000000000; i++) {
// lol
}
completer.complete('finally');
});
return completer.future;
}
Instead of trying to extend a future, you just need to 'use' the future.
import 'dart:async';
void main() {
print('Let\'s get started');
ArtificialWait waitPoint = new ArtificialWait();
Future<String> future = waitPoint.letsWait();
// and how about printing the return here?
print('something fast');
}
class ArtificialWait {
Future<String> letsWait => new Future<String>(_letsWait);
String _letsWait() {
for (var i = 0; i < 5000000000; i++) {
// lol
}
return 'finally';
}
}
Generally a future can be constructed without using a completer except in certain circumstances. The default constructor for Future will automatically wrap your passed function (which takes no arguments) in a Timer.run() to perform it asynchronously.

How to implement dynamic properties in Dart?

I would like to be able to back a dynamic property with a Map using a lookup in noSuchMethod(). However the latest changes makes the incoming property reference name unavailable. I can understand the minification scenario requiring us to use Symbols rather than Strings for names, but this makes implementing serializable dynamic properties difficult. Anyone have good ideas on how to approach this problem?
I can't use String names since the String names are not fixed between calls to the minifier. (This would completely break serialization)
You can access the original name with MirrorSystem.getName(symbol)
So a dynamic class could look like :
import 'dart:mirrors';
class A {
final _properties = new Map<String, Object>();
noSuchMethod(Invocation invocation) {
if (invocation.isAccessor) {
final realName = MirrorSystem.getName(invocation.memberName);
if (invocation.isSetter) {
// for setter realname looks like "prop=" so we remove the "="
final name = realName.substring(0, realName.length - 1);
_properties[name] = invocation.positionalArguments.first;
return;
} else {
return _properties[realName];
}
}
return super.noSuchMethod(invocation);
}
}
main() {
final a = new A();
a.i = 151;
print(a.i); // print 151
a.someMethod(); // throws
}
You could do something like this:
import 'dart:json' as json;
main() {
var t = new Thingy();
print(t.bob());
print(t.jim());
print(json.stringify(t));
}
class Thingy {
Thingy() {
_map[const Symbol('bob')] = "blah";
_map[const Symbol('jim')] = "oi";
}
final Map<Symbol, String> _map = new Map<Symbol, String>();
noSuchMethod(Invocation invocation) {
return _map[invocation.memberName];
}
toJson() => {
'bob': _map[const Symbol('bob')],
'jim': _map[const Symbol('jim')]};
}
Update - dynamic example:
import 'dart:json' as json;
main() {
var t = new Thingy();
t.add('bob', 'blah');
t.add('jim', 42);
print(t.bob());
print(t.jim());
print(json.stringify(t));
}
class Thingy {
final Map<Symbol, String> _keys = new Map<Symbol, String>();
final Map<Symbol, dynamic> _values = new Map<Symbol, dynamic>();
add(String key, dynamic value) {
_keys[new Symbol(key)] = key;
_values[new Symbol(key)] = value;
}
noSuchMethod(Invocation invocation) {
return _values[invocation.memberName];
}
toJson() {
var map = new Map<String, dynamic>();
_keys.forEach((symbol, name) => map[name] = _values[symbol]);
return map;
}
}
If you only need "dynamic properties", it should be enough to use Symbols as keys in the Map. If you also want to serialize that map, then you need to keep track of the original String names and use those for serialization. When deserializing, you'd have to create new Symbols from those Strings.
Note that all these scenarios (and basically everything that involves new Symbol) require a compiler to create a mapping of original names to the minified ones and put this mapping into the program, which of course makes it bigger.
Thanks for the solution of #Alexandre Ardhuin, I made some modification to make it runnable.
import 'dart:mirrors';
class object {
final _properties = new Map<String, Object>();
object();
object.from(Map<String, Object> initial) {
initial.entries.forEach((element) => _properties[element.key] = element.value);
}
noSuchMethod(Invocation invocation) {
if (invocation.isAccessor) {
final realName = MirrorSystem.getName(invocation.memberName);
if (invocation.isSetter) {
// for setter realname looks like "prop=" so we remove the "="
final name = realName.substring(0, realName.length - 1);
_properties[name] = invocation.positionalArguments.first;
return;
} else {
return _properties[realName];
}
}
return super.noSuchMethod(invocation);
}
#override
String toString() {
return _properties.toString();
}
}
main() {
// we can't use var or object type here, because analysis will consider
// https://dart.dev/tools/diagnostic-messages#undefined_setter
// The setter 'i' isn't defined for the type 'object'
// So dynamic is required here!
dynamic a = object.from({'a': 123, 'b': 234});
a.i = 151;
print(a); // print {a: 123, b: 234, i: 151}
try {
a.someMethod(); // throws NoSuchMethodError
} catch (e) {
print(e);
}
}

Resources