I'm trying to use a custom Coder so that I can do some transforms, but I'm having trouble getting the PCollection to use my custom coder, and I suspect (???) it's because it's wrapped in a KV. Specifically:
Pipeline p = Pipeline.create ...
p.getCoderRegistry().registerCoder(MyClass.class, MyClassCoder.class);
...
PCollection<String> input = ...
PCollection<KV<String, MyClass>> t = input.apply(new ToKVTransform());
When I try to run something like this, I get a java.lang.ClassCastException and a stacktrace that includes a SerializableCoder instead of MyClassCoder like I would expect.
[error] at com.google.cloud.dataflow.sdk.coders.SerializableCoder.decode(SerializableCoder.java:133)
[error] at com.google.cloud.dataflow.sdk.coders.SerializableCoder.decode(SerializableCoder.java:50)
[error] at com.google.cloud.dataflow.sdk.coders.KvCoder.decode(KvCoder.java:95)
[error] at com.google.cloud.dataflow.sdk.coders.KvCoder.decode(KvCoder.java:42)
I see that the answer to another, somewhat related question (Using TextIO.Write with a complicated PCollection type in Google Cloud Dataflow) says to map everything to strings, and use that to pass stuff around PCollections. Is that really the recommended way??
(Note: the actual code is in Scala, but I'm pretty sure it's not a Scala <=> Java issue so I've translated it into Java here.)
Update to include Scala code and more background:
So this is the actual exception itself (should have included this at the beginning):
java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.HashMap$SerializationProxy to field com.example.schema.Schema.keyTypes of type scala.collection.immutable.Map in instance of com.example.schema.Schema
Where com.example.schema.Schema is:
case class Schema(id: String, keyTypes: Map[String, Type])
And lastly, the SchemaCoder is:
class SchemaCoder extends com.google.cloud.dataflow.sdk.coders.CustomCoder[Schema] {
def decode(inputStream: InputStream, context: Context): Schema = {
val ois = new ObjectInputStream(inputStream)
val id: String = ois.readObject().asInstanceOf[String]
val javaMap: java.util.Map[String, Type] = ois.readObject().asInstanceOf[java.util.Map[String, Type]]
ois.close()
Schema(id, javaMap.asScala.toMap)
}
def encode(schema: Schema, outputStream: OutputStream, context: Context): Unit = {
val baos = new ByteArrayOutputStream()
val oos = new ObjectOutputStream(baos)
oos.writeObject(schema.id)
val javaMap: java.util.Map[String, Type] = schema.keyTypes.asJava
oos.writeObject(javaMap)
oos.close()
val encoded = new String(Base64.encodeBase64(baos.toByteArray()))
outputStream.write(encoded.getBytes())
}
}
====
Edit2: And here's what ToKVTransform actually looks like:
class SchemaExtractorTransform extends PTransform[PCollection[String], PCollection[Schema]] {
class InferSchemaFromStringWithKeyFn extends DoFn[String, KV[String, Schema]] {
override def processElement(c: DoFn[String, KV[String, Schema]]#ProcessContext): Unit = {
val line = c.element()
inferSchemaFromString(line)
}
}
class GetFirstFn extends DoFn[KV[String, java.lang.Iterable[Schema]], Schema] {
override def processElement(c: DoFn[KV[String, java.lang.Iterable[Schema]], Schema]#ProcessContext): Unit = {
val idAndSchemas: KV[String, java.lang.Iterable[Schema]] = c.element()
val it: java.util.Iterator[Schema] = idAndSchemas.getValue().iterator()
c.output(it.next())
}
}
override def apply(inputLines: PCollection[String]): PCollection[Schema] = {
val schemasWithKey: PCollection[KV[String, Schema]] = inputLines.apply(
ParDo.named("InferSchemas").of(new InferSchemaFromStringWithKeyFn())
)
val keyed: PCollection[KV[String, java.lang.Iterable[Schema]]] = schemasWithKey.apply(
GroupByKey.create()
)
val schemasOnly: PCollection[Schema] = keyed.apply(
ParDo.named("GetFirst").of(new GetFirstFn())
)
schemasOnly
}
}
This problem doesn't reproduce in Java; Scala is doing something differently with types that breaks Dataflow coder inference. To work around this, you can call setCoder on a PCollection to set its Coder explicitly, such as
schemasWithKey.setCoder(KvCoder.of(StringUtf8Coder.of(), SchemaCoder.of());
Here's the Java version of your code, just to make sure that it's doing approximately the same thing:
public static class SchemaExtractorTransform
extends PTransform<PCollection<String>, PCollection<Schema>> {
class InferSchemaFromStringWithKeyFn extends DoFn<String, KV<String, Schema>> {
public void processElement(ProcessContext c) {
c.output(KV.of(c.element(), new Schema()));
}
}
class GetFirstFn extends DoFn<KV<String, java.lang.Iterable<Schema>>, Schema> {
private static final long serialVersionUID = 0;
public void processElement(ProcessContext c) {
c.output(c.element().getValue().iterator().next());
}
}
public PCollection<Schema> apply(PCollection<String> inputLines) {
PCollection<KV<String, Schema>> schemasWithKey = inputLines.apply(
ParDo.named("InferSchemas").of(new InferSchemaFromStringWithKeyFn()));
PCollection<KV<String, java.lang.Iterable<Schema>>> keyed =
schemasWithKey.apply(GroupByKey.<String, Schema>create());
PCollection<Schema> schemasOnly =
keyed.apply(ParDo.named("GetFirst").of(new GetFirstFn()));
return schemasOnly;
}
}
Related
For instance, in Javascript I can do something like:
class Foo {
x = 'baz';
bar() {
const someVar = 'x';
console.log(this[someVar]);
// Output: 'baz';
}
}
Hopefully that's relatively clear - it boils down to accessing a member variable by another variable's contents. How is this achieved in Dart?
This is not trivial in Dart. Dart doesn't have a syntax to access class properties with [].
There are a couple of approaches though:
Mirrors:
https://api.dartlang.org/stable/2.6.1/dart-mirrors/dart-mirrors-library.html
Basically you have access to everything and offers the biggest freedom. You can check what properties a class has, access them via names and so on. Big disadvantage is that the generated JS (if targeting web) will be huge. Flutter doesn't support it at all.
Reflectable
To deal with the large generated JS, you can use package:reflectable. Never tried it with Flutter. It's a bit more to set up and start using bit it works.
Dart only solution 1
You can overload [] operator on a class:
class Foo {
final _backing = <String, String>{
'foo': 'bar'
};
operator [](String val) {
return _backing[val];
}
}
void main() {
final inst = Foo();
print(inst['foo']);
}
Dart only solution 2
Just use a map :) Well sort of... If you are dealing with complex types and you want to add some extra functionality to your map, you can do something like this:
import 'dart:collection';
class StringMap extends Object with MapMixin<String, String> {
final _backing = <String, String>{};
#override
String operator [](Object key) {
return _backing[key];
}
#override
void operator []=(String key, String value) {
_backing[key] = value;
}
#override
void clear() {
_backing.clear();
}
#override
Iterable<String> get keys => _backing.keys;
#override
String remove(Object key) {
return _backing.remove(key);
}
}
I've got the following code in a console application:
import 'dart:mirrors';
void main() {
final foo = Foo();
final mirror = reflect(foo);
final instanceMirror = mirror.invoke(#test, []);
print(instanceMirror);
}
class Foo {
int get test {return 42;}
}
When I run it I get an exception:
Exception has occurred.
NoSuchMethodError (NoSuchMethodError: Class 'int' has no instance method 'call'.
Receiver: 42
Tried calling: call())
If I set a breakpoint on test then it is hit before the exception, so it's definitely invoking the property.
Why is an exception being thrown?
UPDATE: ultimately what I am trying to achieve is to grab the values of all properties in an object. Per #mezoni's answer, it seems I need to treat properties as fields rather than methods (the opposite of C#, incidentally). However, it's still not entirely clear why or how to enumerate all fields. The best I've gotten is this:
import 'dart:mirrors';
void main() {
final foo = Foo();
final mirror = reflect(foo);
for (var k in mirror.type.instanceMembers.keys) {
final i = mirror.type.instanceMembers[k];
if (i.isGetter && i.simpleName != #hashCode && i.simpleName != #runtimeType) {
final instanceMirror = mirror.getField(i.simpleName);
print("${MirrorSystem.getName(i.simpleName)}: ${instanceMirror.reflectee}");
}
}
}
class Foo {
int get someOther {
return 42;
}
int get test {
return someOther + 13;
}
}
Please try this code:
import 'dart:mirrors';
void main() {
final foo = Foo();
final mirror = reflect(foo);
final instanceMirror = mirror.getField(#test);
print(instanceMirror.reflectee);
}
class Foo {
int get test {
return 42;
}
}
So basically, I have a situation where I want to inject primitive types into a class (i.e. a String and an Integer). You can think of a URL and port number for an application as example inputs. I have three components:
Now say I have a class, which does take in these params:
public class PrimitiveParamsDIExample {
private String a;
private Integer b;
public PrimitiveParamsDIExample(String a, Integer b) {
this.a = a;
this.b = b;
}
}
So my question here is simple. How do I inject a and b into class PrimitiveParamsDIExample?
In general, this is also asking how to inject parameters that are decided on runtime as well. If I have a and b above, read from STDIN or from an input file, they're obviously going to be different from run to run.
All the more, how do I do the above within the HK2 framework?
EDIT[02/23/15]: #jwells131313, I tried your idea, but I'm getting the following error (this one for the String param; similar one for int):
org.glassfish.hk2.api.UnsatisfiedDependencyException: There was no object available for injection at Injectee(requiredType=String,parent=PrimitiveParamsDIExample,qualifiers
I set up classes exactly as you did in your answer. I also overrode the toString() method to print both variables a and b in PrimitiveParamsDIExample. Then, I added the following in my Hk2Module class:
public class Hk2Module extends AbstractBinder {
private Properties properties;
public Hk2Module(Properties properties){
this.properties = properties;
}
#Override
protected void configure() {
bindFactory(StringAFactory.class).to(String.class).in(RequestScoped.class);
bindFactory(IntegerBFactory.class).to(Integer.class).in(RequestScoped.class);
bind(PrimitiveParamsDIExample.class).to(PrimitiveParamsDIExample.class).in(Singleton.class);
}
}
So now, I created a test class as follows:
#RunWith(JUnit4.class)
public class TestPrimitiveParamsDIExample extends Hk2Setup {
private PrimitiveParamsDIExample example;
#Before
public void setup() throws IOException {
super.setupHk2();
//example = new PrimitiveParamsDIExample();
example = serviceLocator.getService(PrimitiveParamsDIExample.class);
}
#Test
public void testPrimitiveParamsDI() {
System.out.println(example.toString());
}
}
where, Hk2Setup is as follows:
public class Hk2Setup extends TestCase{
// the name of the resource containing the default configuration properties
private static final String DEFAULT_PROPERTIES = "defaults.properties";
protected Properties config = null;
protected ServiceLocator serviceLocator;
public void setupHk2() throws IOException{
config = new Properties();
Reader defaults = Resources.asCharSource(Resources.getResource(DEFAULT_PROPERTIES), Charsets.UTF_8).openBufferedStream();
load(config, defaults);
ApplicationHandler handler = new ApplicationHandler(new MyMainApplication(config));
final ServiceLocator locator = handler.getServiceLocator();
serviceLocator = locator;
}
private static void load(Properties p, Reader r) throws IOException {
try {
p.load(r);
} finally {
Closeables.close(r, false);
}
}
}
So somewhere, the wiring is messed up for me to get an UnsatisfiedDependencyException. What have I not correctly wired up?
Thanks!
There are two ways to do this, but one isn't documented yet (though it is available... I guess I need to work on documentation again...)
I'll go through the first way here.
Basically, you can use the HK2 Factory.
Generally when you start producing Strings and ints and long and scalars like this you qualify them, so lets start with two qualifiers:
#Retention(RUNTIME)
#Target( { TYPE, METHOD, FIELD, PARAMETER })
#javax.inject.Qualifier
public #interface A {}
and
#Retention(RUNTIME)
#Target( { TYPE, METHOD, FIELD, PARAMETER })
#javax.inject.Qualifier
public #interface B {}
then write your factories:
#Singleton // or whatever scope you want
public class StringAFactory implements Factory<String> {
#PerLookup // or whatever scope, maybe this checks the timestamp?
#A // Your qualifier
public String provide() {
// Write your code to get your value...
return whatever;
}
public void dispose(String instance) {
// Probably do nothing...
}
}
and for the Integer:
#Singleton // or whatever scope you want
public class IntegerBFactory implements Factory<Integer> {
#PerLookup // or whatever scope, maybe this checks the timestamp?
#B // Your qualifier
public Integer provide() {
// Write your code to get your value...
return whatever;
}
public void dispose(String instance) {
// Probably do nothing...
}
}
Now lets re-do your original class to accept these values:
public class PrimitiveParamsDIExample {
private String a;
private int b;
#Inject
public PrimitiveParamsDIExample(#A String a, #B int b) {
this.a = a;
this.b = b;
}
}
Note I changed Integer to int, well... just because I can. You can also just use field injection or method injection in the same way. Here is field injection, method injection is an exercise for the reader:
public class PrimitiveParamsDIExample {
#Inject #A
private String a;
#Inject #B
private int b;
public PrimitiveParamsDIExample() {
}
}
There are several ways to bind factories.
In a binder: bindFactory
Using automatic class analysis: addClasses
An EDSL outside a binder: buildFactory
I was wondering if is possible to create an instance of a generic type in Dart. In other languages like Java you could work around this using reflection, but I'm not sure if this is possible in Dart.
I have this class:
class GenericController <T extends RequestHandler> {
void processRequest() {
T t = new T(); // ERROR
}
}
I tried mezonis approach with the Activator and it works. But it is an expensive approach as it uses mirrors, which requires you to use "mirrorsUsed" if you don't want to have a 2-4MB js file.
This morning I had the idea to use a generic typedef as generator and thus get rid of reflection:
You define a method type like this: (Add params if necessary)
typedef S ItemCreator<S>();
or even better:
typedef ItemCreator<S> = S Function();
Then in the class that needs to create the new instances:
class PagedListData<T>{
...
ItemCreator<T> creator;
PagedListData(ItemCreator<T> this.creator) {
}
void performMagic() {
T item = creator();
...
}
}
Then you can instantiate the PagedList like this:
PagedListData<UserListItem> users
= new PagedListData<UserListItem>(()=> new UserListItem());
You don't lose the advantage of using generic because at declaration time you need to provide the target class anyway, so defining the creator method doesn't hurt.
You can use similar code:
import "dart:mirrors";
void main() {
var controller = new GenericController<Foo>();
controller.processRequest();
}
class GenericController<T extends RequestHandler> {
void processRequest() {
//T t = new T();
T t = Activator.createInstance(T);
t.tellAboutHimself();
}
}
class Foo extends RequestHandler {
void tellAboutHimself() {
print("Hello, I am 'Foo'");
}
}
abstract class RequestHandler {
void tellAboutHimself();
}
class Activator {
static createInstance(Type type, [Symbol constructor, List
arguments, Map<Symbol, dynamic> namedArguments]) {
if (type == null) {
throw new ArgumentError("type: $type");
}
if (constructor == null) {
constructor = const Symbol("");
}
if (arguments == null) {
arguments = const [];
}
var typeMirror = reflectType(type);
if (typeMirror is ClassMirror) {
return typeMirror.newInstance(constructor, arguments,
namedArguments).reflectee;
} else {
throw new ArgumentError("Cannot create the instance of the type '$type'.");
}
}
}
I don't know if this is still useful to anyone. But I have found an easy workaround. In the function you want to initialize the type T, pass an extra argument of type T Function(). This function should return an instance of T. Now whenever you want to create object of T, call the function.
class foo<T> {
void foo(T Function() creator) {
final t = creator();
// use t
}
}
P.S. inspired by Patrick's answer
2022 answer
Just came across this problem and found out that although instantiating using T() is still not possible, you can get the constructor of an object easier with SomeClass.new in dart>=2.15.
So what you could do is:
class MyClass<T> {
final T Function() creator;
MyClass(this.creator);
T getGenericInstance() {
return creator();
}
}
and when using it:
final myClass = MyClass<SomeOtherClass>(SomeOtherClass.new)
Nothing different but looks cleaner imo.
Here's my work around for this sad limitation
class RequestHandler {
static final _constructors = {
RequestHandler: () => RequestHandler(),
RequestHandler2: () => RequestHandler2(),
};
static RequestHandler create(Type type) {
return _constructors[type]();
}
}
class RequestHandler2 extends RequestHandler {}
class GenericController<T extends RequestHandler> {
void processRequest() {
//T t = new T(); // ERROR
T t = RequestHandler.create(T);
}
}
test() {
final controller = GenericController<RequestHandler2>();
controller.processRequest();
}
Sorry but as far as I know, a type parameter cannot be used to name a constructor in an instance creation expression in Dart.
Working with FLutter
typedef S ItemCreator<S>();
mixin SharedExtension<T> {
T getSPData(ItemCreator<T> creator) async {
return creator();
}
}
Abc a = sharedObj.getSPData(()=> Abc());
P.S. inspired by Patrick
simple like that.
import 'dart:mirrors';
void main(List<String> args) {
final a = A<B>();
final b1 = a.getInstance();
final b2 = a.getInstance();
print('${b1.value}|${b1.text}|${b1.hashCode}');
print('${b2.value}|${b2.text}|${b2.hashCode}');
}
class A<T extends B> {
static int count = 0;
T getInstance() {
return reflectClass(T).newInstance(
Symbol(''),
['Text ${++count}'],
{Symbol('value'): count},
).reflectee;
}
}
class B {
final int value;
final String text;
B(this.text, {required this.value});
}
Inspired by Patrick's answer, this is the factory I ended up with.
class ServiceFactory<T> {
static final Map<Type, dynamic> _cache = <String, dynamic>{};
static T getInstance<T>(T Function() creator) {
String typeName = T.toString();
return _cache.putIfAbsent(typeName, () => creator());
}
}
Then I would use it like this.
final authClient = ServiceFactory.getInstance<AuthenticationClient>(() => AuthenticationClient());
Warning: Erik made a very good point in the comment below that the same type name can exist in multiple packages and that will cause issues. As much as I dislike to force the user to pass in a string key (that way it's the consumer's responsibility to ensuring the uniqueness of the type name), that might be the only way.
What is the best pattern to use to convert objects from Javascript to their Dart class counter parts?
// car.dart
import 'part.dart';
class Car {
String paintColor;
List<Part> parts;
}
// part.dart
class Part {
String name;
String SKU;
}
// main.dart
import 'dart:html';
import 'dart:js';
import 'car.dart';
void main() {
var body = document.querySelector('body');
body.addEventListener('carSelect', loadCarHandler, false);
}
void loadCarHandler(event) {
// this is the contents of a CustomEvent from outside dart
// goal is to convert it into Car and Parts
LinkedHashMap obj = event.detail;
/*
this is what the `obj` looks like inside the debugger
obj = _LinkedHashMap
:paintColor = 'Red'
:parts = List[2]
0 = _LinkedHashMap
:name = 'Wheel'
:SKU = 'Z123
1 = _LinkedHashMap
:name = 'Tire'
:SKU = 'Z456'
*/
}
Should I do a conversion in the handler?
Allow the constructor to take a LinkedHashMap and convert it there?
Create a factory?
Is there something built into Dart I'm not aware of that would handle this?
What is the preferred dart way of handling this?
There are several libraries that allow to create Dart object from JSON datas. See morph, dartson or serialization.
You can also avoid mirrors by adding a constructor like this :
class Car {
String paintColor;
List<Part> parts;
Car();
Car.fromJson(json)
: paintColor = json['paintColor'],
parts = json['parts'].map((e) => new Part.fromJson(e)).toList();
}
class Part {
String name;
String SKU;
Part();
Part.fromJson(json)
: name = json['name'],
SKU = json['SKU'];
}