dymamic Schemas and nested Maps in Avro - avro

Question

I'm new to Avro, and am trying to write some code to serialize some nested objects.
The structure of the objects looks like this:
class Parcel {
String recipe;
Map<Integer, PluginDump> dumps;
}
class PluginDump {
byte[] state;
Map<String, Param> params;
}
class Param {
Type type; //can be e.g. StringType, BooleanType, etc
Object value;
}
So I can't use a static avro schema - each PluginDump will have a different schema depending on the types within it.
I have written some code which can generate a Schema based on an individual PluginDump.
So when serializing a Parcel, how do I 'put' each PluginDump entry?
Here is my code:
Schema parcelSchema = AvroHelper.getSchema(p);
GenericRecord parcelRecord = new GenericData.Record(parcelSchema);
parcelRecord.put("recipe", p.getRecipe().toJson());
for (Map.Entry<Integer, PluginDump> entry : p.getDumps().entrySet()) {
PluginDump dump = entry.getValue();
Integer uid = entry.getKey();
Schema dumpSchema = AvroHelper.getSchema(dump);//will be different for each PluginDump
parcelRecord.put(????
Any ideas?
I have a feeling my approach is wrong, but I can't find any examples in the documentation of dynamic schema generation or nested maps.

Answer 1

1 When you get GenericRecord parcelRecord = new GenericData.Record(parcelSchema); you have two fields in your record: recipe and dumps, so you can't iterate through the dumps, you must put prepared map with dumps in the second field of record, just like you did it for recipe: parcelRecord.put("dumps", dumps);. But in this case, you'll get ClassCastException, because PluginDump cannot be cast to org.apache.avro.generic.IndexedRecord, so you need to put in parcelRecord a Map of GenericRecords. Also you need this for Map<String, Param> params, cause Param cannot be cast to IndexedRecord too.
2 Then, I think that its better to use Lists instead of Maps, cause avro not very good enough to work with Maps with different types of keys and values.
3 About the Param class: if you will use auto-generated schema, Param class will be presented like this.
"type": "record",
"name": "Param",
"fields": [
{
"name": "type",
"type": {
"type": "record",
"name": "Type",
"namespace": "java.lang.reflect",
"fields": []
}
},
{
"name": "value",
"type": {
"type": "record",
"name": "Object",
"namespace": "java.lang",
"fields": []
}
}
]
As far as avro uses java.lang.reflect, you will lose type field after deserialization, avro will not know what type it was.
If you want to generate avro-schema manually for each Param, considering its type, you can do something like this (I used ClassUtils.getClass from apache commons-lang3, cause standart Class.forName method doesn't always work properly):
public Schema getParamSchema() throws ClassNotFoundException {
List<Schema.Field> fields = new ArrayList<>();
fields.add(new Schema.Field("key", Schema.create(Schema.Type.STRING), "Doc: key field", (Object) null));
Schema.Field f = new Schema.Field("type", ReflectData.get().getSchema(ClassUtils.getClass(((Class) this.type).getName())), "Doc: type field", (Object) null);
f.addProp("java-class", ((Class) this.type).getName());
fields.add(f);
fields.add(new Schema.Field("value", ReflectData.get().getSchema(value.getClass()), "Doc: value field", (Object) null));
return Schema.createRecord(((Class) this.type).getName() + "Param", "Doc: param record", this.getClass().getPackage().getName(), false, fields);
}
But in this case, avro will throw ClassCastException, because it can't cast Class to Boolean, Integer etc. I always had a lot of problems working with avro and java Types and Classes.
So the best advice i think will be to change you model (Parcel, PluginDump and Param i mean) to have less problems with avro. For example you can store type name like a string, and get a Type with reflection after deserializing.