Could you please tell me, if it's possible or not to implement such kind of schema evolution in Avro? So, V1 message could be converted into V2 automatically.
V1:
{ "title": "Some title" }
convert to V2:
{ "title": [ { "value": "Some title", "type": "main" } ] }
Assuming that old "title" field should be treated as "main" title in the new schema.
Or alternative option of V2:
{ "title": { "main": "Some title" } }
So, old value is transformed into map with "main" key.
Is it possible to do just by using Avro schema capabilities?
Related
I describe the structure of the outgoing JSON in the model class, however I cannot make the output as a list.
My model class:
class versions_info(BaseModel):
""" List of versions """
version : str = Field(..., title="Version",example="2.1.1")
url : str = Field(..., title="Url",example="https://ocpi.wedwe.ww/ocpi/2.1.1/")
And in the documentation I see:
However, I need it to be displayed as:
[
{
"version": "2.1.1",
"url": "https://www.server.com/ocpi/2.1.1/"
},
{
"version": "2.2",
"url": "https://www.server.com/ocpi/2.2/"
}
]
What am I doing wrong?
You can indicate that you're returning a List by wrapping the response_model you've defined in List[<model>].
So in your case it'd be:
#app.get('/foo', response_model=List[versions_info])
I have on query parameter which is little bit complex and i have my own syntax to make that value. Its has more then one variable to make one complete string value.
Let suppose name of parameter is index which has row and column like to make this value 20:30
index = { row: 20, col:30 }
index2 = { row: 20, col:30, chr: 15 }
Now i wanted to make it as
example.com?index=20:30
example.com?index2=20:30:15
Can someone tell me how can i define this in swagger ?
Thank you.
Make your swagger parameter a string and in your code behind handle the splitting into multiple variables...
I do exactly that here:
http://turoapi.azurewebsites.net/swagger/ui/index#/Echo/Echo_Get
"parameters": [
{
"name": "location",
"in": "query",
"description": "SoFL= 26.16,-80.20",
"required": true,
"type": "string"
},
That location is (Latitude,Longitude) and I split it with a C# TypeConverter
...and the request looks like:
http://turoapi.azurewebsites.net/api/Echo?location=26.16,-80.20
The code for that WebApi is here:
https://github.com/heldersepu/TuroApi
I'm new to Avro, and am trying to write some code to serialize some nested objects.
The structure of the objects looks like this:
class Parcel {
String recipe;
Map<Integer, PluginDump> dumps;
}
class PluginDump {
byte[] state;
Map<String, Param> params;
}
class Param {
Type type; //can be e.g. StringType, BooleanType, etc
Object value;
}
So I can't use a static avro schema - each PluginDump will have a different schema depending on the types within it.
I have written some code which can generate a Schema based on an individual PluginDump.
So when serializing a Parcel, how do I 'put' each PluginDump entry?
Here is my code:
Schema parcelSchema = AvroHelper.getSchema(p);
GenericRecord parcelRecord = new GenericData.Record(parcelSchema);
parcelRecord.put("recipe", p.getRecipe().toJson());
for (Map.Entry<Integer, PluginDump> entry : p.getDumps().entrySet()) {
PluginDump dump = entry.getValue();
Integer uid = entry.getKey();
Schema dumpSchema = AvroHelper.getSchema(dump);//will be different for each PluginDump
parcelRecord.put(????
Any ideas?
I have a feeling my approach is wrong, but I can't find any examples in the documentation of dynamic schema generation or nested maps.
1 When you get GenericRecord parcelRecord = new GenericData.Record(parcelSchema); you have two fields in your record: recipe and dumps, so you can't iterate through the dumps, you must put prepared map with dumps in the second field of record, just like you did it for recipe: parcelRecord.put("dumps", dumps);. But in this case, you'll get ClassCastException, because PluginDump cannot be cast to org.apache.avro.generic.IndexedRecord, so you need to put in parcelRecord a Map of GenericRecords. Also you need this for Map<String, Param> params, cause Param cannot be cast to IndexedRecord too.
2 Then, I think that its better to use Lists instead of Maps, cause avro not very good enough to work with Maps with different types of keys and values.
3 About the Param class: if you will use auto-generated schema, Param class will be presented like this.
"type": "record",
"name": "Param",
"fields": [
{
"name": "type",
"type": {
"type": "record",
"name": "Type",
"namespace": "java.lang.reflect",
"fields": []
}
},
{
"name": "value",
"type": {
"type": "record",
"name": "Object",
"namespace": "java.lang",
"fields": []
}
}
]
As far as avro uses java.lang.reflect, you will lose type field after deserialization, avro will not know what type it was.
If you want to generate avro-schema manually for each Param, considering its type, you can do something like this (I used ClassUtils.getClass from apache commons-lang3, cause standart Class.forName method doesn't always work properly):
public Schema getParamSchema() throws ClassNotFoundException {
List<Schema.Field> fields = new ArrayList<>();
fields.add(new Schema.Field("key", Schema.create(Schema.Type.STRING), "Doc: key field", (Object) null));
Schema.Field f = new Schema.Field("type", ReflectData.get().getSchema(ClassUtils.getClass(((Class) this.type).getName())), "Doc: type field", (Object) null);
f.addProp("java-class", ((Class) this.type).getName());
fields.add(f);
fields.add(new Schema.Field("value", ReflectData.get().getSchema(value.getClass()), "Doc: value field", (Object) null));
return Schema.createRecord(((Class) this.type).getName() + "Param", "Doc: param record", this.getClass().getPackage().getName(), false, fields);
}
But in this case, avro will throw ClassCastException, because it can't cast Class to Boolean, Integer etc. I always had a lot of problems working with avro and java Types and Classes.
So the best advice i think will be to change you model (Parcel, PluginDump and Param i mean) to have less problems with avro. For example you can store type name like a string, and get a Type with reflection after deserializing.
I'm working on JSON-LD serialization, and ideally I would like to have a #context which I can add to the existing GeoJSON output (together with some #ids and #types), so that both the Turtle output and the JSON-LD output will normalize to the same triples.
Data is organized as follows: each object/feature has an ID and a name, and data on one or more layers. Per layer, there is a data field, which contains a JSON object.
Example GeoJSON output:
{
"type": "FeatureCollection",
"features": [
{
"type": "Feature",
"properties": {
"id": "admr.nl.appingedam",
"name": "Appingedam",
"layers": {
"cbs": {
"data": {
"name": "Appingedam",
"population": 1092
}
},
"admr": {
"data": {
"name": "Appingedam",
"gme_code": 4654,
"admn_level": 3
}
}
}
},
"geometry": {…}
}
]
}
Example Turtle output:
<admr.nl.appingedam>
a :Node ;
dc:title "Appingedam" ;
:createdOnLayer <layer/admr> ;
:layerData <admr.nl.appingedam/admr> ;
:layerData <admr.nl.appingedam/cbs> .
<admr.nl.appingedam/admr>
a :LayerData ;
:definedOnLayer <layer/admr> ;
<layer/admr/name> "Appingedam" ;
<layer/admr/gme_code> "4654" .
<layer/admr/admn_level> "3" .
<admr.nl.appingedam/cbs>
a :LayerData ;
:definedOnLayer <layer/cbs> ;
<layer/cbs/name> "Appingedam" ;
<layer/cbs/population> "1092" ;
The properties object does not have its own URI. Is there a way to create a JSON-LD context which takes the contents of the properties into account, but further 'ignores' its precence?
Answered by Gregg Kellogg on JSON-LD mailing list:
This is something that keeps coming up: having a transparent layer,
that basically folds properties up a level. This was discussed during
the development of JSON-LD, but ultimately it was rejected.
I don't see any prospects for doing something in the short-term, but
it could be revisited in a possible future WG chartered with revising
the spec. Feedback like this is quite useful.
In the mean time, you can play with different JSON-LD encodings that
match your RDF though tools like http://json-ld.org/playground and my
own http://rdf.greggkellogg.net/distiller.
Gregg
Template
{#person alias=root}{alias.value}: {name}, {age}{/person}
data:
{
"root": {value:"MR."},
"person": {
"name": "Larry",
"age": 45
}
}
Expected output:
MR. Larry, 45
Actual output:
: Larry, 45
I'm trying to alias an object like shown above. But its not working. Please have a look into this fiddle http://jsfiddle.net/G86mu/1/.
If i replace {value:"MR."} with a string say "root":"Mr." and change my template to
{#person alias=root}{alias}: {name}, {age}{/person}
the output is as expected. Please let me know how do i alias an object
The reason this isn't working is because the context within Dust is not the same as the JSON you pass in to dust.render. Internally, Dust wraps your JSON so that it can include params, globals, and blocks in the context.
So, you are not adding alias to the current context, as you might assume. Instead, you are adding alias one level above your current context. Although the representation isn't exactly accurate, it should be helpful for explanation purposes:
// Incorrect:
{
"root": {
"value": "MR."
},
"person": {
// Current context
"alias": {
"value": "MR."
},
"name": "Larry",
"age": "45"
}
}
// (more) correct:
{
"root": {
"value": "MR."
},
"alias": {
"value": "MR."
"person": {
// Current context
"name": "Larry",
"age": "45"
}
}
}
When the context is viewed in this way, it makes sense why {#person alias=root}{alias.value}: {name}, {age}{/person} will not work. When using the dot-notation inside of a reference (as in {alias.value}, dust starts in the current context and goes down. Since there is no "alias" object inside of the current context, dust gives up, and you get an empty string.
However, if when you don't use the dot-notation, dust starts at the current context and searches up. The first time it finds a match, it will use that match. So, for your example, you could use the following to get your expected output.
{#person alias=root}{#alias}{value}{/alias}: {name}, {age}{/person}
Alternatively, if you could use:
{#person aliasVal=root.value}{aliasVal}: {name}, {age}{/person}