faust Record does not deserialize avro Union type correctly - avro

I'm using faust-streaming and python-schema-registry-client to serialize faust.Record classes to avro.
When I try to deserialize a Union of two complex types however faust cannot reconstruct the correct records and only presents a dictionary instead of a faust.Record class.
I have the following code and schema:
schema_union = schema.AvroSchema({
'type': 'record',
'name': 'UnionRecord',
'fields': [
{
'name': 'entity',
'type': [
'null',
{
'name': 'company',
'type': 'record',
'fields': [
{'name': 'name_company', 'type': 'string'},
]
},
{
'name': 'person',
'type': 'record',
'fields': [
{'name': 'name_person', 'type': 'string'},
]
}
]
}
]
})
client = SchemaRegistryClient(url='http://localhost:8081')
faust_serializer_union = FaustSerializer(client, 'schema-union', schema_union)
class Company(faust.Record):
name_company: str
class Person(faust.Record):
name_person: str
class UnionRecord(
faust.Record,
validation=True,
serializer=faust_serializer_union
):
entity: Optional[Union[Company, Person]] # Union is typing.Union
record = UnionRecord(entity=Company(name_company='name'))
If I now try to serialize record and then deserialize it:
out = record.dumps() # == b'\x00\x00\x00\x00\x13\x02\x08name'
UnionRecord.loads(out)
I get this:
<UnionRecord: entity={'name_company': 'name'}>
Whereas I expect to get this:
<UnionNoneRecord: entity=<Company: name_company='name'>>
If I remove the Union type and alter the schema so that I can have a faust.Record that has only this field: Optional[Company], I do get the correct deserialization.

Related

toJson() invalid when recursively encode tree data structure

I have a Dart class that I am using as a node class for a tree data structure.
My goal here is to encode objects of this class and its child nodes recursively.
I have a toJson() method that takes the child Nodes List and calls jsonencode on them.
class Node{
String name;
Map<String, String> attributes;
List<Node> children = List<Node>();
Node(this.name, attributes) {
this.attributes = attributes;
this.children = List<Node>();
}
Node.fromJson(Map<dynamic,dynamic> _map) {
this.name = _map['name'];
this.children = new List<Node>();
this.attributes = _map['attributes'][0];
for(var i = 0; i < _map['children'].length;i++){
Node temp = new Node.fromJson(_map['children'][i]);
this.addChild(temp);
}
}
Map<String, dynamic> toJson() => {
'name': name,
'attributes': [attributes],
'children': [
...this.children.map(jsonEncode)
]
};
}
I have a unit test i created to test this functionality:
Node nodeMap = {
"name": "Name",
"attributes": [
{"#htag1": "tagval1"}
],
"children": [
{
"name": "NameChild1",
"attributes": [
{"#htag2": "tagval2"}
],
"children": []
},
{
"name": "NameChild2",
"attributes": [
{"#htag3": "tagval3"}
],
"children": []
}
]
};
UNode unodeInst = new UNode.fromJson(nodeMap);
// Act
var nodeCreate = nodeInst.toJson();
// Assert
expect(nodeCreate, equals(nodeMap));
Here is the output of my unit test
Expected: {
'name': 'Name',
'attributes': [{'#htag1': 'tagval1'}],
'children': [
{
'name': 'NameChild1',
'attributes': [{'#htag2': 'tagval2'}],
'children': []
},
{
'name': 'NameChild2',
'attributes': [{'#htag3': 'tagval3'}],
'children': []
}
]
}
Actual: {
'name': 'Name',
'attributes': [{'#htag1': 'tagval1'}],
'children': [
'{"name":"NameChild1","attributes":[{"#htag2":"tagval2"}],"children":[]}',
'{"name":"NameChild2","attributes":[{"#htag3":"tagval3"}],"children":[]}'
]
}
Which: at location ['children'][0] is '{"name":"NameChild1","attributes":[{"#htag2":"tagval2"}],"children":[]}' which expected a map
As you see its not encoding my object correctly.
I believe this is happening because when i reclusively call jsonencode this method returns a string that is placed into the children array.
I believe part of my problem is that i dont fully understand the d diffrence between jsonencode() and toJson().
It is my understanding that jsonencode() calls toJson().. but jsonencode() returns a string and toJson() returns a Map<String, dynamic>.. so i think what i want here is to call toJson() recursively and not jsonencode.
Does this sound correct?
But i cannot figure out how to do this on a list in this situation.
I have tried the following
...this.children.map(this.toJson())
but i get "The argument type 'Map<String, dynamic>' can't be assigned to the parameter type 'dynamic Function(Node)'"
...this.children.forEach((element) {element.toJson()})
but i get "Spread elements in list or set literals must implement 'Iterable'"
Does this mean i have to implement the Iterable interface in my class?
You're just using the map method incorrectly. Use the following instead.
[
...this.children.map((e) => e.toJson())
]
It's also unnecessary to use spread with a literal list or use this. You can simplify the code to just
children.map((e) => e.toJson()).toList()

How to make each parameter appear in a table row?

I have a device that get some telemetry data via REST API. Some of the data that it received is in the following format:
{
...
parameters: [
{
'name': 'parameter1',
'grade': '2',
'info': 'some informtion'
},
{
'name': 'parameter2',
'grade': '1',
'info': 'some informtion'
},
...
]
}
what I want to do is to visualize the data in the following way:
name | grade | info
---------------------------------------
parameter1 | 2 | some information
parameter2 | 1 | some information
... | ... | ...
now if I break down each parameter and send it to the device separately it will override the previous one.
How can I make that?
Figured out a way to do this:
Go to 'Widget Bundle' and create a new widget bundle.
Create a new widget of type 'Latest values'.
Here we have CSS/HTML section and a Javascript section.
HTML section:
<div class="my-data-table">
</div>
Javascript section:
self.defaultList = [
{
'id': 1,
'name': 'name 1',
'grade': 123,
'description': 'This is a description'
},
{
'id': 2,
'name': 'name 2',
'grade': 456,
'description': 'More description'
},
{
'id': 3,
'name': 'name 3',
'grade': 789,
'description': 'Even more description'
}
];
self.createTable = function(data) {
const columnNames = Object.keys(data[0]);
let tableHeadContent = $('<tr></tr>');
columnNames.forEach((columName) => {
tableHeadContent.append('<td>' + columName + '</td>');
});
let tableHead = $('<thead></thead>').append(tableHeadContent);
let tableBody = $('<tbody></tbody>');
data.forEach((currentElement, index) => {
const vals = Object.values(currentElement);
let currentRow = $('<tr></tr>');
vals.forEach((val) => {
currentRow.append('<td>' + val + '</td>');
});
tableBody.append(currentRow);
});
return $('<table></table>').append(tableHead).append(tableBody);
}
self.onInit = function() {
let currentList = [...self.defaultList];
if(self.ctx.defaultSubscription.data[0].data.length !== 0) {
currentList = JSON.parse(self.ctx.defaultSubscription.data[0].data[0][1]);
}
let currentTable = self.createTable(currentList);
$('.my-data-table', self.ctx.$container).append(currentTable);
}
self.onDataUpdated = function() {
self.ctx.detectChanges();
}
What you need to pay attention to and understand is self.ctx.defaultSubscription.data, when you visualize your data with a certain widget you subscribe the data to the widget. self.ctx.defaultSubscription.data give you access to the data, you can console.log() it to see how it is structured.
The self.defaultList is for the preview and when you set this widget to a specific data it will use that data.
There may be other way to do this but this is how I did it.

How to get a list of map objects from list of strings dart

I have the following list with phone numbers:
listofnumbers = ['01225','03933']
and a list of map objects as:
List mapofobjects = [
{
'name': 'John Doe',
'phone': {
'numbers': ['03323', '02333'],
'verified': true
},
'uid': '2BDNDD',
'createdat': 'today..'
},
{
'name': 'Mary Doe',
'phone': {
'numbers': ['03933', '39939'], // matches 03933 in listofnumbers
'verified': true
},
'uid': '1BDNDD',
'createdat': 'today..'
},
{
'name': 'Vincin Doe',
'phone': {
'numbers': ['01225', '59939'], // matches 01225 in listofnumbers
'verified': true
},
'uid': 'XBDNDD',
'createdat': 'today..'
}
];
How can I convert the listofnumbers into a list of map objects using each listofnumbers item as join.
I should get something like this for the listofnumbers of two numbers:
finalList = List mapofobjects = [
{
'name': 'John Doe',
'phone': {
'numbers': ['03323', '02333'],
'verified': true
},
'uid': '1BDNDD',
'createdat': 'today..'
},
{
'name': 'Vincin Doe',
'phone': {
'numbers': ['01225', '59939'],
'verified': true
},
'uid': 'XBDNDD',
'createdat': 'today..'
}
];
With each object matching/replacing a listofnumbers of item when phone['numbers'] contains the item.
You can use two forEach to get this result, like this:
List finallist = [];
listofnumbers.forEach((element) {
mapofobjects.forEach((e) => {
if (e['phone']['numbers'].contains(element))
finallist.add(e)
});
});
print(finallist.length.toString());
the result is: 2
You can do something like this:
final finalList = [
...mapofobjects.where((dynamic object) =>
object['phone']['numbers'].any((phone) => listofnumbers.contains(phone)))
];
finalList.forEach(print);
// {name: Mary Doe, phone: {numbers: [03933, 39939], verified: true}, uid: 1BDNDD, createdat: today..}
// {name: Vincin Doe, phone: {numbers: [01225, 59939], verified: true}, uid: XBDNDD, createdat: today..}

Nested/Sub Tables with PDFMake

How do I use nested/sub tables with PDFmake? I've tried simply putting in multiple tables but that doesn't automatically repeat the top level table's header for page breaks.
This code is a simplified example of using a sub-table. It is adapted from tables section of the pdfmake playground (wasn't easy to find via Google searching).
Paste the following into: http://pdfmake.org/playground.html
// playground requires you to assign document definition to a variable called dd
var dd = {
content: [
{ text: 'A simple table with nested elements', style: 'subheader' },
'It is of course possible to nest any other type of nodes available in pdfmake inside table cells',
{
style: 'tableExample',
table: {
headerRows: 1,
body: [
['Column 1', 'Column 2'],
[
{
stack: [
'Let\'s try an unordered list',
{
ul: [
'item 1',
'item 2'
]
}
]
},
[
'or a nested table',
{
table: {
body: [
[ 'Col1', 'Col2', 'Col3'],
[ '1', '2', '3'],
[ '1', '2', '3']
]
},
}
]
]
]
}
},
]
}
I need to achieve almost similar functionality like above. I have tried a lot, but not getting it right. How to create a json array that would produce an pdf output using pdfmake? I need to create a table within a table:
var dd = {
content: [
{
text: 'A simple table with nested elements',
style: 'subheader'
},
'It is of course possible to nest any other type of nodes available in pdfmake inside table cells',
{
style: 'tableExample',
table: {
headerRows: 1,
body: [
['Column 1', 'Column 2'],
[
[
'or a nested table',
{
table: {
body: [
[ 'Col1', 'Col2', 'Col3'],
[ '1', '2', '3'],
[ '1', '2', '3']
]
},
}
]
]
]
}
},
]
}

Requesting list of embedded objects

I have items endpoint which contains a list of embedded images. The scheme looks like:
_schema = {
'name': required_string, # group name
'description': {
'type': 'string',
'maxlength': 140,
},
'images': {
'type': 'list',
'scheme': {
'type': 'objectid',
'data_relation': {
'resource': 'images',
'embeddable': True,
'field': '_id',
}
},
}
}
So I'm trying to make a request to the items endpoint to get embedded objects
/items/549ae47f4fb9041305403292?embedded={"images":1}
But instead of embedded images I receive just the regular object with the list of images _ids.
Here is an example of object:
{
"_updated": "Wed, 24 Dec 2014 16:06:23 GMT",
"name": "New Item",
"images": [
"549ae47f4fb904130540328b",
"549ae47f4fb904130540328e",
"549ae47f4fb9041305403291"
],
"_created": "Wed, 24 Dec 2014 16:06:23 GMT",
"_id": "549ae47f4fb9041305403292",
"_etag": "949e3b731823bb2c08682ba4b6696b86856ef941",
"description": "The best item ever"
}
I tried to convert images ids in list to objectids, but it doesn't help. Any ideas why it doesn't work? Thanks
You have an incorrect schema definition. Replace scheme with schema when defining the images list:
_schema = {
'name': required_string, # group name
'description': {
'type': 'string',
'maxlength': 140,
},
'images': {
'type': 'list',
'schema': { # this was 'scheme' in your def
'type': 'objectid',
'data_relation': {
'resource': 'images',
'embeddable': True,
'field': '_id',
}
},
}
}
It will then properly embed your list of images.

Resources