Avro java serialization as json not working correctly - avro

I have a simple avro schema, from which I generated a java class using the avro-maven-plugin.
The avro schema is as follows:
{
"type": "record",
"name": "addressGeo",
"namespace": "com.mycompany",
"doc": "Best record address and list of geos",
"fields": [
{
"name": "version",
"type": "int",
"default": 1,
"doc": "version the class"
},
{
"name": "eventType",
"type": "string",
"default": "addressGeo",
"doc": "event type"
},
{
"name": "parcelId",
"type": "long",
"doc": "ParcelID of the parcel. Join parcelid and sequence with ParcelInfo"
},
{
"name": "geoCodes",
"type": {"type": "array", "items": "com.mycompany.geoCode"},
"doc": "Multiple Geocodes, with restrictions information"
},
{
"name": "brfAddress",
"type": ["null", "com.mycompany.address"],
"doc": "Address cleansed version of BRF"
}
]
}
If I construct a simple object using the builder, and serialize it using json, I get the following output:
{
"version": 1,
"eventType": {
"bytes": [
97,
100,
100,
114,
101,
115,
115,
71,
101,
111
],
"length": 10,
"string": null
},
"parcelId": 1,
"geoCodes": [
{
"version": 1,
"latitude": 1,
"longitude": 1,
"geoQualityCode": "g",
"geoSourceTypeID": 1,
"restrictions": "NONE"
}
],
"brfAddress": {
"version": 1,
"houseNumber": "1",
"houseNumberFraction": null,
"streetDirectionPrefix": null,
"streetName": "main",
"streetSuffix": "street",
"streetDirectionSuffix": null,
"fullStreetAddress": "1 main street, seattle, wa, 98101",
"unitPrefix": null,
"unitNumber": null,
"city": "seattle",
"state": "wa",
"zipCode": "98101",
"zipPlusFour": null,
"addressDPV": "Y",
"addressQualityCode": "good",
"buildingNumber": "1",
"carrierRoute": "t",
"censusTract": "c",
"censusTractAndBlock": "b",
"dataCleanerTypeID": 1,
"restrictions": "NONE"
}
}
Note the output of the eventType field. It is coming through as an array of bytes whereas the type of the field is a CharSequence.
Any idea why serialization is doing this? It works fine for other types that are strings.
I am using google-gson to serialize the object to json.

You might be working with a older version of avro, that uses CharSequence. Ideally string type should be java String type. I would suggest to update the avro version or have a look at this one - Apache Avro: map uses CharSequence as key

Related

My EF Core 6 OData endpoint is returning whole infrastructure instead of model?

My EF Core application is returning what I imagine is context information in response instead of model. When calling the endpoint with expand option. This is the result returned:
{
"$id": "1",
"instance": null,
"container": {
"$id": "2",
"name": "SurveyQuestionValidations",
"value": null,
"next0": {
"$id": "3",
"name": "Id",
"value": 1,
"autoSelected": true
},
"autoSelected": false
},
"model": {
"$id": "4",
"schemaElements": [
{
"$id": "5",
"declaredKey": [
{
"$id": "6",
"defaultValueString": null,
"propertyKind": 1,
"type": {
"isNullable": false,
"definition": {
"$id": "7",
"name": "Int32",
"namespace": "Edm",
"typeKind": 1,
"primitiveKind": 9,
"schemaElementKind": 1,
"fullName": "Edm.Int32"
}
},
"declaringType": {
"$ref": "5"
},
"name": "Id"
}
],
"schemaElementKind": 1,
"namespace": "Data.Models",
"name": "SurveyQuestion",
"fullName": "Data.ModelsSurveyQuestion",
"typeKind": 2,
"hasStream": false,
"isAbstract": false,
"isOpen": false,
"declaredProperties": [ ...etc
This only happens when using expand. And the query is really slow but I believe it is related.

How to fix parsing TOML file error in Rust? [duplicate]

I am trying to poll the GitHub API for issues and print them out. To do so, I need to deserialize a nested JSON structure that I receive from a cURL GET request.
I am trying to get the url for all the objects in the items array:
{
"total_count": 4905,
"incomplete_results": false,
"items": [
{
"url": "https://api.github.com/repos/servo/saltfs/issues/789",
"repository_url": "https://api.github.com/repos/servo/saltfs",
"labels_url":
"https://api.github.com/repos/servo/saltfs/issues/789/labels{/name}",
"comments_url": "https://api.github.com/repos/servo/saltfs/issues/789/comments",
"events_url": "https://api.github.com/repos/servo/saltfs/issues/789/events",
"html_url": "https://github.com/servo/saltfs/issues/789",
"id": 293260512,
"number": 789,
"title": "Stop setting $CARGO_HOME to its default value",
"user": {
"login": "SimonSapin",
"id": 291359,
"avatar_url": "https://avatars0.githubusercontent.com/u/291359?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/SimonSapin",
"html_url": "https://github.com/SimonSapin",
"followers_url": "https://api.github.com/users/SimonSapin/followers",
"following_url": "https://api.github.com/users/SimonSapin/following{/other_user}",
"gists_url": "https://api.github.com/users/SimonSapin/gists{/gist_id}",
"starred_url": "https://api.github.com/users/SimonSapin/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/SimonSapin/subscriptions",
"organizations_url": "https://api.github.com/users/SimonSapin/orgs",
"repos_url": "https://api.github.com/users/SimonSapin/repos",
"events_url": "https://api.github.com/users/SimonSapin/events{/privacy}",
"received_events_url": "https://api.github.com/users/SimonSapin/received_events",
"type": "User",
"site_admin": false
},
"labels": [
{
"id": 341722396,
"url": "https://api.github.com/repos/servo/saltfs/labels/E-easy",
"name": "E-easy",
"color": "02e10c",
"default": false
}
],
"state": "open",
"locked": false,
"assignee": null,
"assignees": [
],
"milestone": null,
"comments": 0,
"created_at": "2018-01-31T18:16:09Z",
"updated_at": "2018-01-31T18:16:49Z",
"closed_at": null,
"author_association": "MEMBER",
"body": "In `buildbot/master/files/config/environments.py` we set `CARGO_HOME` to Cargo’s default value. Now that `mach` does not set it (since https://github.com/servo/servo/pull/19395), this has no effect. We can remove these lines.",
"score": 1.0
},
{
"url": "https://api.github.com/repos/servo/servo/issues/19916",
"repository_url": "https://api.github.com/repos/servo/servo",
"labels_url": "https://api.github.com/repos/servo/servo/issues/19916/labels{/name}",
"comments_url": "https://api.github.com/repos/servo/servo/issues/19916/comments",
"events_url": "https://api.github.com/repos/servo/servo/issues/19916/events",
"html_url": "https://github.com/servo/servo/issues/19916",
"id": 293237180,
"number": 19916,
"title": "Use a macro to create null-terminated C strings",
"user": {
"login": "jdm",
"id": 27658,
"avatar_url": "https://avatars1.githubusercontent.com/u/27658?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/jdm",
"html_url": "https://github.com/jdm",
"followers_url": "https://api.github.com/users/jdm/followers",
"following_url": "https://api.github.com/users/jdm/following{/other_user}",
"gists_url": "https://api.github.com/users/jdm/gists{/gist_id}",
"starred_url": "https://api.github.com/users/jdm/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/jdm/subscriptions",
"organizations_url": "https://api.github.com/users/jdm/orgs",
"repos_url": "https://api.github.com/users/jdm/repos",
"events_url": "https://api.github.com/users/jdm/events{/privacy}",
"received_events_url": "https://api.github.com/users/jdm/received_events",
"type": "User",
"site_admin": false
},
"labels": [
{
"id": 89384911,
"url": "https://api.github.com/repos/servo/servo/labels/C-assigned",
"name": "C-assigned",
"color": "02d7e1",
"default": false
},
{
"id": 15997664,
"url": "https://api.github.com/repos/servo/servo/labels/E-easy",
"name": "E-easy",
"color": "02e10c",
"default": false
},
{
"id": 135307111,
"url": "https://api.github.com/repos/servo/servo/labels/I-cleanup",
"name": "I-cleanup",
"color": "e11d21",
"default": false
}
],
"state": "open",
"locked": false,
"assignee": null,
"assignees": [
],
"milestone": null,
"comments": 3,
"created_at": "2018-01-31T17:04:06Z",
"updated_at": "2018-01-31T22:03:56Z",
"closed_at": null,
"author_association": "MEMBER",
"body": "When we write them by hand (eg. `b\"some string\\0\"`), we invariably get them wrong in ways that are tricky to notice (https://github.com/servo/servo/pull/19915). We should use a macro like this instead:\r\n```rust\r\nmacro_rules! c_str {\r\n ($str:expr) => {\r\n concat!($str, \"\\0\").as_bytes()\r\n }\r\n}\r\n```\r\nThis would allow us to write code like `(c_str!(\"PEParseDeclarationDeclExpected\"), Action::Skip)` instead of https://github.com/emilio/servo/blob/d82c54bd3033cc3277ebeb4854739bebe4e20f2f/ports/geckolib/error_reporter.rs#L237. We should be able to clean up all of the uses in that file.\r\n\r\nNo need to run any automated tests; if it builds with `./mach build-geckolib`, then it's good enough for a pull request.",
"score": 1.0
}
]}
My request function makes a cURL request and receives the above JSON. I then use serde_json to deserialize the JSON
main.rs
extern crate serde;
#[macro_use]
extern crate serde_derive;
extern crate serde_json;
mod engine;
mod server;
use engine::request;
use std::string::String;
use self::serde_json::{Error, Value};
#[derive(Serialize, Deserialize)]
struct obj {
items: Vec<String>,
}
fn main() {
let output_jn: String = request(
"https://api.github.com/search/issues?q=is:issue+label:e-easy",
).to_string(); //gets json structure as string
let json: obj = serde_json::from_str(&output_jn).unwrap();
for elem in json.iter() {
println!("{:?}", elem);
}
}
I get the following error message
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value:
ErrorImpl { code: Message("invalid type: map, expected a sequence"),
line: 1, column: 0 }', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
I'm certain I'm making a stupid mistake in deserializing my JSON structure, I've tried a number of permutations and combinations but I couldn't get anything to work.
Have a look at this part of your JSON input data:
{
...
"items": [
{
...
"title": "Stop setting $CARGO_HOME to its default value",
...
}
]
}
The top-level data structure is a JSON map, so in Rust this will be represented as a struct. I will use your name Obj.
The top-level JSON map has a key called "items" so in Rust this will be a field items inside the Obj struct.
The value of "items" in the map is a JSON array, so in Rust let's use a Vec.
Each element in the JSON array is a JSON map so in Rust we need a struct for those. We can call it Issue.
Each issue has a JSON key called "title" so this will be a field title inside the Issue struct.
The value of "title" is a JSON string so we can use Rust's String type for the field.
#[derive(Deserialize, Debug)]
struct Obj {
items: Vec<Issue>,
}
#[derive(Deserialize, Debug)]
struct Issue {
title: String,
}
fn main() {
let j = /* get the JSON data */;
let issues = serde_json::from_str::<Obj>(j).unwrap();
for i in issues.items {
println!("{:#?}", i);
}
}

Getting error object field starting or ending with a [.]

While inserting the below document in an Elasticsearch index:
{
"id": "122223334444",
"name": "Mei",
"url": "mei-2019-tamil",
"alternate_urls": [
"mei-2019-tamil",
"sa-baskaran-aishwarya-rajesh-untitled"
],
"type": "Movie",
"poster": "ed3e439b-1ac1-45fe-a915-a5dae60257df",
"poster_url": "//assets.appserver.com/ed3e439b-1ac1-45fe-a915-a5dae60257df",
"alternate_names": [
"Mei",
"SA Baskaran - Aishwarya Rajesh Untitled"
],
"popularity": 0.2,
"info": {
"running_time": 0,
"cpl_types": [
"teaser",
"feature"
],
"has_cpls": true,
"genres": [
"Drama",
"Thriller"
],
"international_release_date": null,
"country_specific_release_dates": {},
"international_film_status": "CS",
"country_specific_film_statuses": {
"IN": "CS",
"CN": "CS",
"": "CS",
"SG": "CS"
},
"country_specific_certifications": {},
"language": "Tamil",
"synopsis": "A thriller film directed by SA Baskaran, starring Aishwarya Rajesh in the lead role.",
"schedules": {
"cities": [],
"countries": []
},
"featured": 0,
"movie_rating": 0,
"cast": [
{
"id": "05ffe715-db60-4947-a45a-99722537571c",
"name": "Aishwarya Rajesh",
"url": "aishwarya-rajesh",
"role": "Actress",
"poster": "65ab15b6-d54a-4965-95d5-38a839cee17d",
"poster_url": "//assets.appserver.com/65ab15b6-d54a-4965-95d5-38a839cee17d",
"type": "Person"
}
],
"crew": [
{
"id": "d9354648-5f48-4bf0-9a00-3de8c4d7a8d0",
"name": "SA Baskaran",
"url": "sa-baskaran",
"role": "Director",
"poster": null,
"poster_url": null,
"type": "Person"
}
]
},
"published": true
}
I'm getting the following Message:
Error: object field starting or ending with a [.] makes object
resolution ambiguous:
However, there is no value that is starting with .
I am clueless as regards which key is causing this issue.
Please help me identify and fix this issue.
ElasticSearch Version: 5.6.14
I am trying to index ES from a rails app using chewy gem.
Values of the JSON could be empty. But if the keys are empty then ES throws an error while indexing. Remove "":{} from the JSON and index again.
Reference: https://discuss.elastic.co/t/object-field-starting-or-ending-with-a-makes-object-resolution-ambiguous/123351

Swift Alamofire get data to equal

Hi Im get json data with Alamofire and get like this:
{
"prices": [
{
"id": 1,
"value": 1.327,
"stationId": 24,
"type": 0,
"score": 5
},
{
"id": 2,
"value": 1.319,
"stationId": 25,
"type": 0,
"score": 4
},...],
"stations": [
{
"id": 24,
"name": "...",
"address": "...",
"brandId": 1,
"location": ".."
},
{
"id": 25,
"name": "..",
"address": "..",
"brandId": 1,
"location": ".."
},..],
"brands": [
{
"id": 6,
"name": "AGIP"
},
{
"id": 2,
"name": "EKO"
}, ...]
How can I get all data with "type": 0
And then when get all data with type compare id from prices stations and brands and put to array or dictionary
You could map your response using this pod AlamofireObjectMapper and then you can filter this object using for cycle.

Creating an avro schema for an array with multiple record types?

I am creating an avro schema for a JSON payload that appear to have an array of multiple objects. I'm not sure exactly how to represent this in the schema. The key in question is content:
{
"id": "channel-id",
"name": "My Channel with a New Title",
"description": "Herpy me derpy merpus herpsum ner berp berps derp ter tee",
"privacyLevel": "<private|org>",
"planId": "some-plan-id",
"owner": "a-user-handle",
"curators": [
"user-handle-1",
"user-handle-2"
],
"members": 5,
"content": [
{
"id": "docker",
"slug": "docker",
"index": 1,
"type": "path"
},
{
"id": "such-linkage",
"slug": "such-linkage",
"index": 2,
"type": "external-link",
"details": {
"url": "http://some-dank-link.com",
"title": "My Dank Link",
"contentType": "External Link",
"level": "Beginner",
"duration": "PT34293H33M9S"
}
},
{
"id": "21f1e812-b10a-40df-8b52-3a1d05fc215c",
"slug": "windows-azure-storage-in-depth",
"index": 3,
"type": "course"
},
{
"id": "7c346c05-6416-42dd-80b2-d5e758de7926",
"slug": "7c346c05-6416-42dd-80b2-d5e758de7926",
"index": 4,
"type": "project"
}
],
"imageUrls": ["https://url/to/an/image", "https://url/to/another/image"],
"analyticsEnabled": true,
"orgDiscoverable": false,
"createdDate": "2015-12-31T01:23:45+00:00",
"archiveDate": "2015-12-31T01:23:45+00:00",
"messagePublishedAt": "2015-12-31T01:23:45+00:00"
}
If you are asking if it is possible create an array with different kind of records, it is. Avro support this through union. it would looks like .
{
"name": "myRecord",
"type":"record",
"fields":[
{
"name":"myArrayWithMultiplesTypes",
"type":{
"type": "array",
"items":[
{
"name":"typeOne",
"type":"record",
"fields":[
{"name":"name", "type":"string"}
]
},
{
"name":"typeTwo",
"type":"record",
"fields":[
{"name":"id", "type":"int"}
]
}
]
}
}
]
}
If you already have the records defined previously, then it could look like this:
{
"name": "mulitplePossibleTypes",
"type": [
"null",
{
"type": "array",
"items": [
"com.xyz.kola.cloud.events.itemmanager.Part",
"com.xyz.kola.cloud.events.itemmanager.Document",
"com.xyz.kola.cloud.events.itemmanager.DigitalModel",
"com.xyz.kola.cloud.events.itemmanager.Interface"
]
}
]
},

Resources