How do I set the arrayofmap in avro schema as optional field. The below schema is working, however, if this field is missing in the data, then the parsing is failing with org.apache.avro.AvroTypeException: Error converting field - quantities and.Caused by: org.apache.avro.AvroTypeException: Expected array-start. Got VALUE_NULL
` I just want to make sure the deserialization of data goes through whether the field is present in the data or not.
{
"name":"quantities",
"type":{
"items":{
"type":"map",
"values":"string"
},
"type":"array"
},
"default" : null,
}
Just found a solution myself. this will make the array of map fields optional in the avro schema
{
"name": "quantities",
"type": ["null",
{
"type": "array",
"items": {
"type": "map",
"values": "string"
}
}
],
"default": null,
}
Related
I am facing a strange issue that I extract a schema of api response and added json file in my serenity project. While validating schema, what ever the schema provided the test was passing, moreover if I changed any type of key like I change the data-type of any key value correct schema( like changed the name data-type from string to integer) then test failed.
Scenario:
My API response:
{
"name":"Alex",
"age" : 20,
"city":"New York"
}
My Schema for this API: Test Passed which is ok
{
"type": "object",
"properties": {
"name": {
"type": "string"
},
"age": {
"type": "integer"
},
"city": {
"type": "string"
}
},
"required": [
"name",
"age",
"city"
]
}
If I changed the schema from correct to wrong that is remove any key value pair the test even passed which is not correct
{
"type": "object",
"properties": {
"name": {
"type": "string"
},
"city": {
"type": "string"
}
},
"required": [
"name",
"city"
]
}
Moreover if I write only " { } " in the Schema file the test passed
The method I am using for validation is matchesJsonSchemaInClassPath
The schema validation only checks the data-types that are coming for values in JSON are correct by matching with schema. For the data validation there is another method in Serenity BDD that is VerifyResponseData.ofTheresponse(jsonobj)
This works for me
I am looking to convert JSON to Avro without altering the shape of the data
A nested field in the JSON contains a variable number of keys, which are never known in advance. The record that branches off of each of these unknown nodes however is of known, well-defined shape.
An example of this input data is as shown below
{
"customers": {
"paul_ince": {
"name": "Mr Paul Ince",
"age": 54
},
"kim_kardashian": {
"name": "Ms Kim Kardashian",
"age": 41
},
"elon_musk": {
"name": "Elon Musk, Technoking of Tesla",
"age": 50
}
}
Now it would be more avro friendly of course to have customers lead to an array of the form
{
"customers": [
{
"customer_name": "paul_ince",
"name": "Mr Paul Ince",
"age": 54
},
{
...
}
]
}
But this would violate my constraint that the shape of the input data be unchanged.
This problem seems to manifest itself frequently if I rely on data from external sources or scrapes, or preexisting data that was never created for Avro.
In my head, the schema would look something like the below,
{
"fields": [
{
"name": "customers",
"type": {
"type": "record",
"name": "customers",
"fields": [
{
"name": $customer_name,
"type": {
"type": "record",
"name": $customer_name,
"fields": [
{
"name": "name",
"type": "string",
},
{
"name": "age",
"type": "int"
}
]
}
}
]
}
}
]
}
where $customer_name is an assignment value, defined on read. Just asking the question it feels like this violates fundamental avro but I must use Avro and I strongly desire to maintain the input shape of the data. It would be highly impractical to modify this, not least given how frequently this problem appears and how large and varied the data I need to transfer from JSON to Avro is
I want to create an Avro schema for schema registry for the following Typescript code:
export type Value = {
[key: string]:
| Value
| Value[]
| string
| number;
};
It's a recursive map type. I know it is possible to create a recursive record like below, but it's a different use case.
export type Node = {
value: number;
leafs: Node[];
}
I tried different approaches, including named types and schema references, but all resulted in validation errors when publishing a schema.
A simplified schema (excluding the recursive array) that is desired but invalid looks like this:
{
"type": "record",
"name": "Value",
"namespace": "com.namespace",
"fields": [
{ "name": "itemValues", "type": { "type": "map", "values": ["string", "int", "itemValues"] } }
]
}
Most of variations of this schema result in an error: org.apache.avro.SchemaParseException: Undefined name: "itemValues"
I could not find examples of similar scenarios and wondered if it's even possible to create one like this? The limitation for that would most likely be the lack of named union and map types in Avro.
Update
An example JSON that I want to achieve:
{
"itemValues": {
"validA": "sth",
"validB": [],
"validC": 8,
"recursiveProperty": {
"anyMap": { "sth": "else" }
}
}
}
The problem with your schema is that your values list is ["string", "int", "itemValues"], but the parser is complaining because you have told it there should be some type itemValues and you haven't defined one. The only type you have defined is Value.
Here's the fixed schema (including adding the array of Value as one of the potential types:
{
"type": "record",
"name": "Value",
"namespace": "com.namespace",
"fields": [
{ "name": "itemValues", "type": { "type": "map", "values": ["string", "int", "Value", {"type": "array", "items": "Value"}] } }
]
}
I have a column that stores json. I am trying to make sure that only an array of objects can be stored in this column as described in the json schema below. The schema is working except for that I am able to save the attribute show as a string when it should be forced to be a boolean. For example, [{"name"=>"primary_phone", "show"=> "some text"}] is saving correctly but it shouldn't. How do I enforce that show must be a boolean?
{
"type": "array",
"items": {
"definitions": {
"name": { "type": "string" },
"show": {"type": "boolean"}
},
"required": ["name", "show"]
}
}
Your schema under "items" is invalid. Perhaps you meant "properties" instead of "definitions"?
I am completely new to Avro serialization and I am trying to get my head around how complex types are defined.
I am puzzled by how Avro generates the Enums in Java.
{
"type":"record",
"namespace":"com.example",
"name": "Customer",
"doc":"Avro Schema for our Customer",
"fields":[
{"name":"first_name","type":"string","doc":"First Name of Customer"},
{"name":"last_name","type":"string","doc":"Last Name of Customer"},
{"name":"automated_email","type":"boolean","doc":"true if the user wants marketing email", "default":true},
{
"name": "customer_type",
"type": ["null",
{
"name": "Customertype",
"type": "enum",
"symbols": ["OLD","NEW"]
}
]
}
]
}
Notice the customer_type field. If I give null, then in my generated sources I get the correct Enum type which is :
private com.example.Customertype customer_type;
But the moment I remove the null value and define customer_type in the following way:
{
"name": "customer_type",
"type": [
{
"name": "Customertype",
"type": "enum",
"symbols": ["OLD","NEW"]
}
]
}
The declaration changes to :
private Object customer_type;
What does that "null" string signify ? Why is it important ?
I have tried looking through the AVRO specification but nothing has given me a clear cut answer why this is working the way it is.
I am using the AVRO Maven plugin.
Any beginner resources for AVRO will also be appreciated.
Thank you.
If you are going to remove the null, you should remove the [ and ] brackets (because it is no longer a union).
So your customer_type schema should look like this:
{
"name": "customer_type",
"type": {
"name": "Customertype",
"type": "enum",
"symbols": ["OLD","NEW"]
}
}