I would like to create an AVRO schema that uses fields from another schema without creating a nested record. Take the following example:
Schema A:
{
"type": "record",
"namespace": "example",
"name": "SchemaA",
"fields": [
{
"name": "foo",
"type": "string"
},
{
"name": "bar",
"type": "string"
}
]
}
I want to create Schema B which uses the fields from Schema A and adds an additional field, i.e. the result should be:
{
"type": "record",
"namespace": "example",
"name": "SchemaB",
"fields": [
{
"name": "foo",
"type": "string"
},
{
"name": "bar",
"type": "string"
},
{
"name": "abc",
"type": "string"
}
]
}
Is it possible to create this Schema B by referencing the fields of Schema A?
Note: I do not want to create a nested field like:
{
"type": "record",
"namespace": "example",
"name": "SchemaB",
"fields": [
{
"name": "SchemaA",
"type": "example.SchemaA"
},
{
"name": "abc",
"type": "string"
}
]
}
Related
Question 1
I'm wondering whether below schema is valid or not for an Avro schema. Note that it is missing name in the first object of fields array.
{
"name": "AgentRecommendationList",
"type": "record",
"fields": [
{
"type": {
"type": "array",
"items": {
"name": "friend",
"type": "record",
"fields": [
{
"name": "Name",
"type": "string"
},
{
"name": "phoneNumber",
"type": "string"
},
{
"name": "email",
"type": "string"
}
]
}
}
}
]
}
Which actually designed to target below kind of data
[
{
"Name": "1",
"phoneNumber": "2",
"email": "3"
},
{
"Name": "1",
"phoneNumber": "2",
"email": "3"
},
{
"Name": "1",
"phoneNumber": "2",
"email": "3"
}
]
Based on reading below, seems like array without name like this are not permitted
Avro Schema failure
There is no way to define and avro schema with an array without a field name.
https://avro.apache.org/docs/current/spec.html#schema_complex
name: a JSON string providing the name of the field (required), and
I'm suspecting that below is the correct ones
{
"name": "AgentRecommendationList",
"type": "record",
"fields": [
{
"name": "friends",
"type": {
"type": "array",
"items": {
"name": "friend",
"type": "record",
"fields": [
{
"name": "Name",
"type": "string"
},
{
"name": "phoneNumber",
"type": "string"
},
{
"name": "email",
"type": "string"
}
]
}
}
}
]
}
And it should have a data like below, in order to do the avro conversion successfully
{
"friends": [
{
"Name": "1",
"phoneNumber": "2",
"email": "3"
},
{
"Name": "1",
"phoneNumber": "2",
"email": "3"
},
{
"Name": "1",
"phoneNumber": "2",
"email": "3"
}
]
}
Question 2
Does below schema is a valid schema? This target the array without name in first example...
{
"name": "AgentRecommendationList",
"type": "array",
"items": {
"name": "friend",
"type": "record",
"fields": [
{
"name": "Name",
"type": "string"
},
{
"name": "phoneNumber",
"type": "string"
},
{
"name": "email",
"type": "string"
}
]
}
}
I will appreciate if anyone can confirm my understanding... thanks!
For question 1...
Everything you have written is right. The first schema, as you mentioned, is not valid because each field within a record needs to have a name. The corrected schema is valid and the corrected data is right for the updated schema.
For question 2...
The schema in question two is valid, but the AgentRecommendationList name will get ignored. Arrays don't have names. This might sound strange after looking at the examples in question one, but in those the name is part of the field specification, not the array.
so im trying to parse an object with this avro schema.
object is like:
myInfo: {size: 'XL'}
But Its behaving like the record type doesn't actually exist and im getting a undefined type name: data.platform_data.test_service.result.record at Function.Type.forSchema for it.
schema looks like:
"avro": {
"metadata": {
"loadType": "full",
"version": "0.1"
},
"schema": {
"name": "data.platform_data.test_service.result",
"type": "record",
"fields": [
{
"name": "myInfo",
"type": "record",
"fields": [{
"name": "size",
"type": {"name":"size", "type": "string"}
}]
}
]
}
}
I should mention im also using avsc for this. Anybody have any ideas? I've tried pretty much all combinations but afaik the only way of parsing out an objct like this is with record
Playing around with the schema, I found that "type": "record" is a problem. I moved it to nested definition. And it worked. Seems like description here is little bit confusing.
Change
Before:
{
"name": "myInfo",
"type": "record",
"fields": [{
"name": "size",
"type": {"name":"size", "type": "string"}
}]
}
After:
{
"name": "myInfo",
"type": {
"type": "record",
"name": "myInfo",
"fields": [
{
"name": "size",
"type": {"name":"size", "type": "string"}
}
]
}
}
Updated schema which is working:
{
"name": "data.platform_data.test_service.result",
"type": "record",
"fields": [
{
"name": "myInfo",
"type": {
"type": "record",
"name": "myInfo",
"fields": [
{
"name": "size",
"type": {"name":"size", "type": "string"}
}
]
}
}
]
}
To make a record attribute nullable, process is same as any other attribute. You need to union with "null" (as show in below schema):
{
"name": "data.platform_data.test_service.result",
"type": "record",
"fields": [
{
"name": "myInfo",
"type": [
"null",
{
"type": "record",
"name": "myInfo",
"fields": [
{
"name": "size",
"type": {
"name": "size",
"type": "string"
}
}
]
}
]
}
]
}
The complete schema is the following:
{
"type": "record",
"name": "envelope",
"fields": [
{
"name": "before",
"type": [
"null",
{
"type": "record",
"name": "row",
"fields": [
{
"name": "username",
"type": "string"
},
{
"name": "timestamp",
"type": "long"
}
]
}
]
},
{
"name": "after",
"type": [
"null",
"row"
]
}
]
}
I wanted to programmatically extract the following sub-schema:
{
"type": "record",
"name": "row",
"fields": [
{
"name": "username",
"type": "string"
},
{
"name": "timestamp",
"type": "long"
}
]
}
As you see, field "before" is nullable. I can extract it's schema by doing:
schema.getField("before").schema()
But the schema is not a record as it contains NULL at the beginning(UNION type) and I can't go inside to fetch schema of "row".
["null",{"type":"record","name":"row","fields":[{"name":"username","type":"string"},{"name":"tweet","type":"string"},{"name":"timestamp","type":"long"}]}]
I want to fetch the sub-schema because I want to create GenericRecord out of it. Basically I want to create two GenericRecords "before" and "after" and add them to the main GenericRecord created from full schema.
Any help will be highly appreciated.
Good news, if you have a union schema, you can go inside to fetch the list of possible options:
Schema unionSchema = schema.getField("before").schema();
List<Schema> unionSchemaContains = unionSchema.getTypes();
At that point, you can look inside the list to find the one that corresponds to the Type.RECORD.
I have some avro data like this which is printed in terminal.
{"cust_status_id":0, "cust_status_description":{"string":" Approved"}}
The avro schema which I have created is like
{
"namespace": "com.thp.report.model",
"type": "record",
"name": "PraStatusMaster",
"fields": [
{
"name": "cust_status_id",
"type": "int"
},
{
"name": "cust_status_description",
"type": "string",
"avro.java.string": "String"
}
]
}
Is the schema correct??
Correct schema for your json is the following one:
{
"name": "PraStatusMaster",
"type": "record",
"namespace": "com.thp.report.model",
"fields": [
{
"name": "cust_status_id",
"type": "int"
},
{
"name": "cust_status_description",
"type": {
"name": "cust_status_description",
"type": "record",
"fields": [
{
"name": "string",
"type": "string"
}
]
}
}
]
}
I am trying to create two Avro schemas using the avro-tools-1.7.4.jar create schema command.
I have two JSON schemas which look like this:
{
"name": "TestAvro",
"type": "record",
"namespace": "com.avro.test",
"fields": [
{"name": "first", "type": "string"},
{"name": "last", "type": "string"},
{"name": "amount", "type": "double"}
]
}
{
"name": "TestArrayAvro",
"type": "record",
"namespace": "com.avro.test",
"fields": [
{"name": "date", "type": "string"},
{"name": "records", "type":
{"type":"array","items":"com.avro.test.TestAvro"}}
]
}
When I run the create schema on these two files the first one works fine and generates the java. The second one fails every time. It does not like the array items when I try and use the first Schema as the type. This is the error I get:
Exception in thread "main" org.apache.avro.SchemaParseException: Undefined name: "com.test.avro.TestAvro"
at org.apache.avro.Schema.parse(Schema.java:1052)
Both files are located in the same path directory.
Use the below avsc file:
[{
"name": "TestAvro",
"type": "record",
"namespace": "com.avro.test",
"fields": [
{
"name": "first",
"type": "string"
},
{
"name": "last",
"type": "string"
},
{
"name": "amount",
"type": "double"
}
]
},
{
"name": "TestArrayAvro",
"type": "record",
"namespace": "com.avro.test",
"fields": [
{
"name": "date",
"type": "string"
},
{
"name": "records",
"type": {
"type": "array",
"items": "com.avro.test.TestAvro"
}
}
]
}]