Avro schema evolution testing and questions - avro

Question

With the following Avro schemas defined and test code, I have a couple questions when considering Avro schema evolution and how the first version of the Avro data can be stored and later retrieved using the second version of the schema. In my example, Person.avsc represents the first version, and PersonWithMiddleName.avsc represents the second version, where we have added a middleName attribute.
Is there a way to store the Avro schema and the binary encoded data as a byte array in Java? We are wanting to store our Avro objects to DynamoDB, and we'd like to store the Avro data as a blob with the schema stored alongside it (just like it is when stored to a file)? As reference, look at my Test Output below (the binary contents didn't copy, so the line just reads The Person is now serialized to a byte array: JoeCool) and compare what gets stored when Person is serialized to a byte array vs. when it is written out during the test to the person.avro file. As you can see, it appears as though the schema is only written out with the file and not with the byte array.
Is the AvroTypeException I encounter during my test truly expected as I have indicated with my comments in the catch block of the test? In this case, I have serialized the Person object as JSON and tried to deserialize it as PersonWithMiddleName.
Java Test Code
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.IOException;
import org.apache.avro.AvroTypeException;
import org.apache.avro.file.DataFileReader;
import org.apache.avro.file.DataFileWriter;
import org.apache.avro.io.BinaryDecoder;
import org.apache.avro.io.DatumReader;
import org.apache.avro.io.DecoderFactory;
import org.apache.avro.io.Encoder;
import org.apache.avro.io.EncoderFactory;
import org.apache.avro.io.JsonDecoder;
import org.apache.avro.specific.SpecificDatumReader;
import org.apache.avro.specific.SpecificDatumWriter;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class SchemaEvolutionTest {
Logger log = LoggerFactory.getLogger(this.getClass());
#Test
public void createAndReadPerson() {
// Create the Person using the Person schema
var person = new Person();
person.setFirstName("Joe");
person.setLastName("Cool");
log.info("Person has been created: {}", person);
SpecificDatumWriter<Person> personSpecificDatumWriter =
new SpecificDatumWriter<Person>(Person.class);
DataFileWriter<Person> dataFileWriter = new DataFileWriter<Person>(personSpecificDatumWriter);
try {
dataFileWriter.create(person.getSchema(), new File("person.avro"));
dataFileWriter.append(person);
dataFileWriter.close();
} catch (IOException e) {
Assertions.fail();
}
log.info("Person has been written to an Avro file");
// ******************************************************************************************************
// Next, read as Person from the Avro file using the Person schema
DatumReader<Person> personDatumReader =
new SpecificDatumReader<Person>(Person.getClassSchema());
var personAvroFile = new File("person.avro");
DataFileReader<Person> personDataFileReader = null;
try {
personDataFileReader = new DataFileReader<Person>(personAvroFile, personDatumReader);
} catch (IOException e1) {
Assertions.fail();
}
Person personReadFromFile = null;
while (personDataFileReader.hasNext()) {
// Reuse object by passing it to next(). This saves us from
// allocating and garbage collecting many objects for files with
// many items.
try {
personReadFromFile = personDataFileReader.next(person);
} catch (IOException e) {
Assertions.fail();
}
}
log.info("Person read from the file: {}", personReadFromFile.toString());
// ******************************************************************************************************
// Read the Person from the Person file as PersonWithMiddleName using only the
// PersonWithMiddleName schema
DatumReader<PersonWithMiddleName> personWithMiddleNameDatumReader =
new SpecificDatumReader<PersonWithMiddleName>(PersonWithMiddleName.getClassSchema());
DataFileReader<PersonWithMiddleName> personWithMiddleNameDataFileReader = null;
try {
personWithMiddleNameDataFileReader =
new DataFileReader<PersonWithMiddleName>(personAvroFile, personWithMiddleNameDatumReader);
} catch (IOException e1) {
Assertions.fail();
}
PersonWithMiddleName personWithMiddleName = null;
while (personWithMiddleNameDataFileReader.hasNext()) {
// Reuse object by passing it to next(). This saves us from
// allocating and garbage collecting many objects for files with
// many items.
try {
personWithMiddleName = personWithMiddleNameDataFileReader.next(personWithMiddleName);
} catch (IOException e) {
Assertions.fail();
}
}
log.info(
"Now a PersonWithMiddleName has been read from the file that was written as a Person: {}",
personWithMiddleName.toString());
// ******************************************************************************************************
// Serialize the Person to a byte array
byte[] personByteArray = new byte[0];
ByteArrayOutputStream personByteArrayOutputStream = new ByteArrayOutputStream();
Encoder encoder = null;
try {
encoder = EncoderFactory.get().binaryEncoder(personByteArrayOutputStream, null);
personSpecificDatumWriter.write(person, encoder);
encoder.flush();
personByteArray = personByteArrayOutputStream.toByteArray();
} catch (IOException e) {
log.error("Serialization error:" + e.getMessage());
}
log.info("The Person is now serialized to a byte array: {}", new String(personByteArray));
// ******************************************************************************************************
// Deserialize the Person byte array into a Person object
BinaryDecoder binaryDecoder = null;
Person decodedPerson = null;
try {
binaryDecoder = DecoderFactory.get().binaryDecoder(personByteArray, null);
decodedPerson = personDatumReader.read(null, binaryDecoder);
} catch (IOException e) {
log.error("Deserialization error:" + e.getMessage());
}
log.info("Decoded Person from byte array {}", decodedPerson.toString());
// ******************************************************************************************************
// Deserialize the Person byte array into a PesonWithMiddleName object
PersonWithMiddleName decodedPersonWithMiddleName = null;
try {
binaryDecoder = DecoderFactory.get().binaryDecoder(personByteArray, null);
decodedPersonWithMiddleName = personWithMiddleNameDatumReader.read(null, binaryDecoder);
} catch (IOException e) {
log.error("Deserialization error:" + e.getMessage());
}
log.info(
"Decoded PersonWithMiddleName from byte array {}", decodedPersonWithMiddleName.toString());
// ******************************************************************************************************
// Serialize the Person to JSON
byte[] jsonByteArray = new byte[0];
personByteArrayOutputStream = new ByteArrayOutputStream();
Encoder jsonEncoder = null;
try {
jsonEncoder =
EncoderFactory.get().jsonEncoder(Person.getClassSchema(), personByteArrayOutputStream);
personSpecificDatumWriter.write(person, jsonEncoder);
jsonEncoder.flush();
jsonByteArray = personByteArrayOutputStream.toByteArray();
} catch (IOException e) {
log.error("Serialization error:" + e.getMessage());
}
log.info("The Person is now serialized to JSON: {}", new String(jsonByteArray));
// ******************************************************************************************************
// Deserialize the Person JSON into a Person object
JsonDecoder jsonDecoder = null;
try {
jsonDecoder =
DecoderFactory.get().jsonDecoder(Person.getClassSchema(), new String(jsonByteArray));
decodedPerson = personDatumReader.read(null, jsonDecoder);
} catch (IOException e) {
log.error("Deserialization error:" + e.getMessage());
}
log.info("Decoded Person from JSON: {}", decodedPerson.toString());
// ******************************************************************************************************
// Deserialize the Person JSON into a PersonWithMiddleName object
try {
jsonDecoder =
DecoderFactory.get()
.jsonDecoder(PersonWithMiddleName.getClassSchema(), new String(jsonByteArray));
decodedPersonWithMiddleName = personWithMiddleNameDatumReader.read(null, jsonDecoder);
} catch (AvroTypeException ae) {
// Do nothing. We expect this since JSON didn't serialize anything out.
log.error(
"An AvroTypeException occurred trying to deserialize Person JSON back into a PersonWithMiddleName. Here's the exception: {}",ae.getMessage());
} catch (Exception e) {
log.error("Deserialization error:" + e.getMessage());
}
}
}
Person.avsc
{
"type": "record",
"namespace": "org.acme.avro_testing",
"name": "Person",
"fields": [
{
"name": "firstName",
"type": ["null", "string"],
"default": null
},
{
"name": "lastName",
"type": ["null", "string"],
"default": null
}
]
}
PersonWithMiddleName.avsc
{
"type": "record",
"namespace": "org.acme.avro_testing",
"name": "PersonWithMiddleName",
"fields": [
{
"name": "firstName",
"type": ["null", "string"],
"default": null
},
{
"name": "middleName",
"type": ["null", "string"],
"default": null
},
{
"name": "lastName",
"type": ["null", "string"],
"default": null
}
]
}
Test Output
Person has been created: {"firstName": "Joe", "lastName": "Cool"}
Person has been written to an Avro file
Person read from the file: {"firstName": "Joe", "lastName": "Cool"}
Now a PersonWithMiddleName has been read from the file that was written as a Person: {"firstName": "Joe", "middleName": null, "lastName": "Cool"}
The Person is now serialized to a byte array: JoeCool
Decoded Person from byte array {"firstName": "Joe", "lastName": "Cool"}
Decoded PersonWithMiddleName from byte array {"firstName": "Joe", "middleName": null, "lastName": "Cool"}
The Person is now serialized to JSON: {"firstName":{"string":"Joe"},"lastName":{"string":"Cool"}}
Decoded Person from JSON: {"firstName": "Joe", "lastName": "Cool"}
An AvroTypeException occurred trying to deserialize Person JSON back into a PersonWithMiddleName. Here's the exception: Expected field name not found: middleName
person.avro
Objavro.schema�{"type":"record","name":"Person","namespace":"org.acme.avro_testing","fields":[{"name":"firstName","type":["null","string"],"default":null},{"name":"lastName","type":["null","string"],"default":null}]}

Answer 1

For question one, I'm not a Java expert, but in Python instead of writing to an actual file, there is the concept of a file-like object that has the same interface as a file, but just writes to a byte buffer. So for example, instead doing this:
file = open(file_name, "wb")
# use avro library to write to file
file.close()
You can do something like this:
from io import BytesIO
bytes_interface = BytesIO()
# use bytes_interface the same way you would the previous "file" object
byte_output = bytes_interface.getvalue()
So that final byte_output would be the bytes that normally would have been written to a file but now is just a byte buffer that could be stored anywhere. Does Java have some concept like this? Alternatively I'm assuming there is some way in Java to read the file contents back into a byte buffer if you absolutely have to go through the process of writing an actual temporary file.
For question two, I think you are hitting the same problem mentioned in this Jira ticket: https://issues.apache.org/jira/browse/AVRO-2890. Currently the JSON decoder expects the schema the data was written with and can't do any sort of schema evolution with a different schema than the data was written with.