I cannot find a tool for checking whether avro schemas follow the avro spec. How can I check this?
I have generated a 300kB avro schema, but I do not know whether its syntax is according to the avro spec. The avro spec has quite a few quirks, like naming conventions, handling records in records and other quirks.
What is more, if I try my avro schema using a tool that validates json according to an avro schema, I get an unspecified error or crash.
Is there a tool, maybe a JSON Schema validator that can validate the avro schema itself?
If you want to validate the schema definition, then it's enough to try to build an object that is generated by the schema definition. You can do it in a unit test for example and when you will run the unit test, an exception will be thrown if you will not respect the avro schema definition.
If you want to check for compatibility issues, then you can use the Schema Registry REST API, more details here.
Best regards!
Related
I am a consumer and want to use SpecificRecord in Avro for type safety to avoid manually mapping things from a GenericRecord. As I understand things it makes little sense for me to still integrate with the schema registry because I could not do anything about the new schema in my consumer code anyways. Note that the schema registry has the full transitive mode enabled for any schema changes (I know that default values can still be changed and might lead to breaking changes).
Why use schema registry together with SpecificRecord in consumer code?
I can answer my own question, because we tried it, and it’s impossible to use the schema registry together with SpecificRecord if you want to have schema evolution. Your code simply breaks at runtime, because it cannot find the required version. Hence, the only way to work with Avro and schema evolution is via GenericRecord, and integrating with the schema registry in case of SpecificRecord provides no value.
Compare this with Protobuf, where you have both, generated code and schema evolution.
I am looking at openapi-generator to see if I can create an OpenAPI schema from an avro schema using the tool. Looking at the documentation, I have not been able to figure out how to provide the parameters to be able to do that. How can I achieve this?
Or, is there another way to get OpenAPI schema from an avro schema either using any online tool or programmatically and how?
The reason I need to do schema to schema conversion is that we use kafka streaming for message transmission and do not have a REST endpoint that could be used with Swagger that would generate the schema. Also, the generated message may not contain all attributes from the avro schema and doesn't identify the optional / mandatory attributes.
Thank you
GORM is clever enough to generate your SQL database schema as you run the app. But how can I get it to populate it with default records at the same time? e.g. reference data or whatever. Spring Boot mentions something about having a data.sql on your classpath, which sounds like the kind of thing needed.
I would like to disable my rails application's oracle database partitioning, but :
I don't know how to tell whether my app is using database partitioning
I don't know how to find the place my app use partitioning, since I didn't write most of the application's code myself
Can I just system search the code base for the keyword 'Partition' and look for any result that has the key word partition in raw SQL statement?
How should I go about this?
Thanks!
Update:
I have 2 answers below and they seem to understand my question differently
I am confused now as well. I want to disable the partition feature of my Oracle Database (https://www.oracle.com/technetwork/database/options/partitioning/overview/index.html), does that means I cannot use the 'partition by' keyword (Oracle "Partition By" Keyword) anymore?
Partitioning is done at the schema level declaratively. Usually, one would not expect the application code to directly need anything specific to use partitioning since it is done at the data definition level. You can connect to the schema owner account and check the data dictionary views USER_PART_TABLES for partitioned tables owned by the user and USER_PART_INDEXES for the indexes.
I'm looking for a way to convert an XML Schema definition file into an ActiveRecord modeled database. Does anyone know of a tool that happens to do this?
So far the best way I've found is to first load the XSD into an RDBMS like postgres or mysql and then have rails connect to do a rake db:schema:dump. This however, only leaves me with a database without rails Models.
What would be the best way to import/load this xsd based database into rails?
Did you try gem magic_model_generator to generate the model from the db?
See:
http://magicmodels.rubyforge.org/magic_model_generator/
Also, how did you load the XSD into your RDBMS? There seems to be plenty of discussion here:
How can I create database tables from XSD files?
Convert XSD into SQL relational tables
Update:
I used Xml Spy (30 free trail, on windows, ug) to build tables from the xsd in a mysql db, then ran magic_model_generator on it to create ActiveRecord classes. It appears to have worked as you would expect, generating validators on the fields based on the the db schema.