I am using Confluent kafka in C# to consume Messages, these messages are formatted as hexadecimal strings from which the Schema fingerprint is extracted. How to get the schema from the schema fingerprint in C#? am I missing something?
I'm a bit late, but wanted to answer your question because I also could not find a definitive answer on this topic. This is what I found out:
The confluent schema registry uses ids to identify schemas.
When an event is serialized with the confluent schema registry (see e.g. here for the Java implementation for Avro), the schema id is embedded into the data: <Magic Byte><Schema ID><Payload>
Other serialization mechanisms that embedd schema fingerprints into the data, as e.g. Avro Single-Object are not supported by the schema registry. See this discussion on Github.
If you can only encodings with a schema fingerprint (i.e. where the schema registry is not available on serialization), the best option is to provide a cache on the consumer side, that maps fingerprint -> schema. This way the consumer can retrieve the fingerprint from the encoded event, lookup the schema from the cache and then decode the event using the retrieved schema.
The short answer: you can't recover the schema from the fingerprint.
The long one: fingerprint is a hash of your schema FingerPrint avro documentation. It's not designed to store the schema.
Fingerprint permits to identify which schema has been used to encode the message. It's used by some Schema registry and store all managed schemas in a map <FingerPrint, Schema>.
Related
I am working on a library that will consume and process JSON from a Graph API (the Users/Mail interface) and I wanted to build in a process to validate the JSON coming from Graph before I try to process it.
I wanted to use a JSON Schema to validate (https://www.newtonsoft.com/json/help/html/JsonSchema.htm) but I can't seem to find the schema for any response, but these are the important ones for me
https://graph.microsoft.com/v1.0/me/messages/{messageID}/attachments
https://graph.microsoft.com/v1.0/me/mailFolders
https://graph.microsoft.com/v1.0/me/messages
Can someone either bust my bubbly or point me toward where I can find the schema files?
Try Graph metadata used for generating client code files.
In my poc, I am using Spring cloud stream and connected to confluent schema registry. I can see schema's registered on schema registry, so I dont want to create own avro schema. I was able to get payload converted to POJO using application/json. Now I am trying to do the same with avro to Pojo conversion. But it doesnt seem to be working.
I am setting content type as ''contentType: application/*+avro''.
Any sample application, will be helpful.
I am looking for a way to export breeze entities on the server side to a json string which a breezejs manager can import from the client side. I looked all over the breeze APIs (both public and internal source code) but I couldn't find an obvious way of achieving this. There is a possibility of getting the desired results by using BreezeSharp (a .NET breeze client) on the server side but I would like to see if this is achievable with using the breeze server APIs only.
First you need to determine the shape of the bundle to be imported, i.e. something that manager.importEntities will understand. I don't think the format is documented, but you can reverse-engineer it by using:
var exported = manager.exportEntities(['Customer', 'Product'], {asString:true, includeMetadata:false});
Then pretty-print the value of exported to see the data format. See EntityManager.exportEntities for more info.
Once you have that, you can re-create it on the server. In C#, you can build it up using Dictionary and List objects, then serialize it using Json.NET.
An alternative approach would be to have your webhook just tell the client to initiate a query to retrieve the data from the server.
Riak supports auto generated keys when storing an object:
http://wiki.basho.com/Basic-Riak-API-Operations.html:
Store a new object and assign random key #
If your application would rather leave key-generation up to Riak, issue a POST request to
the bucket URL instead of a PUT to a bucket/key pair: POST
/riak/bucket If you don’t pass Riak a “key” name after the bucket, it
will know to create one for you.
is it possible to do the same when using the java client?
it seems that key must be provided when storing an object.
Edited to update: This is now available in the Java client. It was added in the 1.0.7 client release. See: https://github.com/basho/riak-java-client/pull/168
Unfortunately ... right now the Java client doesn't support this.
Someone has opened an issue for this: https://github.com/basho/riak-java-client/issues/141
I agree that it needs to be added. We've got a number of things we're working on at the moment for the Riak 1.2 release that are slightly higher priority, but I hope to work on this and get it added in the near future.
I have one type provider that connects to the network to retrieve data.
And produce (the fiction we call) 'static type' through type providers mechanism.
Of course, I might not always be connected. I might be raging in a private jet with satellite connection down.
Has anyone experience building an "offline type provider" which take (somehow) a type (from a type provider) as an input, stores its definition on disk, and provides you later with said type definition for easy access while on your way to Koh Phangan ?
Since types are not allowed as parameter to TP, I was thinking in providing assembly name + type name to be offlined.
You can enhance your original type provider to work both in online and offline modes. I.e. provider tries to connect to data source and fetch schema, if successful schema is cached on disk (in some format that provider can understand). After that provider exposes types using schema information on disk. If for some reason connection to data source is not available - provider checks if cached schema exists and if yes - uses it. For example standard type providers (LINQ2SQL or EF) allow you to specify schema file that can be used if direct connection to database is not possible.
This is a tricky aspect of writing F# type providers. But I think the main problem is that when you're developing in a private jet and you're using type providers to access some external data source, then you won't be able to access the data.
Schema caching -
If the type provider supports some form of schema caching (i.e. by storing the schema in a XML file like LINQ to SQL mentioned by #desco), then you'll be able to write some code and compile it, but you still won't be able to test the code. I think this makes schema caching less useful for the private-jet scenario. However, it is useful in scenario where you build code on a build server that does not have access to the schema.
Local data - For the private-jet scenario, you probably need some sort of local data (or a subset), to be actually able to test the code you write and then you can often point the type provider to your local copy (database, CSV or XML file etc.).
Meta-provider - I think the idea of having meta-provider is pretty cool - it should work to some extent - you would be able to cache the schema, but you probably wouldn't be able to cache the data (perhaps the values of properites, but I guess methods would not work). I think it should be possible to just pass the name of the provider to mock as an argument to your meta-provider. Something like:
type CachedDB =
SchemaCachingProvider<"FSharp.Data.TypeProviders.dll", "SqlDataConnection", "..">
I'm not aware of any plans for doing something like this, but if you started, I'm sure the FSharpX people would be interested in looking at it :-).