The following field is causing my Kafka Stream application to fail when it tries to produce message.
{
"name" : "TS",
"type" : {
"type" : "long",
"logicalType" : "timestamp-millis"
},
"doc" : "date time indicating the timestamp of the position (in UTC)"
}
Below is the stack trace when the Stream tries to write the message to output topic
Caused by: org.apache.kafka.common.errors.SerializationException: Error serializing Avro message
Caused by: org.apache.avro.AvroRuntimeException: Unknown datum type org.joda.time.DateTime: 2018-02-12T06:14:37.000+05:00
at org.apache.avro.generic.GenericData.getSchemaName(GenericData.java:772)
at org.apache.avro.specific.SpecificData.getSchemaName(SpecificData.java:302)
at org.apache.avro.generic.GenericData.resolveUnion(GenericData.java:737)
at org.apache.avro.generic.GenericDatumWriter.resolveUnion(GenericDatumWriter.java:205)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:123)
at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:87)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:156)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:62)
at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:101)
The Serdes used to produce the message is instanitated as
final Map<String, String> serdeConfig = Collections.singletonMap(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG,"http://localhost:8081/");
final Serde<GenericRecord> valueGenericAvroSerde = new GenericAvroSerde();
valueGenericAvroSerde.configure(serdeConfig, false); // `false` for record values
Related
I have this GraphQL query:
query QuoteRequests($page: Int!) {
getQuoteRequestsList(page: $page) {
vehicle
body
licensePlate
vin
quality
currency
items
}
}
It generates this sample result:
{
"data": {
"getQuoteRequestsList": [
{
"vehicle": "1997 TOYOTA AVALON 4 DR ",
"body": "Sedan",
"licensePlate": "RHJ456",
"vin": "XBGGDFYYREAXVJJJD",
"quality": [
"GENUINO"
],
"currency": "USD",
"items": [
{
"uid": "74355f85-5312-9999-8acd-709ceccda00a",
"name": "Doble cero que es",
"description": "no me interesa",
"quantity": 11
},
{
"uid": "66db6fe0-1044-4d58-8454-5e51ab7a313f",
"name": "El arenero",
"description": "el duende verde",
"quantity": 2
}
]
},
]
}
}
The items data is a JSON type, and when trying to fetch that data by using Ferry package with
client.request(GQuoteRequestsReq((b) => b..vars.page = 0)).listen((response) => print(response.data.toString()));
I get a Null result, but if I leave items out of the query, I get a no-null result. I have no problem if I run that query using the HTTP package.
Is this is an error from Package? or do I need to configure something?
Thanks.
UPDATE 1
This is definitely an error because if I try to make a hot-reload in Flutter I get this error in debug console:
Reloaded 1 of 1700 libraries in 860ms.
E/flutter ( 2286): [ERROR:flutter/lib/ui/ui_dart_state.cc(209)] Unhandled Exception: Deserializing '[__typename, Query, getQuoteRequestsList, [{__typename: GetQuoteRequestsRecor...' to 'GQuoteRequestsData' failed due to: Deserializing '[{__typename: GetQuoteRequestsRecord, uid: 0bf6709f-7ab7-464e-8ee3-6a94e46f05...' to 'BuiltList<GQuoteRequestsData_getQuoteRequestsList>' failed due to: Deserializing '[__typename, GetQuoteRequestsRecord, uid, 0bf6709f-7ab7-464e-8ee3-6a94e46f057...' to 'GQuoteRequestsData_getQuoteRequestsList' failed due to: Deserializing '[{uid: 16870250-5acb-4c23-a7e4-f4e23bbd23ad, name: Doble cero que es, descrip...' to 'GJSON' failed due to: type 'List<dynamic>' is not a subtype of type 'String?' in type cast
E/flutter ( 2286): #0 BuiltJsonSerializers._deserialize
package:built_value/src/built_json_serializers.dart:178
E/flutter ( 2286): #1 BuiltJsonSerializers.deserialize
package:built_value/src/built_json_serializers.dart:124
It's clear it is fetching the expected result but it cannot deserialize properly, then it returns a null result.
Is there any workaround?
UPDATE 2
I've tried this solution but I'm getting same error.
In my case, I've followed these steps, I've
installed built_value package,
edited build.yaml file,
run the command flutter packages pub run build_runner build
The answer can be found in this issue.
I am trying a transformer in spring cloud stream to consume a json and generate an avro schema record. Can you give me an idea as to what is the right way to right this?
After following all the documentation I wrote the following transformer
#Bean
public Function<String, GenericRecord> transformMaestroDirectSearchRequest() {
return value -> {
try {
GenericRecord genericRecord = TransformJsonToAvro.convertJsonToAvro(
value, transformJsonToAvro.getMaestroDirectSearchSchema());
return genericRecord;
} catch (Exception e){
e.printStackTrace();
return null;
}
};
}
But I keep getting this error when I run the integration test
nested exception is org.springframework.messaging.converter.MessageConversionException: Could not write JSON: Not an enum:
#Test
public void maestroSearchRequestTransformTest() throws IOException, URISyntaxException {
GenericMessage<byte[]> inputMessage =
new GenericMessage<>(Utils.getJsonFile("Maestro_direct_Req.json").getBytes());
this.input.send(inputMessage);
Message<byte[]> receive = this.output.receive();
System.out.println(receive.getPayload());
}
Any ideas on how I can resolve this?
I have already verified that the conversion logic for creating the GenericRecord works well, and there is nothing wrong there (i.e. a valid GenericRecord Object is generated)
Here is the full stack trace
2020-11-18 13:36:31.112 ERROR 89354 --- [ main] o.s.integration.handler.LoggingHandler : org.springframework.messaging.MessageHandlingException: error occurred during processing message in 'MethodInvokingMessageProcessor' [org.springframework.integration.handler.MethodInvokingMessageProcessor#16fc5622]; nested exception is org.springframework.messaging.converter.MessageConversionException: Could not write JSON: Not an enum: {"type":"record","default":"null"}]}],"default":"null"}]} (through reference chain: org.apache.avro.generic.GenericData$Record["schema"]->org.apache.avro.Schema$RecordSchema["enumSymbols"])
at org.springframework.messaging.converter.MappingJackson2MessageConverter.convertToInternal(MappingJackson2MessageConverter.java:289)
at org.springframework.cloud.stream.converter.ApplicationJsonMessageMarshallingConverter.convertToInternal(ApplicationJsonMessageMarshallingConverter.java:69)
at org.springframework.messaging.converter.AbstractMessageConverter.toMessage(AbstractMessageConverter.java:198)
at org.springframework.cloud.function.context.config.NegotiatingMessageConverterWrapper.toMessage(NegotiatingMessageConverterWrapper.java:125)
at org.springframework.cloud.function.context.config.NegotiatingMessageConverterWrapper.toMessage(NegotiatingMessageConverterWrapper.java:139)
at org.springframework.messaging.converter.CompositeMessageConverter.toMessage(CompositeMessageConverter.java:83)
at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.convertValueToMessage(SimpleFunctionRegistry.java:705)
at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.convertOutputValueIfNecessary(SimpleFunctionRegistry.java:667)
at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.doApply(SimpleFunctionRegistry.java:600)
at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:443)
at org.springframework.cloud.function.context.catalog.SimpleFunctionRegistry$FunctionInvocationWrapper.apply(SimpleFunctionRegistry.java:431)
at org.springframework.cloud.stream.function.PartitionAwareFunctionWrapper.apply(PartitionAwareFunctionWrapper.java:71)
at org.springframework.cloud.stream.function.FunctionConfiguration$FunctionWrapper.apply(FunctionConfiguration.java:609)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:129)
at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:112)
at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:55)
at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:387)
at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:92)
at org.springframework.expression.spel.ast.SpelNodeImpl.getTypedValue(SpelNodeImpl.java:117)
at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:375)
at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:171)
at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:156)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.invokeExpression(MessagingMethodInvokerHelper.java:637)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.fallbackToInvokeExpression(MessagingMethodInvokerHelper.java:630)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.processInvokeExceptionAndFallbackToExpressionIfAny(MessagingMethodInvokerHelper.java:614)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.invokeHandlerMethod(MessagingMethodInvokerHelper.java:585)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.processInternal(MessagingMethodInvokerHelper.java:477)
at org.springframework.integration.handler.support.MessagingMethodInvokerHelper.process(MessagingMethodInvokerHelper.java:355)
at org.springframework.integration.handler.MethodInvokingMessageProcessor.processMessage(MethodInvokingMessageProcessor.java:108)
... 93 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Not an enum: {"type":"record",..... removed real json ,"default":"null"}]}],"default":"null"}]} (through reference chain: org.apache.avro.generic.GenericData$Record["schema"]->org.apache.avro.Schema$RecordSchema["enumSymbols"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:397)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:356)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:316)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:763)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper.writeValue(ObjectMapper.java:3058)
at org.springframework.messaging.converter.MappingJackson2MessageConverter.convertToInternal(MappingJackson2MessageConverter.java:271)
... 125 more
Caused by: org.apache.avro.AvroRuntimeException: Not an enum: {"type":"record",... removed real json...."default":"null"}]}],"default":"null"}]}
at org.apache.avro.Schema.getEnumSymbols(Schema.java:206)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:689)
at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)
... 133 more
I am using FSharp.Data.GraphQL for implementing GraphQL server. The documentation is not very clear about error handling. In the following mutation resolver, I wish to return errors if user provides invalid values for email and password:
let login =
Define.AsyncField(
name = "login",
typedef = Nullable Output.Token,
description = "Login using email and password",
args = [
Define.Input("email", String)
Define.Input("password", String)
],
resolve =
fun ctx _ ->
// ... some more code related to db and validation
let customErr: exn = Error.InvalidUserPassword
// This statement has not effect
ctx.AddError customErr
// Should I use raise to explicitly throw exception
// raise customErr
Async.result None
)
Looking into the source code, I found out AddError method on ResolveFieldContext. But it seems to produce no effect on the final result of the mutation. The output is always without any error:
{
"documentId": 1234,
"data": {
"login": null
},
// The errors array is not available
"errors": []
}
Should I raise the exception to add the error handling? And, if that is the case, how can I go about adding multiple error codes as part of single query or mutation since I can raise only one exception?
What is the meaning of the following error message:
I am attempting to convert the dogfood.nt to its rdf/xml representation form, what does the StackOverflow message indicate ?
<j.12:Person rdf:about="http://data.semanticweb.org/person/rich-keller">
<j.12:name>Rich Keller</j.12:name>
<rdfs:label>Rich Keller</rdfs:label>
<j.3:affiliation rdf:resource="http://data.semanticweb.org/organization/nasa-ames-research-center"/>
<j.4:holdsRole rdf:resource="http://data.semanticweb.org/conference/iswc/2005/pc-member-at-iswc2005-research-track"/>
</j.12:PersException in thread "main" java.lang.StackOverflowError
at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568)
at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
at java.util.regex.Pattern$BranchConn.match(Pattern.java:4568)
at java.util.regex.Pattern$GroupTail.match(Pattern.java:4717)
at java.util.regex.Pattern$Curly.match0(Pattern.java:4272)
at java.util.regex.Pattern$Curly.match(Pattern.java:4234)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
at java.util.regex.Pattern$Branch.match(Pattern.java:4604)
at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)
at java.util.regex.Pattern$Branch.match(Pattern.java:4604)
at java.util.regex.Pattern$Branch.match(Pattern.java:4602)
Following is the code snippet used:
Model model11 = ModelFactory.createDefaultModel();
InputStream is1 = FileManager.get().open("dogfood4.nt");
if (is1 != null) {
model11.read(is1, null, "N-TRIPLE");
model11.write(os1, "RDF/XML");
} else {
System.err.println("cannot read file ");;
}
I am using the semantic dogfood n-triples.
The following are the contents of config/default_mapping.json:
{
"_default_" : [
{
"int_template" : {
"match": "*",
"match_mapping_type": "int",
"mapping": {
"type": "string"
}
}
]
}
Want i want ES to do is to pick out all numbers from my logs and map them as strings.
Use case-
After clearing all indexes- curl -XDELETE 'http://localhost:9200/_all', i run this to send the following to ES (through fluentd's tailf plugin)-
echo "{\"this\" : 134}" >> /home/user/logs/program-data/logs/tiger/tiger.log
Elastic happily creates the initial indexes. Now, to test weather my default_mapping works, i send a string at the value where i previously sent an int.
echo "{\"this\" : \"ABC\"}" >> /home/user/logs/program-data/logs/tiger/tiger.log
Exception caught by ES-
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [this]
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:398)
at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:618)
at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:471)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:513)
at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:457)
at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:342)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:155)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)
Caused by: java.lang.NumberFormatException: For input string: "ABC"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:438)
at java.lang.Long.parseLong(Long.java:478)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.longValue(AbstractXContentParser.java:89)
What could be wrong here?
Update-
My default_mapping.json now looks like-
{
"_default_": {
"dynamic_templates": [
{
"string_template": {
"match": "*",
"mapping": {
"type": "string"
}
}
}
]
}
}
First of all, I'd suggest not to use file system based configuration or mappings. Just do it via api.
Your mapping is malformed, as you have the type name (_default_) but you don't specify that what you are submitting is a dynamic template.
As for the content, I'd remove that match_mapping_type if you want to map everything as a string.