I started kafka connector using following command:
./bin/connect-standalone etc/schema-registry/connect-avro-standalone.properties etc/kafka-connect-postgres/connect-postgres.properties
Serialization props in the connect-avro-standalone.properties is:
key.converter=io.confluent.connect.avro.AvroConverter
key.converter.schema.registry.url=http://localhost:8081
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=http://localhost:8081
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
I've created a java backend which listen to this kafka stream topic and its able to get the data from postgres with each add/update/delete.
But the data is coming in some unknown encoding format and that's why ican't read the data correctly.
Here is the relevant code snippet:
properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
Serdes.String().getClass().getName());
properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
Serdes.ByteArray().getClass().getName());
StreamsBuilder streamsBuilder = new StreamsBuilder();
final Serde<String> stringSerde = Serdes.String();
final Serde<byte[]> byteSerde = Serdes.ByteArray();
streamsBuilder.stream(Pattern.compile(getTopic()), Consumed.with(stringSerde, byteSerde))
.mapValues(data -> {
System.out.println("->"+new String(data));
return data;
});
I'm confused on where and what I need to change; in the avro connector prop or in the java side code
Your Kafka Connect config here means that the messages on the Kafka topic will be Avro serialised:
value.converter=io.confluent.connect.avro.AvroConverter
Which means that you need to deserialise using Avro in your Streams app. See here for more details: https://docs.confluent.io/current/streams/developer-guide/datatypes.html#avro
Related
I have a thingsboard UI available at "thingsboard.MYDOMAIN.io:8080". I'm using NodeJS and MQTT.js. I can connect using the sample code on that page. But I don't know what URL i would use in that connect string. I've tried using that URL and putting MQTT:// on the front of it to no avail.
It's just one device coming into a thingsboard. I wouldn't think i would need to add anything other than the default thingsboard.
If someone could tell what the expect default URL would be?
The default MQTT port number is 1883 so something like
mqtt://thingsboard.mydomain.io:1883
would be a good starting point. You should even be able to leave the port number off e.g.
mqtt://thingsboard.mydomain.io
Connection code should be something like this:
var mqtt = require('mqtt');
const thingsboardHost = "127.0.0.1";
const ACCESS_TOKEN = "**********";
// Initialization of mqtt client using Thingsboard host and device access token
console.log('Connecting to: %s using access token: %s', thingsboardHost, ACCESS_TOKEN);
var client = mqtt.connect('mqtt://'+ thingsboardHost, { username: ACCESS_TOKEN });
The default MQTT port number is 1883, but you don't have to manually set that value. Take a look at the source code.
https://github.com/mqttjs/MQTT.js
I am working on a Spring Cloud Stream Kafka application. I have added only consumers to consume messages from topics and deliver them to a third party using FIX protocol.
It is working fine till this point, but now the third party sends back the response and I would like to produce them to a new topic. When I added a Supplier in my existing code, it starts behaving weirdly. bootstrap.servers config changes from remoteHost broker to localhost and started giving below error:
[AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established> Broker may not be available.
error would come if trying to connect localhost as there isn't any Kafka setup.
Below is my application.yml file:
spring.cloud.stream.function.definition: amerData;emeaData;ackResponse #added new ackResponse here
spring.cloud.stream.kafka.streams:
binder:
brokers: remoteHost:9092
configuration:
schema.registry.url: remoteHost:8081
default.key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
default.value.serde: io.confluent.kafka.streams.serdes.avro.SpecificAvroSerde
bindings:
ackResponse-out-0: #new addition
producer.configuration:
key.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
value.serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
spring.cloud.stream.bindings:
amerData-in-0:
destination: topic1
emeaData-in-0:
destination: topic2
ackResponse-out-0: #new addition
destination: topic3
and tried possible options for Supplier -> Supplier<String> ackResponse() or Supplier<Message<String>> ackResponse()
It only doesn't change remoteHost to localhost when I am doing Supplier<KStream<String,String>> ackResponse(), then bootstrap.servers show the configured remote one, but this isn't correct and I can't write the received response (mostly a string or json) like this to a Kafka topic.
I did configure my consumers as Consumer<KStream<String, AVROPOJO1>> amerData() and Consumer<KStream<String, AVROPOJO2>> emeaData() as per need & they work fine.
Am I missing or messing up something? Can't we have producer/consumer both in the same spring cloud stream application? Using Streambridge also couldn't solve this. Could someone help?
If you are adding a Supplier bean as you have done, it becomes a regular producer that is using the MessageChannel based Kafka binder. You need to add the regular Kafka binder in your project (spring-cloud-stream-binder-kafka). The bindings for that should be under spring.cloud.stream.kafka.bindings. I see that you have it defined above under spring.cloud.stream.kafka.streams.bindings. I wonder if that is the issue?
I have a streaming topic in Json with 50 fields. I try to create another stream with 1 field using KSQL from the topic as below:
create stream data (timeGMT string) with (kafka_topic='json_data', value_format='json');
The stream was created successfully, however no data returns from below KSQL query:
select * from data;
This is running on KSQL 5.0.0
There are a few things to check, including:
Is there any data in the topic?
PRINT 'json_data' FROM BEGINNING;
Have you told KSQL to read from the beginning of the topic?
SET 'auto.offset.reset' = 'earliest';
Are there messages in your topic that aren't JSON or can't be parsed? Check the KSQL Server log for errors.
You can see more information on these, and other troubleshooting tips, in this blog.
I am trying to use Jena to write to a local free standalone GraphDB (version 8.5.0) repository.
What I have tried
(1) Direct use from Jena
I used this Jena 3.7.0 code snippet:
String strInsert =
"INSERT DATA {"
+ "<http://dbpedia.org/resource/Grace_Hopper> "
+ "<http://dbpedia.org/ontology/birthDate>"
+ " \"1906-12-9\"^^<http://www.w3.org/2001/XMLSchema#date> .}";
UpdateRequest updateRequest = UpdateFactory.create(strInsert);
UpdateProcessor updateProcessor = UpdateExecutionFactory.createRemote(updateRequest,
"http://localhost:7200/repositories/PersonData");
updateProcessor.execute();
which results in the following exception
org.apache.jena.atlas.web.HttpException: 415 -
at org.apache.jena.riot.web.HttpOp.exec(HttpOp.java:1091)
at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:718)
at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:501)
at org.apache.jena.riot.web.HttpOp.execHttpPost(HttpOp.java:459)
at org.apache.jena.sparql.modify.UpdateProcessRemote.execute(UpdateProcessRemote.java:81)
at org.graphdb.jena.tutorial.SimpleInsertQueryExample.main(SimpleInsertQueryExample.java:91)
On the GraphDB side I get the following error:
[INFO ] 2018-06-29 11:33:05,605 [repositories/PersonData | o.e.r.h.s.ProtocolExceptionResolver] Client sent bad request ( 415)
org.eclipse.rdf4j.http.server.ClientHTTPException: Unsupported MIME type: application/sparql-update
(2) GraphDB via Jena Fuseki
As an alternative I explored the GraphDB documentation, which states that it is possible to access GraphDB using the Jena Joseki, now Fuseki, server. But for that Fuseki needs to be configured to read the GraphDB as a Jena dataset and then accessed via a Ontotext Jena adapter com.ontotext.jena.SesameDataset. But I can find no GraphDB libraries that inlude this class.
(3) Accessing GraphDB using RDF4J
Accessing GraphDB from RDF4J works without issues:
Repository repository = new HTTPRepository(GRAPHDB_SERVER, REPOSITORY_ID);
repository.initialize();
RepositoryConnection repositoryConnection = repository.getConnection();
repositoryConnection.begin();
Update updateOperation = repositoryConnection.prepareUpdate(QueryLanguage.SPARQL, strInsert);
updateOperation.execute();
try {
repositoryConnection.commit();
} catch (Exception e) {
if (repositoryConnection.isActive())
repositoryConnection.rollback();
}
My Question
Is there a way to access GraphDB efficiently from Jena? I have seen this related SO question, but I was hoping for a better approach.
GraphDB implements standard SPARQL 1.1 endpoints according the RDF4J protocol.
http://localhost:7200/repositories/PersonData - SPARQL query endpoint, which does not support "application/sparql-update"
http://localhost:7200/repositories/PersonData/statements - SPARQL update endpoint
Try changing your code to point to the update endpoint:
UpdateProcessor updateProcessor = UpdateExecutionFactory.createRemote(updateRequest,
"http://localhost:7200/repositories/PersonData/statements");
The Jena adapter to GraphDB is no longer supported.
FWIW not an answer to "how to connect with Jena", but the code you use to access GraphDB via the RDF4J API is more complicated than it needs to be. You can simply do this:
Repository repository = new HTTPRepository(GRAPHDB_SERVER, REPOSITORY_ID);
repository.initialize();
try(RepositoryConnection conn = repository.getConnection()) {
conn.prepareUpdate(strInsert).execute();
}
It will auto-commit and also automatically roll back on connection close if necessary.
I used Swagger Yaml to describe an endpoint and generate the mock server. The existing endpoint (that I'm mocking) doesn't follow RESTful principles 100%, so I simply want to overwrite the response that is returned by the mock server. The simple server code is shown below:
var swagger = require('swagger-server');
var server = swagger('map-cache.yaml');
var port = 7072;
server.post('/map-qa_trunk/v2/getData', function(req, res, next) {
var foo = {
err : 123,
msg : "error message"
};
res.json(foo);
});
server.listen(port, function() {
console.log('Map Cache Mock Server is now running at http://localhost:' + port);
});
In the Yaml definition, there is an object defined called MapResponseData, how do I create an instance of this object so that I can populate it as needed and return in the res.json()? Something similar to below:
var response = getMapResponseData(); // don't know what this call should be
response.fieldA = 123;
res.json(response);
I am guessing this should be possible, since Swagger parsed the YAML file and is aware of all definitions that were specified.
Try outputting the request object to console.log to see if you can find reference to the swagger definition. Another option would be to pull the parsed swagger definition from the yaml file (using js-yaml for example) and extracting from there.
However, my best advice is to use swagger-tools instead of swagger-server. The swagger-server package is alpha version and has fewer downloads, revisions, and users than swagger-tools. Benefit of swagger-tools is that it will be actively maintained and there is a larger community that can support you. To convert your project to swagger-tools, use swagger.io > Swagger Editor > Online Editor > Paste yaml in left pane > Generate Server > Node.js
In swagger-tools the entire Swagger Yaml definition is contained in each request object:
req.swagger.swaggerObject
and you can pull the response object definitions from that as needed.