I'm using confluentinc/cp-kafka-connect docker image.
I'm trying to send JSON file to kafka, with elasticsearch id.
{"_id":10000725, "_source": {"createdByIdentity":"tu_adminn","createdBy":"Admin Testuser"}}
here is my connector
{
"name": "test-connector",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"tasks.max": "1",
"topics": "andrii",
"key.ignore": "false",
"schema.ignore": "true",
"connection.url": "http://elasticsearch:9200",
"type.name": "test-type",
"name": "elasticsearch-sink"
}
}
When i'm using key.ignore = true it's generates some weird id.
How can i pass exactly my id and source?
Per the docs:
If you specify key.ignore=true then Kafka Connect will use a composite key of your message's kafka topic, partition, and offset -- this is the "weird id" that you're seeing.
If you want to use your own ID for the created Elasticsearch document, you can set key.ignore=false and Kafka Connect will use the key of the Kafka message as the ID.
If your Kafka message does not have the appropriate key for what you want to do, you will need to set it. One option is to use something like KSQL:
CREATE STREAM target AS SELECT * FROM source PARTITION BY _id
Disclaimer: I work for Confluent, the company behind the open-source KSQL project
Related
I have created a new Container instance in Azure. Below are the steps.
Step:1- I created a new Cognitive Services (A Language Service) and used its "Key" and "Endpoint" value inside Container Instance
Step:2- I created a new Container Instance, and provide it all the required information as mentioned in the below article.
https://learn.microsoft.com/en-us/azure/container-instances/container-instances-quickstart-portal
but I changed the PORT 80 to "5001" and Image "mcr.microsoft.com/azure-cognitive-services/textanalytics/healthcare:latest".
Below are env variable I used
{
"name": "Eula",
"value": "accept"
},
{
"name": "RAI_TERMS",
"value": "accept"
},
{
"name": "Billing",
"value": "XXXXXXXXXXXXXXXXXXXXXXXXXXX"
},
{
"name": "ApiKey",
"value": "4a46537f51f64765864cabc20318bdcc"
},
{
"name": "enablelro",
"value": "true"
}
Finally it was created and deployed successfully. Now I tried to access it via below url
http://FQDN:5001/Demo/
FQDN--> qualified domain name is used in the url
its not accessible though instance is up and running properly.
It doesn't matter from which port you are trying to access. instead of using this url http://FQDN:5001/Demo/ would suggest you please use FDQN or IP address of container instance.
Using the complete FQDN when identifying something is the way it is supposed to be.
You can refer this thread same i have reprod related to your question. In which i have used FDQN to access the Conatainer Instance.
I have a gateway fromKhomp manufacturer which delivers packages in the following format (SenML):
message: [
{
"bn": "000D6FFFFE642E09",
"bt": 1611339204
},
{
"n": "model",
"vs": "nir21z"
},
{
"n": "Geladeira Temp",
"u": "Cel",
"v": 4.0
}
When I connect to the Thingsboard platform, the internal GW/Parser breaks as an array before the Input in the Root Rule Chain, and threats as individual packets, but since the first position in this array corresponds to the device ID (MAC) I need to have the whole message to be parser in a script. Does anyone know a way to get the information before the GW parses the message?
If you're using Thingsboard CE then I think you will need to first forward the data to a middleware service to restructure the payload. If you are familiar with AWS Lambda you can do it there.
It would just be a simple script that takes an input payload, restructures, and then forwards to your Thingsboard deployment.
If you're using Thingsboard PE then you can use Integration/Data Converters to do this.
I followed the Microsoft docs for deploying a storage account edge module and my module is currently running in my VM.
However, I cannot connect the Python SDK or the storage explorer to it.
My container create options are:
{
"Env": [
"localstorage",
"92RhvUXR59Aa8h90LSHC7w=="
],
"HostConfig": {
"Binds": [
"blobvolume:/blobroot"
],
"PortBindings": {
"11002/tcp": [
{
"HostPort": "11002"
}
]
}
}
}
My module twin settings are
{
"deviceAutoDeleteProperties": {
"deleteOn": false,
"deleteAfterMinutes": 0,
"retainWhileUploading": true
},
"deviceToCloudUploadProperties": {
"uploadOn": true,
"uploadOrder": "OldestFirst",
"cloudStorageConnectionString": "<<my connection string was here :) >>",
"storageContainersForUpload": {
"eds-container-pcu": {
"target": "eds-container-pcu"
}
},
"deleteAfterUpload": true
}
}
iotedge list says
blobstorage running Up 5 hours mcr.microsoft.com/azure-blob-storage:latest
I also checked the volume. I put a file into the "BlockBlob" folder in my volume and then I went into the shell of my container and under /blobroot I could find that file.
I also copied the cloud storage connection string and used that for a file upload and that also worked. I could upload a text file into my cloud storage account successfully.
My connection string for my local storage account would be that, no? I created it by myself like the docs said:
DefaultEndpointsProtocol=http;BlobEndpoint=http://localhost:11002/localstorage;AccountName=localstorage;AccountKey=92RhvUXR59Aa8h90LSHC7w==;
With that connection string I cannot connect anything. Not the SDK, nor the Storage Explorer.
Python code would be:
CONNECTION_STRING = "DefaultEndpointsProtocol=http;BlobEndpoint=http://localhost:11002/localstorage;AccountName=localstorage;AccountKey=92RhvUXR59Aa8h90LSHC7w==;"
blobClient = BlobClient.from_connection_string(CONNECTION_STRING, "eds-container-pcu", "test123.txt")
# Doesn't work with or without this line
blobClient._X_MS_VERSION = '2017-04-17'
blobClient.upload_blob(data="Hello World", blob_type="BlockBlob")
And I get:
HttpResponseError: Server encountered an internal error. Please try again after some time.
RequestId:fef9066d-323c-47e3-8e6f-ba006557ee65
Time:2021-04-08T14:33:23.7892893Z
ErrorCode:InternalError
Error:None
ExceptionDetails:None
ExceptionMessage:'s' cannot be null
StackTrace:AsyncHelper.ArgumentNullRethrowableException: 's' cannot be null
---> System.ArgumentNullException: Value cannot be null. (Parameter 's')
at System.Convert.FromBase64String(String s)
at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStorageAccount..ctor(WStorageStamp stamp, String accountName)
at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStorageStamp.Microsoft.Cis.Services.Nephos.Common.Storage.IStorageStamp.CreateAccountInstance(String accountName, ITableServerCommandFactory tableServerCommandFactory)
at Microsoft.Cis.Services.Nephos.Common.Storage.PerRequestStorageManager.CreateAccountInstance(String accountName)
at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.GetResourceAccountPropertiesImpl(String accountName, Boolean isAdminAccess, TimeSpan timeout, AsyncIteratorContext`1 context)+MoveNext()
at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)
--- End of inner exception stack trace ---
at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.EndGetResourceAccountProperties(IAsyncResult asyncResult)
at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.ProcessImpl(AsyncIteratorContext`1 async)+MoveNext()
Why could that be? Storage explorer is also not connecting and the lines of code above are working fine with my cloud storage connection string.
A colleague pointed out that my local account key is too short. As written in the docs, the key must have a length of 64 bytes. Connection with SDK or StorageExplorer is fine then. My code also was missing the creation of a local container but the error message will tell about this.
in your sample for the container create options, I am missing the keys (LOCAL_STORAGE_ACCOUNT_NAME and LOCAL_STORAGE_ACCOUNT_KEY). Did you specify only the values?
"Env": [
"LOCAL_STORAGE_ACCOUNT_NAME=localstorage",
"LOCAL_STORAGE_ACCOUNT_KEY=92RhvUXR59Aa8h90LSHC7w=="
],
When I deploy a module with the settings, I can connect via Storage Explorer.
Did you look at the logs of the blob storage module?
I try to set up a TTN based LoRaWAN Monitoring of my Gateways and devices inside a FIWARE-Environment. Therefore it would be essential to access data not in payload_field of the MQTT-Broker of TTN.
I wonder if it is possible to access field like counter, port, app_id and metadata.
I did not find a possibility yet. Does any of you face the same problem and got a solution to this challenge?
I use the following relevant FIWARE-components in a docker environment:
fiware/orion:2.2.0
fiware/iotagent-lorawan:1.2.3
mongo:3.6.8
If you need to receive metadata directly from LoRaWAN, you will have to customize the code within the LoRaWAN IoT Agent - this just passes measures by default, but the IoT Agent node lib interface is capable of receiving metadata as well.
Alternatively a recent PR Request for the IoT Agent node lib allows for additional static metadata to be added during the provisioning stage and sent as part of the requests to the context broker. You would need to use the latest development code base as the library hasn't been ported to the LoRaWAN IoT Agent yet - amend the iotagent-node-lib dependency in the package.json as shown:
"dependencies": {
...
"iotagent-node-lib": "git://github.com/telefonicaid/iotagent-node-lib.git#master",
...
},
... etc
The documentation can be found here
Attributes with metadata are provisioned with an additional parameter as shown:
"attributes": [
{"object_id": "s", "name": "state", "type":"Text"},
{"object_id": "l", "name": "luminosity", "type":"Integer",
"metadata":{
"unitCode":{"type": "Text", "value" :"CAL"}
}
}
I deployed Prometheus on my cluster as well as cAdvisor and Grafana. It works tremendously well. I get all the data I need on Grafana's UI.
I started using Prometheus Java API in order to use this data. For example get the CPU usage and if it has a certain value something will be done.
What I display on Grafana is the Container CPU usage for each container. Now I would like to get that information with the Java API if possible (or something if not). But of course the PromQL queries aren't usable from a Java program (from what I tried but I may be wrong).
I thought of several ways:
Clone the cAdvisor project and directly implement what I want to do in Go
Create a bash script with the docker stat command that would get me the container and CPU usage associated
Or maybe there is actually a way to send PromQL queries.
For instance we get the metric by its name via Java or the Prometheus interface:
ex: node_cpu would get me some data.
But if I want something more precise, I need to send a request, for example irate(node_cpu{job="prometheus"}[5m]) which is not possible via Java.
Is there a way for me to get more precise metrics ?
Prometheus supports REST API requests, which are language-agnostic. You just need to send an HTTP request with your query and process the response.
see an example below, copied from their site.
the following HTTP GET request:
http://localhost:9090/api/v1/query?query=up&time=2015-07-01T20:10:51.781Z
returns something like this:
{
"status" : "success",
"data" : {
"resultType" : "vector",
"result" : [
{
"metric" : {
"__name__" : "up",
"job" : "prometheus",
"instance" : "localhost:9090"
},
"value": [ 1435781451.781, "1" ]
},
{
"metric" : {
"__name__" : "up",
"job" : "node",
"instance" : "localhost:9100"
},
"value" : [ 1435781451.781, "0" ]
}
]
}
}
lots more details, here