Can't connect to local storage account - azure-iot-edge

I followed the Microsoft docs for deploying a storage account edge module and my module is currently running in my VM.
However, I cannot connect the Python SDK or the storage explorer to it.
My container create options are:
{
"Env": [
"localstorage",
"92RhvUXR59Aa8h90LSHC7w=="
],
"HostConfig": {
"Binds": [
"blobvolume:/blobroot"
],
"PortBindings": {
"11002/tcp": [
{
"HostPort": "11002"
}
]
}
}
}
My module twin settings are
{
"deviceAutoDeleteProperties": {
"deleteOn": false,
"deleteAfterMinutes": 0,
"retainWhileUploading": true
},
"deviceToCloudUploadProperties": {
"uploadOn": true,
"uploadOrder": "OldestFirst",
"cloudStorageConnectionString": "<<my connection string was here :) >>",
"storageContainersForUpload": {
"eds-container-pcu": {
"target": "eds-container-pcu"
}
},
"deleteAfterUpload": true
}
}
iotedge list says
blobstorage running Up 5 hours mcr.microsoft.com/azure-blob-storage:latest
I also checked the volume. I put a file into the "BlockBlob" folder in my volume and then I went into the shell of my container and under /blobroot I could find that file.
I also copied the cloud storage connection string and used that for a file upload and that also worked. I could upload a text file into my cloud storage account successfully.
My connection string for my local storage account would be that, no? I created it by myself like the docs said:
DefaultEndpointsProtocol=http;BlobEndpoint=http://localhost:11002/localstorage;AccountName=localstorage;AccountKey=92RhvUXR59Aa8h90LSHC7w==;
With that connection string I cannot connect anything. Not the SDK, nor the Storage Explorer.
Python code would be:
CONNECTION_STRING = "DefaultEndpointsProtocol=http;BlobEndpoint=http://localhost:11002/localstorage;AccountName=localstorage;AccountKey=92RhvUXR59Aa8h90LSHC7w==;"
blobClient = BlobClient.from_connection_string(CONNECTION_STRING, "eds-container-pcu", "test123.txt")
# Doesn't work with or without this line
blobClient._X_MS_VERSION = '2017-04-17'
blobClient.upload_blob(data="Hello World", blob_type="BlockBlob")
And I get:
HttpResponseError: Server encountered an internal error. Please try again after some time.
RequestId:fef9066d-323c-47e3-8e6f-ba006557ee65
Time:2021-04-08T14:33:23.7892893Z
ErrorCode:InternalError
Error:None
ExceptionDetails:None
ExceptionMessage:'s' cannot be null
StackTrace:AsyncHelper.ArgumentNullRethrowableException: 's' cannot be null
---> System.ArgumentNullException: Value cannot be null. (Parameter 's')
at System.Convert.FromBase64String(String s)
at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStorageAccount..ctor(WStorageStamp stamp, String accountName)
at Microsoft.AzureStack.Services.Storage.FrontEnd.WossProvider.WStorageStamp.Microsoft.Cis.Services.Nephos.Common.Storage.IStorageStamp.CreateAccountInstance(String accountName, ITableServerCommandFactory tableServerCommandFactory)
at Microsoft.Cis.Services.Nephos.Common.Storage.PerRequestStorageManager.CreateAccountInstance(String accountName)
at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.GetResourceAccountPropertiesImpl(String accountName, Boolean isAdminAccess, TimeSpan timeout, AsyncIteratorContext`1 context)+MoveNext()
at AsyncHelper.AsyncIteratorContextBase.ExecuteIterator(Boolean inBegin)
--- End of inner exception stack trace ---
at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.EndGetResourceAccountProperties(IAsyncResult asyncResult)
at Microsoft.Cis.Services.Nephos.Common.Protocols.Rest.BasicHttpProcessorWithAuthAndAccountContainer`1.ProcessImpl(AsyncIteratorContext`1 async)+MoveNext()
Why could that be? Storage explorer is also not connecting and the lines of code above are working fine with my cloud storage connection string.

A colleague pointed out that my local account key is too short. As written in the docs, the key must have a length of 64 bytes. Connection with SDK or StorageExplorer is fine then. My code also was missing the creation of a local container but the error message will tell about this.

in your sample for the container create options, I am missing the keys (LOCAL_STORAGE_ACCOUNT_NAME and LOCAL_STORAGE_ACCOUNT_KEY). Did you specify only the values?
"Env": [
"LOCAL_STORAGE_ACCOUNT_NAME=localstorage",
"LOCAL_STORAGE_ACCOUNT_KEY=92RhvUXR59Aa8h90LSHC7w=="
],
When I deploy a module with the settings, I can connect via Storage Explorer.
Did you look at the logs of the blob storage module?

Related

AzureBlobStorageOnIoTEdge: Error Target container connection not specified, upload turned off

My local blob storage is not uploading blobs to my cloud storage account. It reports back
"configurationValidation": {
"deviceAutoDeleteProperties": {
"deleteOn": {
"Status": "Success"
},
"deleteAfterMinutes": {
"Status": "Warning",
"Message": "Auto Delete after minutes value not specified, auto deletion turned off."
},
"retainWhileUploading": {
"Status": "Success"
}
},
"deviceToCloudUploadProperties": {
"uploadOn": {
"Status": "Success"
},
"cloudStorageAccountName": {
"Status": "Error",
"Message": "Target container connection not specified, upload turned off."
},
"cloudStorageAccountKey": {
"Status": "Error",
"Message": "Target container connection not specified, upload turned off."
},
"uploadOrder": {
"Status": "Success"
},
"deleteAfterUpload": {
"Status": "Success"
}
}
},
I am pretty sure that it should work. My desired properties are
"deviceToCloudUploadProperties": {
"uploadOn": true,
"uploadOrder": "OldestFirst",
"cloudStorageConnectionString": "DefaultEndpointsProtocol=https;AccountName=*****;AccountKey=******;EndpointSuffix=core.windows.net",
"storageContainersForUpload": {
"***": {
"target": "***"
}
},
"deleteAfterUpload": true
}
The container exists locally and on the cloud site. I copied the primary connection string from my local storage account into the configuration. The local storage is working, I can see that my container was created and contains data but it doesn't want to synchronize with the cloud. Why is it saying "Target container connection not specified, upload turned off."? It sounds like this part is missing
"storageContainersForUpload": {
"***": {
"target": "***"
}
},
but obviously it is not.
I'm using the latest docker image of this service. Is there any chance to use an older version? Some months ago I could make it work already. I tried to use a different version like mcr.microsoft.com/azure-blob-storage:1.4.0 but it doesn't accept any other tags than latest.
Thx!
The difference between my working version of the local blob storage module and my non working version was that the non working version was deployed by a deployment plan. In the deployment plan you cannot just paste the module twin settings of the documentation of the blob storage on IoT edge like https://learn.microsoft.com/en-us/azure/iot-edge/how-to-deploy-blob?view=iotedge-2020-11
You need to split the configuration into two parts where the first part looks like this
and the second part looks like that
And that totally makes sense. If you want to update your modules you probably want to keep your configuration because there might have been some changes which were made e.g. by a customer. This gives you the possibilty to add some properties to your inital configuration later without changing anything what was already configured. In fact every device can keep its individual configuration at any time.
My wrongly configured reported proterties were hidden in the suggested default path "properties.desired.settings" and thus the edge runtime could not find it.

IoTAgent-LoRaWAN is apparently not working as expected

I was trying to provisioning the IoTAgent-LoRaWAN using the TTN credentials, I'm following the official docs and this is my POST request:
{
"devices": [
{
"device_id": "{{node}}",
"entity_name": "LORA-N-0",
"entity_type": "LoraDevice",
"timezone": "Europe/Madrid",
"attributes": [
{
"object_id": "potVal",
"name": "Pot_Value",
"type": "Number"
}
],
"internal_attributes": {
"lorawan": {
"application_server": {
"host": "eu.thethings.network",
"username": "{{TTN_app_id}}",
"password": "{{TTN_app_pw}}",
"provider": "TTN"
},
"dev_eui": "{{TTN_dev_eui}}",
"app_eui": "{{TTN_app_eui}}",
"application_id": "{{TTN_app_id}}",
"application_key": "{{TTN_app_skey}}"
}
}
}
]
}
Obviously I'm using Postman to manage all those HTTP requests in a collection and I've setup a few environment variables that are:
{{node}} -> the device ID node_0
{{TTN_app_id}} -> my app id which I've chosen dendrometer
{{TTN_app_pw}} -> the application access key shown in the picture (It can be found in the same view than the Application Overview; https://console.thethingsnetwork.org/applications/<application_id>)
{{TTN_dev_eui}} and {{TTN_app_eui}} also shown in the following picture (regarding to device; I think these are not sensitive info because TTN is not hiding them, that's because I'm posting the picture)
{{TTN_app_skey}} -> The Application Session Key also shown in the following picture (the last one)
The point is ... once I've provisioned IoTAgent using that request, the docker-compose logs -f iot-agent shows the following errors:
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.689Z","level":"info","message":"New message in topic"}
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.690Z","level":"info","message":"IOTA provisioned devices:"}
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.691Z","level":"info","message":"Decoding CaynneLPP message:+XQ="}
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.691Z","level":"error","message":"Error decoding CaynneLPP message:Error: Invalid CayennLpp buffer size"}
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.691Z","level":"error","message":"Could not cast message to NGSI"}
So I think there is something not working properly. That's my docker-compose.yml, btw http://ix.io/2pWd
However I don't think the problem is caused by docker, all containers are working as expected apparently because I can request their versions and I don't see error messages in the logs.
Also ... I feel the docs like incomplete, I'd like more info, about how to subscribe those provisioned devices with OrionCB (?) or Delete them (that's not shown in the docs, although is just a DELETE request to the proper URL.)
Anyway ... What I'm doing wrong? Thank you all.
EDIT: I feel like there is something wrong in the IoTAgent itself, there is a typo in the following error messages:
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.691Z","level":"info","message":"Decoding CaynneLPP message:+XQ="}
fiware-iot-agent | {"timestamp":"2020-06-23T11:45:53.691Z","level":"error","message":"Error decoding CaynneLPP message:Error: Invalid CayennLpp buffer size"}
Because it isn't CaynneLPP but CayenneLPP. I've also opened an issue in its GitHub repo but I don't expect they answer any time soon. I actually feel like this project has been abandoned.
It's apparently a problem with encoding, I was using the encoding method suggested by arduinio-lmic library but FIWARE does work under CayenneLPP data model. So I'm going to try replace that encoding method.
Thank you all anyway and specially to #arjan

FIWARE Metadata in IoTAgent

I try to set up a TTN based LoRaWAN Monitoring of my Gateways and devices inside a FIWARE-Environment. Therefore it would be essential to access data not in payload_field of the MQTT-Broker of TTN.
I wonder if it is possible to access field like counter, port, app_id and metadata.
I did not find a possibility yet. Does any of you face the same problem and got a solution to this challenge?
I use the following relevant FIWARE-components in a docker environment:
fiware/orion:2.2.0
fiware/iotagent-lorawan:1.2.3
mongo:3.6.8
If you need to receive metadata directly from LoRaWAN, you will have to customize the code within the LoRaWAN IoT Agent - this just passes measures by default, but the IoT Agent node lib interface is capable of receiving metadata as well.
Alternatively a recent PR Request for the IoT Agent node lib allows for additional static metadata to be added during the provisioning stage and sent as part of the requests to the context broker. You would need to use the latest development code base as the library hasn't been ported to the LoRaWAN IoT Agent yet - amend the iotagent-node-lib dependency in the package.json as shown:
"dependencies": {
...
"iotagent-node-lib": "git://github.com/telefonicaid/iotagent-node-lib.git#master",
...
},
... etc
The documentation can be found here
Attributes with metadata are provisioned with an additional parameter as shown:
"attributes": [
{"object_id": "s", "name": "state", "type":"Text"},
{"object_id": "l", "name": "luminosity", "type":"Integer",
"metadata":{
"unitCode":{"type": "Text", "value" :"CAL"}
}
}

Push own id. Confluent kafka connect elasticsearch docker

I'm using confluentinc/cp-kafka-connect docker image.
I'm trying to send JSON file to kafka, with elasticsearch id.
{"_id":10000725, "_source": {"createdByIdentity":"tu_adminn","createdBy":"Admin Testuser"}}
here is my connector
{
"name": "test-connector",
"config": {
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"tasks.max": "1",
"topics": "andrii",
"key.ignore": "false",
"schema.ignore": "true",
"connection.url": "http://elasticsearch:9200",
"type.name": "test-type",
"name": "elasticsearch-sink"
}
}
When i'm using key.ignore = true it's generates some weird id.
How can i pass exactly my id and source?
Per the docs:
If you specify key.ignore=true then Kafka Connect will use a composite key of your message's kafka topic, partition, and offset -- this is the "weird id" that you're seeing.
If you want to use your own ID for the created Elasticsearch document, you can set key.ignore=false and Kafka Connect will use the key of the Kafka message as the ID.
If your Kafka message does not have the appropriate key for what you want to do, you will need to set it. One option is to use something like KSQL:
CREATE STREAM target AS SELECT * FROM source PARTITION BY _id
Disclaimer: I work for Confluent, the company behind the open-source KSQL project

Using characters such as ) in Bluemix runtime environment variables

I've got a ruby on rails app running on Bluemix. With this app I use a couple of services, one of which is Object Storage.
Logically, I want to put the credentials that I use for each environment (dev and prod) in the environment variables that you can specify in the runtime tab within Bluemix.
I want to put a password like this in there:
23aSeefae,,)ewFe
The runtime environment is not accepting the ) sign.
It says:
I have tried double quotes, single quotes and I have tried to escape the ) sign with a backslash.
Any help would be appreciated. Is there any way in which I can store my variables outside of my app and within the Bluemix environment instead?
PS: password is not a real password.
You have to bind (connect) your Object Service instance to your application in Bluemix so the VCAP_SERVICES environment variable is automatically created for you.
Here is an example of a VCAP_SERVICES env variable for an application binding Object Storage service instance (I have modified some data for security reasons):
{
"Object-Storage": [
{
"credentials": {
"auth_url": "https://identity.open.softlayer.com",
"project": "object_storage_a92583b3_329e_4ed8_8918_xxx",
"projectId": "7f1f5659d21340dfaa4568dxxxx",
"region": "dallas",
"userId": "abcdefghxxxxxxxxxxxxx",
"username": "admin_3ff9bf1e187e7fa02e28c96232dxxxxxxx",
"password": "BF_0_)s3#xxxXXbY^",
"domainId": "79fc08601744486abf930000000000",
"domainName": "761111",
"role": "admin"
},
"syslog_drain_url": null,
"label": "Object-Storage",
"provider": null,
"plan": "standard",
"name": "app-object-storage",
"tags": [
"storage",
"ibm_release",
"ibm_created"
]
}
]
}
You can then read this as JSON object in your ruby code, for example:
vcap_services = JSON.parse(ENV['VCAP_SERVICES'])
credentials = vcap_services["Object-Storage"][0]["credentials"]
password = credentials["password"]
I've gotten help from the Bluemix support as well now. This is by far most easy way to do what I want:
You can set environment variables through the Cloud Foundry command line interface.
cf set-env <APP_NAME> <ENV_VAR_NAME> <ENV_VAR_VALUE>
You will have to restage your app before you can use them.

Resources