Azure digital twin gives example of using Time Series Insights. Looking for steps how to persist Sensor telemetry data into cold storage such as big data storage or cosmosdb for later retrieval for the business application.
We are currently implementing such a system and I made a few tests, among which create an endpoint of type "Event Hub" (through the API) and then configuring the "Capture" feature to put the collected data into AVRO files in a Data Lake. This works but may not be the most ideal solution for what we need so I'll explore streaming data from the IoT Hub to a SQL DB... Now I need to access that IoT Hub that was created through the API and is not available in the Azure Portal... Will keep you posted.
For future reference linking the Azure Digital Twins User Voice entry
Alina Stanciu commented · December 20, 2018 12:53 PM ·
Azure Digital Twins events and messages could be routed to Azure Event Hubs, Azure Service Bus topics, or Azure Event Grid for further processing.
For warm analytics, you could send the data to an Event Hub and from there to Azure Time Series Insights (TSI). TSI added recently cold storage as well (they are public preview for cold storage).
If you need more advanced analytics on cold storage, you can forward the data from Event Hub to Azure Data Lake (ADL). Our customers are storing today warm and cold data from sensor & spaces into TSI and ADL. We are looking into integrating our graph modeling with TSI and ADL so modeling will be defined in one place Digital Twins and will be discovered and recognized in downstream services.
I used a custom Edge Module routing messages to IoT Hub, then set up a Stream Analytics Job with IoT hub as input and an azure sql instance as the output. It was pretty painless.
https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-manage-job
https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-stream-analytics-query-patterns#query-example-send-data-to-multiple-outputs
Related
Can https://volttron.org/ be used as a 'protocol translator gateway' software which can take data from PLC over BACnet/Modbus and can forward it to Azure IoT hub?
Where can I get demo for Volttron? I am new to python and unable to understand the code present related to Volttron. Office hours videos present on Volttron sites are not indicating if Volttron is a choice for my scenario shown in the image.
Please guide/help. Thank you.
Yes, VOLTTRON can be configured to retrieve bacnet traffic. The picture you have drawn can be accomplished.
See
https://github.com/VOLTTRON/volttron
And
https://volttron.readthedocs.io/en/main/
For instructions on how to use the platform.
In the picture volttron could actually be in either the PLC Controller (assuming it has enough capabilities) or the protocol translator gateway or both. There is an mqtt facility in VOLTTRON to do the last step to the Azure IOT Hub as well, though it has not been tested with that interface.
Can anyone explain what is the benefit of adopting google cloud pub/sub service in a streaming pipeline?
I saw one of the event streaming pipeline example showcased, and it was using pub/sub to ingest the events data before connecting to the google cloud data flow service to transform it. Why does it not connect to the events data directly through data flow?
Thanks.
Dataflow will need a source to get the data from. If you are using a streaming pipeline you can use different options as a source and each of them will have its own characteristics that may fit your scenario.
With Pub/Sub you can easily publish events using a client library or directly the API to a topic, and it will guarantee at least once delivery of that message.
When you connect it with Dataflow streaming pipeline, you can have a resilient architecture (Pub/Sub will keep sending the message until Dataflow acknowledge that it has processed it) and a near real-time processing. In addition, Dataflow can use Pub/Sub metrics to scale up or down depending on the number of the messages in the backlog.
Finally, Dataflow runner uses an optimized version of the PubSubIO connector which provides additional features. I suggest checking this documentation that describes some of these features.
Is there any dashboard solutions available to communicate OPC Server directly (OLE for Process Control + Used in Industrial Standards)?
If so please suggest the dashboard vendor name.
I am using https://thingsboard.io/, an open-source IOT platform. The IOT Gateway service gathers the time-stamped data (telemetry) from OPC UA servers on my network. Thingsboard stores all the telemetry in a open-source, fault tolerant database, and provides web-based, customizable dashboards for visualization.
Check out the live demo.
We are considering using Azure offline data sync for our app which usually has very sporadic connectivity (In most cases users sync their data once a day). Thing is that mobile app needs to hold a lot of data (tens of thousands of products). Currently we have our own sync solution which works fine with sqlite.
My question is, do you have any experience or thoughts about performance of Azure offline data sync? Will it be able to handle really large datasets?
Thanks you
Azure mobile service is the cloud version of popular Microsoft sync framework. This is a light weight json API which tracks changes between local and remote data store. It transfers only changed rows, hence data traffic will be minimum. But when you sync very first time and you have huge data, it might be a problem.
You could overcome this problem by carefully designing your database structure. Azure SDK provides api to sync table by table, which gives you enough flexibility to choose what to sync and not.
The type of content isn't really important for this question, but let's just say I wanted to implement a (native mobile) shopping list app that allowed multiple users to collaborate on a shared list.
How are sync features like this usually implemented that work automatically (without explicit user interaction)? Is the preferred way to pull every few seconds to check for newer versions and update if necessary, or is it possible to push changes?
A polling solution would be (relatively) easy to implement I guess using something like AWS, Google App Engine or even from scratch on a LAMP stack and REST. But I'm worried about traffic resulting from continuous polling.
Would it be practical to try to implement this using push updates? If so, what technologies, services or design principles should I look into? Is something like this possible with AWS or Google App Engine? Or is pulling (and reducing traffic as much as possible) the way to go?
On app engine you should look into the channel API. From the overview:
The Channel API creates a persistent connection between your application and Google
servers, allowing your application to send messages to JavaScript clients in real time without the use of polling. This is useful for applications that are designed to update the user about new information immediately or where user input is immediately broadcast to other users. Some examples include collaborative applications, multi-player games, and chat rooms. In general, using Channel API is a better choice than polling in situations where updates can't be predicted or scripted, such as when relaying information between human users or from events not generated systematically.
You can use a few of Amazon Web Services to create an effective and responsive service.
If you check out the IOS SDK that you can download from AWS site, you can find in it an example for a service that is using such services: S3_SimpleDB_SNS_SQS_Demo
First you can use SQS, which is the queueing service, which has long polling that will help you to lower the number of requests.
Second you can use SNS, which is the notification (pub/sub) service. It is integrated with SQS, and you can subscribe queues to listen to notifications.
These services (and others) are accessible through the iOS SDK, as well as with other SDKs (Java, .NET, Android...) and REST and SOAP APIs.