We have a customer who has deployed a number of iotedge transparent gateways and keeps routing data from tons of leaf devices to cloud.
Recently they noticed the output (edge to IoT Hub) cannot catch up the input on part of the edge devices, which is causing a severe latency issue for their messages.
Here's the information of the built-in metrics on edgeHub,named 8B:
edgehub_queue_length
8B: 981061
edgehub_message_send_duration_seconds
8B: ~110ms
{quantile="0.1"} 0.0632608
{quantile="0.5"} 0.1136008
{quantile="0.9"} 0.127605
{quantile="0.99"} 0.2449048
edgehub_message_process_duration_seconds
8B: 0.5-2.0 ms
We would like to clarify two questions:
What is the recommended network latency for iotedge gateway?
Are there any other methods we can do to improve the output throughput of
edgeHub?
I have two type of devices one support IoT JSON agent and another Ultralight Iot Agent. What approach is best should i used both IoT agent separately or there is something like IoT manager. What in future i have another device which support another IoT Agent like LoRaWan .
I have tried IoT manager but could not understand it is only for device provisioning or actual device data transfer also.
IoT Manager acts as a proxy for actual IOTAs, so depending on some parameters in the request (I don't remember exactly which ones, but they should be described in the IOTAs documentation) redirects the request to one or another IOTA agent.
But of course, you can implement this proxy using another piece of software (e.g. ngix with some king of URL-based or port-based redirection) if you want.
I have a customer with IoT Edge deployed to manufacturing plants in remote areas with spotty internet. They have leaf devices sending messages to IOT Edge and then to IoT Hub. They frequently have small outages (5, 10, 15 minutes). They often need to make timely decisions based on the data that makes it to IOT Hub from the plants. They've noticed, if they have a 15 minute outage, it can take anywhere from 15-30 minutes afterwards for IOT Edge to catch up.
Besides network speed itself, what are the factors that would influence that.. For example
- if we were hitting throttling based on their number of iot hub units, would that be surfaced in the edgeHub logs?
- if disk, network, etc can keep up, does edgeHub pretty much upload data as fast as possible (given throttling), or are there any other limits imposed by default?
- What is the default connection retry policy in edgeHub? is the same exponential backoff policy in the C# SDK? If so, could that be the case that if I have a 15 minute outage, that it's taking edgeHub a while after network recovery to 'try again'? If so, is that policy configurable in edgeHub? (via ENV variable or something?)
Any other things to check?
I am looking at a project that is trying to connect a number of IPad type devices, so each can pass information to the other members of this group.
For example, given six devices, if Dev1 moves to a new position, that position information should be propagated to Dev2-6. Similarly, if Dev2 moves, the information is passed to Dev1, Dev3-6.
It would seem to me that this might be doable by
Having a Message Broker with a MQTT topic to which all devices subscribe.
Writing the information to the broker
Then reading, or better still, pushing the data to other devices (perhaps using shadow devices).
Is this a reasonable approach? (I don't quite see how to write to the message broker).
Could binary data be sent?
Any ideas on how it would scale?
Just asking one silly question, hope someone can answer this.
I'm bit confused regarding MQTT broker. Basically, the confusion is, there are so many things being used for data storing, transfer and processing (like Flume, HDInsight, Spark etc). So, when and why I need to use one MQTT broker?
If I would like to use Windows 10 IoT application with HiveMQ, from where can I get the details? how to use it? How I get benefit out of this MQTT broker? Can I not send data from my IoT application directly using Azure or HDFS? So, how MQTT broker fits into it or helping me to achieve something?
I'm new to all these and tried to find some tutorials, however, I'm not getting anything proper. Please explain it in more details or give some tutorials for this?
MQTT is a client-server protocol for pub-sub based transport that has a comparatively small overhead, and thus applicable to mobile and IoT applications (unlike Flume, etc.). The MQTT broker is basically a server that handles messaging to/from MQTT clients and among them. The functionality pretty much stops at the transport layer, even though various MQTT add-ons exist.
If you are looking to implement a solution that would reliably transfer data from your IoT devices to the back-end system for processing, I would suggest you take a look into Kaa open-source IoT platform. It goes much further than MQTT by providing not only the transport layer, suitable for low-power IoT devices, but also a solid chunk of the application level logic (including the object bindings for your application-level data structures, temporary data persistence, etc.).
Here is a link to a webinar that explains how to build a scalable IoT analytics system with Kaa and Spark in less than an hour.
This is an architectural choice. IoT applications are possible without MQTT but there are some advantages when using MQTT. If you are completely new to MQTT, take a look at this in-depth MQTT series: http://forkbomb-blog.de/2015/all-you-need-to-know-about-mqtt
Basically the main architectural advantage is publish / subscribe designed for low-latency, high throughput (mobile) communication with minimal protocol overhead (which is important if bandwidth is at a premium). You can completely decouple consumers and producers.
HDFS is the (distributed) Hadoop file system and is the foundation for Map / Reduce processing. It is not comparable to a MQTT broker. The MQTT broker could write to the HDFS, though (in case of HiveMQ with a custom plugin).
Basically MQTT is a protocol while the products you are mentioning are, well, products which solve completely different problems:
Flume is basically used for log aggregation at scale. You won't use MQTT for that, at least there is not too much advantage because this is typically done in backend applications.
Spark and Hadoop shine at Big Data crunching. They are a framework and not a ready to use solution. They are not really comparable to MQTT. Often MQTT brokers like HiveMQ are used in conjunction with these, Spark / Hadoop for data processing and HiveMQ for communication.
I hope this helps you getting started. Best would be to read about typical use cases of all these technologies, this is a bit too broad for a single SO answer.
MQTT is a data transport, so the usual thing I have to compare it with is HTTP. HTTP has two important characteristics, a) It goes from one point to another, b) It is request/response, so only one end can start a data transfer. MQTT connects many end points to many end points, and either end can start a data transfer. So, if you have just one device and only one service or person that will ever access it, and only by polling, then HTTP is great. MQTT means many devices can post data to many services or people, AND the other way around. Your question assumes that your data is always going to land up in some sort of data store, but many interactions are about events and responding to them immediately, like ringing a doorbell, or lowering the landing gear. In these cases you will often want to both record the data, and have an immediate action occur, like your phone making a doorbell noise.
Finally, you send data to MQTT semantically, rather than by IP address.
This means that your services subscribes to /mikeshouse/doorbell rather than polling 192.168.22.4, which is a huge gain once you have a number of devices.