I am trying to "transparently" intercept and modify incoming device messages. Say for example three devices send data to the Edge Hub at 20 messages per second each, I want to apply a moving average to this data and then send it upstream at a rate of 1 message per second each, while retaining the original sender information visible from the Hub and without using message properties or something alike that would require further configuration on the hub data entrypoints.
I would like to do this in a mostly transparent fashion, as if the device itself was directly connected to the IoT Hub, while retaining the Modules sinks and outputs so that multiple modules of this kind can be easily stacked merely by adapting the routes. By example linking default input to module1 sink, module1 output to module2 sink and finally module2 output to upstream.
How can this be achieved? Preferably using the Node SDK's.
Related
We are using dapr along with an input binding to consume IoT Events from Azure Event Hub.
The event hub has 32 partitions and the sender uses the device id as partition key.
The controller receiving the events delegates them to dapr Actors for processing.
Now my expectation would be, that the messages in each partition are processed in parallel, resulting in parallel requests to the controller receiving the events allowing for up to 32 concurrent event batches to be processed.
However tests show that the events are received synchronously. The consequent event is received just after the full processing of the previous event has completed.
Now i know that actors process messages one after the other by design but i did not read anything like this for input bindings according to the documentation.
Is there anything i'm completely missing? Otherwise i could not image how this system could scale.
We are using dapr 0.11 along with ASP.NET Core 3.1 running in an AKS cluster.
I just got an answer on GitHub: https://github.com/dapr/components-contrib/issues/759
So basically this confirms my observation based on the current version of the dapr Event Hub Binding.
However, you could manually configure the partition ID per dapr component. So to read from 32 Partitions concurrently you currently would have to use 32 dapr components each reading from a separate partition.
Quote from above GitHub Issue:
[...] you can specify a Dapr component per partition ID: https://docs.dapr.io/operations/components/setup-bindings/supported-bindings/eventhubs/
If you set the partitionID field on the component metadata, the consumer will pull all messages for that partition ID.
Otherwise, it'll pull from all partitions.
Is there any recommended way to inspect/plot the numeric values that are being sent through the ports between drake systems in real-time?. (something similar to rqt_plot in ROS). Apart from the SignalLogger or writing and wiring custom individual plotting Systems, is there any method to access the port values internally?
There's nothing as nice as rqt_plot as far as I know.
If you are able to alter your Diagram before calling DiagramBuilder::Build, you could add an LcmScopeSystem onto any vector-valued output port and then the port's contents will be transmitted on an LCM channel. You can add multiple scopes, but you currently have to add them one by one, ahead of time.
Once the data is onto an LCM channel, then you could use the provided drake-lcm-spy program which has the ability to show (very rudimentary) live plots:
cd drake
bazel build //lcmtypes:drake-lcm-spy
bazel-bin/lcmtypes/drake-lcm-spy &
Also tangentially related would be https://github.com/RobotLocomotion/drake/issues/5857, though that is not on any near-term roadmap.
One of the new features of MQTT 5 is the shared subscriptions feature, which allows client-side load balancing between multiple workers, so that multiple workers can be responsible for handling messages, but every message is only ever sent to a single server.
By default, this works with a round-robin approach, but I am in the need of a slightly more advanced scenario:
What I want is some kind of routing, so that one of the messages' properties gets used as some kind of routing key. I.e., I want multiple workers to be responsible for the messages, but all messages with value X in their routing key property should always go to the same worker, and all messages with Y should do as well. The workers for X and Y may be different, but all messages with X should always go to the same one.
Question 1: Is this even possible with MQTT 5? If so, what is the term I need to look for? I tried googling for this, but wasn't really successful (mainly, I guess, because I don't know exactly what to look for).
Now, supposed this is possible: How can I then handle cases where nodes join or leave? Then I still want only a single node to be responsible, so it would be great if the assignment was not statically, but could be adjusted dynamically (or even better, would adjust itself automatically). However, what I strictly need to avoid is that two messages with X ever go to different servers at the same time.
Question 2: Supposed, this is not possible – what alternatives do I have to MQTT 5?
You don't at a protocol level. That is the whole point of a shared subscription to distribute the incoming messages evenly across all the subscribers.
This also goes against the pub/sub paradigm, that messages are published to a topic not an individual subscriber.
If you want to route messages differently publish them to different topics. There is nothing to stop you republishing a message on a separate topic based on it's meta data once it's been received by a client if needed.
I have a virtual socketCAN network. How do I block a particular ID from being sent on the network?
If a node is connected to a CAN bus, at the lowest level it cannot be prevented from sending any message externally.
However, there are 3 things that can be done:
Add a gateway - a device that separates the bus into multiple small buses and passes messages from each sub-buses to the others, it does not prevent any node from sending a message, but it will not pass it to the others. This solution have a few clear drawbacks - it requires a separate device with multiple CAN interfaces (up to the number of nodes on the bus), it adds a delay for each message, and it renders the ACK bit unusable.
Apply filters for the received messages in each node. Again, this will not prevent sending the message, but will drop the load on the nodes. Most CAN controllers have hardware support for filtering by ID or a bit mask of ID.
There are some CAN controllers that can block the sending of messages, again, this will require adding such controller and setting it up for each node in the CAN bus.
I would like to hear your insights about an IoT data ingesting case. In AWS IoT hub, thing shadows are virtual representation of physical ones. What i understood from the figure below is whenever a thing sends a data to platform via a message broker, thing shadows and rule engine portions get the same sensor data concurrently and process it.
Are my conclusions correct ?
Things shadow system is subscribed to message broker and gets sensor data, updates their shadow actors. Shadow side is also responsible for storing sensor data such an event sourcing mechanism.
The thing shadow system does not perform any rules, it is just for performing event sourcing and keeping last known state in virtual thing actors.
The same sensor data also is an inbound data to rules engine. Rules engine is just and ECA (event condition action) type system that handle streaming data and decides what it will do with them. This means every incoming data eventually will be processed in rules engine portion.
Below are my comments to your conclusions.
What i understood from the figure below is whenever a thing sends a
data to platform via a message broker, thing shadows and rule engine
portions get the same sensor data concurrently and process it.
Changes in the thing shadow can trigger an action registered in the rule engine. There are specific topics associated with a thing shadow that you can subscribe the rule engine to, in order to perform one or many action(s) in response.
Things shadow system is subscribed to message broker and gets sensor
data, updates their shadow actors. Shadow side is also responsible for
storing sensor data such an event sourcing mechanism.
You can update the device shadow by using the REST API, or dedicated MQTT topics to publish on specific shadow topics. The shadow does not constitute an event-sourcing system by itself, but a representation of the data model associated with a physical device, as you said.
You can however create a rule that listens for changes on one or more shadow instances, and register the changes into DynamoDB for instance, in a time-series manner. You'll then have an event-sourcing system allowing you to store the previous states, or changes, sent by a device during an arbitrary amount of time.
The thing shadow system does not perform any rules, it is just for
performing event sourcing and keeping last known state in virtual
thing actors.
The thing shadow keeps the desired and reported state of a physical device in the cloud. It does not execute rules, but emits messages on MQTT topics when events happen within the shadow. These messages can then be captured by the rules engine to execute actions.
The same sensor data also is an inbound data to rules engine. Rules
engine is just and ECA (event condition action) type system that
handle streaming data and decides what it will do with them. This
means every incoming data eventually will be processed in rules engine
portion.
The rules engine does not listen by default on an MQTT topic, and hence, on data sent by devices to the Device Gateway. You must register in the rules engine the topics you'd like to listen to along with their associated actions.
Other than that, the rules engine allows you to describe your rules in ANSI SQL, meaning that you are able to specify the origin of your data (the FROM in your SQL statement), the specific fields in a JSON payload you are interested in capturing (SELECT), and an optional condition specifying on what condition the rule should be triggered (WHERE).
An example of a rule listening on the fictive topic device/+/telemetry and interested in capturing all the fields in the received payload would be :
SELECT * FROM device/+/telemetry
Note how the + can be used as a placeholder for any device identifier for instance.