My aim is to measure MQTT device-to-device message latency (not throughput) and I'm looking for feedback on my code-hacks. The setup is simple; just one device serving as two end-points (old Linux PC with two terminal sessions; one running the subscriber and the other running the publisher sample app) and the default broker at tcp://m2m.eclipse.org:1883). I inserted time-capturing code-fragments into the C-language publish/subscribe sample apps on the src/samples folder.
Below are the changes. Please provide feedback.
Changes to the subscribe sample app (MQTTAsync_subscribe.c)
Inserted the lines below at the top of the msgarrvd (message arrived) function
//print arrival time
struct timeval tv;
gettimeofday (&tv, NULL);
printf("Message arrived: %ld.%06ld\n", tv.tv_sec, tv.tv_usec);
Changes to the publish sample app (MQTTAsync_publish.c)
Inserted the lines below at the top of the onSend (callback) function
struct timeval tv;
gettimeofday (&tv, NULL);
printf("Message with token value %d delivery confirmed at %ld.%06ld\n",
response->token, tv.tv_sec, tv.tv_usec);
With these changes (after subtracting the time message arrived at the subscriber from the time that the delivery was confirmed at the publisher), I get a time anywhere between 1 millisecond and 0.5 millisecond.
Questions
Does this make sense as a rough benchmark on latency?
Is this the round-trip time?
Is the round-trip time in the right ball-park? Should be less? more?
Is it the one-way time?
Should I design the latency benchmark in a different way? I need a rough measurements (I'm comparing with XMPP).
I'm using the default QoS value (1). Should I change it?
The publisher takes a finite amount of time to connect (and disconnect). Should these be added?
The 200ms latency is high ! Can you please upload your code here ?
Does this make sense as a rough benchmark on latency?
-- Yes it makes sense. But a better approach is to make an automated time subtract with subscribed message and both synchronized to NTP.
Is this the round-trip time? Is it the one-way time?
-- Messages got published - you received ACK for publisher and same message got transferred to subscribed client.
Is the round-trip time in the right ball-park? Should be less? more?
-- It should be less.
Should I design the latency benchmark in a different way? I need a rough measurements (I'm comparing with XMPP).
I'm using the default QoS value (1). Should I change it?
-- Try with QoS 0 ( fire and forget )
The publisher takes a finite amount of time to connect (and disconnect). Should these be added?
-- Yes. It needs to be added but this time should be very small.
Related
I am new to Beam/Dataflow and am trying to figure out if it is suited to this problem. I am trying to keep a running sum of which types of messages are currently backlogged in a queueing system. The system uses a monotonically increasing offset number to order messages: producers learn the number when the send a message, and consumers track the watermark offset as they process each message in FIFO order. This pipeline would have two inputs: counts from the producers and watermarks from the consumers.
The queue producer would regularly flush a batch of count metrics to Beam:
(type1, offset, count)
(type2, offset, count)
...
where the offset was the last offset the producer wrote for typeN, and count is how many typeN messages it enqueued in the current batch period.
The queue consumer will regularly send its latest consumed watermark offset. The effect this should have is to invalidate any counts that have an offset lower than this consumer watermark.
The output of the pipeline is the sum of all counts with a higher offset than the largest consumer watermark yet seen, grouped by message type. (snapshotted every 5 minutes or so.)
(Of course there would be 100k message "types", hundreds of producer servers, occasional 2-hour periods where the consumer doesn't report an advancing watermark, etc.)
Is this doable? That this pipeline would need to maintain and scan an unbounded-ish history of count records is the part that seems maybe unsuited to Beam.
One possible approach would be to model this as two timeseries (left , right) where you want to match left.timestamp <= right.timestamp. You can do this using the State and Timer API.
In order to achieve this unbounded, you will need to be working within a GlobalWindow. Important note in the Global Window there is no expiry of the state, so you will need to make sure to do Garbage Collection on your left and right streams. Also data will arrive in the onprocess unordered, so you will need to make use of Event Time timers to do the actual work.
Very roughly:
onProcess(){
Store data in BagState.
Setup Event time timer to go off
}
OnTimer(){
Do your buiss logic.
}
This is a lot easier with Apache Beam > 2.24.0 as OrderedListState has been added.
Although the timeseries use case is different from the one in this question, this talk from the 2019 Beam summit also has some pointers (but does not make use of OrderedListState, which was not available at the time);
State and Timer API and Timeseries
We would like to use Graphite to plot values related to events such as "a packet of N messages has been published". When no packet is published, no code is run at all and so we cannot send zero to Graphite.
Essentially, we would like to compute some kind of publication rate per second.
Here are some sample data that we send to Graphite (with added timestamps):
2016-11-28 14:46:33.6338Z api.message.publication.count:100
2016-11-28 15:01:36.0780Z api.message.publication.count:12
2016-11-28 15:01:36.9911Z api.message.publication.count:1
2016-11-28 15:01:37.0679Z api.message.publication.count:100
Between 14:46:33 and 15:01:36, no messages were sent. However, between 15:01:36 and 15:01:37, 13 messages were sent (reported as two values, 12 and 1).
I've tried the summarize() function but it does not give results that make sense to me, i.e. I cannot correlate what I'm sending to Graphite and what is displayed by Graphite. Moreover, it seems that summarize() does not support 1-second intervals (I've tried "1second" and "1s" for the interval parameter).
The perSecond() function computes a rate of change (i.e. a derivative) but what we're sending is already a kind of derivative (maybe it's closer to a Dirac delta?) so it doesn't make sense in our context.
Are we completely off, or is there a way to make this work with Graphite?
Edit: I guess we need to add an aggregation stage to our data. Would Carbon aggregation fit the bill here?
It turns out that we were already sending our metrics to statsd, which supports aggregation via the c metric type, and a few other nifty things: https://github.com/etsy/statsd/blob/master/docs/metric_types.md
I'm pretty new to using GNURadio and I'm having trouble recovering the data from a signal that I've saved into a file. The signal is a carrier frequency of 56KHz with a frequency shift key of +/- 200hz at 600 baud.
So far, I've been able to demodulate the signal that looks similar to the signal I get from the source:
I'm trying to get this into a repeating string of 1s and 0s (the whole telegram is 38 bytes long and it continuously repeats). I've tried to use a clock recovery block in order to have only one byte per sample, but I'm not having much luck. Using the M&M clock recovery block, the whole telegram sometimes comes out correct, but it is not consistent. I've tried to adjust the omega and Mu values, but it doesn't seem to help that much. I've also tried using the Polyphase Clock sync, but I keep getting a runtime error of 'please specify a filter'. Is this asking me to add a tap? what tap would i use?
So I guess my overall question would be: What's the best way to get the telegram out of the demodulated fsk signal?
Again, pretty new at this so please let me know if I've missed something crucial. GNU flow graph below:
You're recovering the bit timing, but you're not recovering the byte boundaries – that needs to happen "one level higher", eg. by a well-known packet format with a defined preamble that you can look for.
I am trying to drive down the current consumption of the contiki os running on the CC2538 development kit.
I would like to operate the device from a CR2032 with a run life of 2 years. To achieve this I would need an average current less than 100uA.
However when I run the following at 3V, I get the following results:
contiki/examples/hello-world = 0.4mA - 2mA
contiki/examples/er-rest-example/er-example-client = 27mA
contiki/examples/er-rest-example/er-example-server = 27mA
thingsquare websocket example = 4mA
I have also designed my own target platform based on the cc2538 and get similar results.
I have read the guide at https://github.com/contiki-os/contiki/blob/648d3576a081b84edd33da05a3a973e209835723/platform/cc2538dk/README.md
and have ensured that in the contiki-conf.h file:
- LPM_CONF_ENABLE 1
- LPM_CONF_MAX_PM 2
Can anyone give me some pointers as to how I can get the current down. It would be most appreciated.
Regards,
Shane
How did you measure the current?
You have to be aware that using a basic ampere meter to measure the current consumption of contiki-os wouldn't give you relevant results. The system is turning on/off the radio at a relative high rate (8Hz by default) in order to perform the CCA. This might not be very easy to catch for an ampere meter.
To have an idea of the current consumption when the device is in deep sleep (and then make calculation to determine the averaged current consumption), I'd rather put the device in the PM state before the program reach the infinite while loop. I used the following code to do that:
lpm_enter();
REG(SYS_CTRL_PMCTL) = SYS_CTRL_PMCTL_PM2;
do { asm("wfi"::); } while(0);
leds_on(LEDS_RED); // should not reach here
while(1){
...
On the CC2538, the CCA check consumes about 10-15mA and last approximately 2ms. When the radio transmit a packet, it consume 25mA. Have a look at this post: Contiki UDP packet transmission duration with CC2538.
Furthermore, to save a little more current, turn off the serial com:
#define CC2538_CONF_QUIET 1
Are you using the SmartRF board? If you want to make proper current measurement with this board, you have to remove every jumpers: P486, P487, P411 and P408. Keep only the jumpers of BTN_SEL and the RESET signals.
I am developing some data analysis algorithms on top of Storm and have some questions about the internal design of Storm. I want to simulate a sensor data yielding and processing in Storm, and therefore I use Spout to push sensor data into the succeeding bolts at a constant time interval via setting a sleep method in nextTuple method of Spout. But from the experiment results, it appeared that spout didn't push data at the specified rate. In the experiment, there was no bottleneck bolt in the system.
Then I checked some material about the ack and nextTuple methods of Storm. Now my doubt is if the nextTuple method is called only when the previous tuples are fully processed and acked in the ack method?
If this is true, does it means that I cannot set a fixed time interval to emit data?
Thx a lot!
My experience has been that you should not expect Storm to make any real-time guarantees, including in your case the rate of tuple processing. You can certainly write a spout that only emits tuples on some time schedule, but Storm can't really guarantee that it will always call on the spout as often as you would like.
Note that nextTuple should be called whenever there is room available for more pending tuples in the topology. If the topology has free capacity, I would expect Storm to try to fill it up if it can with whatever it can get.
I had a similar use-case, and the way I accomplished it is by using TICK_TUPLE
Config tickConfig = new Config();
tickConfig.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 15);
...
...
builder.setBolt("storage_bolt", new S3Bolt(), 4).fieldsGrouping("shuffle_bolt", new Fields("hash")).addConfigurations(tickConfig);
Then in my storage_bolt (note it's written in python, but you will get an idea) i check if message is tick_tuple if it is then execute my code:
def process(self, tup):
if tup.stream == '__tick':
# Your logic that need to be executed every 15 seconds,
# or what ever you specified in tickConfig.
# NOTE: the maximum time is 600 s.
storm.ack(tup)
return