How to write a lua-program for tarantool that will perform some tasks regularly (e.g. once per 10 minutes) in background?
The first way
Use fibers. Fibers - is a set of instructions which are executed with cooperative multitasking. Fibers managed by the fiber package are associated with a user-supplied function called the fiber function. A fiber has three possible states: running, suspended or dead.
Example
fiber.create(function()
while true do
-- Let say you have space with tree index.
-- Where each row index is timestamp + interval.
-- So, here you can get lower/upper bound by current timestamp e.g.
-- space:select{fiber.now()} -- get expired tasks
fiber.sleep(1) -- interval
end
end)
The second way
Use expirationd - https://github.com/tarantool/expirationd
Related
i read about flink`s window assigners over here: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-assigners , but i cant find any solution for my problem.
as part of my project i need a windowing that the timer will start given the first element of the key and will be closed and set ready for processing after X minutes. for example:
first keyA comes at (hh:mm:ss) 00:00:02, i want all keyA will be windowing until 00:01:02, and then the timer of 1 minutes will start again only when keyA will be given as input.
Is it possible to do something like that in flink? is there a workaround?
hope i made it clear enough.
Implementing keyed windows that are aligned with the first event, rather than with the epoch, is quite difficult, in general, which I believe is why this isn't supported by Flink's window API. The problem is that with an out-of-order stream using event time processing, as earlier events arrive you may need to revise your notion of when the window began, and when it should end. For example, if the first keyA arrives at 00:00:02, but then some time later an event with keyA arrives with a timestamp of 00:00:01, now suddenly the window should end at 00:01:01, rather than 00:01:02. And if the out-of-orderness is large compared to the window length, handling this becomes quite complex -- imagine, for example, that the event from 00:00:01 arrives 2 minutes after the event from 00:00:02.
Rather than trying to implement this with the window API, I would use a KeyedProcessFunction. If you only need to support processing time windows, then these concerns about out-of-orderness do not apply, and the solution can be fairly simple. It suffices to keep one object in keyed state, which might be a list holding all of the events in the window, or a counter or other aggregator, depending on what you're trying to accomplish.
When an event arrives, if the state (for this key) is null, then there is no open window for this key. Initialize the state (i.e., create a new, empty list, or set the counter to zero), and create a Timer to fire at the appropriate time. Then regardless of whether the state had been null, add the incoming event to the state (i.e., append it to the list, or increment the counter).
When the timer fires, emit the window's result and reset the state to null.
If, on the other hand, you want to do this with event time windows, first sort the stream and then use the same approach. Note that you won't be able to handle late events, so plan your watermarking accordingly (reducing the likelihood of late events to a manageable level), or go for a more complex implementation.
Is there any way to measure the performance of a Redis Lua script?
I have a lua script and I ended up to a slightly different implementation and I am wondering if there is any way to measure which of the two implementations is faster.
You can call Redis' TIME command to perform in-script "benchmarking". Something like the following should work:
local start = redis.call('TIME')
-- your logic here
local finish = redis.call('TIME')
return finish[1]-start[1]
I read in the comments that someone mensioned finish[2]-start[2] which is not a good idea because [2] has "the amount of microseconds already elapsed in the current second" and not the entire timestamp (so if we finish in a different second, this calculation will fail.)
based on: https://redis.io/commands/TIME
to get time as microseconds, I would do:
local start = redis.call('TIME')
-- your logic here
local finish = redis.call('TIME')
return (finish[1]-start[1])*1000000+(finish[2]-start[2])
I am developing some data analysis algorithms on top of Storm and have some questions about the internal design of Storm. I want to simulate a sensor data yielding and processing in Storm, and therefore I use Spout to push sensor data into the succeeding bolts at a constant time interval via setting a sleep method in nextTuple method of Spout. But from the experiment results, it appeared that spout didn't push data at the specified rate. In the experiment, there was no bottleneck bolt in the system.
Then I checked some material about the ack and nextTuple methods of Storm. Now my doubt is if the nextTuple method is called only when the previous tuples are fully processed and acked in the ack method?
If this is true, does it means that I cannot set a fixed time interval to emit data?
Thx a lot!
My experience has been that you should not expect Storm to make any real-time guarantees, including in your case the rate of tuple processing. You can certainly write a spout that only emits tuples on some time schedule, but Storm can't really guarantee that it will always call on the spout as often as you would like.
Note that nextTuple should be called whenever there is room available for more pending tuples in the topology. If the topology has free capacity, I would expect Storm to try to fill it up if it can with whatever it can get.
I had a similar use-case, and the way I accomplished it is by using TICK_TUPLE
Config tickConfig = new Config();
tickConfig.put(Config.TOPOLOGY_TICK_TUPLE_FREQ_SECS, 15);
...
...
builder.setBolt("storage_bolt", new S3Bolt(), 4).fieldsGrouping("shuffle_bolt", new Fields("hash")).addConfigurations(tickConfig);
Then in my storage_bolt (note it's written in python, but you will get an idea) i check if message is tick_tuple if it is then execute my code:
def process(self, tup):
if tup.stream == '__tick':
# Your logic that need to be executed every 15 seconds,
# or what ever you specified in tickConfig.
# NOTE: the maximum time is 600 s.
storm.ack(tup)
return
I am trying to find out a way to increase the computation time of a function to 1 second without using the sleep function in xilinx microblaze, using the xilkernel.
Hence, may i know how many iterations do i need to do in a simple for loop to increase the computation time to 1 second?
You can't do this reliably and accurately. If you want do a bodge like this, you'll have to calibrate it yourself for your particular system as Microblaze is so configurable, there isn't one right answer. The bodgy way is:
Set up a GPIO peripheral, set one of the pins to '1', run a loop of 1000 iterations (make sure the compiler doesn't optimise it away!) set the pins to '0'. Hang a scope off that pin (you're doing work on embedded systems, you do have a scope, right?) and see how long it takes to run the loop.
But the right way to do it is to use a hardware timer peripheral. Even at a very simple level, you could clear the timer at the start of the function, then poll it at the end until it reaches whatever value corresponds to 1 second. This will still have some imperfections, but given that you haven't specified how close to 1 sec you need to be, it is probably adequate.
I capture data with tshark and save certain data from the packet header to process them in order to detect some incedants in the network. I saved the data in a table in my lua program (which is running in the cmd with tshark using the command (-Xlua_script:))
and now i want to process the data of each minute alone while capturing is running. It's an online processing. Firstly:Any body knows if this could be implemented?Secondly I need a timer, I don't know how to do this, and i want a way that i can take the data in the tables to process them, reset the tables to get the new data of the next minute without losing any data.
Any suggestions or ideas??
there isn't the concept of a 'timer' in lua like some other languages, where you can create one and set up an event handler and have your main program notified when the timer goes off... however you can periodically check os.clock() to determine how long its been since you've done some processing and if a minute has elapsed, go ahead and process the data.
something like this might be what you need:
lastTimeProcessed = os.clock()
function IsTimeToProcess(currentTime)
span = currentTime - lastTimeProcessed
if span >= 60 then
lastTimeProcessed = currentTime
return true
end
return false
end
while true do
if IsTimeToProcess(os.clock()) then
-- process some data here
end
-- otherwise do another round of whatever you're doing
end