is it possible to detect the absence of a certain event type within a given time window without using any other event types in Esper?
Thanx ;)
You can take a look at the solution patterns here for some ideas:
http://www.espertech.com/esper/solution-patterns#absence-1
Here is an adapted example of detecting an absence of an event, after it has been fired once:
select * from pattern [every EventX -> (timer:interval(10 sec) and not EventX)];
This will fire only once, if after an EventX is received, no other EventX arrives within 10 seconds...
This example:
select * from pattern [every (timer:interval(10 sec) and not EventX)];
Will fire every 10 seconds if no EventX has been receive during that time (and keep firing like this forever).
Outer joins are also a good way to detect the absence of an event.
Related
i read about flink`s window assigners over here: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#window-assigners , but i cant find any solution for my problem.
as part of my project i need a windowing that the timer will start given the first element of the key and will be closed and set ready for processing after X minutes. for example:
first keyA comes at (hh:mm:ss) 00:00:02, i want all keyA will be windowing until 00:01:02, and then the timer of 1 minutes will start again only when keyA will be given as input.
Is it possible to do something like that in flink? is there a workaround?
hope i made it clear enough.
Implementing keyed windows that are aligned with the first event, rather than with the epoch, is quite difficult, in general, which I believe is why this isn't supported by Flink's window API. The problem is that with an out-of-order stream using event time processing, as earlier events arrive you may need to revise your notion of when the window began, and when it should end. For example, if the first keyA arrives at 00:00:02, but then some time later an event with keyA arrives with a timestamp of 00:00:01, now suddenly the window should end at 00:01:01, rather than 00:01:02. And if the out-of-orderness is large compared to the window length, handling this becomes quite complex -- imagine, for example, that the event from 00:00:01 arrives 2 minutes after the event from 00:00:02.
Rather than trying to implement this with the window API, I would use a KeyedProcessFunction. If you only need to support processing time windows, then these concerns about out-of-orderness do not apply, and the solution can be fairly simple. It suffices to keep one object in keyed state, which might be a list holding all of the events in the window, or a counter or other aggregator, depending on what you're trying to accomplish.
When an event arrives, if the state (for this key) is null, then there is no open window for this key. Initialize the state (i.e., create a new, empty list, or set the counter to zero), and create a Timer to fire at the appropriate time. Then regardless of whether the state had been null, add the incoming event to the state (i.e., append it to the list, or increment the counter).
When the timer fires, emit the window's result and reset the state to null.
If, on the other hand, you want to do this with event time windows, first sort the stream and then use the same approach. Note that you won't be able to handle late events, so plan your watermarking accordingly (reducing the likelihood of late events to a manageable level), or go for a more complex implementation.
I read that, "..The ordering operator has to buffer all elements it receives. Then, when it receives a watermark it can sort all elements that have a timestamp that is lower than the watermark and emit them in the sorted order. This is correct because the watermark signals that not more elements can arrive that would be intermixed with the sorted elements..." - https://cwiki.apache.org/confluence/display/FLINK/Time+and+Order+in+Streams
Hence, it seems that the watermark serves as a signal to the following operator, for beginning processing. I guess, that's what also a Trigger does. What's the difference between the two?
You can think of watermarks as special records that tell an operator what (event-) time it is. When an operator receives a watermark, it compares the watermark with its current time and other watermarks it received from different stream partitions. Depending on the comparison, the operator advances its own clock.
Some operators register timers (windows, time-based joins, custom implementations). An operator triggers a timer when the clock of the operator passes the time for which the timer was registered.
So, watermarks and timers are two different things. Watermarks tell an operator what time it is and the operator triggers a timer at the right point in time.
A Watermark can be thought of as an assertion that an event time stream is now complete up to a particular timestamp. When a Watermark is processed by an operator it will cause the firing of any relevant event time timers. The operators that use EventTimeTimers are EventTimeWindows and ProcessFunctions.
Triggers are part of the window API and define when Windows will produce results. An EventTimeTrigger wraps around an event time timer that is called when an suitably large Watermark is processed, indicating that the window is now complete.
How to write a lua-program for tarantool that will perform some tasks regularly (e.g. once per 10 minutes) in background?
The first way
Use fibers. Fibers - is a set of instructions which are executed with cooperative multitasking. Fibers managed by the fiber package are associated with a user-supplied function called the fiber function. A fiber has three possible states: running, suspended or dead.
Example
fiber.create(function()
while true do
-- Let say you have space with tree index.
-- Where each row index is timestamp + interval.
-- So, here you can get lower/upper bound by current timestamp e.g.
-- space:select{fiber.now()} -- get expired tasks
fiber.sleep(1) -- interval
end
end)
The second way
Use expirationd - https://github.com/tarantool/expirationd
Can NSTimer be used to fire a series of events. For instance for effect:
Its kick off Click start to toss
create random number
wait 5 seconds show result
wait 3 seconds start the match?
You can use it to repeat at a given interval, but not a variable one. If you really wanted to wait 5 seconds and then then wait another 3 seconds you'd probably want to chain timers. So, when the first timer fires and calls a message, that message creates a second timer with a different time interval.
This is actually a case where the Prototype Pattern would apply: make an NSTimer and set it up with all the properties you want, and then clone that object each time you need to make another. Or you could just make a factory. Objective C does not have a clone, but the NSCoding protocol is actually a workable and proper way of doing cloning (unlike Java's broken (and abandoned) clone interface).
I am using Complex Event Processing (Esper) technology to provide a real-time candlestick calculations in my system. I am doing fine with calculating values, however I find it difficult to ensure that candle window starts at full minutes (for one minute candle) and ends before the next minute starts (i.e. candle 1[06:00.000 - 06:00.999], candle 2[06:01.000 - 06:01.999], etc... ).
Is there a pattern or command in Esper's query language that is able to provide such functionality?
I'd appreciate constructive comments and directions.
In Esper you can use a pattern to fire every minute at the zero second, i.e.
insert into TriggerEvent select * from pattern[pattern[every timer:interval(1 min).]
// named window to hold candle data, compute next candle
on TriggerEvent select * from NamedWindowCandle ....
// delete old data
on TriggerEvent delete from NamedWindowCandle
-rg
Local time is often different from exchange time, also there is latency in delivering tick data. Minute bars are often computed using exchange timestamp. The exchange timestamp value must be extracted from tick events. New minute bar event is sent when the tick timestamps enter new minute.