I am trying to create a pattern in EPL, but i do not get why it gets matched only one. Here is what I did on the web page Esper EPL online. The EPL I input in the left textbox is:
create schema StockTick(signature string, source string, destination string, action string);
#Name('Monitor') select * from StockTick;
#Name('Out') select * from pattern [A=StockTick -> B=StockTick(B.source = A.source and B.destination = A.destination and B.action='block') where timer:within(2 seconds)];
and the events I input in the middle textbox are:
StockTick={signature='sig1', source ='s1', destination ='d1'}
t=t.plus(0.4 seconds)
StockTick={source ='s1', destination ='d1', action='block'}
t=t.plus(0.4 seconds)
StockTick={signature='sig2', source ='s2', destination ='d2'}
t=t.plus(0.4 seconds)
StockTick={source ='s2', destination ='d2', action='block'}
t=t.plus(0.4 seconds)
The output I get in the right box:
At: 2001-01-01 08:00:00.000
Statement: Monitor
Insert
StockTick={signature='sig1', source='s1', destination='d1', action=(null)}
At: 2001-01-01 08:00:00.400
Statement: Monitor
Insert
StockTick={signature=(null), source='s1', destination='d1', action='block'}
Statement: Out
Insert
stmt2_pat_0_1={A={StockTick={signature='sig1', source='s1', destination='d1', action=(null)}}, B={StockTick={signature=(null), source='s1', destination='d1', action='block'}}}
At: 2001-01-01 08:00:00.800
Statement: Monitor
Insert
StockTick={signature='sig2', source='s2', destination='d2', action=(null)}
At: 2001-01-01 08:00:01.200
Statement: Monitor
Insert
StockTick={signature=(null), source='s2', destination='d2', action='block'}
So:
after 0 seconds, the Monitor statement gets fired (fine!)
after 0.4 seconds, the Monitor statement gets fired again (fine!)
after 0.4 seconds, the Out statement gets fired (fine!)
after 0.8 seconds, the Monitor statement gets fired again (fine!)
after 1.2 seconds, the Monitor statement gets fired again (fine!)
What I do not understand is why after 1.2 seconds I do not get fired the Out statement again, as I would expect.
Please note that if I try the first two inserts alone and the second two inserts alone, in both cases I have the Monitor statement fired twice and the Out statement fired once. The problem arises when I concatenate the four insert statements.
Thanks to another forum, I managed to solve the problem. I missed the every keyword in defining the pattern, so, after updating the rule as follows:
#Name('Out') select * from pattern [every A=StockTick -> B=StockTick(B.source = A.source and B.destination = A.destination and B.action='block')];
it worked like a charm:
At: 2001-01-01 08:00:00.000
Statement: Monitor
Insert
StockTick={signature='sig1', source='s1', destination='d1', action=(null)}
At: 2001-01-01 08:00:00.400
Statement: Monitor
Insert
StockTick={signature=(null), source='s1', destination='d1', action='block'}
Statement: Out
Insert
stmt2_pat_0_0={A={StockTick={signature='sig1', source='s1', destination='d1', action=(null)}}, B={StockTick={signature=(null), source='s1', destination='d1', action='block'}}}
At: 2001-01-01 08:00:00.800
Statement: Monitor
Insert
StockTick={signature='sig2', source='s2', destination='d2', action=(null)}
At: 2001-01-01 08:00:01.200
Statement: Monitor
Insert
StockTick={signature=(null), source='s2', destination='d2', action='block'}
Statement: Out
Insert
stmt2_pat_0_0={A={StockTick={signature='sig2', source='s2', destination='d2', action=(null)}}, B={StockTick={signature=(null), source='s2', destination='d2', action='block'}}}
Related
we have a strange issue, where esper query is only partial executed...
select cseShutDownAlarm.INSTANCEID as INSTANCEID, swtDownAlarm.source
from Alarm(severity.getValue()=5, eventType.getValue() = 'SWT_SWITCH_DOWN').win:time(60 sec) as swtDownAlarm,
sql:stormdb['select id as INSTANCEID, source as SOURCE
from Alarm where EVENTTYPE_VALUE =\'cseShutDownNotify\' and source = ${swtDownAlarm.source} and severity !=5'] as cseShutDownAlarm,
sql:stormdb['insert into wcsdba.dyinggasp (parameter, value) VALUES (\'cseShutDownAlarm.SOURCE\', ${cseShutDownAlarm.SOURCE}) '],
sql:stormdb['insert into wcsdba.dyinggasp (parameter, value) VALUES (\'swtDownAlarm.source\', ${swtDownAlarm.source}) '] where swtDownAlarm.source = cseShutDownAlarm.SOURCE
and in the DB, we see that:
SQL> /
PARAMETER VALUE
cseShutDownAlarm.SOURCE 172.16.148.48
cseShutDownAlarm.SOURCE 172.16.148.48
SQL>
but second source (swtDownAlarm.source) is not printed...
If I switch the order then the other one will be inserted only.
any reason why it does not execute both inserts? also the rest of the condition is not checked... as the source on both are identical, but condition not fulfilled.
Thanks,
fcbman
Joins are meant to be the place for event stream joins against SQL database for purpose of querying. The "insert into" isn't a query and would never return rows. And if an inner join doesn't return rows it ends early (outer join would be right then)
Given chain of insert-into statements, for example:
on GenericEvent insert into rule_A_stream select *;
on rule_A_stream
insert into rule_B_stream select * where condition1
insert into rule_C_stream select * where condition2;
on rule_B_stream
insert into rule_D_stream select * where condition3
insert into rule_E_stream select * where condition4;
on rule_E_stream
insert into rule_F_stream select * where condition4
insert into rule_G_stream select * where condition5;
How can we terminate Event processing immediately once rule_E_stream is reached and return the results? (termination means stopping event propagation so no following statements should not be invoked)
Is it possible to stop processing on Subscriber/Listener side?
Considering using context with “terminated by” Event:
create context Ctx initiated by InitEvent terminated by TerminateEvent
I wonder is it applicable in our case?
The #Drop annotation (aka. prioritized execution) terminates. An example would be:
#Drop #Priority(1) select * from rule_E_stream(drop_conditions_here_if_any)
#Priority(0) on rule_E_Stream ....
Listeners don't stop things in Esper in general. But listeners can reinsert (aka. route) events as a result of their own filter or transformation they may do.
I have a question about using the output clause in combination with a named window, patterns and the insert into statement.
My goal is to detect the absence of an event, store it in a named window and when the events start coming again select and delete the row and use that as an "online" indicator (see Esper - detect event after absence)
I somehow want to be able limit the rate of events when there are multiple offline - online events in a short period of time (disable false positives). I thought the output clause could help here but when I use that on the insert into statement no events are stored in the named window. Is this the right approach or is there an other way to limit the events in this scenario?
This is my code in Esper EPL online:
create schema MonitorStats(id string, time string, host string, alert string);
create window MonitorWindow.win:keepall() as select id, alert, time, host from MonitorStats;
insert into MonitorWindow select a.id as id, 'offline' as alert, a.time as time, a.host as host from pattern
[every a=MonitorStats(id='1234') -> (timer:interval(25 sec) and not MonitorStats(id=a.id))];
on pattern[every b=MonitorStats(id='1234') -> (timer:interval(25 sec) and MonitorStats(id=b.id))]
select and delete b.id as id, 'online' as alert, b.time as time, b.host as host from MonitorWindow as win
where win.id = b.id;
You may insert output events into another stream and use "output first every".
insert into DetectedStream select ....;
select * from DetectedStream output first every 10 seconds;
Let's consider a simple object with the same representation in a SQL database with properties(columns¨): Id, UserId,Ip.
I would like to prepare a query that would generate event in case that one user logs in from 2 IP adresses (or more) within 1 hour period.
My SQL looks like:
SELECT id,user_id,ip FROM w_log log
LEFT JOIN
(SELECT user_id, count(distinct ip) AS ip_count FROM w_log GROUP BY user_id) ips
ON log.user_id = ips.user_id
WHERE ips.ip_count > 1
Transformation to EPL:
SELECT * FROM LogEntry.win:time(1 hour) logs LEFT INNER join
(select UserId,count(distinct Ip) as IpCount FROM LogEntry.win:time(1 hour)) ips
ON logs.UserId = ips.UserId where ips.IpCount>1
Exception:
Additional information: Incorrect syntax near '(' at line 1 column 100,
please check the outer join within the from clause near reserved keyword 'select'
UPDATE:
I was successfuly able to create a schema, named window and insert data into it (or update it). I would like to increase the counter when a new LogEvent arrives in the .win:time(10 seconds) and decrease it when the event is leaving the 10 seconds window. Unfortunately the istream() doesn't seem to provide the true/false when the event is in remove stream.
create schema IpCountRec as (ip string, hitCount int)
create window IpCountWindow.win:time(10 seconds) as IpCountRec
on LogEvent.win:time(10 seconds) log
merge IpCountWindow ipc
where ipc.ip = log.ip
when matched and istream()
then update set hitCount = hitCount + 1
when matched and not istream()
then update set hitCount = hitCount - 1
when not matched
then insert select ip, 1 as hitCount
Is there something I missed?
In EPL I don't think it is possible to put a query into the from-part. You can change using "insert into". An EPL alternative is also a named window or table.
The problem is pretty simple: extract only the not exists records from 2 different streams using Esper engine.
ID exists in streamA but NOT EXITS in streamB.
In SQL it would look like this:
SELECT *
FROM tableA
WHERE NOT EXISTS (SELECT *
FROM tableB
WHERE tableA.Id = tableB.Id)
I've tried it Esper style but it doesn't work:
SELECT *
FROM streamA.win:ext_timed(timestamp, 5 seconds) as stream_A
WHERE NOT EXSITS
(SELECT stream_B.Id
FROM streamB.win:ext_timed(timestamp, 5 seconds) as stream_B
WHERE stream_A.Id = stream_B.Id)
Sadly if stream_A.Id inserted before stream_B.id than it will answer the query conditions and the query won't work.
Any suggestions on how to identify "ID exists in streamA but NOT EXITS in streamB" using Esper?
One simple way is to time-order the stream, so that A and B are timestamp ordered before sending events in.
Or you could delay A such as this query:
select * from pattern [every a=streamA -> timer:interval(1 sec)] as delayed_a
where not exists (... where delayed_a.a.id = b.id)
There is no need for an externally timed window for streamA. For externally timed behavior in general use external timer events.