I'm trying to off set my kapacitor query to look for a period of 5 mins but start 10 secs in the past. so the query would be for (now - 10s) to (now - 5m10s). This is to offset the the delay in data.
dbrp "telegraf"."Raw"
var process_load = batch
|query('''SELECT sum("ProcessTime")
FROM "telegraf"."Raw".Document
WHERE ("Application" = 'AMP' AND "Environment" = 'test' )
''')
.period(5m)
.every(10s)
.groupBy( 'GroupID' , 'Thread' , time(5m))
|alert()
.stateChangesOnly()
.warn(lambda: "sum" > 90000)
.crit(lambda: "sum" > 240000)
.log('/tmp/document.test.log')
Currently the influxdb is getting it's data from telegraf but the delay is causing the query to send a warning message even when it should be at critical. currently it sends an warning and a critical switching between the two. It should get a critical and then stay there until a document thread stops.
found the issue I needed to add the .align() and the .aligngroup() to the query node
Related
Good afternoon,
I have created the following tickscript with a standard tickstack setup.
Which includes: InfluxDB(latest version) and kapacitor(latest version):
dbrp "derivatives"."default"
var data = batch
|query('select sum(value) from "derivatives"."default".derivative_test where time > now() - 10m')
.every(1m)
.period(2m)
var slope = data
|derivative('value')
.as('slope')
.unit(2m)
slope
|eval(lambda: ("slope" - "value") / "value")
.as('percentage')
|alert()
.crit(lambda: "percentage" <= -50)
.id('derivative_test_crit')
.message('{{ .Level }}: DERIVATIVE FOUND!')
.topic('derivative')
// DEBUGGING
|influxDBOut()
.database('derivatives')
.measurement('derivative_logs')
.tag('sum', 'sum')
.tag('slope', 'slope')
.tag('percentage', 'percentage')
But every time i want to define it i get the following message:
batch query is not allowed to request data from "derivatives"."autogen"
I never had this problem before with stream's but every batch tick script i write returns the same message.
My kapacitor user has full admin privs and i am able to get the data via a curl request, does anyone have any idea what could possibly be the problem here?
My thanks in advance.
Change this
dbrp "derivatives"."default"
var data = batch
|query('select sum(value) from "derivatives"."default".derivative_test where time > now() - 10m')
to this:
dbrp "derivatives"."autogen"
var data = batch
|query('select sum(value) from "derivatives"."autogen".derivative_test where time > now() - 10m')
It might not be obvious, but the retention policy is most likely incorrect.
If you run SHOW RETENTION POLICIES on the derivatives database you will see the RP's. I suspect you have an RP of autogen, which is the default RP. However "default" doesn't normally exist as an RP unless you create it, it just signifies that it is the default RP, if that makes sense?
RP Documentation might help clear it up Database Documentation.
default autogen RP
I'd like to calculate the delta values for a series of measurements stored in an InfluxDB. The values are readings from an electricity meter taken every 5 minutes. The values increase over time. Here is subset of the data to give you an idea (commands shown below are executed in the InfluxDB CLI):
> SELECT "Haushaltstromzaehler - cnt" FROM "myhome_measurements" WHERE time >= '2018-02-02T10:00:00Z' AND time < '2018-02-02T11:00:00Z'
name: myhome_measurements
time Haushaltstromzaehler - cnt
---- --------------------------
2018-02-02T10:00:12.610811904Z 11725.638
2018-02-02T10:05:11.242021888Z 11725.673
2018-02-02T10:10:10.689827072Z 11725.707
2018-02-02T10:15:12.143326976Z 11725.736
2018-02-02T10:20:10.753357056Z 11725.768
2018-02-02T10:25:11.18448512Z 11725.803
2018-02-02T10:30:12.922032896Z 11725.837
2018-02-02T10:35:10.618788096Z 11725.867
2018-02-02T10:40:11.820355072Z 11725.9
2018-02-02T10:45:11.634203904Z 11725.928
2018-02-02T10:50:11.10436096Z 11725.95
2018-02-02T10:55:10.753853952Z 11725.973
Calculating the differences in the InfluxDB CLI is pretty straightforward with the difference() function. This gives me the electricity consumed within the 5 minutes intervals:
> SELECT difference("Haushaltstromzaehler - cnt") FROM "myhome_measurements" WHERE time >= '2018-02-02T10:00:00Z' AND time < '2018-02-02T11:00:00Z'
name: myhome_measurements
time difference
---- ----------
2018-02-02T10:05:11.242021888Z 0.03499999999985448
2018-02-02T10:10:10.689827072Z 0.033999999999650754
2018-02-02T10:15:12.143326976Z 0.02900000000045111
2018-02-02T10:20:10.753357056Z 0.0319999999992433
2018-02-02T10:25:11.18448512Z 0.03499999999985448
2018-02-02T10:30:12.922032896Z 0.033999999999650754
2018-02-02T10:35:10.618788096Z 0.030000000000654836
2018-02-02T10:40:11.820355072Z 0.03299999999944703
2018-02-02T10:45:11.634203904Z 0.028000000000247383
2018-02-02T10:50:11.10436096Z 0.02200000000084401
2018-02-02T10:55:10.753853952Z 0.02299999999922875
Where I struggle is getting this to work in a continuous query. Here is the command I used to setup the continuous query:
CREATE CONTINUOUS QUERY cq_Haushaltstromzaehler_cnt ON myhomedb
BEGIN
SELECT difference(sum("Haushaltstromzaehler - cnt")) AS "delta" INTO "Haushaltstromzaehler_delta" FROM "myhome_measurements" GROUP BY time(1h)
END
Looking in the InfluxDB log file I see that no data is written in the new 'delta' measurement from the continuous query execution:
...finished continuous query cq_Haushaltstromzaehler_cnt, 0 points(s) written...
After much troubleshooting and experimenting I now understand why no data is generated. Setting up a continuous query requires to use the GROUP BY time() statement. This in turn requires to use an aggregate function within the differences() function. The problem now is that the aggregate function returns only one value for the time period specified by GROUP BY time(). Obviously, the differences() function cannot calculate a difference from just one value. Essentially, continuous query executes a command like this:
> SELECT difference(sum("Haushaltstromzaehler - cnt")) FROM "myhome_measurements" WHERE time >= '2018-02-02T10:00:00Z' AND time < '2018-02-02T11:00:00Z' GROUP BY time(1h)
>
I'm now somewhat clueless as to how to make this work and appreciate any advice you might have.
Does it help using the last aggregate function? Not tested this as a cq yet.
Select difference(last(T1_Consumed)) AS T1_Delta, difference(last(T2_Consumed)) AS T2_Delta
from P1Data
where time >= 1551648871000000000 group by time(1h)
DIFFERENCE() would calculate delta from the "aggregated" value taken from previous group, not within current group.
So fill free to use selector function there - since your counters seemed to be cumulative, LAST() should be working well.
I’m trying to aggregate data over the previous week hourly, and compute summary statistics like median, mean, etc.
I’m using something like this:
var weekly_median = batch
|query('''SELECT median("duration") as week_median
FROM "db"."default"."profiling_metrics"''')
.period(1w)
.every(1h)
.groupBy(*)
.align()
|influxDBOut()
.database('default')
.measurement('summary_metrics')
The query works as expected, except that when recording and replaying data to test with
kapacitor record batch -task medians -past 30d
kapacitor replay -task medians -recording $rid -rec-time
the data is missing for the last period (1 week in this case). If I change the period to 1 day, all data is replayed except the final day’s worth.
Is this a problem with my tickscript, the way I’m recording data, or the way I’m replaying it?
I see, I need to do the aggregation in Kapacitor, not Influx. This seems to be a known issue, but finding documentation on it was tricky. https://github.com/influxdata/kapacitor/issues/1257 and https://github.com/influxdata/kapacitor/issues/1258 were helpful. The solution is to instead do something like:
var weekly_median = batch
|query('''SELECT "duration"
FROM "db"."default"."profiling_metrics"
WHERE "result" =~ /passed/''')
.period(1w)
.every(1h)
.groupBy(*)
.align()
|median('duration')
.as('week_median')
|influxDBOut()
.database('default')
.measurement('summary_metrics')
Is there a simple test to make sure I have proper influxdb communication?
My configuration looks like this
influxHost = influxhost:8086
smtpHost = mail:25
emailFrom = user#domain.com
template cpu {
body = `Alert definition:
Name: {{.Alert.Name}}
Crit: {{.Alert.Crit}}
Tags:{{range $k, $v := .Tags}}
{{$k}}: {{$v}}{{end}}
`
subject = cpu idle at {{.Alert.Vars.q | .E}} on {{.Tags.host}}
}
notification default {
email = user#domain.com
next = default
timeout = 1h
}
On the bosun expression evulator I am doing
influx("db",'''SELECT mean(usage_idle) FROM "cpu" group by host''',"10m","","2m")
I keep getting
influx: did not get a valid result from InfluxDB
Make sure you have the correct influx database and that there is data in the specified time range. I usually try from the admin site first:
Then insert the query into the influx(...) expression
Bosun will add the time conditions to the WHERE and GROUP BY clauses as needed, so the full influxql generated should be something like:
SELECT mean(usage_idle) FROM cpu WHERE time >= '2016-12-07 20:00:00' AND time <= '2016-12-07 20:10:00' GROUP BY host,time(2m)
If it still doesn't work try SELECT * FROM cpu on the admin page to see what data is in the table (telegraf has gone thru a few changes). Also note in the recent versions you probably want to add cpu = 'cpu-total' to the WHERE clause to get the overall average.
I have a non real time Esper configuration where I feed a stream that I read from a file. I'm trying to create an expression that computes a statistic over the entire stream and outputs one value at the very end. Esper has semantics for forcing a view to output every X seconds, for instance, but is there a semantic for asking the view or the engine to "flush" the output when you know there are no more events to feed.
Turns out that at least one way to do this is to use the output clause with a variable trigger.
The expression would be:
select count(*) as totalCount from events output last when OutputSummary = true
The OutputSummary variable would be initialized like so:
epConfiguration.addVariable("OutputSummary", Boolean.class, "false");
When you're ready to flush, set the variable to true like so:
epRuntime.setVariableValue("OutputSummary", true);
long currentTime = epService.getEPRuntime().getCurrentTime();
epRuntime.sendEvent(new CurrentTimeEvent(currentTime));
It's necessary to send another time event to force the expression to evaluate.
When output requires at every 60 sec then the expression would be:
select emplyee_id from employees output snapshot every 60 sec
and when the output requires at every 10000 events then the expression would be:
select emplyee_id from employees output snapshot every 10000 events