I'd like to calculate the delta values for a series of measurements stored in an InfluxDB. The values are readings from an electricity meter taken every 5 minutes. The values increase over time. Here is subset of the data to give you an idea (commands shown below are executed in the InfluxDB CLI):
> SELECT "Haushaltstromzaehler - cnt" FROM "myhome_measurements" WHERE time >= '2018-02-02T10:00:00Z' AND time < '2018-02-02T11:00:00Z'
name: myhome_measurements
time Haushaltstromzaehler - cnt
---- --------------------------
2018-02-02T10:00:12.610811904Z 11725.638
2018-02-02T10:05:11.242021888Z 11725.673
2018-02-02T10:10:10.689827072Z 11725.707
2018-02-02T10:15:12.143326976Z 11725.736
2018-02-02T10:20:10.753357056Z 11725.768
2018-02-02T10:25:11.18448512Z 11725.803
2018-02-02T10:30:12.922032896Z 11725.837
2018-02-02T10:35:10.618788096Z 11725.867
2018-02-02T10:40:11.820355072Z 11725.9
2018-02-02T10:45:11.634203904Z 11725.928
2018-02-02T10:50:11.10436096Z 11725.95
2018-02-02T10:55:10.753853952Z 11725.973
Calculating the differences in the InfluxDB CLI is pretty straightforward with the difference() function. This gives me the electricity consumed within the 5 minutes intervals:
> SELECT difference("Haushaltstromzaehler - cnt") FROM "myhome_measurements" WHERE time >= '2018-02-02T10:00:00Z' AND time < '2018-02-02T11:00:00Z'
name: myhome_measurements
time difference
---- ----------
2018-02-02T10:05:11.242021888Z 0.03499999999985448
2018-02-02T10:10:10.689827072Z 0.033999999999650754
2018-02-02T10:15:12.143326976Z 0.02900000000045111
2018-02-02T10:20:10.753357056Z 0.0319999999992433
2018-02-02T10:25:11.18448512Z 0.03499999999985448
2018-02-02T10:30:12.922032896Z 0.033999999999650754
2018-02-02T10:35:10.618788096Z 0.030000000000654836
2018-02-02T10:40:11.820355072Z 0.03299999999944703
2018-02-02T10:45:11.634203904Z 0.028000000000247383
2018-02-02T10:50:11.10436096Z 0.02200000000084401
2018-02-02T10:55:10.753853952Z 0.02299999999922875
Where I struggle is getting this to work in a continuous query. Here is the command I used to setup the continuous query:
CREATE CONTINUOUS QUERY cq_Haushaltstromzaehler_cnt ON myhomedb
BEGIN
SELECT difference(sum("Haushaltstromzaehler - cnt")) AS "delta" INTO "Haushaltstromzaehler_delta" FROM "myhome_measurements" GROUP BY time(1h)
END
Looking in the InfluxDB log file I see that no data is written in the new 'delta' measurement from the continuous query execution:
...finished continuous query cq_Haushaltstromzaehler_cnt, 0 points(s) written...
After much troubleshooting and experimenting I now understand why no data is generated. Setting up a continuous query requires to use the GROUP BY time() statement. This in turn requires to use an aggregate function within the differences() function. The problem now is that the aggregate function returns only one value for the time period specified by GROUP BY time(). Obviously, the differences() function cannot calculate a difference from just one value. Essentially, continuous query executes a command like this:
> SELECT difference(sum("Haushaltstromzaehler - cnt")) FROM "myhome_measurements" WHERE time >= '2018-02-02T10:00:00Z' AND time < '2018-02-02T11:00:00Z' GROUP BY time(1h)
>
I'm now somewhat clueless as to how to make this work and appreciate any advice you might have.
Does it help using the last aggregate function? Not tested this as a cq yet.
Select difference(last(T1_Consumed)) AS T1_Delta, difference(last(T2_Consumed)) AS T2_Delta
from P1Data
where time >= 1551648871000000000 group by time(1h)
DIFFERENCE() would calculate delta from the "aggregated" value taken from previous group, not within current group.
So fill free to use selector function there - since your counters seemed to be cumulative, LAST() should be working well.
Related
i have an issue with influxdb query.
when I query normally:
SELECT "distance" FROM "testdb"."autogen"."laserdistance" WHERE time > timeA AND time < timeB
I will get back all the data without error.
if I query with either mean() or median():
SELECT mean("distance") AS "mean_distance" FROM "testdb"."autogen"."laserdistance" WHERE time > timeA AND time < timeB GROUP BY time(some_interval) FILL(null)
I will sure get back some null value in a very inconsistent pattern(sometime a lot, sometime not so much).
I understand that FILL(null) is the one responsible for the null issue to fill in the interval that don't have value.
is there a way to get the nearest value instead of fill it with null?
Use FILL(linear):
linear - Reports the results of linear interpolation for time intervals with no data.
Doc: https://docs.influxdata.com/influxdb/v1.7/query_language/data_exploration/#group-by-time-intervals-and-fill
I’m trying to aggregate data over the previous week hourly, and compute summary statistics like median, mean, etc.
I’m using something like this:
var weekly_median = batch
|query('''SELECT median("duration") as week_median
FROM "db"."default"."profiling_metrics"''')
.period(1w)
.every(1h)
.groupBy(*)
.align()
|influxDBOut()
.database('default')
.measurement('summary_metrics')
The query works as expected, except that when recording and replaying data to test with
kapacitor record batch -task medians -past 30d
kapacitor replay -task medians -recording $rid -rec-time
the data is missing for the last period (1 week in this case). If I change the period to 1 day, all data is replayed except the final day’s worth.
Is this a problem with my tickscript, the way I’m recording data, or the way I’m replaying it?
I see, I need to do the aggregation in Kapacitor, not Influx. This seems to be a known issue, but finding documentation on it was tricky. https://github.com/influxdata/kapacitor/issues/1257 and https://github.com/influxdata/kapacitor/issues/1258 were helpful. The solution is to instead do something like:
var weekly_median = batch
|query('''SELECT "duration"
FROM "db"."default"."profiling_metrics"
WHERE "result" =~ /passed/''')
.period(1w)
.every(1h)
.groupBy(*)
.align()
|median('duration')
.as('week_median')
|influxDBOut()
.database('default')
.measurement('summary_metrics')
select SUM(value)
from /measurment1|measurment2/
where time > now() - 60m and host = 'hostname' limit 2;
Name: measurment1
time sum
---- ---
1505749307008583382 4680247
name: measurment2
time sum
---- ---
1505749307008583382 3004489
But is it possible to get value of SUM(measurment1+measurment2) , so that I see only o/p .
Not possible in influx query language. It does not support functions across measurements.
If this is something you require, you may be interested in layering another API on top of influx that do this, like Graphite via Influxgraph.
For the above, something like this.
/etc/graphite-api.yaml:
finders:
- influxgraph.InfluxDBFinder
influxdb:
db: <your database>
templates:
# Produces metric paths like 'measurement1.hostname.value'
- measurement.host.field*
Start the graphite-api/influxgraph webapp.
A query /render?from=-60min&target=sum(*.hostname.value) then produces the sum of value on tag host='hostname' for all measurements.
{measurement1,measurement2}.hostname.value can be used instead to limit it to specific measurements.
NB - Performance wise (of influx), best to have multiple values in the same measurement rather than same value field name in multiple measurements.
I'm trying to leverage my moderate SQL-knowledge for InfluxQL, but I'm missing something(s) about the nature of timeseries db.
Use case
I write a measurements from our issue tracker, when an issue is updated:
issue_updated,project=facebook,ticket=fb1,assignee=coolman status="todo"
Problem
Given this returns rows of issues statuses:
SELECT status
FROM "issue_updated"
If this was SQL (fiddle) I would use COUNT(and then add the WHERE time > NOW() - 1Y GROUP BY time(5m)). However the following gives me Mixing aggregate and non-aggregate queries is not supported
SELECT status, count(status) as 'Count'
FROM "issue_updated"
Can someone give some guidance here? ta
Sounds like what you're looking for is the ability to group by a field value which isn't currently supported.
From what I can tell, if you modify your schema a bit, it should be possible to do what you're looking. Instead of
issue_updated,project=facebook,ticket=fb1,assignee=coolman status="todo"
Do
issue_updated,project=facebook,ticket=fb1,assignee=coolman,status=todo value=1
then
SELECT count(value) FROM "issue_updated" WHERE time > now() - 52w GROUP BY status
name: issue_updated
tags: status=other
time count
---- -----
1449523659065350722 1
name: issue_updated
tags: status=todo
time count
---- -----
1449523659065350722 2
should work.
I have a series, disk, that contains a path (/mnt/disk1, /mnt/disk2, etc) and total space of a disk. It also includes free and used values. These values are updated at a specified interval. What I would like to do, is query to get the sum of the total of the last() of each path. I would also like to do the same for free and for used, to get a aggregate of the total size, free space, and used space of all of my disks on my server.
I have a query here that will get me the last(total) of all the disks, grouped by its path (for distinction):
select last(total) as total from disk where path =~ /(mnt\/disk).*/ group by path
Currently, this returns 5 series, each containing 1 row (the latest) and the value of its total. I then want to take the sum of those series, but I cannot just wrap the last(total) into a sum() function call. Is there a way to do this that I am missing?
Carrying on from my comment above about nested functions.
Building a toy example:
CREATE DATABASE FOO
USE FOO
Assuming your data is updated at intervals greater than[1] every minute:
CREATE CONTINUOUS QUERY disk_sum_total ON FOO
BEGIN
SELECT sum("total") AS "total_1m" INTO disk_1m_total FROM "disk"
GROUP BY time(1m)
END
Then push some values in:
INSERT disk,path="/mnt/disk1" total=30
INSERT disk,path="/mnt/disk2" total=32
INSERT disk,path="/mnt/disk3" total=33
And wait more than a minute. Then:
INSERT disk,path="/mnt/disk1" total=41
INSERT disk,path="/mnt/disk2" total=42
INSERT disk,path="/mnt/disk3" total=43
And wait a minute+ again. Then:
SELECT * FROM disk_1m_total
name: disk_1m_total
-------------------
time total_1m
1476015300000000000 95
1476015420000000000 126
The two values are 30+32+33=95 and 41+42+43=126.
From there, it's trivial to query:
SELECT last(total_1m) FROM disk_1m_total
name: disk_1m_total
-------------------
time last
1476015420000000000 126
Hope that helps.
[1] Picking intervals smaller than the update frequency prevents minor timing jitters from making all the data being accidentally summed twice for a given group. There might be some "zero update" intervals, but no "double counting" intervals. I typically run the query twice as fast as the updates. If the CQ sees no data for a window, there will be no CQ performed for that window, so last() will still give the correct answer. For example, I left the CQ running overnight and pushed no new data in: last(total_1m) gives the same answer, not zero for "no new data".