I need to develop an InfluxDB Time Series. The time series needs to contain the following information:
time - The time the data was recorded. This will use the InfluxDB Time Field.
value - The value for the time series. A simple integer.
date - A separate date associated with the value. This date has nothing to do with the "time" field. It will be used to help narrow down queries.
My current line of thinking is to save the above "date" field as a separate "column" in the time series so that I can use the "where" clause to filter the data using that date. However I am struggling with how to represent that. Does InfluxDB support any kind of date or date/time fields? For the "time" field it seems to just use milliseconds. However if I try the same in a field with a different name, then the normal time queries don't work. So for example:
select * from myseries where time > now() - 1d
The above query will work just fine.
vs
select * from myseries where date > now() - 1d
This query will fail with an error because it doesn't seem to know how to treat "date" as a time value.
Is there a better representation for dates in this scenario?
InfluxDB data types can be only be one of: floats, ints, bools, or strings. The time field is a special exception.
You can use integers representing count-since-epoch for the date field. Nice convenient functions like now() don't seem work for that though (using v0.13):
insert test_dates date_int=1573405622000000000i,desc="years from now"
insert test_dates date_int=1373405661000000000i,desc="years ago"
Testing that with now()
select * from test_dates where date_int > now()
gives:
name: test_dates
time date_int desc
1473404302801938064 1573405622000000000 years from now
1473404315927493772 1373405661000000000 years ago
And:
select * from test_dates where date_int < now()
gives:
name: test_dates
time date_int desc
1473462286404084162 1573405622000000000 years from now
1473462286408231540 1373405661000000000 years ago
Seems every date_int is somehow both greater than and less than now()
So the comparison isn't a syntax error if you use integers, but doesn't work the way we'd like.
One way to solve this is to create your own date-to-int conversion in the front-end app. Then a date comparison on the front-end is an int comparison within InfluxDB. Clunky, but those are the data types we have.
The date as stored in InfluxDB could be a single epoch-based int, or store separate int fields for year, month, day within InfluxDB. The queries are bigger and slower in the latter case, but it's easier to read and debug.
Good luck!
Related
I am building a rails app, where the user picks up a date from a date picker and a time from the time picker. Both the date and time have been formatted using moment js to show the date and time in the following way:
moment().format('LL'); //January 23,2017
moment().format('LTS'); //1:17:54 PM
I read this answer with guidelines about selection of a proper column type.
Is there documentation for the Rails column types?
Ideally, I should be using :date, :time or :timestamp for this. But since the dates are formatted, should I be using :string instead?
Which would be the correct and appropriate column type to use in this situation?
If you want to store a time reference in your database you should use one of the types the database offers you. I'll explain this using MySQL (which is the one I have used the most) but the explanation should be similar in other database servers.
If you use a timestamp column you will be using just 4 bytes of storage, which is always a good new since it makes smaller indexes, uses less memory in temporal tables during the internal database operations and so on. However, timestamp has a smaller range than datetime so you will only be able to store values from year 1970 up to year 2038 more or less
If you use datetime you will be able to store a wider range (from year 1001 to year 9999) with the same precision (second). The bad consequence is that a higher range needs more memory, making it a bit slower.
There are some other differences between these two column types that don't fit in this answer, but you should keep an eye on before deciding.
If you use varchar, which is the default column type for text attributes in Ruby on Rails, you will be forced to convert from text to datetime and vice-versa every time you need to use that field. In addition, ordering or filtering on that column will be very inefficient because the database will need to convert all strings into dates before filtering or sorting, making it impossible to use indexes on that column.
If you need sub-second precision, you can use bigint to meet your requirements, as MySQL does not provide a date specific type for this purpose
In general, I recommend using timestamp if your application requirements fit the timestamp limitation. Otherwise, use datetime, but I strongly discourage you to use varchar for this purpose.
EDIT: Formatting
The way you store dates in database is completely different from the way you display it to the user. You can create a DateTime object using DateTime.new(year, month, day, hour, minute, second) and assign that object to your model. By the time you save it into database, ActiveRecord will be in charge of converting the DateTime object into the appropiate database format.
In order to display a value that is already stored in database in a specific format (in a view, API response, etc.) you can hava a look at other posts like this one.
You can have a timestamp column in your database, and then parse the request to a ruby datetime object like this:
d = Time.parse(params[:date])
t = Time.new(params[:time])
dt = DateTime.new(d.year, d.month, d.day, t.hour, t.min, t.sec, t.zone)
#now simply use dt to your datetime column
On Postgres you can save a ruby DateTime object straight into a postgres timestamp field, e.g
User.first.update_attribute('updated_at', dt )
Another option is to concatenate your date and time strings into one and then u can do a one-liner:
User.last.update_attribute('created_at', Time.parse('January 23,2017 1:17:54 PM'))
I'm pretty sure this will work on MySQL datetime or timestamp as well.
Credit to david grayson Ruby: combine Date and Time objects into a DateTime
Is it possible to write a InfluxDB query that will give me the number of milliseconds since the last entry in a time series? I'd like to add a single-stat panel in Grafana displaying how old the data is.
I don't think it is possible since you are not able to query the time alone. A influxdb query needs at least one non-time field in a query. You could workaround that by double saving the time in a extra field which you are able to query alone.
But you still want to use now() - "the extra time field". But as far as I found out you also can't use now() inside grafana.
Update: there is a [Feature-Request] now on grafanas github. Make sure to vote it up so it gets implemented one day: https://github.com/grafana/grafana/issues/6710
Update 2: The feature got finaly implemented -> See my answer here: How to show "33 minutes ago" on Grafana dashboard with InfluxDB?
Background
I have a ror application which is continuously recording and showing on a web site real time sensor data. Then I have a table called sensors which has unique list of all sensors and stores latest values.
I also have another table histories which dumps all the sensor values ever received for each sensor.
So the relation is "sensor has many histories" , the time_stamp col records the creation time stamp.
Not all sensors update at same interval or frequency.
Problem
Now I want to take a input time stamp from user, a date and time in past, and show what the sensors were showing at that time. For example say i want to see what all sensors looked like at 2 PM yesterday, once i have this time stamp from user, how do i retrieve one sensors value closest to input time stamp from the history table.
I am looking to add a method in Sensor model, which will take time_stamp as argument, and retrive the value closest to input time_stamp from the history table.
What is they simplest way to write this Active record query?
Thanks
Shaunak
Just sort the histories according to the difference between the passed timestamp and the history timestamp (absolute value so it can go in either direction), and return the top result (that's the one with the smallest difference).
sensor.histories.order("ABS(time_stamp - #{params[:time_stamp].to_i})").first
Note that for this query I am assuming you are using MySQL (because I'm using a MySQL method ABS) and I am also assuming that the time_stamp field is stored as unix timestamp and the user input likewise. If the database storage or input is in a different format, you'll have to use different date arithmetic functions. See http://dev.mysql.com/doc/refman/5.5/en/date-and-time-functions.html for details. Or if you are not using MySQL, see the docs for the database you are using.
Also note that I am using .to_i to sanitize my data for the query. If your data is in a different format, you may need to sanitize it a different way.
To make this more efficient, limit it to time_spans within the maximum possible range. If sensors take data every 10 minutes or more frequently (never less than 10 minutes apart between readings), then a range of greater than 10 minutes on each side will do. Something like below. Here, 600 = 10 (minutes) * 60 (seconds):
sensor.histories.where("time_stamp >= ? AND time_stamp <= ?", params[:time_stamp].to_i - 600, params[:time_stamp].to_i + 600).order("ABS(time_stamp - #{params[:time_stamp].to_i})").first
It is simple to convert this to a model method:
class Sensor < ActiveRecord::Base
def history_at(time_stamp)
self.histories.where("time_stamp >= ? AND time_stamp <= ?", time_stamp.to_i - 600, time_stamp.to_i + 600).order("ABS(time_stamp - #{time_stamp.to_i})").first
end
end
I want to store the date of an event in my database, but I want to do so without storing informations about seconds or anything smaller than seconds. Using Rails, in my migration I have the option to create a date column or a datetime column, the first one of which is too less accurate, and the second one is too much (up to the second and less). Which type should I choose to store such a date? Currently I'm using datetime and setting the seconds to a fixed value (e.g. 0) manually each time some date is set in the model.
Something like this:
self.date ||= Time.now.change(:sec => 0)
Am I totally out of track? Should I just use an integer field for each component of the date instead? (year, month, day, etc...) Or is datetime the correct type but I'm not understanding the purpose of it? (I think it's meant for timestamps and such things where seconds matter)
datetime is the correct type. And be sure to store it without time zone at time zone UTC:
http://derickrethans.nl/storing-date-time-in-database.html
At your option, use an SQL trigger to round your date to the minute on insert/update. It'll simplify your ruby code.
I want to let users specify a date that may or may not include a day and month (but will have at least the year.) The problem is when it is stored as a datetime in the DB; the missing day/month will be saved as default values and I'll lose the original format and meaning of the date.
My idea was to store the real format in a column as a string in addition to the datetime column. Then I could use the string column whenever I have to display the date and the datetime for everything else. The downside is an extra column for every date column in the table I want to display, and printing localized dates won't be as easy since I can't rely on the datetime value... I'll probably have to parse the string.
I'm hoping I've overlooked something and there might be an easier way.
(Note I'm using Rails if it matters for a solution.)
As proposed by Jhenzie, create a bitmask to show which parts of the date have been specified. 1 = Year, 2 = Month, 4 = Day, 8 = Hour (if you decide to get more specific) and then store that into another field.
The only way that I could think of doing it without requiring extra columns in your table would be to use jhenzie's method of using a bitmask, and then store that bitmask into the seconds part of your datetime column.
in your model only pay attention to the parts you care about. So you can store the entire date in your db, but you coalesce it before displaying it to the user.
The additional column could simple be used for specifying what part of the date time has been specified
1 = day
2 = month
4 = year
so 3 is day and month, 6 is month and year, 7 is all three. its a simple int at that point
If you store a string, don't partially reinvent ISO 8601 standard which covers the case you describe and more:
http://en.wikipedia.org/wiki/ISO_8601
Is it really necessary to store it as a datetime at all ? If not stored it as a string 2008 or 2008-8 or 2008-8-1 - split the string on hyphens when you pull it out and you're able to establish how specific the original input was
I'd probably store the datetime and an additional "precision" column to determine how to output it. For output, the precision column can map to a column that contains the corresponding formatting string ("YYYY-mm", etc) or it can contain the formatting string itself.
I don't know a lot about DB design, but I think a clean way to do it would be with boolean columns indicating if the user has input month and day (one column for each). Then, to save the given date, you would:
Store the date that the user input in a datetime column;
Set the boolean month column if the user has picked a month;
Set the boolean day column if the user has picked a day.
This way you know which parts of the datetime you can trust (i.e. what was input by the user).
Edit: it also would be much easier to understand than having an int field with cryptic values!
The informix database has this facility. When you define a date field you also specify a mask of the desired time & date attributes. Only these fields count when doing comparisons.
With varying levels of specificity, your best bet is to store them as simple nullable ints. Year, Month, Day. You can encapsulate the display logic in your presentation model or a Value Object in your domain.
Built-in time types represent an instant in time. You can use the built in types and create a column for precision (Year, Month, Day, Hour, Etc.) or you can create your own date structure and use nulls (or another invalid value) for empty portions.
For ruby at least - you could use this gem - partial-date
https://github.com/58bits/partial-date