We are looking into using Grafana to use as a dashboard to display some graphs and whatnot. But apparently all data in Grafana is with a timestamp and our data runs at night and we give it all the same timestamp.
In our mysql db it's like this:
2015-11-04 2015-11-05 xxx yyyy zzz
2015-11-04 2015-11-05 aaa bbb ccc
...
So can we use grafana for this? We don't need to be able to see the difference during the day, just total numbers. We don't want to overwrite our records however with eachother.
Thanks.
Related
Is it possible to write a InfluxDB query that will give me the number of milliseconds since the last entry in a time series? I'd like to add a single-stat panel in Grafana displaying how old the data is.
I don't think it is possible since you are not able to query the time alone. A influxdb query needs at least one non-time field in a query. You could workaround that by double saving the time in a extra field which you are able to query alone.
But you still want to use now() - "the extra time field". But as far as I found out you also can't use now() inside grafana.
Update: there is a [Feature-Request] now on grafanas github. Make sure to vote it up so it gets implemented one day: https://github.com/grafana/grafana/issues/6710
Update 2: The feature got finaly implemented -> See my answer here: How to show "33 minutes ago" on Grafana dashboard with InfluxDB?
I am having a hard time to figure out how to deal with the following problem:
Our company is publishing posts to social media platforms. Those posts are stored within the database once they where successfully postet.
We want to provide a dashboard showing an overview of how many posts the user published over a time period grouped by minutes, hours and days.
I want to display the results as a time series graph.
This would work fine, but it gets very tricky once I have to support multiple time zones when I do aggregation/grouping by days. (apparently posts around midnight belong to different days depending on which time zone you are)
My current solution builds the postgres query using rails ActiveRecord. The problem I am facing is that I am struggling to deal with the timezone conversions...
Also I am not particular good at postgres...
The current implementation essentially looks like this (I removed irrelevant code):
Publication.select(
%{date_trunc('#{interval}',
published_at::timestamptz at time zone interval '#{time_zone_offset}')::timestamptz as time,
count(published_at)})
.where(%(published_at BETWEEN
timestamptz '#{start_date}' AND
timestamptz '#{end_date}'))
.group("1")
.order('time').limit(LIMIT)
For example:
I have one publication at 2016-03-15 10:19:24.219258 (Thats how it is stored inside the database therefore UTC time)
I create the following query:
SELECT date_trunc('hour',
published_at::timestamptz at time zone interval '+01:00')::timestamptz as time,
count(published_at) FROM "publications" WHERE (published_at BETWEEN
timestamptz '2016-03-15 10:00:00 +0100' AND
timestamptz '2016-03-15 12:00:00 +0100') GROUP BY 1
;
Which results in:
time | count
------------------------+-------
2016-03-15 10:00:00+01 | 1
(1 row)
Which should be:
time: "2016-03-15 10:00:00 UTC" or "2016-03-15 11:00:00+01" ( i don't care about the time zone representation but this is simply the wrong result)
Anybody knows what I am doing wrong here?
The main problem I got stuck is that I want to be able to group/aggregate publications per day, with respect to the time zone of the user requesting the query.
I don't care which time zone is returned as the front end can transform it to the user time zone.
Any feedback, help, or answer is highly appreciated.
Many thanks
Thanks to the discussion I had with devanand one solution is to split up the code and handle the daily interval with the query used in the question.
For the other intervals I use the following query:
Publication.select(
%{date_trunc('#{interval}',
published_at::timestamptz) as time,
count(published_at)})
.where(%(published_at BETWEEN
timestamptz '#{start_date}' AND
timestamptz '#{end_date}'))
.group('1')
.order('time').limit(LIMIT)
I am not happy with the solution though as it feels more like a workaround to me
I have some eventlog data in HDFS that, in its raw format, looks like this:
2015-11-05 19:36:25.764 INFO [...etc...]
An external table points to this HDFS location:
CREATE EXTERNAL TABLE `log_stage`(
`event_time` timestamp,
[...])
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\t'
LINES TERMINATED BY '\n'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
For performance, we'd like to query this in Impala. The log_stage data is inserted into a Hive/Impala Parquet-backed table by executing a Hive query: INSERT INTO TABLE log SELECT * FROM log_stage. Here's the DDL for the Parquet table:
CREATE TABLE `log`(
`event_time` timestamp,
[...])
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
The problem: when queried in Impala, the timestamps are 7 hours ahead:
Hive time: 2015-11-05 19:36:25.764
Impala time: 2015-11-06 02:36:25.764
> as.POSIXct("2015-11-06 02:36:25") - as.POSIXct("2015-11-05 19:36:25")
Time difference of 7 hours
Note: The timezone of the servers (from /etc/sysconfig/clock) are all set to "America/Denver", which is currently 7 hours behind UTC.
It seems that Impala is taking events that are already in UTC, incorrectly assuming they're in America/Denver time, and adding another 7 hours.
Do you know how to sync the times so that the Impala table matches the Hive table?
Hive writes timestamps to Parquet differently. You can use the impalad flag -convert_legacy_hive_parquet_utc_timestamps to tell Impala to do the conversion on read. See the TIMESTAMP documentation for more details.
This blog post has a brief description of the issue:
When Hive stores a timestamp value into Parquet format, it converts local time into UTC time, and when it reads data out, it converts back to local time. Impala, however on the other hand, does no conversion when reads the timestamp field out, hence, UTC time is returned instead of local time.
The impalad flag tells Impala to do the conversion when reading timestamps in Parquet produced by Hive. It does incur some small cost, so you should consider writing your timestamps with Impala if that is an issue for you (though it likely is minimal).
On a related note, as of Hive v1.2, you can also disable the timezone conversion behaviour with this flag:
hive.parquet.timestamp.skip.conversion
"Current Hive implementation of parquet stores timestamps to UTC, this flag allows skipping of the conversion on reading parquet files from other tools."
This was added in as part of https://issues.apache.org/jira/browse/HIVE-9482
Lastly, not timezone exactly, but for compatibility of Spark (v1.3 and up) and Impala on Parquet files, there's this flag:
spark.sql.parquet.int96AsTimestamp
https://spark.apache.org/docs/1.3.1/sql-programming-guide.html#configuration
Other: https://issues.apache.org/jira/browse/SPARK-12297
be VERY careful with the answers above due to https://issues.apache.org/jira/browse/IMPALA-2716
As for now, the best workaround is not to use TIMESTAMP data type and store timestamps as strings.
As mentioned in
https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_timestamp.html
You can use ----use_local_tz_for_unix_timestamp_conversions=true and --convert_legacy_hive_parquet_utc_timestamps=true to match Hive results.
The first one ensures it converts to local timezone when you use any datetime function. You can set it as Impala Daemon startup options as mentioned in this document.
https://docs.cloudera.com/documentation/enterprise/5-6-x/topics/impala_config_options.html
I want to answer the question: what time window my servers are least used?
My idea is to have a 24h histogram (with 10min buckets) with the count of requests made to my server. In other words, I want to ignore date portion in datetime aggregations.
How do I create such graph in Kibana 4?
Right, this is a bit confusing for me, so I'm going to try and explain from the top!
I have a rails web app. It's an internal company app and will only be used in the UK.
One of the things the app does is manage meetings.
Meetings have a date & time when they start. There's a date/time picker on the form which allows the user to pick the date & time the meeting is for. I save this date AS IS into the database. All meetings last 2 hours, so the end time is simply start + 2 hours.
Example:
2013-06-23 6:45PM in the form is stored in the db as 2013-06-23 18:45:00
2013-12-23 6.45pm in the form is stored in the db as 2013-12-23 18:45:00
Note that the first date is during Daylight Savings (BST) and the second is during GMT. I don't actually care whether it is GMT or BST: the meeting happens at that time, absolutely.
Inside the rails webapp, I simply print out the exact date & time from the DB - formatted nicely, of course!
Now, at some point I send an email to the organiser of the meeting, and the person they're meeting with. This email tells them the the date & time of the meeting etc, and also includes an iCal (.ics) file for them to put into their (Outlook usually, but also Apple or gmail) calendar.
The issue I am having is that (using the above examples) Outlook shows the meetings like this:
Meeting #1: Start: 23/06/2013 7:45pm, End: 23/06/2013 9:45pm
Meeting #2: Start: 23/12/2013 6:45pm, End: 23/12/2013 8:45pm
Note that it has adjusted the first one because of the BST/GMT thing.
The text of the .ics file contains this code:
Meeting #1:
BEGIN:VCALENDAR
...
DTEND:20130623T204500Z
DTSTART:20130623T184500Z
...
END:VCALENDAR
Meeting #2:
BEGIN:VCALENDAR
...
DTEND:20131223T204500Z
DTSTART:20131223T184500Z
...
END:VCALENDAR
So I am encoding the dates/times using the Z timezone (UTC). I understand this is why Outlook mis converting the UTC time into the BST time for #1 and leaving #2 alone (because GMT == UTC)
My question is: how do I stop this happening? I want the time the meeting is scheduled for to be the absolute, actual time, regardless of GMT/BST: 6:45pm
Should I be storing the date-times as UTC in the DB? How would this be done (I assume it would apply to all dates, not just meeting start dates). And how to re-convert them back into the actual datetime when I display them in the webapp?
Extra:
I have an entry in my initializers/time_formats.rb like this:
:ical => "%Y%m%dT%H%M00Z"
So dates come out like "20130623T184500Z". I use this when building the ics. And this I think is the issue - if the date/time is during BST I don't want to be using Z, but something else?
Your problem is your date/time format. You have:
DTSTART:20130623T184500Z
in your .ics file and this corresponds to 19:45 BST (as British summer time is UTC+1).
There are a few things you should do. First, you can simply remove the 'Z' from the end of your dates. This means that the times inherit the timezone of the calendar, or the underlying application.
This will work assuming that the machines which are running Outlook are all in the Europe/London timezone. If not, or if you want to be a bit safer, you should also specify the following after your BEGIN: VCALENDAR line:
X-WR-TIMEZONE:Europe/London
This specifies the default timezone for all dates which are not specified explicitly.
Finally, if this does not work for any reason then you need to define your datetimes explicitly. First you need to add a timezone definition for Europe/London to the calendar. The info you need is available at http://www.tzurl.org/zoneinfo-outlook/Europe/London.ics. Then you need to ensure that all datetimes are of the format:
DTSTART;TZID=Europe/London:20130623T184500
This last approach is the best, as it means that if your requirements expand to other timezones you will be able to handle them relatively easily.
Sorry to answer this myself, but in case anyone else runs into this here's what I found was the cause of my particular issue. Note that the answer above re timezones also makes sense!
My rails app is storing UTC datetimes in the DB (as is default)
But, it also thought it's own timezone was UTC, which also seems to be the default.
The upshot of that is essentially it was storing local dates, local to UTC anyway. Changing the app to know it was sitting in Europe/London made it so the dates in the DB are all now accurately UTC (meaning, they're an hour off if I'm currently in BST)
I can now use the Z datetime format in iCals, and outlook and the rails app both convert the UTC date back into the actual datetime for the viewing-user's locale (Europe/London for everyone at the moment). This is what I wanted.