I am having a hard time to figure out how to deal with the following problem:
Our company is publishing posts to social media platforms. Those posts are stored within the database once they where successfully postet.
We want to provide a dashboard showing an overview of how many posts the user published over a time period grouped by minutes, hours and days.
I want to display the results as a time series graph.
This would work fine, but it gets very tricky once I have to support multiple time zones when I do aggregation/grouping by days. (apparently posts around midnight belong to different days depending on which time zone you are)
My current solution builds the postgres query using rails ActiveRecord. The problem I am facing is that I am struggling to deal with the timezone conversions...
Also I am not particular good at postgres...
The current implementation essentially looks like this (I removed irrelevant code):
Publication.select(
%{date_trunc('#{interval}',
published_at::timestamptz at time zone interval '#{time_zone_offset}')::timestamptz as time,
count(published_at)})
.where(%(published_at BETWEEN
timestamptz '#{start_date}' AND
timestamptz '#{end_date}'))
.group("1")
.order('time').limit(LIMIT)
For example:
I have one publication at 2016-03-15 10:19:24.219258 (Thats how it is stored inside the database therefore UTC time)
I create the following query:
SELECT date_trunc('hour',
published_at::timestamptz at time zone interval '+01:00')::timestamptz as time,
count(published_at) FROM "publications" WHERE (published_at BETWEEN
timestamptz '2016-03-15 10:00:00 +0100' AND
timestamptz '2016-03-15 12:00:00 +0100') GROUP BY 1
;
Which results in:
time | count
------------------------+-------
2016-03-15 10:00:00+01 | 1
(1 row)
Which should be:
time: "2016-03-15 10:00:00 UTC" or "2016-03-15 11:00:00+01" ( i don't care about the time zone representation but this is simply the wrong result)
Anybody knows what I am doing wrong here?
The main problem I got stuck is that I want to be able to group/aggregate publications per day, with respect to the time zone of the user requesting the query.
I don't care which time zone is returned as the front end can transform it to the user time zone.
Any feedback, help, or answer is highly appreciated.
Many thanks
Thanks to the discussion I had with devanand one solution is to split up the code and handle the daily interval with the query used in the question.
For the other intervals I use the following query:
Publication.select(
%{date_trunc('#{interval}',
published_at::timestamptz) as time,
count(published_at)})
.where(%(published_at BETWEEN
timestamptz '#{start_date}' AND
timestamptz '#{end_date}'))
.group('1')
.order('time').limit(LIMIT)
I am not happy with the solution though as it feels more like a workaround to me
Related
I'm having a trouble when develop multiple timezones website.
Currently I'm storing time in UTC after some researches and it is working fine in most cases.
But there is one case that I couldn't find solution for it:
There are two kinds of user in two countries which are United States
and Thailand.
User in Thailand is worker (A).
User in US is manager (B).
When A starts working, their activities logged into our system and B
can watch those via a monitoring screen on web app and they can choose
the date on that.
Example user A starts working at 8 AM on 23 June with mobile
app, when B chooses 23 June date on the monitoring screen, they
can see the activities of user on Thailand on 23 June (because the results is queried by UTC time), but the
problem is he should see the activities on 22 June instead of 23
June because the time in Thailand is faster than United States 12
hours.
How can I show to user B activities of user A when he chooses the date 22 June?
You've not asked about any particular technology stack or implementation, so I can only answer from a general perspective.
Concepts worth understanding:
Thailand has a single time zone, which has an offset of UTC+7 all year.
The US has multiple time zones, whose offsets range from UTC-10 to UTC-4, depending on what part of the country you are referring to, whether or not daylight saving time is in effect, and whether or not a particular location observes daylight saving time. (Most of the country does, but all of Hawaii and much of Arizona does not.)
A "date" is just a year, month, and day on a calendar, but the time that which a date is observed is different depending on the time zone of the observer. There is a good visualization of this at everytimezone.com.
In your situation, you will have to decide the behavior you want depending on the specific needs of your application:
Do you want the period shown to represent all activities on the date as observed by the person choosing the date? If so, then determine the start of the current date and the start of the next date in the local time zone of the person selecting the date. Convert those to UTC, and query for all events in that UTC time range.
Example:
Example Activity Time: 2018-06-23T18:00:00+07:00 (Asia/Bangkok)
Stored as UTC: 2018-06-23T11:00:00Z
Date Selected: 2018-06-23 (America/New_York)
Local Range: [2018-06-23T00:00:00-04:00 , 2018-06-24T00:00:00-04:00 )
UTC Range: [2018-06-23T04:00:00Z , 2018-06-24T04:00:00Z )
Query: ... where ActivityUTC >= '2018-06-23 04:00:00' and ActivityUTC < '2018-06-24 04:00:00'
Or, do you want the date selected to always represent the date of the activity in the time zone of the person who recorded that activity, regardless of the time zone of the viewer? If so, then store that local date in a separate date-only column and just query on it without regard to time zone.
Example:
Example Activity Time: 2018-06-23T18:00:00+07:00 (Asia/Bangkok)
Local Date Stored: 2018-06-23
Date Selected: 2018-06-23
Query: ... where ActivityLocalDate = '2018-06-23'
Note, you might still store the UTC date and time in some other field, but it isn't relevant for this particular query.
From prior experience in the time and attendance industry, I can say that if it were me I would want the second option - as workers are typically paid based on their own time zones, not on those of their manager. However their are indeed edge cases and you'll have to decide for yourself which approach best matches your business requirements.
This Answer is specific to MySQL.
If you want B to see what A's clock says, use DATETIME; it will say 8AM.
If you want B to see A logging in in the middle of the night, use TIMESTAMP.
(This extends to A vs B, and to date as well as clock.)
Twice a year, DATETIME has a hiccup between 2AM and 3AM if there is a switch between standard and daylight-savings time.
I know that I can get browser timezone name: moment.tz.guess()
How can I guess timezone for custom date? e.g. moment("2020-12-30T14:17:40+11:00").tz.guess() ?
You can't. There are 9 different location-based zones that use +11:00 year round, and 5 more that use it for daylight saving time.
You might be able to eliminate a few if the date and time put it into a time zone where daylight time is not in effect and the standard time doesn't match, but from there, the best you could do would be to pick one at random.
See "time zone != offset" in the timezone tag wiki.
Moment-timezone can guess the browser's time zone because it can query the offset for multiple different timestamps, algorithmically reducing the possibilities. Even then, there are often several that could match, and thus the most likely one is picked (based on population statistics). In the end, it's still a guess.
I'm looking to display data across more than one time zone in the same view in a Rails app for a time and attendance system. A bit of context:
We make electronic time clocks. People but them in their businesses. Staff clock in and out of work and it records their hours.
The time clock pushes the time that someone clocked in/out to our API as a unix time (for example, our Javascript time clock implementation grabs the clock in time like so: moment().unix()). The API then stores this in a Postgres database as a timestamp without time zone.
When a user logs in to the site, an around_filter sets the appropriate time zone for the request based on a setting for this user's organisation.
The problem occurs if we have an organisation that spans multiple time zones. For example, a business that has an office in every Australian capital city will span three time zones (more during DST). However, there will be one person in a central office who will need to check data across the organisation - we'll call them our manager.
Suppose our manager is based in Sydney, and it's 11am. They manage three offices - one in Sydney, one in Brisbane (an hour behind Sydney during DST), and one in Adelaide (half an hour behind Sydney during DST). Staff clocked in at the three offices at 9am in their local times. So, on the manager's dashboard, all the times of the clock-ins should show up as 9am. However, the current implementation (using an around_filter) will show the times as 9am, 8am, and 8:30am, respectively, because they will be offset using the Sydney time zone.
There is a layer of filtering applied to staff from different cities, so it is possible to tell the system that person A is from Sydney, person B is from Adelaide, and person C is from Brisbane. The issue - which I'd like advice on - is how best to get Rails to display offset to different time zones as efficiently as possible.
Bonus credit: as well as showing times, we also need to read input. For example, someone may have clocked in 5 minutes early, and their timesheet needs to be corrected. If a local manager (ie. someone in Brisbane) corrects the timesheet for a Brisbane employee then that should be relatively easy to manage - given we know they are in Brisbane, we can just set the request's time zone to Brisbane and let ActiveRecord do the offsetting for us. But if the general manager (who is based in Sydney but manages all time zones) wants to make the change, then we need to be able to correctly convert their input back into UTC based on their time zone. Any suggestions on how best to do this would be wonderful.
Concrete example of the issue
In my database, my clock_ins table looks like this:
user_id (integer) | time (timestamp without time zone)
------------------|-----------------------------------
1 | "2012-09-25 22:00:00.0"
2 | "2012-09-25 22:30:00.0"
3 | "2012-09-25 23:00:00.0"
And my users table looks like this:
user_id (integer) | time_zone (varchar)
------------------|-----------------------------------
1 | "Sydney"
2 | "Adelaide"
3 | "Brisbane"
(this is a simplification, in reality there is another join between a user and their time zone)
If we apply each user's time zone to the time of their clock in, we find they are all at 9am local time. ie. 2012-09-25 23:00:00.0 at UTC is 2012-09-26 09:00:00.0 in Brisbane (+1000). The general approach in Rails is to use an around_filter to set the time zone for a request; if I did that here, each of the times would be displayed half an hour apart, which is not correct. So I'm looking on advice on best practices when working with times from various zones.
The simplest way I can see is to use the Time.use_zone method when rendering your times. e.g.
Time.use_zone('Sydney') { Time.current }
Time.use_zone(person.office.time_zone) { person.clock_ins.last.time_stamp }
This "Allows override of Time.zone locally inside supplied block; resets Time.zone to existing value when done."
Right, this is a bit confusing for me, so I'm going to try and explain from the top!
I have a rails web app. It's an internal company app and will only be used in the UK.
One of the things the app does is manage meetings.
Meetings have a date & time when they start. There's a date/time picker on the form which allows the user to pick the date & time the meeting is for. I save this date AS IS into the database. All meetings last 2 hours, so the end time is simply start + 2 hours.
Example:
2013-06-23 6:45PM in the form is stored in the db as 2013-06-23 18:45:00
2013-12-23 6.45pm in the form is stored in the db as 2013-12-23 18:45:00
Note that the first date is during Daylight Savings (BST) and the second is during GMT. I don't actually care whether it is GMT or BST: the meeting happens at that time, absolutely.
Inside the rails webapp, I simply print out the exact date & time from the DB - formatted nicely, of course!
Now, at some point I send an email to the organiser of the meeting, and the person they're meeting with. This email tells them the the date & time of the meeting etc, and also includes an iCal (.ics) file for them to put into their (Outlook usually, but also Apple or gmail) calendar.
The issue I am having is that (using the above examples) Outlook shows the meetings like this:
Meeting #1: Start: 23/06/2013 7:45pm, End: 23/06/2013 9:45pm
Meeting #2: Start: 23/12/2013 6:45pm, End: 23/12/2013 8:45pm
Note that it has adjusted the first one because of the BST/GMT thing.
The text of the .ics file contains this code:
Meeting #1:
BEGIN:VCALENDAR
...
DTEND:20130623T204500Z
DTSTART:20130623T184500Z
...
END:VCALENDAR
Meeting #2:
BEGIN:VCALENDAR
...
DTEND:20131223T204500Z
DTSTART:20131223T184500Z
...
END:VCALENDAR
So I am encoding the dates/times using the Z timezone (UTC). I understand this is why Outlook mis converting the UTC time into the BST time for #1 and leaving #2 alone (because GMT == UTC)
My question is: how do I stop this happening? I want the time the meeting is scheduled for to be the absolute, actual time, regardless of GMT/BST: 6:45pm
Should I be storing the date-times as UTC in the DB? How would this be done (I assume it would apply to all dates, not just meeting start dates). And how to re-convert them back into the actual datetime when I display them in the webapp?
Extra:
I have an entry in my initializers/time_formats.rb like this:
:ical => "%Y%m%dT%H%M00Z"
So dates come out like "20130623T184500Z". I use this when building the ics. And this I think is the issue - if the date/time is during BST I don't want to be using Z, but something else?
Your problem is your date/time format. You have:
DTSTART:20130623T184500Z
in your .ics file and this corresponds to 19:45 BST (as British summer time is UTC+1).
There are a few things you should do. First, you can simply remove the 'Z' from the end of your dates. This means that the times inherit the timezone of the calendar, or the underlying application.
This will work assuming that the machines which are running Outlook are all in the Europe/London timezone. If not, or if you want to be a bit safer, you should also specify the following after your BEGIN: VCALENDAR line:
X-WR-TIMEZONE:Europe/London
This specifies the default timezone for all dates which are not specified explicitly.
Finally, if this does not work for any reason then you need to define your datetimes explicitly. First you need to add a timezone definition for Europe/London to the calendar. The info you need is available at http://www.tzurl.org/zoneinfo-outlook/Europe/London.ics. Then you need to ensure that all datetimes are of the format:
DTSTART;TZID=Europe/London:20130623T184500
This last approach is the best, as it means that if your requirements expand to other timezones you will be able to handle them relatively easily.
Sorry to answer this myself, but in case anyone else runs into this here's what I found was the cause of my particular issue. Note that the answer above re timezones also makes sense!
My rails app is storing UTC datetimes in the DB (as is default)
But, it also thought it's own timezone was UTC, which also seems to be the default.
The upshot of that is essentially it was storing local dates, local to UTC anyway. Changing the app to know it was sitting in Europe/London made it so the dates in the DB are all now accurately UTC (meaning, they're an hour off if I'm currently in BST)
I can now use the Z datetime format in iCals, and outlook and the rails app both convert the UTC date back into the actual datetime for the viewing-user's locale (Europe/London for everyone at the moment). This is what I wanted.
for a while I´m trying to understand how this timezone times will work, and I had a question:
Today in my country, we are in Daylight saving time (GMT-2).
So the user of my application enter a time, like 11:00AM and post the form.
Far as I know, rails will convert this date to UTC and save in the database (mysql in my case), like: 01:00PM UTC.
When I recover that record, I had to convert to local time to display. Ok?
My question is, lets suppose that this date/time represents a date/time in future, when my country gets out from summer time (GMT-3). Rails will save 01:00PM UTC? Today, in Daylight saving time, how will be the local time? And in the future, how will be this local time?
Basically, I always need to display to user 11:00AM.
thanks.
There are several places where timezone can come into play: the operating system (or probably user account) default setting, the database server, Rails environment.rb.
The key is to make sure all dates are stored with UTC time zone, then displayed in whatever your local timezone is. It sounds like you're doing that.
So your question seems to boil down to "if it's Daylight time, I want to offset by -3 hours, else offset by -2 hours". The Rails time extensions let you determine your current offset like Time.zone.now.utc_offset, and Time#dst? tells you if it's Daylight Savings Time with those two you can conditionally subtract the extra hour (3600 hundred seconds).
7 months after you asked, but perhaps skip_time_zone_conversion_for_attributes= will help - it tells AcitveRecord not to convert timezones on storage or retrieval. See ActiveRecord Timestamp which shows the example:
class Topic < ActiveRecord::Base
self.skip_time_zone_conversion_for_attributes = [:written_on]
end