Background
Article model with default created_at column
Rails config.time_zone = 'Warsaw'
I've got an article with created_at = local time 2012-08-19 00:15 (2012-08-18 22:15 in UTC).
Goal
To receive all articles created in 2012-08-19 (in local time).
My (not working properly) solution
Article.where(
"date_trunc('day', created_at AT TIME ZONE '#{Time.zone.formatted_offset}')
= '#{Date.civil(2012, 8, 19)}'"
)
Which generates SQL:
SELECT "articles".* FROM "articles"
WHERE (date_trunc('day', created_at AT TIME ZONE '+01:00') = '2012-08-19')
And returns an empty set. But if I run the same query in psql it returns an article ... which confuses me.
Question
What am I doing wrong and how to fix it?
Goal: To receive all articles created in 2012-08-19 (in local time).
'+01:00' (like you use it) is a fixed time offset and cannot take DST (Daylight Saving Time) into account. Use a time zone name for that (not an abbreviation). These are available in PostgreSQL:
SELECT * FROM pg_timezone_names;
For Warsaw this should be 'Europe/Warsaw'. The system knows the bounds for DST from its stored information and applies the according time offset.
Also, your query can be simplified.
As created_at is a timestamp [without time zone], the values saved reflect the local time of the server when the row was created (saved internally as UTC timestamp).
There are basically only two possibilities, depending on the time zone(s) of your client.
Your reading client runs with the same setting for timezone as the writing client: Just cast to date.
SELECT *
FROM articles
WHERE created_at::date = '2012-08-19';
Your reading client runs with a different setting for timezone than the writing client: Add AT TIME ZONE '<tz name of *writing* client here>'. For instance, if that was Europe/Warsaw, it would look like:
...
WHERE (created_at AT TIME ZONE 'Europe/Warsaw')::date = '2012-08-19';
The double application of AT TIME ZONE like you have it in your posted answer should not be necessary.
Note the time zone name instead of the abbreviation. See:
Time zone names with identical properties yield different result when applied to timestamp
If you span multiple time zones with your application ..
.. set the column default of created_at to now() AT TIME ZONE 'UTC' - or some other time zone, the point being: use the same everywhere.
.. or, preferably, switch to timestamptz (timestamp with time zone).
Linked answer helped. I have to run following query:
SELECT *
FROM articles
WHERE (created_at AT TIME ZONE 'UTC' AT TIME ZONE 'CEST')::date = '2012-08-19';
This question would need the exact definition of the column created_at (what data type exactly?)
Rails always creates created_at column as timestamp without time zone. So I have to make the first AT TIME ZONE 'UTC' to say dbms that this timestamp is at UTC, and the second one to display date at CEST zone.
Related
I am running PostgreSQL 9.6.6 on x86_64-pc-linux-gnu and my time zone is set to 'UTC'.
Does anyone know why the results of the following SELECT statements are different?
A)
SELECT timezone('EST', '2017-12-21');
timezone
---------------------
2017-12-20 19:00:00
B)
SELECT timezone('-05', '2017-12-21');
timezone
---------------------
2017-12-21 05:00:00
According to the pg_timezone_names table -05 should have the same offset as EST... Any thoughts? Thanks.
https://www.postgresql.org/docs/current/static/view-pg-timezone-names.html
The view pg_timezone_names provides a list of time zone names that are
recognized by SET TIMEZONE
and further:
utc_offset interval Offset from UTC (positive means east of Greenwich)
when you set timezone to 'EST' - you declare that your client is in EST time zone, thus returned time will be adjusted for your tz:
t=# select '2017-12-21'::timestamptz;
timestamptz
------------------------
2017-12-21 00:00:00-05
(1 row)
the interval match utc_offset from pg_timezone_names and isequal -05, so it works as expected. (indeed in EST will be 5 hours less then UTC) same result if you set timezone to '-05'.
Both -05 and EST give same result for SET TIMEZONE as described in docs.
Now you answer reconciles with docs on using interval: https://www.postgresql.org/docs/current/static/functions-datetime.html#FUNCTIONS-DATETIME-ZONECONVERT
In these expressions, the desired time zone zone can be specified
either as a text string (e.g., 'PST') or as an interval (e.g.,
INTERVAL '-08:00').
following these rules it works as well:
t=# select '2017-12-21'::timestamptz at time zone 'EST';
timezone
---------------------
2017-12-20 19:00:00
(1 row)
t=# select '2017-12-21'::timestamptz at time zone interval '-05:00';
timezone
---------------------
2017-12-20 19:00:00
(1 row)
but further, docs say:
In the text case, a time zone name can be specified in any of the ways
described in Section 8.5.3.
which is https://www.postgresql.org/docs/current/static/datatype-datetime.html#DATATYPE-TIMEZONES
PostgreSQL allows you to specify time zones in three different forms:
recognized time zone names are listed in the pg_timezone_names
recognized abbreviations are listed in the pg_timezone_abbrevs
POSIX-style time zone specifications of the form STDoffset or STDoffsetDST
(formatting mine)
and lastly:
One should be wary that the POSIX-style time zone feature can lead to
silently accepting bogus input...Another issue to keep in mind is that
in POSIX time zone names, positive offsets are used for locations west
of Greenwich. Everywhere else, PostgreSQL follows the ISO-8601
convention that positive timezone offsets are east of Greenwich.
TL;DR
So in short - when you define '-05' as text (not interval) input for timezone() function or AT TIME ZONE directive (effectively same) Postgres thinks this is an attempt to use POSIX style time zone and thus inverts sign, thus you get "opposite" result...
a simple demonstration of this documented inversion:
t=# select '2017-12-21'::timestamptz at time zone '05';
timezone
---------------------
2017-12-20 19:00:00
(1 row)
Okay I think I found an answer to my own question:
According to the PostgreSQL docs, section 9.9.3 at the following link https://www.postgresql.org/docs/9.6/static/functions-datetime.html
In these expressions, the desired time zone zone can be specified either as a text string (e.g., 'PST') or as an interval (e.g., INTERVAL '-08:00'). In the text case, a time zone name can be specified in any of the ways described in Section 8.5.3.
So using the INTERVAL syntax, the following appears to work:
SELECT timezone(INTERVAL '-05:00', '2017-12-21');
timezone
---------------------
2017-12-20 19:00:00
I think it is still curious, what exactly SELECT timezone('-05', '2017-12-21'); means, as the following also provides the expected result (with the addition of a TZ offset):
SELECT timezone('-05', '2017-12-21'::timestamp);
timezone
------------------------
2017-12-20 19:00:00+00
I'm working on an app in which users can log in and select a site which may be in a different time zone. Because the app is showing data that is relevant to that site I've decided to set the default timezone for the app to be the site timezone via NSTimeZone.setDefaultTimeZone. This works great except when I select data out of our sqlite db via FMDB (I don't think FMDB has anything to do with it) and use strftime with 'localtime'. While our data is stored by the minute in UTC using epochs, we often need to show summations by day, weekday or month in the local time zone. For example to get averages by weekday:
select strftime('%w',datetime(epoch,'localtime')),avg(value)
from values
where siteId = 'x'
group by 1
order by 2 desc
The 'localtime' that it's using is the system local time and not the default time zone for the app where all NSDate calls respect the default time zone. There does not to be any other timezone options for strftime other than localtime and UTC and the current work arounds are pretty slow requiring multiple SQL roundtrips where this should easily be handled in 1 query as above.
Can anyone tell me how sqlite on iOS determines 'localtime'? Is there a way to force it to use a different i.e. defaultTimeZone?
As you have seen, SQLite doesn't use NSDate or the app's local timezone. The datetime function converts with a Modifier. Say you had a DB that stores as GMT (I think this is the case for your app):
sqlite> create table mytable (id int, time datetime);
sqlite> insert into mytable values (1, CURRENT_TIMESTAMP);
sqlite> select time from mytable;
2016-06-24 19:05:36 <- THIS IS GMT
sqlite> select datetime(time, 'localtime') from mytable;
2016-06-24 15:05:36 <- THIS IS LOCAL TIME
In this example (and yours) 'localtime' is the Modifier. From the sqlite docs:
Modifiers
The time string can be followed by zero or more modifiers that alter
date and/or time. Each modifier is a transformation that is applied to
the time value to its left. Modifiers are applied from left to right;
order is important. The available modifiers are as follows.
NNN days
NNN hours
NNN minutes
NNN.NNNN seconds
NNN months
NNN years
start of month
start of year
start of day
weekday N
unixepoch
localtime
utc
So you cannot directly convert to the local value. However, because you can use these modifers your app can get your local GMT offset from NSDate:
if let myZone = NSTimeZone(abbreviation: "EST")
{
NSTimeZone.setDefaultTimeZone(myZone)
var offset = (myZone.secondsFromGMT)/3600 as Int
var offsetModifer = "\(offset) hours"
}
Then you can execute your sqlite query as so (building the query using offsetModifer which translates to -4 hours in the example here:
sqlite> select datetime(time, '-4 hours') from mytable;
2016-06-24 15:05:36
I'm working on a Rails application which stores all dates to PostgreSQL as "TIMESTAMP WITHOUT TIME ZONE". (Rails handles the time zone on the application layer which for this application is "Europe/Berlin".) Unfortunately, Daylight Savings Time (DST) becomes an issue.
The simplified "projects" table has the following columns:
started_at TIMESTAMP WITHOUT TIME ZONE
duration INTEGER
Projects start at started_at and run for duration days.
Now, say there's only one project which starts on 2015-01-01 at 10:00. Since this is "Europe/Berlin" and it's January (no DST), the record looks like this on the database:
SET TimeZone = 'UTC';
SELECT started_at from projects;
# => 2015-01-01 09:00:00
It should end on 2015-06-30 at 10:00 (Europe/Berlin). But it's summer now, so DST applies and 10:00 in "Europe/Berlin" is now 08:00 in UTC.
Due to this, finding all projects for which the duration has elapsed by use of the following query does not work for projects which start/end across DST boundaries:
SELECT * FROM projects WHERE started_at + INTERVAL '1 day' * duration < NOW()
I guess it would be best if the above WHERE did the calculation in timezone "Europe/Berlin" rather than "UTC". I've tried a few things with ::TIMESTAMTZ and AT TIME ZONE none of which has worked.
As a side note: According to the PostgreSQL docs, + INTERVAL should deal with '1 day' intervals differently from '24 hours' intervals when it comes to DST. Adding days ignores DST, so 10:00 always stays 10:00. When adding hours on the other hand, 10:00 may become 09:00 or 11:00 if you cross the DST boundary one way or another.
Thanks a lot for any hints!
I think you've got two strategies for avoiding headache:
Let Rails handle everything to do with Timezones, so Postgres doesn't have to at all
or
Let Postgres handle everything to do with Timezones, so Rails doesn't have to at all
Mixing the two will always be a pain, and is basically what's causing your problems now. I'd go with strategy 1 (let Rails handle it). To do this, your Postgres database should store a start time, and a finish time, both in UTC. duration may be a thing in your user interface still, but if a user enters a start time and a duration, then you should calculate a finish time, and store that finish time in your database. The start time the users enters, and the finish time that you calculate in your app, with both be timezone-specific, and you just let Rails handle the conversion to UTC when it saves to the database.
Your query would then be simply:
SELECT * FROM projects WHERE finished_at < NOW()
(BTW, You could also store the duration in your database, but it's superfluous, since it can be calculated from the start time and finish time)
I've created a function which calculates ended_at by adding duration days to started_at honoring DST changes of a given time zone. Both started_at and ended_at, however, are in UTC and therefore play nice with Rails.
It turns started_at (timestamp without time zone, implicit UTC by Rails) to a timestamp with time zone UTC, then to the given time zone, adds the duration and returns the timestamp without time zone (implicit UTC).
# ended_at(started_at, duration, time_zone)
CREATE FUNCTION ended_at(timestamp, integer, text = 'Europe/Zurich') RETURNS timestamp AS $$
SELECT (($1::timestamp AT TIME ZONE 'UTC' AT TIME ZONE $3 + INTERVAL '1 day' * $2) AT TIME ZONE $3)::timestamp
$$ LANGUAGE SQL IMMUTABLE SET search_path = public, pg_temp;
With this function, I can omit having to add ended_at as an explicit column which would have to be kept in sync. And it's easy to use:
SELECT ended_at(started_at, duration) FROM projects
I have a time field in table "timestamp without time zone". When record is saved to database, the utc time might be a different day compared to the local time. However, I need to group the records by date. Hence, I am doing something like this:
result = transmissions.joins(:report).where('reports.time::timestamp::date = ?', record.time.to_date)
The problem is if the utc date is on a different date than local time, then that record is not included in result. Any ideas how to get the right result?
And apparently I cannot change the "without time zone" either:
Rails database-specific data type
It says:
"concluded that the default ActiveRecord datetime and timestamp column types in schema migrations cannot be modified to force PostgreSQL to use timestamp with time zone."
So I have no idea how to group by date, as obviously something like this is wrong:
Unit.where(id: 1100).first.reports.order("DATE(time)").group("DATE(time)").count
=> {"2013-12-14"=>19, "2013-12-15"=>5}
That return value is completely wrong. All 25 records should be on 2013-12-14 and 0 records on 2013-12-15.
Assuming your records are timestamped with a particular UTC offset, you can try passing in the start and end times of the date in question in UTC format to your query:
result = transmissions.joins(:report).where('reports.time >= ? AND reports.time < ?', record.time.midnight.utc, (record.time.midnight + 1.day).utc)
Explanation:
midnight is a Rails method on an instance of Time that returns the Time object that represents midnight on the date of the original Time object. Similarly, record.time.midnight + 1.day returns the Time object representing midnight of the following day. Then, converting both Time objects – which are presumably timestamped in a standard UTC offset – to UTC creates a time period representing midnight-to-midnight for the system timezone in UTC format (not midnight in UTC time), which is precisely what you're seeking to query.
How about something like result = transmissions.joins(:report).where('reports.time >= ? AND reports.time <= ?', record.time.beginning_of_day.utc, record.time.end_of_day.utc)
The .utc part may not be necessary.
I want to be able to test the behaviour of a scheduler component across different timezones. However, the functionality to trigger scheduled behaviour uses time based queries within postgres:
e.g.
# find reminders which have not been sent for "today" in the local date
Schedule.where('evening_reminder_last_sent_on_local_date !=
DATE( NOW() AT TIME ZONE time_zone )')
I would like to be able to test this behaviour in RSpec and ensure that it plays correctly through the day and that if I send a Japanese user a reminder at 1am UTC on the 25th Dec, then at 10pm UTC, their reminder for "today" will show up as not have been sent (since it's about 7am the next day in Japan).
However, in order to do this I need to be able to set the datetime in postgres. Is this possible?
Please note... this is not about stubbing Rails' time
The challenge is not to stub the time in Rails - I know how to do that. The problem is how to set the time in Postgres.
Postgres uses the date / time setting of the underlying OS (at least on UNIX-like systems). To stage various timestamps, you would have to set the time of the OS, with the date command.
I would certainly not recommend that on a system that does anything else of importance. cronjobs can run haywire and other unpleasant side effects.
Instead, replace the function now() with a user-defined server-side function like:
CREATE OR REPLACE FUNCTION now_test()
RETURNS timestamptz AS $func$SELECT '2013-12-01 12:34'::timestamptz$func$ LANGUAGE SQL;
(The above cast assumes the current time zone setting of the session. Alternatively, you can provide a time zone or time offset with the literal.)
Then you can test your code with:
Schedule.where('evening_reminder_last_sent_on_local_date !=
DATE(now_test() AT TIME ZONE time_zone)')
Modify the above SQL function above with various timestamps and you are good to go.
The previous suggestion is indeed very good but as I wanted to easily test my query with Delorean gem, I came up with this workaround:
Schedule.where('evening_reminder_last_sent_on_local_date !=
DATE( ?::timestamp AT TIME ZONE time_zone )', Time.now)
I took the previous example just for the sake, but I had to manipulate times in Postgres with the now() function. I couldn't just inject my ruby time instead without casting it with ::timestamp.
Moreover, in your case, maybe you could use timestamptz.
I had a similar case except instead of modifying the timezones, I wanted to apply an offset. This let me synchronize postgres's responses to NOW() with my appliction's responses to datetime.now() (tampered via libfaketime).
I ended up with one statement that renamed now() to system_now(), but only if system_now() didn't already exist:
DO $$
DECLARE
found_func pg_proc%rowtype;
BEGIN
SELECT * FROM pg_proc WHERE proname = 'system_now' INTO found_func;
IF FOUND THEN
RAISE NOTICE 'DB Already Faked';
ELSE
RAISE NOTICE'Faking DB Time';
ALTER FUNCTION now RENAME TO system_now;
END IF;
END $$;
...and another one that redefined NOW() to include the offset:
CREATE OR REPLACE FUNCTION now() RETURNS timestamptz
AS $func$
SELECT system_now() + INTERVAL '1225288 seconds';
$func$ LANGUAGE SQL;