I have a test in a Rails 3 test suite that makes a number assertions that compare timestamps that passes on my local machine but fails within our CI pipeline. This test stores a timestamp in a Postgres database timestamp field with the precision of 6 and compares the stored value to the original timestamp, very similar to the following example:
tmp_time = Time.zone.now
u = User.find(1)
u.updated_at = tmp_time
u.save!
u.reload
assert_equal u.updated_at.to_i, tmp_time.to_i # passes...
assert_equal u.updated_at, tmp_time # fails...
assert_equal u.updated_at.to_f, tmp_time.to_f # fails...
I believe the problem relates to Ruby's time representation being of a higher precision than the stored value.
What is the best way of compensating for the slight difference in values due to precision, outside of being less precise in comparisons? We have considered overriding Time.zone.now, but believe that will lead to downstream problems.
Thanks in advance.
The issue is probably not with precision in the database, but rather that a small time passes between when you define tmp_time and when you save.
You can see that the .to_f representation of Time changes instantly:
irb(main):011:0> 2.times.map { Time.now.to_f }
=> [1551755487.5737898, 1551755487.573792]
This difference is usually not visible when you use .to_i because it rounds to the nearest second.
You can use Timecop, as another answer mentions, to get around this:
irb(main):013:0> Timecop.freeze { 2.times.map { Time.now.to_f } }
=> [1551755580.12368, 1551755580.12368]
When you call .save! an actual write to the database occurs. Timestamps are written by the database which updates the actual data stored in updated_at which is not written by the ActiveRecord Ruby object (unless you do so explicitly with u.update_attribute(updated_at: tmp_time) which defeats the point of timestamps in most cases.
So the time in memory at the moment you instantiate the Ruby Time object won't match the time recorded by the database, which will be some nanoseconds later. Converting Time.new.to_i is not very accurate. While .to_f is normally "close enough", equality of time is nearly impossible. This can be illustrated with a multi-threaded example:
#times = []
def test_time
t1 = Thread.new{ #times << Time.now }
t2 = Thread.new{ #times << Time.now }
t1.join; t2.join
end
test_time
puts #times.each(&:to_s)
# they may appear the same depending on default `.to_s` format
# 10/29/2021 2:20PM
# 10/29/2021 2:20PM
#times.map(&:to_i)
=>[1635531636,1635531636] # same problem
#times.map(&:to_f)
=>[1635531636.989422, 1635531636.989532] # here's often enough precision but...
# under the hood times[1] == times[2] will use the most precise nsec method
#times.map(&:nsec)
=>[989422129, 989531969]
See also See docs on Time#nsec
Related
Does Rails really not properly support PostgreSQL's interval data type?
I had to use this Stack Overflow answer from 2013 to create an interval column, and now it looks like I'll need to use this piece of code from 2013 to get ActiveRecord to treat the interval as something other than a string.
Is that how it is? Am I better off just using an integer data type to represent the number of minutes instead?
From Rails 5.1, you can use postgres 'Interval' Data Type, so you can do things like this in a migration:
add_column :your_table, :new_column, :interval, default: "2 weeks"
Although ActiveRecord only treat interval as string, but if you set the IntervalStyle to iso_8601 in your postgresql database, it will display the interval in iso8601 style: 2 weeks => P14D
execute "ALTER DATABASE your_database SET IntervalStyle = 'iso_8601'"
You can then directly parse the column to a ActiveSupport::Duration
In your model.rb
def new_column
ActiveSupport::Duration.parse self[:new_column]
end
More infomation of ISO8601 intervals can be find at https://en.wikipedia.org/wiki/ISO_8601#Time_intervals
I had a similar issue and went with defining reader method for the particular column on the ActiveRecord model. Like this:
class DivingResults < ActiveRecord::Base
# This overrides the same method for db column, generated by rails
def underwater_duration
interval_from_db = super
time_parts = interval_from_db.split(':')
if time_parts.size > 1 # Handle formats like 17:04:41.478432
units = %i(hours minutes seconds)
in_seconds = time_parts
.map.with_index { |t,i| t.to_i.public_send(units[i]) }
.reduce(&:+) # Turn each part to seconds and then sum
ActiveSupport::Duration.build in_seconds
else # Handle formats in seconds
ActiveSupport::Duration.build(interval_from_db.to_i)
end
end
end
This allows to use ActiveSupport::Duration instance elsewhere. Hopefully Rails will start handling the PostgreSQL interval data type automatically in near future.
A more complete and integrated solution is available in Rails 6.1
The current answers suggest overriding readers and writers in the models. I took the alter database suggestion and built a gem for ISO8601 intervals, ar_interval.
It provides a simple ActiveRecord::Type that deals with the serialization and casting of ISO8601 strings for you!
The tests include examples for how to use it.
If there is interest, the additional formats Sam Soffes demonstrates could be included in the tests
Similar to Madis' solution, this one handles fractions of a second and ISO8601 durations:
def duration
return nil unless (value = super)
# Handle ISO8601 duration
return ActiveSupport::Duration.parse(value) if value.start_with?('P')
time_parts = value.split(':')
if time_parts.size > 1
# Handle formats like 17:04:41.478432
units = %i[hours minutes seconds]
in_seconds = time_parts.map.with_index { |t, i| t.to_f.public_send(units[i]) }.reduce(&:+)
ActiveSupport::Duration.build in_seconds
else
# Handle formats in seconds
ActiveSupport::Duration.build(value)
end
end
def duration=(value)
unless value.is_a?(String)
value = ActiveSupport::Duration.build(value).iso8601
end
self[:duration] = value
end
This assumes you setup your database like Leo mentions in his answer. No idea why sometimes they come back from Postgres in the PT42S format and sometimes in the 00:00:42.000 format :/
So I'm running a bit of logic through all of the 50,000+ records in my table (several actually, but we'll just address one here they're all the same operation) and marking boolean indicators:
ActiveRecord::Base.silence do
CoreRevenue.where('core_revenues_usd IS NOT NULL').each do |c|
c.ok = true
c.save
end
end
As you may have noticed, I've already "shut up" the database from outputting a bunch of SQL responses to the console, but there's always the big array dump at the end of the operation that can take sometimes as long as 5-10 seconds.
So I'm wondering if I can stop x where x = CoreRevenue.where('core_revenues_usd IS NOT NULL') from getting dumped after the operation is completed. Thanks.
equally 'exciting' would be an answer explaining why this is not possible due to some sort of lambda calculus computing thing or what have you
You can’t prevent a block from returning, but you can just return something else instead:
ActiveRecord::Base.silence do
CoreRevenue.where('core_revenues_usd IS NOT NULL').each do |c|
c.ok = true
c.save
end
nil
end
I would write that this way:
CoreRevenue.where('core_revenues_usd IS NOT NULL').update_all(:ok, true)
So it will run much faster than your version.
EDIT: Corrected with #John Neagle suggestion
I have the below code snippet:
class Foo
include DataMapper::Resource
property :id, Serial
property :timestamp, DateTime
end
I just want to convert the current time to ms:
class Time
def to_ms
(self.to_f * 1000.0).to_i
end
end
def current_time
time = Time.now
return time.to_ms
end
time = current_time # => 1352633569151
but when I am going to save the Foo with above timestamp, then it can't be saved to the database and I'm not getting any error message.
foo = Foo.new
foo.timestamp = time
foo.save
Any idea?
are you using a correct format for your :datetime property?
should be like:
DateTime.now.to_s
=> "2012-11-11T14:04:02+02:00"
or a "native" DateTime object, without any conversions.
DataMapper will carry to convert it to according values based on adapter used.
also, to have exceptions raised when saving items:
DataMapper::Model.raise_on_save_failure = true
that's a global setting, i.e. all models will raise exceptions.
to make only some model to behave like this:
YourModel.raise_on_save_failure = true
http://datamapper.org/docs/create_and_destroy.html
See "Raising an exception when save fails" chapter
btw, to see what's wrong with your item before saving it, use and item.valid? and item.errors
foo = Foo.new
foo.timestamp = time
if foo.valid?
foo.save
else
p foo.errors
end
I replicated your code and got following error:
#errors={:timestamp=>["Timestamp must be of type DateTime"]}
See live demo here
The PostgreSQL data types would be timestamp or timestamp with time zone. But that contradicts what you are doing. You take the epoch value and multiply by 1000. You'd have to save that as integer or some numeric type.
More about the handling of timestamps in Postgres in this related answer.
I would save the value as timestamp with time zone as is (no multiplication). You can always extract ms out of it if need should be.
If you need to translate the Unix epoch value back to a timestamp, use:
SELECT to_timestamp(1352633569.151);
--> timestamptz 2012-11-11 12:32:49.151+01
Just save "now"
If you actually want to save "now", i.e. the current point in time, then let Postgres do it for you. Just make sure the database server has a reliable local time - install ntp. This is generally more reliable, accurate and simple.
Set the DEFAULT of the timestamp column to now() or CURRENT_TIMESTAMP.
If you want timestamp instead of timestamptz you can still use now(), which is translated to "local" time according to the servers timezone setting. Or, to get the time for a given time zone:
now() AT ZIME ZONE 'Europe/Vienna' -- your time zone here
Or, in your particular case, since you seem to want only three fractional digits: now()::timestamp(3) or CURRENT_TIMESTAMP(3) or CURRENT_TIMESTAMP(3) AT ZIME ZONE 'Europe/Vienna'.
Or, if you define the type of the column as timestamp(3), all timestamp values are coerced to the type and rounded to 3 fractional decimal digits automatically.
So this would be all you need:
CREATE TABLE tbl (
-- other columns
,ts_column timestamp(3) DEFAULT now()
);
The value is set automatically on INSERT, you don't even have to mention the column.
If you want to update it ON UPDATE, add a TRIGGER like this:
Trigger function:
CREATE OR REPLACE FUNCTION trg_up_ts()
RETURNS trigger AS
$BODY$
BEGIN
NEW.ts_column := now();
RETURN NEW;
END
$BODY$ LANGUAGE plpgsql VOLATILE
Trigger:
CREATE TRIGGER log_up_ts
BEFORE UPDATE ON tbl
FOR EACH ROW EXECUTE PROCEDURE trg_up_ts();
Now, everything works automatically.
If that's not what you are after, #slivu's answer seems to cover the Ruby side just nicely.
I'm not familiar with PostgreSQL, but why are you assigning a Fixnum (time) to timestamp (which is a DateTime)? Your model must be failing to convert time to a proper DateTime value before generating the SQL.
Try foo.save!. I'm pretty sure you'll see an error, either reported from PostgreSQL, saying 1352633569151 is not a valid value for the table column, or your model will say it can't parse 1352633569151 to a valid DateTime.
foo.timestamp = Time.now or foo.timestamp = '2012-11-11 00:00:00' is something that'll work.
Why do the following differ?
Time.now.end_of_day == Time.now.end_of_day - 0.days # false
Time.now.end_of_day.to_s == Time.now.end_of_day - 0.days.to_s # true
Because the number of nanoseconds is different:
ruby-1.9.2-p180 :014 > (Time.now.end_of_day - 0.days).nsec
=> 999999000
ruby-1.9.2-p180 :015 > Time.now.end_of_day.nsec
=> 999999998
Like Mischa said, the times differ by nanoseconds. Here is an article on workarounds and fixes for doing this in Rails, specifically for tests like you are doing.
The seemingly most straightforward approach given is to round the times to seconds by appending .to_i, but there are other alternatives.
To expand on Mischa's answer:
From the docs on the Time object: "All times may have fraction. Be aware of this fact when comparing times with each other—times that are apparently equal when displayed may be different when compared."
So your first calculation compares two Time objects, which are different at the nanosecond level, but your second calculation converts both Time objects to Strings, which ignores the nanoseconds and returns true because both String representations match.
I'm working with Time in Rails and using the following code to set up the start date and end date of a project:
start_date ||= Time.now
end_date = start_date + goal_months.months
I then clone the object and I'm writing rspec tests to confirm that the attributes match in the copy. The end dates match:
original[end_date]: 2011-08-24 18:24:53 UTC
clone[end_date]: 2011-08-24 18:24:53 UTC
but the spec gives me an error on the start dates:
expected: Wed Aug 24 18:24:53 UTC 2011,
got: Wed, 24 Aug 2011 18:24:53 UTC +00:00 (using ==)
It's clear the dates are the same, just formatted differently. How is it that they end up getting stored differently in the database, and how do I get them to match? I've tried it with DateTime as well with the same results.
Correction: The end dates don't match either. They print out the same, but rspec errors out on them as well. When I print out the start date and end date, the values come out in different formats:
start date: 2010-08-24T19:00:24+00:00
end date: 2011-08-24 19:00:24 UTC
This usually happens because rspec tries to match different objects: Time and DateTime, for instance. Also, comparable times can differ a bit, for a few milliseconds.
In the second case, the correct way is to use stubbing and mock. Also see TimeCop gem
In the first case, possible solution can be to compare timestamps:
actual_time.to_i.should == expected_time.to_i
I use simple matcher for such cases:
# ./spec/spec_helper.rb
#
# Dir[File.dirname(__FILE__) + "/support/**/*.rb"].each {|f| require f}
#
#
# Usage:
#
# its(:updated_at) { should be_the_same_time_as updated_at }
#
#
# Will pass or fail with message like:
#
# Failure/Error: its(:updated_at) { should be_the_same_time_as 2.days.ago }
# expected Tue, 07 Jun 2011 16:14:09 +0300 to be the same time as Mon, 06 Jun 2011 13:14:09 UTC +00:00
RSpec::Matchers.define :be_the_same_time_as do |expected|
match do |actual|
expected.to_i == actual.to_i
end
end
You should mock the now method of Time to make sure it always match the date in the spec. You never know when a delay will make the spec fail because of some milliseconds. This approach will also make sure that the time on the real code and on the spec are the same.
If you're using the default rspec mock lib, try to do something like:
t = Time.parse("01/01/2010 10:00")
Time.should_receive(:now).and_return(t)
I totally agree with the previous answers about stubbing Time.now. That said there is one other thing going on here. When you compare datetimes from a database you lose some of the factional time that can be in a ruby DateTime obj. The best way to compare date in that way in Rspec is:
database_start_date.should be_within(1).of(start_date)
One gotcha is that Ruby Time objects have nanosecond precision, but most databases have at most microsecond precision. The best way to get around this is to stub Time.now (or use timecop) with a round number. Read the post I wrote about this: http://blog.solanolabs.com/rails-time-comparisons-devil-details-etc/
Depending on your specs, you might be able to use Rails-native travel helpers:
# in spec_helper.rb
config.include ActiveSupport::Testing::TimeHelpers
start_date ||= Time.current.change(usecs: 0)
end_date = start_date + goal_months.months
travel_to start_date do
# Clone here
end
expect(clone.start_date).to eq(start_date)
Without Time.current.change(usecs: 0) it's likely to complain about the difference between time zones. Or between the microseconds, since the helper will reset the passed value internally (Timecop has a similar issue, so reset usecs with it too).
My initial guess would be that the value of Time.now is formatted differently from your database value.
Are you sure that you are using == and not eql or be? The latter two methods use object identity rather than comparing values.
From the output it looks like the expected value is a Time, while the value being tested is a DateTime. This could also be an issue, though I'd hesitate to guess how to fix it given the almost pathological nature of Ruby's date and time libraries ...
One solution I like is to just add the following to spec_helper:
class Time
def ==(time)
self.to_i == time.to_i
end
end
That way it's entirely transparent even in nested objects.
Adding .to_datetime to both variables will coerce the datetime values to be equivalent and respect timezones and Daylight Saving Time. For just date comparisons, use .to_date.
An example spec with two variables:
actual_time.to_datetime.should == expected_time.to_datetime
A better spec with clarity:
actual_time.to_datetime.should eq 1.month.from_now.to_datetime
.to_i produces ambiguity regarding it's meaning in the specs.
+1 for using TimeCop gem in specs. Just make sure to test Daylight Saving Time in your specs if your app is affected by DST.
Our current solution is to have a freeze_time method that handles the rounding:
def freeze_time(time = Time.zone.now)
# round time to get rid of nanosecond discrepancies between ruby time and
# postgres time
time = time.round
Timecop.freeze(time) { yield(time) }
end
And then you can use it like:
freeze_time do
perform_work
expect(message.reload.sent_date).to eq(Time.now)
end