Description
We have a Ruby and Rails application with a web frontend and an API to support web calls from outside the frontend. We use the API in one use case to create and update work orders (resource) from calling an web url and sending the parameters as raw JSON data to our API controller (separate controller with separate route).
One attribute of an work order is 'due_date' which is defined in the model as a field of type DateTime (through Mongoid).
While Rails tries to instantiate the new object of a work order in the controller with the given parameters the transformation into the defined database fieldtypes gets done. For DateTime fields this is done internally by calling “at_with_coercion” in rails/activesupport/lib/active_support/core_ext/time/calculations.rb in line 44 (https://github.com/rails/rails/blob/v5.2.6/activesupport/lib/active_support/core_ext/time/calculations.rb#L44).
Because the API is called from outside there is an use case to set the 'due_date' field with an integer number which contains the amount of milliseconds from epoch time (1970-01-01 01:00:00).
Rails can only handle an integer which contains the amount of seconds from epoch time.
So we tried to monkey patch the internal “at_with_coercion” method but ran into some issues.
Steps to reproduce
Model:
include Mongoid::Document
include Mongoid::Timestamps
field :due_date, type: DateTime
Controller:
def create
#work_order = WorkOrder.new(params.require(:work_order).permit(:name, … , :due_date, … ))
#work_order.save
end
Monkey Patches
lib/core_extensions/time.rb:
class Time
class << self
remove_method :at_without_coercion
remove_method :at_with_coercion
remove_method :at
# Layers additional behavior on Time.at so that ActiveSupport::TimeWithZone and DateTime
# instances can be used when called with a single argument
def at_with_coercion(*args)
return at_without_coercion(*args) if args.size != 1
# Time.at can be called with a time or numerical value
time_or_number = args.first
if time_or_number.is_a?(ActiveSupport::TimeWithZone) || time_or_number.is_a?(DateTime)
at_without_coercion(time_or_number.to_f).getlocal
elsif time_or_number.is_a?(Integer) && time_or_number.to_s.length == 13
time_or_number /= 1000.0
at_without_coercion(time_or_number)
else
at_without_coercion(time_or_number)
end
end
alias at_without_coercion at
alias at at_with_coercion
end
end
config/initializers/monkey_patches.rb:
Dir[Rails.root.join('lib', 'core_extensions', '*.rb')].sort.each do |f|
require f
end
API call:
URL: {{protocol}}://{{subdomain}}.{{url}}/api/v2/work_orders.json
Parameters (raw JSON): {
"work_order":{
"name":"Test workorder",
... ,
"due_date":1669105754783,
...
}
}
Expected behavior
Rails should use our monkey patched method and return the correct formatted Time value.
Actual behavior
If we don’t remove the existing aliases the calls to “at_without_coercion” leads to the internal Rails method alias and we get a StackLevel too deep exception.
If we override the aliases we end up in an infinity loop following a StackLevel to deep exception because the overriden method calls itself multiple times.
System configuration
Rails version: 5.2.6
Ruby version: 2.5.9p229
Mongoid gem version: 7.3.1
Alternatives
If there is no possibility to monkey patch the internal method is there a possibility so Rails can handle an amount of milliseconds from epoch time?
Related
I am currently upgrading a Ruby on Rails app from 4.2 to 5.0 and am running into a roadblock concerning fields that store data as a serialized hash. For instance, I have
class Club
serialize :social_media, Hash
end
When creating new clubs and inputting the social media everything works fine, but for the existing social media data I'm getting:
ActiveRecord::SerializationTypeMismatch: Attribute was supposed to be a Hash, but was a ActionController::Parameters.
How can I convert all of the existing data from ActionController::Parameter objects to simple hashes? Database is mysql.
From the fine manual:
serialize(attr_name, class_name_or_coder = Object)
[...] If class_name is specified, the serialized object must be of that class on assignment and retrieval. Otherwise SerializationTypeMismatch will be raised.
So when you say this:
serialize :social_media, Hash
ActiveRecord will require the unserialized social_media to be a Hash. However, as noted by vnbrs, ActionController::Parameters no longer subclasses Hash like it used to and you have a table full of serialized ActionController::Parameters instances. If you look at the raw YAML data in your social_media column, you'll see a bunch of strings like:
--- !ruby/object:ActionController::Parameters...
rather than Hashes like this:
---\n:key: value...
You should fix up all your existing data to have YAMLized Hashes in social_media rather than ActionController::Parameters and whatever else is in there. This process will be somewhat unpleasant:
Pull each social_media out of the table as a string.
Unpack that YAML string into a Ruby object: obj = YAML.load(str).
Convert that object to a Hash: h = obj.to_unsafe_h.
Write that Hash back to a YAML string: str = h.to_yaml.
Put that string back into the database to replace the old one from (1).
Note the to_unsafe_h call in (3). Just calling to_h (or to_hash for that matter) on an ActionController::Parameters instance will give you an exception in Rails5, you have to include a permit call to filter the parameters first:
h = params.to_h # Exception!
h = params.permit(:whatever).to_h # Indifferent access hash with one entry
If you use to_unsafe_h (or to_unsafe_hash) then you get the whole thing in a HashWithIndifferentAccess. Of course, if you really want a plain old Hash then you'd say:
h = obj.to_unsafe_h.to_h
to unwrap the indifferent access wrapper as well. This also assumes that you only have ActionController::Parameters in social_media so you might need to include an obj.respond_to?(:to_unsafe_hash) check to see how you unpack your social_media values.
You could do the above data migration through direct database access in a Rails migration. This could be really cumbersome depending on how nice the low level MySQL interface is. Alternatively, you could create a simplified model class in your migration, something sort of like this:
class YourMigration < ...
class ModelHack < ApplicationRecord
self.table_name = 'clubs'
serialize :social_media
end
def up
ModelHack.all.each do |m|
# Update this to match your real data and what you want `h` to be.
h = m.social_media.to_unsafe_h.to_h
m.social_media = h
m.save!
end
end
def down
raise ActiveRecord::IrreversibleMigration
end
end
You'd want to use find_in_batches or in_batches_of instead all if you have a lot of Clubs of course.
If your MySQL supports json columns and ActiveRecord works with MySQL's json columns (sorry, PostgreSQL guy here), then this might be a good time to change the column to json and run far away from serialize.
Extending on short's reply - a solution that does not require a database migration:
class Serializer
def self.load(value)
obj = YAML.load(value || "{}")
if obj.respond_to?(:to_unsafe_h)
obj.to_unsafe_h
else
obj
end
end
def self.dump(value)
value = if value.respond_to?(:to_unsafe_h)
value.to_unsafe_h
else
value
end
YAML.dump(value)
end
end
serialize :social_media, Serializer
Now club.social_media will work whether it was created on Rails 4 or on Rails 5.
The reply by #schor was a life-saver, but I kept getting no implicit conversion of nil into String errors when doing the YAML.load(value).
What worked for me was:
class Foo < ApplicationRecord
class NewSerializer
def self.load(value)
return {} if !value #### THIS NEW LINE
obj = YAML.load(value)
if obj.respond_to?(:to_unsafe_h)
obj.to_unsafe_h
else
obj
end
end
def self.dump(value)
if value.respond_to?(:to_unsafe_h)
YAML.dump(value.to_unsafe_h)
else
YAML.dump(value)
end
end
end
serialize :some_hash_field, NewSerializer
end
I gotta admin the Rails team totally blindsided me on this one, a most unwelcome breaking change that doesn't even let an app fetch the "old" data.
The official Ruby on Rails documentation has a section about upgrading between Rails versions that explains more about the error you have:
ActionController::Parameters No Longer Inherits from HashWithIndifferentAccess
Calling params in your application will now return an object instead of a hash. If your parameters are already permitted, then you will not need to make any changes. If you are regardless of permitted? you will need to upgrade your application to first permit and then convert to a hash.
params.permit([:proceed_to, :return_to]).to_h
Run a migration on Rails 4 to prepare the data for Rails 5.
We're going through the exact same thing, except we serialize as ActiveSupport::HashWithIndifferentAccess instead of just Hash, which I recommend doing, but I'll provide my answer here for just a simple Hash.
If you have not yet upgraded to Rails 5, which I hope you haven't and your tests have uncovered this issue, you can run a migration on the Rails 4 branch that will get your data ready for Rails 5.
It essentially re-serializes all of your records from ActionController::Parameters to Hash while in Rails 4 and ActionController::Parameters still inherits from HashWithIndifferentAccess.
class ConvertSerializedActionControllerParametersToHashInClubs < ActiveRecord::Migration
disable_ddl_transaction! # This prevents the locking of the table (e.g. in production).
def up
clubs = Club.where.not( social_media: nil )
total_records = clubs.count
say "Updating #{ total_records } records."
clubs.each.with_index( 1 ) do |club, index|
say "Updating #{ index } of #{ total_records }...", true
club.social_media = club.social_media.to_h
club.social_media_will_change!
club.save
end
end
def down
puts "Cannot be reverse! See backup table."
end
end
If you have multiple columns that need to be converted, it's easy to modify this migration to convert all of the necessary tables and columns.
Depending on when you do this, your data should be ready for Rails 5.
I have ruby on rails app and my controller should process request which creates many objects. Objects data is passed from client via json using POST method.
Example of my request (log from controller):
Processing by PersonsController#save_all as JSON
Parameters: {"_json"=>[{"date"=>"9/15/2014", "name"=>"John"},
{"date"=>"9/15/2014", "name"=>"Mike"}], "person"=>{}}
So i need to save these two users but i have some issues:
How to verify strong parameters here? Only Name and Date attributes can be passed from client
How can I convert String to Date if i use Person.new(params)?
Can i somehow preprocess my json? For example i want to replace name="Mike" to name="Mike User" and only then pass it in my model
I want to enrich params of every person by adding some default parameters, for example, i want to add status="new_created" to person params
First of all I'd name the root param something like "users", then it gives a structure that is all connected to the controller name and the data being sent.
Regarding strong params. The config depends of your rails app version. <= 3.x doesn't have this included so you need to add the gem. If you're on >= 4.x then this is already part of rails.
Next in your controller you need to define a method that will do the filtering of the params you need. I should look something like:
class PeopleController < ApplicationController
def some_action
# Here you can call a service that receives people_params and takes
# care of the creation.
if PeopleService.new(people_params).perform
# some logic
else
# some logic
end
end
private
def base_people_params
params.permit(people: [:name, :date])
end
# Usually if you don't want to manipulate the params then call the method
# just #people_params
def people_params
base_people_params.merge(people: normalized_params)
end
# In case you decided to manipulate the params then create small methods
# that would that separately. This way you would be able to understand this
# logic when returning to this code in a couple of months.
def normalized_params
return [] unless params[:people]
params[:people].each_with_object([]) do |result, person|
result << {
name: normalize_name(person[:name]),
date: normalize_date(person[:date]),
}
end
end
def normalize_date(date)
Time.parse(date)
end
def normalize_name(name)
"#{name} - User"
end
end
If you see that the code starts to get to customized take into a service. It will help to help to keep you controller thin (and healthy).
When you create one reason at the time (and not a batch like here) the code is a bit simpler, you work with hashes instead of arrays... but it's all pretty much the same.
EDIT:
If you don't need to manipulate a specific param then just don't
def normalized_params
return [] unless params[:people]
params[:people].each_with_object([]) do |result, person|
result << {
name: person[:name],
date: normalize_date(person[:date]),
}
end
end
I have built an app that consumes a json api. I removed active record from my app because the data in the api can theoretically change and I don't want to wipe the database each time.
Right now I have a method called self.all for each class that loops through the json creating ruby objects. I then call that method in various functions in order to work with the data finding sums and percentages. This all works fine, but seems a bit slow. I was wondering if there is somewhere I should be storing my .all call rather than instantiating new objects for each method that works with the data.
...response was assign above using HTTParty...
def self.all
puppies = []
if response.success?
response['puppies'].each do |puppy|
accounts << new(puppy['name'],
puppy['price'].to_money,
puppy['DOB'])
end
else
raise response.response
end
accounts
end
# the methods below only accept arguments to allow testing with Factories
# puppies is passed in as Puppy.all
def self.sum(puppies)
# returns money object
sum = Money.new(0, 'USD')
puppies.each do |puppy|
sum += puppy.price
end
sum
end
def self.prices(puppies)
prices = puppies.map { |puppy| puppy.price }
end
def self.names(puppies)
names = puppies.map { |puppy| puppy.name }
end
....many more methods that take an argument of Puppy.all in the controller....
Should I use cacheing? should I bring back active record? or is how I'm doing it fine? Should I store Puppy.all somewhere rather than calling the method each time?
What I guess is happening is that you are making a request with HTTParty every time you call any class method. What you can consider is creating a class variable for the response and a class variable called expires_at. Then you can do some basic caching.
##expires_at = Time.zone.now
##http_response
def make_http_call
renew_http_response if ##expires_at.past?
end
def renew_http_response
# make HTTParty request here
##http_response = # HTTParty response
##expires_at = 30.minutes.from_now
end
# And in your code, change response to ##response
# ie response.success? to ##response.success?
This is all in memory and you lose everything if you restart your server. If you want more robust caching, the better thing to do would probably to look into rails low-level caching
Given documents with datetime and data to be displayed in a graph, how can I return the query results directly without converting from BSON to Ruby and then finally to JSON?
Problem: The time values are stored correctly for the client in BSON, but having Ruby involved turns it into Time objects that I have to do time.to_i * 1000 to store correctly in the returned JSON. In any case, I have no need to transform any of the data, so this just feels like a waste.
I run Rails, Mongoid on Heroku + MongoHQ. I'd like to leave the Rails app doing the authorization of the query, but not converting the response to Ruby objects.
def show_graph
raw_bson = TheModel.all_raw_documents_matching(query_params)
raw_bson.to_json
# Alternatively, this BSON to JSON could be happening client side.
# side. Whatever, just don't convert to ruby objects...
end
Your problem has many point of view.
Instead of BSON deserialization i see more critical how the to_json method handles the serialization of Date and Time ruby object.
The most easy, and tricky way that comes in my mind is to override the as_json method of Time class:
class Time
def as_json(options={})
self.to_i * 1000
end
end
hash = {:time => Time.now}
hash.to_json # "{\"a\":1367329412000}"
You can put it in an initializer. This is a really easy solution but you have to keep in mind that every ruby Time object in your app will be serialized with your custom method. This could be fine, or not, it's really hard to say, some gem could depend on this, or not.
A more secure way is to write a wrapper and call it instead of to_json:
def to_json_with_time
converted_hash = self.attributes.map{|k,v| {k => v.to_i * 1000 if v.is_a?(Time)} }.reduce(:merge)
converted_hash.to_json
end
Finally if you really want to override the way how Mongoid serialize and deserialize your object and if you want skip the BSON process you have to define mongoize and demongoize methods.
Here you can find the documentation: Custom Field Serialization
**UPDATE**
Problem is serialization not deserialization. If you get a raw BSON from a query you still have a string representation of a Time ruby object.
You can't convert BSON directly to JSON because it's not possible to convert a string representation of time, to an integer representation without passing from ruby Time class.
Here an example how to use Mongoid Custom Field Serialization.
class Foo
include Mongoid::Document
field :bar, type: Time
end
class Time
# The instance method mongoize take an instance of your object, and converts it into how it will be stored in the database
def mongoize
self.to_i * 1000
end
# The class method mongoize takes an object that you would use to set on your model from your application code, and create the object as it would be stored in the database
def self.mongoize(o)
o.mongoize
end
# The class method demongoize takes an object as how it was stored in the database, and is responsible for instantiating an object of your custom type.
def self.demongoize(object)
object
end
end
Foo.create(:bar => Time.now) #<Foo _id: 518295cebb9ab4e1d2000006, _type: nil, bar: 1367512526>
Foo.last.as_json # {"_id"=>"518295cebb9ab4e1d2000006", "bar"=>1367512526}
Background
I'm using ledermann-rails-settings (https://github.com/ledermann/rails-settings) on a Rails 2/3 project to extend virtually the model with certain attributes that don't necessarily need to be placed into the DB in a wide table and it's working out swimmingly for our needs.
An additional reason I chose this Gem is because of the post How to create a form for the rails-settings plugin which ties ledermann-rails-settings more closely to the model for the purpose of clean form_for usage for administrator GUI support. It's a perfect solution for addressing form_for support although...
Something that I'm running into now though is properly validating the dynamic getters/setters before being passed to the ledermann-rails-settings module. At the moment they are saved immediately, regardless if the model validation has actually fired - I can see through script/console that validation errors are being raised.
Example
For instance I would like to validate that the attribute :foo is within the range of 0..100 for decimal usage (or even a regex). I've found that with the previous post that I can use standard Rails validators (surprise, surprise) but I want to halt on actually saving any values until those are addressed - ensure that the user of the GUI has given 61.43 as a numerical value.
The following code has been borrowed from the quoted post.
class User < ActiveRecord::Base
has_settings
validates_inclusion_of :foo, :in => 0..100
def self.settings_attr_accessor(*args)
>>SOME SORT OF UNLESS MODEL.VALID? CHECK HERE
args.each do |method_name|
eval "
def #{method_name}
self.settings.send(:#{method_name})
end
def #{method_name}=(value)
self.settings.send(:#{method_name}=, value)
end
"
end
>>END UNLESS
end
settings_attr_accessor :foo
end
Anyone have any thoughts here on pulling the state of the model at this point outside of having to put this into a before filter? The goal here is to be able to use the standard validations and avoid rolling custom validation checks for each new settings_attr_accessor that is added. Thanks!
Here is a newer version that works in the new 2x syntax. Yes it is ugly and does double eval.
This produces namespaces method names and adds them to the attr_accessible list. The names are in the form of "#{namespace}_#{attribute} and can be use in forms. I am monkeying with a ppatch to the gem to do this automatically but I an not there yet.
has_settings do |s|
eval 'def self.settings_accessors(namespace, defaults)
defaults.keys.each do |method_name|
attr_accessible "#{namespace}_#{method_name}"
eval "def #{namespace}_#{method_name}
self.settings(:#{namespace.to_s}).send(:#{method_name})
end
def #{namespace}_#{method_name}=(value)
self.settings(:#{namespace}).send(:#{method_name}=, value)
end
"
end
end'
namespace = :fileshare
defaults = {:media => false, :sit => false, :quota_size => 1}
s.key namespace, :defaults => defaults
self.settings_accessors(namespace, defaults)
end