TL;DR: Rails 5.1, Ruby 2.4.0 is serialising a hash including Time objects with quotes around the string representation of the time. These quotes weren't there in Rails 2.3, Ruby 1.8.7 and break my app; how do I get rid of them?
Context and details
I'm upgrading an app from Rails 2.3, Ruby 1.8.7 to Rails 5.1, Ruby 2.4.0.
I have a ReportService class, which has a report_params constructor argument which takes a hash. Upon creation of these objects, this hash gets serialised in YAML format.
class ReportService < ApplicationRecord
# irrelevant AR associations omitted
serialize :report_params
serialize :output_urls
end
A user submits a form containing details of a report they want to be run, including a string that gets parsed using Time.parse(), which gets passed as a constructor argument; so the code (in procedural form to strip out irrelevant details, with lots of extraneous stuff ommitted) looks like
offset = customer.timezone.nil? ? '+0000' : customer.timezone.formatted_offset(:time => start_date)
params[:date_from] = Time.parse("#{start_date} #{params[:hour_from]}:{params[:min_from]} #{offset}").utc.strftime('%Y-%m-%d %H:%M:%S')
report_args = {...
report_params: { ...
date: params[:date_from]
}
}
ReportService.create(report_args)
When I look in my MYSQL database, I find that my report_params field looks like ... date_from: '2017-12-27 00:00:00' .... The corresponding code in the old version produces a result that looks like ... date_from: 2017-12-27 00:00:00 .... This is a bad thing, because the YAML in that field is getting parsed by a (legacy) Java app that polls the database to check for new entries, and the quotes seem to break that deserialisation (throwing java.lang.Exception: BaseProperties.getdate()); if I manually edit the field to remove the quotes, the app works as expected. How can I prevent these quotation marks from being added?
Rails5.1/Ruby2.4 do it correct, since 2017-12-27 00:00:00 is not a valid yaml value.
The good thing is serialize accepts two parameters, the second one being a serializer class.
So, all you need to do is:
class ReportService < ApplicationRecord
# irrelevant AR associations omitted
serialize :report_params, MyYaml
serialize :output_urls, MyYaml
end
and implement MyYaml, delegating everything, save for date/time to YAML and producing whatever you need for them.
The above is valid for any format of serialized data, it’s completely format-agnostic. Examples.
Related
I am upgrading an app to Rails 4.2 and am running into an issue where nil values in a field that is serialized as an Array are getting interpreted as an empty array. Is there a way to get Rails 4.2 to differentiate between nil and an empty array for a serialized-as-Array attribute?
Top level problem demonstration:
#[old_app]
> Rails.version
=> "3.0.3"
> a = AsrProperty.new; a.save; a.keeps
=> nil
#[new_app]
> Rails.version
=> "4.2.3"
> a = AsrProperty.new; a.save; a.keeps
=> []
But it is important for my code to distinguish between nil and [], so this is a problem.
The model:
class AsrProperty < ActiveRecord::Base
serialize :keeps, Array
#[...]
end
I think the issue lies with Rails deciding to take a shortcut for attributes that are serialized as a specific type (e.g. Array) by storing the empty instance of that type as nil in the database. This can be seen by looking at the SQL statement executed in each app:
[old_app]: INSERT INTO asr_properties (lock_version, keeps)
VALUES (0, NULL)
Note that the above log line has been edited for clarity; there are other serialized attributes that were being written due to old Rails' behavior.
[new_app]: INSERT INTO asr_properties (lock_version)
VALUES (0)
There is a workaround: by removing the "Array" declaration on the serialization, Rails is forced to save [] and {} differently:
class AsrProperty < ActiveRecord::Base
serialize :keeps #NOT ARRAY
#[...]
end
Changing the statement generated on saving [] to be:
INSERT INTO asr_properties (keeps, lock_version) VALUES ('---[]\n', 0)
Allowing:
> a = AsrProperty.new; a.save; a.keeps
=> nil
I'll use this workaround for now, but:
(1) I feel like declaring a type might allow more efficiency, and also prevents bugs by explicitly prohibiting the wrong data type being stored
(2) I'd really like to figure out the "right" way to do it, if Rails does allow it.
So: can Rails 4.2 be told to store [] as its own thing in a serialized-as-Array attribute?
What's going on?
What you're experiencing is due to how Rails 4 treats the 2nd argument to the serialize call. It changes its behavior based on the three different values the argument can have (more on this in the solution). The last branch here is the one we're interested in as when you pass the Array class, it gets passed to the ActiveRecord::Coders::YAMLColumn instance that is created. The load method receives the YAML from the database and attempts to turn it back into a Ruby object here. If the coder was not given the default class of Object and the yaml argument is nil in the case of a null column, it will return a new instance of the class, hence the empty array.
Solution
There doesn't appear to be a simple Rails-y way to say, "hey, if this is null in the database, give me nil." However looking at the second branch here we see that we can pass any object that implements the load and dump methods or what I call the basic coder protocol.
Example code
One of the members of my team built this simple class to handle just this case.
class NullableSerializer < ActiveRecord::Coders::YAMLColumn
def load(yaml)
return nil if yaml.nil?
super
end
end
This class inherits from the same YAMLColumn class provided by ActiveRecord so it already handles the load and dump methods. We do not need any modifications to dump but we want to slightly handle loading differently. We simply tell it to return nil when the database column is empty and otherwise call super to work as if we made no other modification.
Usage
To use it, it simply needs to be instantiated with your intended serialization class and passed to the Rails serialize method as in the following, using your naming from above:
class AsrProperty < ActiveRecord::Base
serialize :keeps, NullableSerializer.new(Array)
# …
end
The "right" way
Getting things done and getting your code shipped is paramount and I hope this helps you. After all, if the code isn't being used and doing good, who cares how ideal it is?
I would argue that Rails' approach is the right way in this case especially when you take Ruby's philosophy of The Principle of Least Surprise into account. When an attribute can possibly be an array, it should always return that type, even if empty, to avoid having to constantly special case nil. I would argue the same for any database column that you can put a reasonable default on (i.e. t.integer :anything_besides_a_foreign_key, default: 0). I've always been grateful to past-Aaron for remembering this most of the time whenever I get an unexpected NoMethodError: undefined method 'whatever' for nil:NilClass. Almost always my special case for this nil is to supply a sensible default.
This varies greatly on you, your team, your app, and your application and it's needs so it's never hard and fast. It's just something I've found helps me out immensely when I'm working on something and wondering if amount could default to 0 or if there's some reason buried in the code or in the minds of your teammates why it needs to be able to be nil.
I am setting up stripe connect with the example from https://github.com/rfunduk/rails-stripe-connect-example and am running into a problem using serialize to store stripe_account_status which should be stored as an array.
This is how it should be stored (Generated from the above example link)
{"details_submitted"=>false, "charges_enabled"=>true, "transfers_enabled"=>false, "fields_needed"=>["legal_entity.first_name", "legal_entity.last_name", "legal_entity.dob.day", "legal_entity.dob.month", "legal_entity.dob.year", "legal_entity.address.line1", "legal_entity.address.city", "legal_entity.address.postal_code", "bank_account"], "due_by"=>nil}
And this is how my application is storing it
{:details_submitted=>false, :charges_enabled=>true, :transfers_enabled=>false, :fields_needed=>["legal_entity.first_name", "legal_entity.last_name", "legal_entity.dob.day", "legal_entity.dob.month", "legal_entity.dob.year", "legal_entity.address.line1", "legal_entity.address.city", "legal_entity.address.postal_code", "bank_account"], :due_by=>nil}
As far as I am concerned everything is set up the same. The only difference is that the first example uses
serialize :stripe_account_status, JSON
and my app just has
serialize :stripe_account_status
The reason for this is that when I add JSON I this error:
JSON::ParserError - 795: unexpected token at '':
I have tried finding out the JSON error including changing the config/initializers/cookies_serializer.rb to use :hybrid but this is giving me the same error.
Could someone point me into the right direction of either fixing the JSON issue OR finding a way to make sure the stripe_account_status is stored as an array correctly.
Below is the methods used to store the array:
if #account
user.update_attributes(
currency: #account.default_currency,
stripe_account_type: 'managed',
stripe_user_id: #account.id,
secret_key: #account.keys.secret,
publishable_key: #account.keys.publishable,
stripe_account_status: account_status
)
end
def account_status
{
details_submitted: account.details_submitted,
charges_enabled: account.charges_enabled,
transfers_enabled: account.transfers_enabled,
fields_needed: account.verification.fields_needed,
due_by: account.verification.due_by
}
end
Thanks I really appreciate any direction you could point me!
When you ask Rails to serialize an attribute on a model, it will default to storing the object as YAML string.
You can ask Rails to serialize differently, as you have noticed by providing a class to do the serialization e.g
serialize :stripe_account_status, JSON
The reason why this isn't working when you add it is because you presumably already have a record in the database using the YAML and so Rails can't parse this as a valid JSON string when reading from the DB. If it's just development data that you don't need, you can delete the records and then use JSON, otherwise you will need to convert the current YAML strings to JSON.
Rails will also symbolize the keys of a hash when parsing a serialized string in the database. This is the only difference between the hashes in your question and shouldn't matter in practise. Should you need String keys for some reason, you can use the #stringify_keys method on the hash provided by Rails.
This is using Rails 4.2.0, Ruby 2.2.0.
What I'd like to do is use the data contained in a fixture object to verify that duplicates are caught before insertion into the same database:
test "identical entries should be impossible to create" do
dup_entry = Entry.new(entries(:test_entry))
assert_not dup_entry.save
end
(where Entry is a well-defined model with a controller method .new, and test_entry is a fixture containing some valid Entry instance.)
Unfortunately, this doesn't work because entries(:test_entry) is going to be an Entry, not a hash accepted by Entry.new.
I know that I can access fixture properties with an expression of the form fixture_objname.property in the associated tests, since whatever is specified in the YAML will automatically be inserted into the database and loaded. The problem with this is that I have to manually retype a bunch of property names for the object I just specified in the YAML, which seems silly.
The documentation also says I can get the actual model instances by adding self.use_instantiated_fixtures = true to the test class. However, there don't seem to be any instance_methods that will dump out the fixture's model instance (test_entry) in a hash format to feed back into the .new method.
Is there an idiomatic way to get what I want, or a different, easier way?
I believe you're looking for something like:
entries(:test_entry).attributes
entries(:test_entry).attributes.class # => Hash
You can also remove properties if needed:
entries(:admin).attributes.except("id")
Hope this helps.
I have a model and in that model I'm generating a more complex field than I've done before. I've serialized hashes and arrays, but this field is the result of Gibberish::RSA.generate_keypair ( https://github.com/mdp/gibberish ). Which is more or less a private/public key pair in a ruby wrapper, to my understanding.
Working from the command line, I can do an update_attributes and the result of the generation gets stored in the text field. When doing rake db:seed or creating an instance, this doesn't work, I get a yaml string that indicates several types of Gibberish objects.
How do I do more complex activerecord serialization beyond hashes and arrays? Or how do I approach a greater understanding of what I'm trying to do?
Code:
def generate_keypair
self.update_attributes(:rsakey => Gibberish::RSA.generate_keypair(1024) )
end
which I call on the associated model creation, basic call the Gibberish wrapper
Then the output I get for the field myresource.rsakey
"--- !ruby/object:Gibberish::RSA::KeyPair\nkey:
!ruby/object:OpenSSL::PKey::RSA {}\ncipher:
!ruby/object:OpenSSL::Cipher::Cipher {}\n"
Updating the attributes works from the rails command line, but not while seeding or creating. Other ways I attempted to add serialize so far have completely ruined the process or the created instances.
EDIT: solved bluntly by just calling 'to_s' on the result of the keypair generation method, which just saves it as a text field that 'works for now' until it needs to be more elegant.
The underlying issue seems to be that the openssl library doesn't implement YAML dumping:
YAML.dump(OpenSSL::PKey::RSA.generate(1024))
#=> "--- !ruby/object:OpenSSL::PKey::RSA {}"
If you are using rails 3.1 you can define custom serializers, like so
class KeySerializer
def dump(key)
key.to_pem
end
def load(data)
data && OpenSSL::Pkey::RSA.new(data)
end
end
Then in your class you can do
class Foo < ActiveRecord::Base
serialize :key, KeySerializer.new
end
I am serializing an object to a database field using ActiveRecord's :serialize functionality in Ruby on Rails
class DrawElement < ActiveRecord::Base
...
serialize :content
end
The reason I'm serializing the objects is that I'm dynamically loading the types from disk using const_missing, so I don't have to setup database tables for them.
def DrawElement.const_missing(const)
require File.join('draw_elements',const.to_s)
draw_class = const_get(const)
return draw_class if draw_class
raise "Draw Element not found #{const.to_s}"
end
So when I want to add a draw element, I do something like this in irb
draw_element.content = DrawElement::Text.new
Everything here works fine
The problem is that when I try to load the object from the database in a fresh session, YAML::load never calls const_missing to require the class definition before loading the file. So all my #content objects come back as YAML::Object
Is there a better way to do this? I'm trying to be able to add new types without having to change the database, or have a has_many_polymorph relationship between DrawElements and a Document.
Ruby on Rails v.2.3.8, Ruby v. 1.8.7
From my experience YAML::load returns a hash. It's up to me to walk through the hash and do something with its contents. Neither load or load_file accept a block to get inside them and influence how the YAML document is parsed.
You could try messing with load_documents or each_document though, because they take a block, but I don't know if you could add additional hash elements that way.