I'm using Faraday to retrieve JSON encoded objects from a web service, but the web service uses horrible key strings for the key:value pairs, like 'Memo' where I want 'description' and 'projectManager' where I want 'manager'.
How can I transform the keys as they are retrieved by Faraday and inserted the result into a Ruby hash?
I'm hoping this is a solved problem, but it sure is hard to find. (I see a lot of posts asking about symbols being converted to strings in JSON.parse() ).
consider mapping the data into Virtus objects
https://github.com/solnic/virtus
class Foo
include Virtus.model
attribute :manager, String
attribute :description, String
def self.map(json)
# you only need to map the ones that differ
new(json.merge!({
manager: json["projectManager"],
description: json["Memo"]
})
end
end
then loop through your data as needed (or create a top level virtus object if it makes sense)
payload = ... # json retrieved by Faraday as array of ruby hashes
foo_list = payload.map do |item|
Foo.map(item)
end
Related
I have this problem with my Rails (5.1.6) application that is running on a PostgreSQL instance.
I have a model with a JSON type column (t.json :meta). The model has a store accessor like
store :meta, accessors: [:title], coder: JSON
The problem is now when I set this value that it shows up in the database as
"{\"title\":\"I am a title\"}"
making it text rather than a JSON value, which in turn makes that I cannot use the JSON query operator (->>) to query my JSON fields. I already tried without the coder option, but this results in it being saved as YAML.
The serialize function also did not change anything for me (adding serialize :meta, JSON)
Any and all help is appreciated!
serialize and store are not intended to be used for native JSON columns. Their purpose is to marshal and un-marshal data into string columns.
This was a "poor mans" JSON storage before native JSON support existed (and was supported by ActiceRecord). Using it on a JSON column will result in a double encoded string as you have noticed.
You don't actually have to do anything to use a JSON column. Its handled by the adapter.
See:
http://api.rubyonrails.org/classes/ActiveRecord/AttributeMethods/Serialization/ClassMethods.html#method-i-serialize
http://guides.rubyonrails.org/active_record_postgresql.html#json-and-jsonb
TL;DR: Rails 5.1, Ruby 2.4.0 is serialising a hash including Time objects with quotes around the string representation of the time. These quotes weren't there in Rails 2.3, Ruby 1.8.7 and break my app; how do I get rid of them?
Context and details
I'm upgrading an app from Rails 2.3, Ruby 1.8.7 to Rails 5.1, Ruby 2.4.0.
I have a ReportService class, which has a report_params constructor argument which takes a hash. Upon creation of these objects, this hash gets serialised in YAML format.
class ReportService < ApplicationRecord
# irrelevant AR associations omitted
serialize :report_params
serialize :output_urls
end
A user submits a form containing details of a report they want to be run, including a string that gets parsed using Time.parse(), which gets passed as a constructor argument; so the code (in procedural form to strip out irrelevant details, with lots of extraneous stuff ommitted) looks like
offset = customer.timezone.nil? ? '+0000' : customer.timezone.formatted_offset(:time => start_date)
params[:date_from] = Time.parse("#{start_date} #{params[:hour_from]}:{params[:min_from]} #{offset}").utc.strftime('%Y-%m-%d %H:%M:%S')
report_args = {...
report_params: { ...
date: params[:date_from]
}
}
ReportService.create(report_args)
When I look in my MYSQL database, I find that my report_params field looks like ... date_from: '2017-12-27 00:00:00' .... The corresponding code in the old version produces a result that looks like ... date_from: 2017-12-27 00:00:00 .... This is a bad thing, because the YAML in that field is getting parsed by a (legacy) Java app that polls the database to check for new entries, and the quotes seem to break that deserialisation (throwing java.lang.Exception: BaseProperties.getdate()); if I manually edit the field to remove the quotes, the app works as expected. How can I prevent these quotation marks from being added?
Rails5.1/Ruby2.4 do it correct, since 2017-12-27 00:00:00 is not a valid yaml value.
The good thing is serialize accepts two parameters, the second one being a serializer class.
So, all you need to do is:
class ReportService < ApplicationRecord
# irrelevant AR associations omitted
serialize :report_params, MyYaml
serialize :output_urls, MyYaml
end
and implement MyYaml, delegating everything, save for date/time to YAML and producing whatever you need for them.
The above is valid for any format of serialized data, it’s completely format-agnostic. Examples.
I have parameters from an external API that formats JSON responses using CamelCase that I want to fit into my Rails app:
{"AccountID"=>"REAWLLY_LONG_HASH_API_KEY_FROM_EXTERNAL_SERVICE",
"ChannelProductDescription"=>"0004", "Currency"=>"CAD",
"CurrentBalance"=> {"Amount"=>"162563.64", "Currency"=>"CAD"}}
Using the below script I converted them to lower case:
data = JSON.parse(response, symbolize_keys: true)
data = {:_json => data} unless data.is_a?(Hash)
data.deep_transform_keys!(&:underscore)
data.deep_symbolize_keys!
Leaving me correctly formatted params like this:
{:account_id=>"REAWLLY_LONG_HASH_API_KEY_FROM_EXTERNAL_SERVICE",
:channel_product_description=>"SAVINGS", :currency=>"CAD",
:current_balance=> {:amount=>"43.00", :currency=>"CAD"}}
I'm trying to map this external API response into a generic Rails model Account, where JSON from this API call will return cleanly as parameters into my database to allow a clean saving interface such as:
#account = Account.create(ParamParser.call(params))
But I ran into a problem with converting :account_id, as that param conflicts with the primary key of my database.
To get around this, my idea is to convert all symbol instances of params[:account_id] into params[:account_key_id], so that those params don't conflict with my databases existing account_id field.
How do I do this, and is there a better approach for consuming external JSON API's than what I've described here?
Hash#deep_transform_keys does this:
Returns a new hash with all keys converted by the block operation.
This includes the keys from the root hash and from all
nested hashes and arrays.
So you could do it in one pass with an appropriate block, something like:
data.deep_transform_keys! do |key|
key = key.underscore.to_sym
key = :account_key_id if(key == :account_id)
key
end
You might as well drop the symbolize_keys: true flag to JSON.parse too, you're changing all the keys anyway so don't bother.
If you're doing this sort of thing a lot then you could write a method that takes a key mapping Hash and gives you a lambda for transforming the keys:
def key_mangler(key_map = { })
->(key) do
key = key.underscore.to_sym
key = key_map[key] if(key_map.has_key?(key))
key
end
end
and then say things like:
data.deep_transform_keys!(&key_mangler(:account_id => :account_key_id))
You might want to use a different name than key_mangler of course but that name is good enough to illustrate the idea.
BTW, if you're sending this JSON into the database then you probably don't need to bother with symbol keys, JSON only uses strings for keys so you'll be converting strings to symbols only for them to be converted back to strings. Of course, if you're symbolizing the keys when pulling the JSON out of the database then you'll probably want to be consistent and use symbols across the board.
In addition to the previous answer...
Unfortunately, there is, to my knowledge, no method on Hash that does this in one operation. I've always accomplished this by brute force, as in:
hash[:new_key] = hash[:old_key]
hash.delete(:old_key)
A shortcut for this, suggested in the comment below by "mu is too short", is:
hash[:new_key] = hash.delete(:old_key)
To illustrate in irb:
2.4.1 :002 > h = { foo: :bar }
=> {:foo=>:bar}
2.4.1 :003 > h[:foo_new] = h.delete(:foo)
=> :bar
2.4.1 :004 > h
=> {:foo_new=>:bar}
There is already similar question to this but I am not satisfied with answers since I am trying to do something more complex.
I have web service which provides list/single objects. Objects are Users, Categories, etc. Here is example of object:
<UserObject name="foo" description="bar" category=<Category name="cat1" description="bar"> locations=[<Location id=1>, <Location id=2>] >
In other words objects are somewhat complex and can be arrays of those objects. I am looking for a way to:
Serialize these object to JSON or Hash string
Send them over HTTP
Deserialize them to OpenStruct objects
Service that is serializing objects is not ROR.
App that is receiving and deserializing objects is ROR.
There must be some generic way to do this, I tried using to_json and JSON.parse but it only de-serializes object to one level. So for example above I would get:
<OpenStruct name="foo" description="bar" category="{\"name\"... JSON STRING}" locations="JSON STRING">
Instead of JSON STRINGs I would like to get objects inside object as it was in original.
Ruby: 1.9.3
Thanks
Take a look at the oj gem. It allows you to serialize and deserialize ruby objects to and from json. It also has the benefit of being very fast.
After looking into oj gem and contacting it's creator Peter Ohler, who was very nice and helped, I was able to get desired effect.
require 'oj'
# user instance is nested instance
json_string = Oj.dump user
# send over http
# de-serialize without domain classes (classes created by Oj gem)
user = Oj.load(json_string, { :auto_define => true })
Thanks to #josh-voigts for letting me know about the gem.
I have an rails app where one of the attributes on an object is a data set which consists of an array of x,y coordinates. I am currently storring this in the sql database using the rails serialize helper :
serialize :data, Array
This converts the array to yaml and then stores it in a string field in the sql database. The problem is that our database is getting really big doing this and we need to keep it smaller. Is it possible to serialize to raw binary instead of a string and store in a blob?, this would dramatically reduce the size and help our problem.
I have had a search for a gem to do this, or even a ruby method that will turn an array in to binary data without much help. Any suggestions would be appreciated.
You may be interested in Array.pack and String.unpack methods. See ruby documentation for it: type ri Array.pack
You may want to use a 'packed_data' attribute in your database, then add accessors to pack/unpack it:
def data
packed_data.unpack('....')
end
def data=(v)
self.packed_data = v.pack('....')
end
To make it more useful, you may store the unpacked form in a variable, but you have to remember to clear it when the packed_data attribute changes, like when you call .reload
before_validation :pack_data
UNPACK_FORMAT = '.....' # See ri Array.pack
def data
#data ||= packed_data.unpack(UNPACK_FORMAT)
end
def data=(v)
#data = v
end
def reload(options=nil)
#data = nil
super
end
def pack_data
self.packed_data = self.data.pack(UNPACK_FORMAT)
true # Because we are in a before_.. callback
end
The format of the magic string used to pack/unpack the data depends on the data you have in your array. The documentation will help you to choose the right one.
I believe the format for pack and unpack will be the same, but don't trust me too much. ;)