I want to parse the JSONP data and save that data into my data base .
I have my jsonp url lets say this >>http://a0.awsstatic.com/pricing/1/ec2/pricing-data-transfer-with-regions.min.js?callback=callback&_=1409722308446
Its not normal json object so how can i parse this json in ruby/ruby on rails and save the data into my database. I want to save the data in table having filed lets say region , name ,type, price .
What are the ways to do the same.
JSONP is a work around for the same origin policy in client side JavaScript. It isn't required for processing data in Ruby. Firstly I would try to find the data available in plain JSON, without the callback.
If, however, that URL is absolutely the only one you can call for this data, then I would strip away the callback using a regular expression and then parse the plain JSON data.
If you have already loaded the contents of that file into a variable called jsonp, then something like this might work:
require 'net/http'
require 'json'
uri = URI.parse('http://a0.awsstatic.com/pricing/1/ec2/rhel-od.min.js?callback=callback&_=1409731896563')
jsonp = Net::HTTP.get(uri)
jsonp.gsub!(/^.*callback\(/, '') # removes the comment and callback function from the start of the string
jsonp.gsub!(/\);$/, '') # removes the end of the callback function
jsonp.gsub!(/(\w+):/, '"\1":')
hash = JSON.parse(jsonp)
then hash will have the parsed JSON from the response.
Please note, this code has no error handling and should be treated as a starting point for your final solution.
[edit] Added the third gsub to change the JavaScript style keys to JSON style keys. This works in this case because the keys all appear to be simple enough to fit that regex.
[edit2] Added way to load the JSONP with Net::HTTP
If what you're really trying to do is parse the Amazon price list JS file into Ruby, there is a better (read: safer--no evals) way to do it:
require 'net/http'
require 'json'
JSON.parse(
Net::HTTP.get(
URI.parse('http://a0.awsstatic.com/pricing/1/ec2/rhel-od.min.js')
).split('callback(')[1].sub(');', '').gsub(/(\w+):/, '"\1":')
)
As the data is as a raw JS object form, it's actually valid Ruby:
require 'json'
data = '{key: "value", key2: 33}'
obj = eval data
obj[:key]
#=> "value"
obj.to_json
# => "{\"key\":\"value\",\"key2\":33}"
You must trust your source completely though - this would allow the running of abstract Ruby code if the data were tampered with - which could in turn run abstract command-line terminal code, using the back-tick operators. This could delete your hard drive for example.
Related
Input is a params string (Input format cannot be changed)
something=1,2,3,4,5&something_else=6,7,8
Expected Output:
something=1,2,3,4,5&something_else=6,7,8
What I am doing:
params = 'something=1,2,3,4,5'
CGI::parse(params)
CGI.unescape(CGI.parse(params).to_query)
And I am getting this as output:
something[]=1,2,3,4,5
When I did CGI::parse(params)
I am getting this : {"something"=>["1,2,3,4,5"]}
which is wrong because it is not an array, something is a string which is "1,2,3,4,5" but it is being converted as array when I did CGI parse.
The reason I need to do CGI parse is because I need to manipulate the url PARAMS.
Is there any other possible way where I can convert it in the right way and maintain the params format?
The CGI module is a complete dinosaur and should probably be thrown in the garbage because of how bad it is, but for some reason it persists in the Ruby core. Maybe some day someone will refactor it and make it workable. Until then, skip it and use something better like URI, which is also built-in.
Given your irregular, non-compliant query string:
query_string = 'something=1,2,3,4,5&something_else=6,7,8'
You can handle this by using the decode_www_form method which handles query-strings:
require 'uri'
decoded = URI.decode_www_form(query_string).to_h
# => {"something"=>"1,2,3,4,5", "something_else"=>"6,7,8"}
To re-encode it you just call encode_www_form and then force unescape to undo what it's correctly doing to handle the , values:
encoded = URI.unescape(URI.encode_www_form(decoded))
# => "something=1,2,3,4,5&something_else=6,7,8"
That should get the effect you want.
I am setting up stripe connect with the example from https://github.com/rfunduk/rails-stripe-connect-example and am running into a problem using serialize to store stripe_account_status which should be stored as an array.
This is how it should be stored (Generated from the above example link)
{"details_submitted"=>false, "charges_enabled"=>true, "transfers_enabled"=>false, "fields_needed"=>["legal_entity.first_name", "legal_entity.last_name", "legal_entity.dob.day", "legal_entity.dob.month", "legal_entity.dob.year", "legal_entity.address.line1", "legal_entity.address.city", "legal_entity.address.postal_code", "bank_account"], "due_by"=>nil}
And this is how my application is storing it
{:details_submitted=>false, :charges_enabled=>true, :transfers_enabled=>false, :fields_needed=>["legal_entity.first_name", "legal_entity.last_name", "legal_entity.dob.day", "legal_entity.dob.month", "legal_entity.dob.year", "legal_entity.address.line1", "legal_entity.address.city", "legal_entity.address.postal_code", "bank_account"], :due_by=>nil}
As far as I am concerned everything is set up the same. The only difference is that the first example uses
serialize :stripe_account_status, JSON
and my app just has
serialize :stripe_account_status
The reason for this is that when I add JSON I this error:
JSON::ParserError - 795: unexpected token at '':
I have tried finding out the JSON error including changing the config/initializers/cookies_serializer.rb to use :hybrid but this is giving me the same error.
Could someone point me into the right direction of either fixing the JSON issue OR finding a way to make sure the stripe_account_status is stored as an array correctly.
Below is the methods used to store the array:
if #account
user.update_attributes(
currency: #account.default_currency,
stripe_account_type: 'managed',
stripe_user_id: #account.id,
secret_key: #account.keys.secret,
publishable_key: #account.keys.publishable,
stripe_account_status: account_status
)
end
def account_status
{
details_submitted: account.details_submitted,
charges_enabled: account.charges_enabled,
transfers_enabled: account.transfers_enabled,
fields_needed: account.verification.fields_needed,
due_by: account.verification.due_by
}
end
Thanks I really appreciate any direction you could point me!
When you ask Rails to serialize an attribute on a model, it will default to storing the object as YAML string.
You can ask Rails to serialize differently, as you have noticed by providing a class to do the serialization e.g
serialize :stripe_account_status, JSON
The reason why this isn't working when you add it is because you presumably already have a record in the database using the YAML and so Rails can't parse this as a valid JSON string when reading from the DB. If it's just development data that you don't need, you can delete the records and then use JSON, otherwise you will need to convert the current YAML strings to JSON.
Rails will also symbolize the keys of a hash when parsing a serialized string in the database. This is the only difference between the hashes in your question and shouldn't matter in practise. Should you need String keys for some reason, you can use the #stringify_keys method on the hash provided by Rails.
So I have this currency .xml file:
http://www.ecb.int/stats/eurofxref/eurofxref-daily.xml
Now, I am wondering, how can I make my rails application read it? Where do I even have to put it and how do I include it?
I am basically making a currency exchange rate calculator.
And I am going to make the dropdown menu have the currency names from the .xml table appear in it and be usable.
First of all you're going to have to be able to read the file--I assume you want the very latest from that site, so you'll be making an HTTP request (otherwise, just store the file anywhere in your app and read it with File.read with a relative path). Here I use Net::HTTP, but you could use HTTParty or whatever you prefer.
It looks like it changes on a daily basis, so maybe you'll only want to make one HTTP request every day and cache the file somewhere along with a timestamp.
Let's say you have a directory in your application called rates where we store the cached xml files, the heart of the functionality could look like this (kind of clunky but I want the behaviour to be obvious):
def get_rates
today_path = Rails.root.join 'rates', "#{Date.today.to_s}.xml"
xml_content = if File.exists? today_path
# Read it from local storage
File.read today_path
else
# Go get it and store it!
xml = Net::HTTP.get URI 'http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml'
File.write today_path, xml
xml
end
# Now convert that XML to a hash. Lots of ways to do this, but this is very simple xml.
currency_list = Hash.from_xml(xml_content)["Envelope"]["Cube"]["Cube"]["Cube"]
# Now currency_list is an Array of hashes e.g. [{"currency"=>"USD", "rate"=>"1.3784"}, ...]
# Let's say you want a single hash like "USD" => "1.3784", you could do a conversion like this
Hash[currency_list.map &:values]
end
The important part there is Hash.from_xml. Where you have XML that is essentially key/value pairs, this is your friend. For anything more complicated you will want to look for an XML library like Nokogiri. The ["Envelope"]["Cube"]["Cube"]["Cube"] is digging through the hash to get to the important part.
Now, you can see how sensitive this will be to any changes in the XML structure, and you should make the endpoint configurable, and that hash is probably small enough to cache up in memory, but this is the basic idea.
To get your list of currencies out of the hash just say get_rates.keys.
As long as you understand what's going on, you can make that smaller:
def get_rates
today_path = Rails.root.join 'rates', "#{Date.today.to_s}.xml"
Hash[Hash.from_xml(if File.exists? today_path
File.read today_path
else
xml = Net::HTTP.get URI 'http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml'
File.write today_path, xml
xml
end)["Envelope"]["Cube"]["Cube"]["Cube"].map &:values]
end
If you do choose to cache the xml you will probably want to automatically clear out old versions of the cached XML file, too. If you want to cache other conversion lists consider a naming scheme derived automatically from the URI, e.g. eurofxref-daily-2013-10-28.xml.
Edit: let's say you want to cache the converted xml in memory--why not!
module CurrencyRetrieval
def get_rates
if defined?(##rates_retrieved) && (##rates_retrieved == Date.today)
##rates
else
##rates_retrieved = Date.today
##rates = Hash[Hash.from_xml(Net::HTTP.get URI 'http://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml')["Envelope"]["Cube"]["Cube"]["Cube"].map &:values]
end
end
end
Now just include CurrencyRetrieval wherever you need it and you're golden. ##rates and ##rates_retrieved will be stored as class variables in whatever class you include this module within. You must test that this persists between calls in your production setup (otherwise fall back to the file-based approach or store those values elsewhere).
Note, if the XML structure changes, or the XML is unavailable today, you'll want to invalidate ##rates and handle exceptions in some nice way...better safe than sorry.
I have a rails app which gets a response from World Weather Online API. I'm using the rest-client gem and the response is in JSON format.
I parse the response using:
parsed_response = JSON.parse(response)
Where parsed_response is obviously a hash.
The data I need are strings inside a hash inside an array inside a hash inside another array inside another hash inside another hash.
The inner-most nested hashes are inside ["hourly"], an array of 8 hashes, each with 20 keys, possessing string values of various weather parameters. Each of these hashes in the array is a different time of day (the forecast is three-hourly, 3*8 = 24hours).
So, for example, if I want the swell height in metres at 9pm, I find it with the following call:
#swell_height = parsed_data["data"]["weather"][0]["hourly"][7]["swellHeight_m"]
Where the 7th element in the array correspond to "time" => "2100"
While I can definitely work with this, I'm curious as to whether there is a more straightforward method of accessing my data, like if it was a database table, I could use active record, something like:
#swell_height = parsed_data.swellHeight_m.where(:time => "2100")
You may want to look at JSONPath. It does exactly what you need. Its syntax is very similar to XPath, but JSONPath works with JSON data (as obvious).
There is a Ruby implementation: https://github.com/joshbuddy/jsonpath
I personally use it in every project where I need to test JSON responses.
I have a large amount of nested data formatted in JSON.
I would like to select for a single element:
{"data": [
{"id": "123456","from": {"name": "Jason Wade","id":
"654321"},"message":
"http://www.youtube.com/watch?v=ZfXHAqBRIEk"
...
How do I efficiently select for
a single element, say "message"?
Is there a simple recommended
method for this? Say convert it to
an array, or something?
Could you point me to some reading for
parsing JSON?
Thanks!
If you are using Rails, see the question in my comment. If you are using pure Ruby, there is a gem called json. Install the gem, and use like this
require 'rubygems'
require 'json'
url = "www.example.com/api?format=json"
response = Net::HTTP.get_response(URI.parse(url))
data = response.body
result = JSON.parse(data)
JSON format and ruby data types have similarity and in your example, the result will be a hash like this
{"data"=>[{"message"=>"http://www.youtube.com/watch?v=ZfXHAqBRIEk", "from"=>{"name"=>"Jason Wade", "id"=>"654321"}, "id"=>"123456"}]}