I have a url which has been supplied which data is updated every 30 mins and wondering if i can save the data to my database as it updates? I'm using Rails 4.2.0.
There a 10 url's all up, each with a different unit number, which needs to be reference to be able to call each data for each unit.
URL structure
http://sitename/cgi-bin/site=1
JSON structure
{"status"=>"ok", "data"=>[{"2014-08-11 11:00:00"=>14.9},{"2014-08-11 11:30:00"=>15.1}]}
With your json response, it can be done with something like this:
json = JSON.parse('{"status"=>"ok", "data"=>[{"2014-08-11 11:00:00"=>14.9},{"2014-08-11 11:30:00"=>15.1}]}') #string representing your json
json['data'].each do |element|
element.each do |key, value|
Model.create(date: key, number: value) # This Model is the name of your model
end
end
If you let me suggest you something, You can send json as:
{"status"=>"ok", "data"=>[{"date" => "2014-08-11 11:00:00", "number" =>14.9},{...}]}
So you can access data like: element['date'] and element['number']
I think you should go for crone jobs. Refer this Whenever gem that provides a clear syntax for writing and deploying cron jobs and you would save your data on every 30 mins into the database.
Related
I'm looking to truncate and save JSON data from an external URL at a certain time (10pm) every day. The URL requires a login which I have access to, but not sure how best to implement this.
The JSON is structured as such:
{
"Region": [{"id":"1","region":"South"}],
"Agent": [{"id":"1","first_name":"Tim","last_name":"Jones"}]
}
I have created the model's Region and Agent to store the data in, matching the fields above.
This is what I have so far in Region model
json = JSON.parse('http://....')
json['Region'].each do |data|
Region.create(
id: data['id'],
region: data['region']
)
My questions are:
Is this possible via the model, or do I have to place this in a
controller
How do I go about truncating and saving at a certain time?
To get the data from an external url, you'll need to use an http client such as Faraday or Httparty
Once you got your json data you can parse and manipulate it however you want.
To automate it at 10pm every day you need to schedule a background job using something like ActiveJob or Sidekiq.
In our Rails app, the user (or we on his behalf) load some data or even insert it manually using a crud.
After this step the user must validate all the configuration (the data) and "accept and agree" that it's all correct.
On a given day, the application will execute some tasks according the configuration.
Today, we already have a "freeze" flag, where we can prevent changes in the data, so the user cannot mess the things up...
But we also would like to do something like hash the data and say something like "your config is frozen and the hash is 34FE00...".
This would give the user a certain that the system is running with the configuration he approved.
How can we do that? There are 7 or 8 tables. The total of records created would be around 2k or 3k.
How to hash the data to detect changes after the approval? How would you do that?
I'm thinking about doing a find_by_user in each table, loop all records and use some fields (or all) to build a string and hash it at the end of the current loop.
After loop all tables, I would have 8 hash strings and would concatenate and hash them in a final hash.
How does it looks like? Any ideas?
Here's a possible implementation. Just define object as an Array of all the stuff you'd like to hash :
require 'digest/md5'
def validation_hash(object, len = 16)
Digest::MD5.hexdigest(object.to_json)[0,len]
end
puts validation_hash([Actor.first,Movie.first(5)])
# => 94eba93c0a8e92f8
# After changing a single character in the first Actors's biography :
# => 35f342d915d6be4e
I am setting up stripe connect with the example from https://github.com/rfunduk/rails-stripe-connect-example and am running into a problem using serialize to store stripe_account_status which should be stored as an array.
This is how it should be stored (Generated from the above example link)
{"details_submitted"=>false, "charges_enabled"=>true, "transfers_enabled"=>false, "fields_needed"=>["legal_entity.first_name", "legal_entity.last_name", "legal_entity.dob.day", "legal_entity.dob.month", "legal_entity.dob.year", "legal_entity.address.line1", "legal_entity.address.city", "legal_entity.address.postal_code", "bank_account"], "due_by"=>nil}
And this is how my application is storing it
{:details_submitted=>false, :charges_enabled=>true, :transfers_enabled=>false, :fields_needed=>["legal_entity.first_name", "legal_entity.last_name", "legal_entity.dob.day", "legal_entity.dob.month", "legal_entity.dob.year", "legal_entity.address.line1", "legal_entity.address.city", "legal_entity.address.postal_code", "bank_account"], :due_by=>nil}
As far as I am concerned everything is set up the same. The only difference is that the first example uses
serialize :stripe_account_status, JSON
and my app just has
serialize :stripe_account_status
The reason for this is that when I add JSON I this error:
JSON::ParserError - 795: unexpected token at '':
I have tried finding out the JSON error including changing the config/initializers/cookies_serializer.rb to use :hybrid but this is giving me the same error.
Could someone point me into the right direction of either fixing the JSON issue OR finding a way to make sure the stripe_account_status is stored as an array correctly.
Below is the methods used to store the array:
if #account
user.update_attributes(
currency: #account.default_currency,
stripe_account_type: 'managed',
stripe_user_id: #account.id,
secret_key: #account.keys.secret,
publishable_key: #account.keys.publishable,
stripe_account_status: account_status
)
end
def account_status
{
details_submitted: account.details_submitted,
charges_enabled: account.charges_enabled,
transfers_enabled: account.transfers_enabled,
fields_needed: account.verification.fields_needed,
due_by: account.verification.due_by
}
end
Thanks I really appreciate any direction you could point me!
When you ask Rails to serialize an attribute on a model, it will default to storing the object as YAML string.
You can ask Rails to serialize differently, as you have noticed by providing a class to do the serialization e.g
serialize :stripe_account_status, JSON
The reason why this isn't working when you add it is because you presumably already have a record in the database using the YAML and so Rails can't parse this as a valid JSON string when reading from the DB. If it's just development data that you don't need, you can delete the records and then use JSON, otherwise you will need to convert the current YAML strings to JSON.
Rails will also symbolize the keys of a hash when parsing a serialized string in the database. This is the only difference between the hashes in your question and shouldn't matter in practise. Should you need String keys for some reason, you can use the #stringify_keys method on the hash provided by Rails.
I am using an API for movies, for the most part I use a gem, but in order to get certain information, I have to use RestClient.get methods.
response = RestClient.get "http://api.themoviedb.org/3/movie/8699/keywords", headers
puts response
If I run this code, it returns this JSON "extract"
{"id":8699,"keywords":[{"id":917,"name":"journalism"},{"id":4411,"name":"sexism"},{"id":6198,"name":"ladder"},{"id":8531,"name":"panda"},{"id":18290,"name":"1970s"},{"id":18298,"name":"tv show in film"},{"id":41390,"name":"mustache"},{"id":165288,"name":"misogynist"},{"id":167241,"name":"newsroom"},{"id":167250,"name":"teleprompter"},{"id":167252,"name":"what happened to epilogue"},{"id":167254,"name":"gang warfare"},{"id":167256,"name":"multiple cameos"},{"id":179430,"name":"aftercreditsstinger"},{"id":179431,"name":"duringcreditsstinger"},{"id":185281,"name":"news spoof"}]}
Now what I need to do is be able to do is turn the above into a rails readable piece of code, so that I can then put it into a database.
More specifically, I would like to take the above information and be able to execute it as such
keywords.each do |keyword|
Keyword.create(name: keyword.name, tmdbid: keyword.id)
end
So that it creates a new Keyword based on each keyword and its name & id based on the JSON File
You can use JSON.parse to turn it into a hash
keywords = JSON.parse(response)["keywords"]
It will be faster if you create all keywords at once by first create correct array of values to be saved. Here is what I will do
keywords = JSON.parse(response)["keywords"].map do |k|
{ name: k["name"], tmdbid: k["id"] }
end
Keyword.create(keywords) #one SQL statement only
Regards
I have a rails app which gets a response from World Weather Online API. I'm using the rest-client gem and the response is in JSON format.
I parse the response using:
parsed_response = JSON.parse(response)
Where parsed_response is obviously a hash.
The data I need are strings inside a hash inside an array inside a hash inside another array inside another hash inside another hash.
The inner-most nested hashes are inside ["hourly"], an array of 8 hashes, each with 20 keys, possessing string values of various weather parameters. Each of these hashes in the array is a different time of day (the forecast is three-hourly, 3*8 = 24hours).
So, for example, if I want the swell height in metres at 9pm, I find it with the following call:
#swell_height = parsed_data["data"]["weather"][0]["hourly"][7]["swellHeight_m"]
Where the 7th element in the array correspond to "time" => "2100"
While I can definitely work with this, I'm curious as to whether there is a more straightforward method of accessing my data, like if it was a database table, I could use active record, something like:
#swell_height = parsed_data.swellHeight_m.where(:time => "2100")
You may want to look at JSONPath. It does exactly what you need. Its syntax is very similar to XPath, but JSONPath works with JSON data (as obvious).
There is a Ruby implementation: https://github.com/joshbuddy/jsonpath
I personally use it in every project where I need to test JSON responses.