This post is more about a coding approach to a problem rather than a problem itself (for a change!).
I have a number of projects that i'm working on the moment that require me to gather sales data from a number of disperate sources.
The data from each vendor is almost always accessed and structured in different ways; best case a nice valid JSON reponse, worst case I screen scrape the data.
As each vendor's source data is so different i've decided that a dedicated Rails-api App that feeds json data to a master sales-data App via it's API is the way forward.
I did look at using Sinatra for each vendor app but my knowledge is with Rails so I can get things done much quicker. I feel a dedicated app for each vendor app is the right approach as these can be maintained independently and should a vendor decide to start feeding their data themselves, I (or another developer) can easily swap things over without having to delve into one massive monothlic sales data gathering app, however, do say if you think dedicated apps doesn't make much sense.
So, as a simple stripped down example, each vendor app is structured around a class like this. Currently i'm just calling the methods via console, but will automate via rake tasks and background workers eventually.
class VendorA < ActiveRecord::Base
def self.get_report
# Uses Mechanize Gem to fetch a report in CSV format
# returns report
end
def self.save_report(report)
# Takes the report from the get_report method and saves it, currently to the app root but eventually this will be S3
# returns the local_report
end
def self.convert_report_to_json(local_report)
# Reads the local report, iterates through it grabbing the fields required for the master sales-data app and constructs a JSON repsonse
# returns a valid JSON repsonse called json_data
end
def self.send_to_master_sales_api(json_data)
# As you can see here I take the repsonse from convert_report_to_json and post it to my sales data App
require 'rest-client'
RestClient.post('http://localhost:3000/api/v1/sales/batch', json_data)
end
end
This works fine with the send_to_master_sales_api method doing what is expected. I haven't yet tested this beyond approx 1000 data objects/lines though.
On the receiving end, at the master sales-data App, things look like this:
module Api
module V1
class SalesController < ApplicationController
before_filter :restrict_access
skip_before_filter :verify_authenticity_token
respond_to :json
def batch
i = 0
clientid = params[:clientid]
objArray = JSON.parse(params[:sales])
objArray.each do |sale|
sale = Sale.new(sale)
sale.clientid = clientid #Required / Not null
sale.save
i +=1
end
render json: {message: "#{i} Sales lines recevied"}, status: 202
end
private
def restrict_access
api_key = ApiKey.find_by_api_key_and_api_secret(params[:api_key],params[:api_secret])
render json: { message: 'Invalid Key - Access Restricted' }, status: :unauthorized unless api_key
end
end
end
end
So, my main question is really about the volume of JSON data you think this approach can handle. As I mentioned above, i've tested with around 1000 lines/objects and it worked fine. But should I start having to process data with one of the vendor apps where i'm approaching say, 10,000, 100,000 or even 1,000,000 lines/objects per day from each source what should I be thinking about in terms of the above two apps being able to cope. Is there something like find_in_batches I could be using to ease the load of receiving the data? Currently I plan to write the master sales-data records to a Postgres DB. While i've limited experience of NoSQL, would writing to MongoDB or similar speed things up at the receiving end?
I realise this isn't a particularly direct question, but I would really appreicate input and thoughts from those with experience of this kind of thing.
Thanks in advance!
Related
I am trying to add the User Appearances example (from the Rails' Guide : https://guides.rubyonrails.org/action_cable_overview.html#example-1-user-appearances ) in my app but I don't understand this part :
# app/channels/appearance_channel.rb
class AppearanceChannel < ApplicationCable::Channel
def subscribed
current_user.appear
end
def unsubscribed
current_user.disappear
end
def appear(data)
current_user.appear(on: data['appearing_on'])
end
def away
current_user.away
end
end
If someone has an explanation for the following sentence : "That appear/disappear API could be backed by Redis, a database, or whatever else." (Just above this part of code in the Rails' Guide).
I try several options, as adding a method "appear" in my model User which change on "true" a database value from my model User, but the subscribed definition call current_user.appear and then the appear definition call current_user.appear(with_param) generates a conflict ...
There is probably something I don't understand but I don't see exactly what is it ...
Thank you very much for your answers.
The sentence about "appear/disappear API backing" - means that ActionCable does not care where and how you are storing and handling users statuses - you may store only a flag or more data in database (like last seen chatroom, last seen time etc.), you may store similar data in redis or any other place you like.
(un)subscribed methods are caller by ActionCable itself upon user (dis)connection to that channel(usually this happens on page load and after navigating away/closing - and while page is open in browser it does not necessary mean that user is actually near their device), while appear/away are actions that are called from clientside js via calling perform("action_name_here") on the channel.
Example assumes that clientside code will detect user presence and send updates.
So I have this plan to use some raspberry PI Zero's to send data to a Rails backend API i'll be designing. This is to get more experience with Rails and just for a fun side project.
Im going to be reading sensor data on the Raspberry Pi's and sending them to my REST API (the data will be stored and then I can go to each "sensors" page on a dashboard and see historical values/timeline of temperature or whatever sensor I decide to use).
Im mostly familiar with HTTP requests...but usually only on the same server. And this made me realize I don't really understand "HOW" they work. (I just know that with Forms I need to use POST/GET for normal pages/etc....,and just general routing involving that).
That being said, I know on the Rails side i'll need to receive the data Probably via a POST request (Sort of like a form being submitted)...but im not sure how I should be sending it? As in what format? How should it be constructed?
I thought this was simple, but then the more I thought about it the more I realized I didn't know. I know that I need to send the data every x intervals (Via a CRON job running a python script) so i'd be sending a POST request to the API right? I guess I understand that a Form on a site in sending a POST request but what exactly are we requesting? And what would be returned from a Form POST request? Just a 200 status ok?
What about in the case of these devices: Sending a POST request to send their sensor data to be saved on the backend, should the server respond back to them?
Thanks, and sorry for the sorta scattered question. Im realizing I know a lot less than when I started learning rails. Just trying to clear up my understanding on http requests.
Send the data as JSON. Building the rails app is not really different from building any other JSON API.
Lets say you set your models up like so:
# app/models/measurement.rb
class Measurement
belongs_to :unit
end
# app/units/unit.rb
class Unit
has_many :measurements
end
You can use a token based authentication scheme to authenticate each PI unit. Or you can use shared secret such as a MAC address of the unit.
You can register each PI unit in the rails app via the rails console or create the routes and controllers for it in the rails app if you want to be thorough.
If you don't want to manually enter the rails db id for each PI unit you can create a "discovery route" where the PI sends its MAC address and gets an id.
To register measurements you just have send a POST request:
# /config/routes.rb
resources :units do
resources :measurements, module: :units, only: [:create, :index]
end
# /app/controllers/units/measurements_controller.rb
module Units
class MeasurementsController
before_action :set_unit
before_action :authenticate_unit!
# POST /units/:unit_id/measurements
def create
#measurement = #unit.measurements.new(measurement_params)
if #measurement.save
head :created, location: #measurement
else
head :unprocessable_entity
end
end
# GET /units/:unit_id/measurements
def index
render json: #unit.measurements
end
private
def authenticate_unit!
# #todo check auth token in header.
# should raise an exception and return 401 unauthorized if not authenticated
end
def set_unit
#unit = Unit.find(params[:id])
end
def measurement_params
params.permit(:temperature, :foo, :bar)
end
end
end
The Pi would send the data to POST /units/:unit_id/measurements and get a 201 - Created response or 422 - Bad entity. Sending a Location header is really optional since the PI is probally not going to do anything with the response.
How exactly the payload is formatted is up to you as an author. This example just uses a "flat" JSON object:
{
temperature: 3,
foo: 2,
bar: 3
}
As i mentioned on the question title, I want to know the best approach to get this to work and i'll need and example ( very simple one as the follow: )
Let's say i have an API which has 1 controller and 1 action for example simplicity
root 'main#index'
and inside the index action i have
def index
#date = Data.today
end
Then i have another rails app which will work for front-end rendering
How can i pass this #date as JSON from the API to the other app to render it ?
Should i have same controller on the other app ?
How can i connect and send http request and receive response ?
Thanks in advance
For such a simple example, you can do something as simple as:
def index
#date = Date.today
respond_to do |format|
format.json #date
end
end
However, you're most likely going to want to deal with more complicated JSON responses, so before long you'll probably want to use something like the Jbuilder gem or ActiveModel Serializers (my preferred approach).
On the other end, your front-end will need to make an HTTP GET request. Lots of ways (and gems) to do this, but one common approach is just to use the built in Net::HTTP class.
require 'net/http'
url = URI.parse('http://backend.dev/main/index')
request = Net::HTTP::Get.new(url.to_s)
response = Net::HTTP.start(url.host, url.port) do |http|
http.request(request)
end
raise response.body.inspect
In your situation, a better approach might be to use the Active Resource gem. This gem allows you to create models that are backed by a REST API rather than a database. For example, if your API app provides basic Create-Read-Update-Destroy actions for a particular model (let's call it Widget) at the following URLs:
GET http://backend.dev/widget # listing of widgets
GET http://backend.dev/widget/1 # Read for widget id: 1
POST http://backend.dev/widget # Create new widget
UPDATE http://backend.dev/widget/1 # Update widget id: 1
DELETE http://backend.dev/widget/1 # Destroy widget id: 1
then in your front-end app you could declare an Active Resource like this:
class Widget < ActiveResource::Base
self.site = "http://backend.dev"
end
which will auto-magically access all of those methods in your API, and behave much like a regular Active Record model. That way, you basically design your front-end app like a "normal" rails app, but using ActiveResource-based models in place of ActiveRecord.
I would note, however, that a more common thing to do these days would be to build your API in Rails, and build your front-end with client-side Javascript, using something like JQuery or Angular to make requests from the API. I'm not sure what you're gaining by splitting API and front-end, where both of them are Rails apps - unless you've got a compelling reason, I'd just build one Rails app that handles both API and front-end, or build a Rails API + Angular (or similar) front-end.
Disclaimer: I'm doing something which may qualify for Code Smell of 2015 Award. Using rails 4.2, no javascript or anything like that.
I have a form into which users input their data. With this data I call a third-party API which will remain nameless. :)
I have no model, I'm not persisting anything. (Part of a larger app, not a one-pager.) Thus when faced with presenting the user with the response, I find myself stuck on how to render the data properly into a view. The response contains an array of hashes which I obviously intend to present the user.
I render the form into widgets/new, etc, create and process the request, etc, but then what?
I thought maybe I could make use of decorators to do my dirty work but not sure how to actually get the user off to the respective view. I don't care which view. Call it a widget_path.
WidgetsController < ApplicationController
def new
render :new
end
def create
# preparing request
...
# data = response, each_serializer, WidgetSerializer, root: false
# data = WidgetDecorator.new(render_serialized(response, WidgetSerializer))
# #data = WidgetDecorator.new(JSON.parse(response))
# redirect_to ??_path ... and take your #data with you
end
end
What do I do?
Your idea of Model is unfortunately corrupted by Rails itself (sorry).
A model is business logic not an ActiveRecord::Base (not necessarily). Controller methods shouldn't be big, ~5 lines long is probably the maximum with a ~100 lines max per controller file. Try to stick with this and it will automatically correct good chunck of code smells.
Anyway, you may handle this with a Model, as a PORO (plain old ruby object).
class MyApiResponse
attr_reader :myapikey
attr_reader :whatever
def initialize(myapikey, whatever)
#myapikey = myapikey
#whatever = whatever
end
def get
#_response ||= JSON.parse(run_api_stuff(myapikey))
end
end
So in controller you would do something like
def create
myapiresponse = MyApiResonse.new(myapikey, whatever)
#response = myapiresponse.get
end
Last but not least, you can't pass what you obtained through the API in the redirect. You are subject to HTTP limits so, you have a limit on GET params size, a limit on session and you can't redirect to a POST. You have 3 options
Best is store last api request for given user in the database and fetch it back through an ID (which will travel through the redirect)
Store it in session if request is really small (and you must ensure it is small!)
Perform the API request again after the redirect, horrible. Otherwise perform the API request only after redirect, not sure if this is an option though
API requests take too long and are costing me money in my Rails integration tests and my application.
I would like to save API responses and then use that data for testing. Are there any good ways to make that happen?
Also, how can I make fewer api calls in production/development? What kind of caching can I use?
If I understand correctly, your rails app is using an external api, like a google/fb/twitter api, this kind of stuff
Caching the views won't work, because it only caches the template, so it doesn't waste time rendering the view again, and it validates that the cache is warm by hashing the data, which the code will still hit the api to verify that the hashes still match
For you the best way is to use a class that does all the api calls, and cache them in rails cache and give that cache a timeout period, because you don't want your cache to be too stale, but in the same time you will sacrifice some accuracy for some money ( like only do a single call every 5, 15, 30 mins, which ever you pick )
Here's a sample of what I have in mind, but you should modify it to match your needs
module ApiWrapper
class << self
def some_method(some_key) # if keys are needed, like an id or something
Rails.cache.fetch("some_method/#{some_key}", expires_in: 5.minutes) do
# assuming ApiLibrary is the external library handler
ApiLibrary.call_external_library(some_key)
end
end
end
end
Then in your code, call that wrapper, it will only contact the external api if the stored value in the cache is expired.
The call will be something like this
# assuming 5 is the id or value you want to fetch from the api
ApiWrapper.some_method(5)
You can read more about caching methods from the rails guide for caching
Update:
I just thought of another way, for your testing (like rspec tests) you could stub the api calls, and this way you'll save the whole api call, unless you are testing the api it self, using to the same api library I wrote above, we can stub the ApiLibrary it self
allow(ApiLibrary).to receive(:some_method).and_return({ data: 'some fake data' })
PS: the hash key data is part of the return, it's the whole hash not just the string.
There is a great gem for this called VCR. It allows you to make a single request and keep response cached, so every time you run the test you will use this saved response.
I would use http://redis.io/ in conjunction with something like jbuilder. So as an example your view would look like:
json.cache! ["cache", "plans_index"] do
json.array! #plans do |plan|
json.partial! plan
end
end
for this controller:
def index
#plans = Plan.all
end
If you have something that is a show page you can cache it like this:
json.cache! ["cache", "plan_#{params["id"]}"] do
json.extract! #plan, :short_description, :long_description,
end