I want to save information about requests to a certain action in a model named Impression.
I assume it's benificial for the visitor's response time to save this info in an after_filter, e.g:
after_filter :save_impression
private
def save_impression
Impression.create!(ip_address: request.remote_ip, controller_name: params[:controller], action_name: params[:action], referer: request.referer)
end
Can this code be optimized or am I doing it right?
A good solution for that would typically involve using a worker. Anything that is not mission critical to the request and that involves complex computing can be deferred and run later by a background job.
Two common implementations of workers are delayed_job and resque.
For example, with resque, you would have a job class in app/jobs/impression_creation_job.rb, containing something like that :
class ImpressionJob
#queue = :impression
def self.perform( attrs )
Impression.create!( attrs )
end
end
And you can call it in your controller like that :
after_filter :save_impression
private
def save_impression
Resque.enqueue( ImpressionJob, ip_address: request.remote_ip, controller_name: params[:controller], action_name: params[:action], referer: request.referer)
end
This will ensure a fast handling on the request part (it just loads data in redis) and will then be processed by a background process (see resque documentation for how to set it up and start workers).
Please note that this will be useful in your case in only two cases :
Your app is always under heavy load or need especially good response time
You do big computations in Impression#before_create or other callbacks
If not matching one of those conditions, it's probably more effective to just let your impression creation in a controller filter : accessing database has a cost, but not that much that a user will feel when you make a single insertion in database.
This will still run before render. To run after the render/redirect, you need to spawn a separate thread.
See this question
Related
Generally any DB/File IO even external HTTP requests are pretty quick, but I am finding slower ones can hold up all my workers (and memory limits how many Ruby instances I can run), and creating large numbers of threads per worker has other issues (with CPU or memory heavy actions clogging up the system).
Can I have Rails process these actions in an async manner (more like NodeJS) or else introduce threads for that action in some way?
Since I want to respond to the original request, neither workers or just spawning another thread myself seems appropriate, since Rails will ensure the original thread sends a response when it returns from the controller.
def my_action
#data1 = get_data("https://slow.com/data") #e.g. Net::HTTP
#data2 = get_data("https://slow.com/data2?group_id=#{#data["id"]}")
render
end
def my_action
get_data("https://slow.com/data").then do |data1| # e.g. internal thread, not sure on other options
get_data("https://slow.com/data2?group_id=#{data["id"]}").then do |data2|
#data1 = data1
#data2 = data2
render # Appears to have no effect
end
end
# Rails does an implicit "render" on return
end
def my_action
Thread.new do # explicit thread just for this request
#data1 = get_data("https://slow.com/data")
#data2 = get_data("https://slow.com/data2?group_id=#{#data["id"]}")
render
end
end
In a Rails application, you're better off relying on an external process to run background jobs rather than using Ruby Threads.
Sidekiq is a pretty standard gem now for this purpose.
If it takes 10 seconds to process a request, and you want to send your response to the original HTTP request, then you've got to hold open that HTTP connection for 10 seconds. You can't get around that. If your server can handle X HTTP connections, and you have X+1 people making these slow requests... someone is going to get blocked.
There are only three possible solutions:
Figure out a way to process the requests faster. This is ideal, if you can do it.
Don't hold open the HTTP connection. Run a background task (using Sidekiq or similar gem) to do the work. When it's done, send it via websocket, or have the client poll for it. It makes your API more complicated for the client, but as a client I'd rather deal with a little complexity than having my requests blocked and maybe time out.
Scale up your server until it can handle the traffic. This is the "throw money at the problem" solution. I generally disapprove of this, since you'll have to keep throwing more money every time demand grows. But if your organization has more money than dev time, it might work for a while.
Those are your options.
API requests take too long and are costing me money in my Rails integration tests and my application.
I would like to save API responses and then use that data for testing. Are there any good ways to make that happen?
Also, how can I make fewer api calls in production/development? What kind of caching can I use?
If I understand correctly, your rails app is using an external api, like a google/fb/twitter api, this kind of stuff
Caching the views won't work, because it only caches the template, so it doesn't waste time rendering the view again, and it validates that the cache is warm by hashing the data, which the code will still hit the api to verify that the hashes still match
For you the best way is to use a class that does all the api calls, and cache them in rails cache and give that cache a timeout period, because you don't want your cache to be too stale, but in the same time you will sacrifice some accuracy for some money ( like only do a single call every 5, 15, 30 mins, which ever you pick )
Here's a sample of what I have in mind, but you should modify it to match your needs
module ApiWrapper
class << self
def some_method(some_key) # if keys are needed, like an id or something
Rails.cache.fetch("some_method/#{some_key}", expires_in: 5.minutes) do
# assuming ApiLibrary is the external library handler
ApiLibrary.call_external_library(some_key)
end
end
end
end
Then in your code, call that wrapper, it will only contact the external api if the stored value in the cache is expired.
The call will be something like this
# assuming 5 is the id or value you want to fetch from the api
ApiWrapper.some_method(5)
You can read more about caching methods from the rails guide for caching
Update:
I just thought of another way, for your testing (like rspec tests) you could stub the api calls, and this way you'll save the whole api call, unless you are testing the api it self, using to the same api library I wrote above, we can stub the ApiLibrary it self
allow(ApiLibrary).to receive(:some_method).and_return({ data: 'some fake data' })
PS: the hash key data is part of the return, it's the whole hash not just the string.
There is a great gem for this called VCR. It allows you to make a single request and keep response cached, so every time you run the test you will use this saved response.
I would use http://redis.io/ in conjunction with something like jbuilder. So as an example your view would look like:
json.cache! ["cache", "plans_index"] do
json.array! #plans do |plan|
json.partial! plan
end
end
for this controller:
def index
#plans = Plan.all
end
If you have something that is a show page you can cache it like this:
json.cache! ["cache", "plan_#{params["id"]}"] do
json.extract! #plan, :short_description, :long_description,
end
I need to make some slow operations on controller action. But it's not necessary to wait this operation for response rendering.
class ProductController < ActionController
def update
slow_operations()
render json: {status: 'ok'}
end
end
Even I move my code after render in Product#update action, it's not reduces response time.
class ProductController < ActionController
def update
render json: {status: 'ok'}
slow_operations()
end
end
How to force to return complete response before executing of slow operations?
Because of the way Rails works, it's still going to do the actual "rendering" after the action is complete - so, as you found out, moving "render" higher in your action doesn't help. What you need to do is shuffle off the long-running operation into a background process. There's lots of gems to do this, including BackgroundRb, Delayed Job and Sidekiq (my personal favourite, largely because it is multi-threaded, cutting down on the number of processes you need to start, and because of its nice web-based admin/monitoring interface).
There's even a Railscast to get you started for most of these, like this one: http://railscasts.com/episodes/366-sidekiq
You need to integrate something like Resque or Girl Friday to offload the slow process to a background task.
I've developed an application that allows our customers to create their own membership protected websites. My application then connects to an outside API service (customer specific api_key/api_url) to sync/update/add data to this other service. Well, I've had an API wrapper written for this other service that has worked up to this point. However, I'm now seeing very random drops where the connection is nil. Here is how I'm currently using the connection:
I have a xml/rpc connection class
class ApiConnection
attr_accessor :api_url, :api_key, :retry_count
def initialize(url, key)
#api_url = url
#api_key = key
#retry_count = 1
end
def api_perform(class_type, method, *args)
server = XMLRPC::Client.new3({'host' => #api_url, 'path' => "/api/xmlrpc", 'port' => 443, 'use_ssl' => true})
result = server.call("#{class_type}.#{method}", #api_key, *args)
return result
end
end
I also have a module that I can include in my models to access and call the api methods
module ApiService
# Set account specific ApiConnection obj
def self.set_account_api_conn(url, key)
if ac = Thread.current[:api_conn]
ac.api_url, ac.api_key = url, key
else
Thread.current[:api_conn] = ApiConnection.new(url, key)
end
end
########################
### Email Service ###
########################
def api_email_optin(email, reason)
# Enables you to opt contacts in
Thread.current[:api_conn].api_perform('APIEmailService', 'optIn', email, reason)
end
### more methods here ###
end
Then in the application controller I create a new ApIConnection object on every request using a before filter which sets the Thread.current[:api_conn]. This is because I have hundreds of customers each with their own api_key and api_url, using the application at the same time.
# In before_filter of application controller
def set_api_connection
Thread.current[:api_conn] = ApiService.set_account_api_conn(url, key)
end
Well my question is that I've read that using Thread.current is not the most ideal way of handling this, and I'm wondering if this is the cause for the ApiConnection to be nil on random requests. So I would like to know how I could better setup this wrapper.
Answer 1
I'd expect that the problem is the next request coming before the connection has finished, and then the before_filter overwrites the connection for the still ongoing connection. I'd try to stay away from threads. It's easier to fork_off, but there's certain caveats to that as well, especially regarding performance.
I try to move logic like this over to a background job of some sort. A common solution is delayed job https://github.com/collectiveidea/delayed_job that way you don't have to mess with threads and it's more robust and easy to debug. You can then start background jobs to asynchronously sync the service whenever somebody logs in.
#account.delay.optin_via_email(email,user)
This will serialize the account, save it to the job queue, where it will be picked up by delayed job unserialized and the method after delay will be called. You can have any number of background jobs, and even some job queues dedicated to certain types of actions (via using job priorities - let's say two bj for high prio jobs and one dedicated to low prio jobs)
Answer 2
Just make it as an object instead
def before_filter
#api_connection = ApiConnection.new(url, key)
end
then you can use that connection in your controller methods
def show
#just use it straight off
#api_connection.api_perform('APIEmailService', 'optIn', email, reason)
# or send the connection as a parameter to some other class
ApiService.do_stuff(#api_connection)
end
Answer 3
The easiest solution might just be to create the api connection whenever you need it
class User < ActiveRecord::Base
def api_connection
# added caching of the the connection in object
# doing this makes taking a block a little pointless but making methods take blocks
# makes the scope of incoming variables more explicit and looks better imho
# might be just as good to not keep #conn as an instance variable
#conn = ApiConnection.new(url, key) unless #conn
if block_given?
yield(#conn)
else
#conn
end
end
end
that way you can easily just forget about the creation of the connection and have a fresh one handy. There might be performance penalities with this but I suspect that they are insignificant unless there's an extra login request
#user.api_connection do { |conn| conn.optin_via_email(email,user) }
I decided to use the singleton design pattern while creating a view helper class. This got me thinking; will the singleton instance survive across requests? This led to another question, Which variables (if any) survive across web requests and does that change depending on deployment? (Fastcgi, Mongrel, Passenger, ...)
I know that Controller instance variables aren't persisted. I know Constants are persisted (or reloaded?). But I don't know about class variables, instance variables on a class, Eigenclasses, ...
The simple answer is none. Each request is treated as an independent event and no state information is carried over apart from what is stored in the user session and any external databases, caches, or file stores. It is best that you design your application with this in mind and not expect things to persist just because you've set them.
The more complicated story is that some things do persist. For example, you can create a class variable on a controller and this will be carried from one request to the next as you might expect. The catch is that this only applies to the singular instance of that controller, as contained within that process, and will not apply to requests served by other processes. If you need caching, make use of the Rails.cache infrastructure and avoid hacking in your own.
A typical production environment is a complicated, ever-changing thing, where processes are created and destroyed constantly and there is no way to determine in advance which process will ultimately end up serving a particular request. As many deployments involve not only multiple processes on a single machine, but multiple machines, there really is no practical way to create application-wide singleton objects.
The best thing you can do is build a layer on top of the caching engine where your singleton object is merely a wrapper to functions that fetch and write from the cache. This gives you the appearance of a singleton object while maintaining inter-process consistency.
I know that this post is old, but for who is looking a solution, it's possible to use Rails.Cache, like this:
class TestEventsController < ApplicationController
require 'httparty'
##cache = ActiveSupport::Cache::MemoryStore.new(expires_in: 5.minutes)
before_action :get_data, only: [:get]
before_action :get_response, only: [:set]
def get
uri = "https://hooks.zapier.com/hooks/catch/zap_id/"
event_id = event_id_generate()
##cache.write(event_id, "")
result = HTTParty.post(uri.to_str,
:body => {id: event_id, data: #data}.to_json,
:headers => {'content-Type' => 'application/json'})
sleep 2
render json: { 'value': ##cache.read(event_id) }, status: 200
end
def set
##cache.write(#id, #value)
render json: { 'value': ##cache.read(#id) }, status: 200
end
def get_data
#data = params["data"]
end
def get_response
#id = params["id"]
#value = params["value"]
end
def event_id_generate
token = SecureRandom.urlsafe_base64(10, false)
end
end
What I'm doing is receive a request in a route, sending a GET to Zapier, and waiting for the answer in another route. Rails opens a new Thread for each request, so I write in the RAM my data in format 'key: value'
Rails.cache in different threads not caching?
The web is a stateless medium. Unless you purposely save data in a session or pass it in a get or post, each web request starts with a blank slate. Any objects created with the current request are destroyed once the page is delivered to the web browser.