The rails app (4) calls an external API using HTTParty. The API is read only. Caching is required as the data does not change often (24h) and the API allow only a limited number of calls per hour.
I guess I need some kind of hash based cache where I will use "params/sent/to/the/api" as key.
Rails tools for caching seems only to be for pages,fragments or SQL.
What should I do to cache calls to an external API?
It'll be something like this. Basically, the Rails.cache.fetch call will wrap your API call. It won't hit the API unless the cache has expired.
class Results
def get(url, params)
Rails.cache.fetch([url, params], :expires => 1.hour) do
HTTParty.get('url/to/api')
end
end
end
Be sure you have a cache set in your environment. Memcache works great for this sort of thing.
Related
ok, first time making an API!
My assumption is that if data needs to be stored on the back end such that it persists across multiple API calls, it needs to be 1) in cache or 2) in a Database. is that right?
I was looking at the code for the gem "google-id-token". it seems to do just what i need for my google login application. My front end app will send the google tokens to the API with requests.
the gem appears to cache the public (PEM) certificates from Google (for an hour by default) and then uses them to validate the Google JWT you provide.
but when i look at the code (https://github.com/google/google-id-token/blob/master/lib/google-id-token.rb) it just seems to fetch the google certificates and put them into an instance variable.
am i right in thinking that the next time someone calls the API, it will have no memory of that stored data and just fetch it again?
i guess its a 2 part question:
if i put something in an #instance_variable in my API, will that data exist when the next API call comes in?
if not, is there any way that "google-id-token" is caching its data correctly? maybe HTTP requests are somehow cached on the backend and therefore the network request doesnt actually happen over and over? can i test this?
my impulse is to write "google-id-token" functionality in a way that caches the google certs using MemCachier. but since i dont know what I'm doing i thought i would ask.? Maybe the gem works fine as is, i dont know how to test it.
Not sure about google-id-token, but Rails instance variables are not available beyond single requests and views (and definitely not from one user's session to another).
You can low-level cache anything you want with Rails.cache.fetch this is put in a block, takes a key name, and an expiration. So it looks like this:
Rails.cache.fetch("google-id-token", expires_in: 24.hours) do
#instance_variable = something
end
If the cache exists and it is not past its expiration date/time, Rails grabs it from the cache; otherwise, it would make your API request.
It's important to note that low-level caching doesn't work with mem_store (the default for development) and so you need to implement a solution with redis or memcached or something like that for development, too. Also, make sure the file tmp/cache.txt exists. You can run rails dev:cache or just touch it to create it.
More on Rails caching
Scenario:
I have a Rails API with a /tasks.json endpoint that serves all the Tasks.
For large Accounts this can sometimes be close to 1,000,000 records.
Sending 1,000,000 records over the wire in JSON generally doesn't work due to a network timeout, etc.
I can easily "paginate" the number of tasks being sent so the endpoint uses something like /tasks.json?limit=10000&page=1 but how does the client know to send this?
Is there a "standard" way to handle large requests like this and break them up naturally into chunks?
Does each client need to handle this on their end manually?
Thanks!
You should use kaminari gem. It handles all requests and paginates it. It works both on Rails Api apps and rails standard apps.
https://github.com/kaminari/kaminari
Ruby on Rails 4.1.4
I made an interface to a Twitch gem, to fetch information of the current stream, mainly whether it is online or not, but also stuff like the current title and game being played.
Since the website has a lot of traffic, I can't make a request every time a user walks in, so instead I need to cache this information.
Cached information is stored as a class variable ##stream_data inside class: Twitcher.
I've made a rake task to update this using cronjobs, calling Twitcher.refresh_stream, but naturally that is not running within my active process (to which every visitor is connecting to) but instead a separate process. So the ##stream_data on the actual app is always empty.
Is there a way to run code, within my currently running rails app, every X minutes? Or a better approach, for that matter.
Thank you for your time!
This sounds like a good call for caching
Rails.cache.fetch("stream_data", expires_in: 5.minutes) do
fetch_new_data
end
If the data is in the cache and is not old then it will be returned without executing the block, if not the block is used to populate the cache.
The default cache store just keeps things in memory so doesn't fix your problem: you'll need to pick a cache store that is shared across your processes. Both redis and memcached (via the dalli gem) are popular choices.
Check out Whenever (basically a ruby interface to cron) to invoke something on a regular schedule.
I actually had a similar problem with using google analytics. Google analytics requires that you have an API key for each request. However the api key would expire every hour. If you requested a new api key for every google analytics request, it'd be very slow per request.
So what I did was make another class variable ##expires_at. Now in every method that made a request to google analytics, I would check ##expires_at.past?. If it was true, then I would refresh the api key and set ##expires_at = 45.minutes.from_now.
You can do something like this.
def method_that_needs_stream_data
renew_data if ##expires_at.past?
# use ##stream_data
end
def renew_data
# renew ##stream_data here
##expires_at = 5.minutes.from_now
end
Tell me how it goes.
I am working on real time data visualization project that consumes twitter streaming api
s'.For processing tweets in a server side that is based on rails Framework.
With twitter ruby gem, i can able to fetch the stream tweets
topics = ["coffee", "tea"]
client.filter(:track => topics.join(",")) do |tweet|
puts tweet.text
end
With this i need to build a JSON API in Rails.
UPDATE: With JSON API, need to integrate with AngularJS. For building API at real-time, whether i need to store it any database or not needed.
I suggest you consider Sinatra to build the API, but you can certainly do it in Rails. Simply when a client makes a REST call to the endpoint defined in routes.rb, the controller method will itself make a REST call to Twitter and then transform and serialize their result to JSON to return to your client.
Just remember that your clients need to send the CSRF token with their requests to your services for Rails to let them through and maintain session.
In JQuery it might look something like this:
$.ajaxSetup({
beforeSend: function(xhr) {
xhr.setRequestHeader('X-CSRF-Token', $('meta[name="csrf-token"]').attr('content'));
}
});
As for memory concerns with the volume of data, that depends on how much data you are retrieving, what you are doing with it, the power of your machine, etc. I wouldn't worry if you aren't hitting the Firehose. Let's worry about a memory issue later if it happens. There are always things you can do like caching results, etc. without using a database.
I have an application with user 'Notifications' think SO or facebook or twitter. However, as notifications won't necessarily change on every page view I decided to save them in memcached.
def get_notification
if current_user
mc = Dalli::Client.new('localhost:11211')
require_dependency 'notification.rb'
#new_notification = mc.get(current_user.id.to_s+'new_notification')
if #new_notification == nil
#new_notification = Notification.getNew(current_user.id)
mc.set(current_user.id.to_s+'notification',#new_notification)
end
end
end
I overlooked the obvious flaw in this implementation. Once the notifications are loaded they would never be refreshed until the user logs out or the cache entry expires. One way to do this is to negate the user's cache entry when a event for a new notification occurs. This would force a new request to the db. Is there any other way to implement this?
Currently you are manually connecting to Memchaced, check if key exists, store content, expire it. But as you may notice this gets tedious & repetitive very soon.
However, Rails Provides you with few patterns that you can use to accomplish this but more easily.
First using Cache Stores option you can instruct rails to use Memchached
config.cache_store = :mem_cache_store, "example.com"
This cache store uses memcached server to provide a
centralized cache for your application. Rails uses the bundled dalli
gem by default. This is currently the most popular cache store for
production websites. It can be used to provide a single, shared cache
cluster with very a high performance and redundancy.
When initializing the cache, you need to specify the addresses for all
memcached servers in your cluster. If none is specified, it will
assume memcached is running on the local host on the default port, but
this is not an ideal set up for larger sites.
The write and fetch methods on this cache accept two additional
options that take advantage of features specific to memcached. You can
specify :raw to send a value directly to the server with no
serialization. The value must be a string or number. You can use
memcached direct operation like increment and decrement only on raw
values. You can also specify :unless_exist if you don't want memcached
to overwrite an existing entry.
Using rails Cache store instead of directly using Dalli allows you to use the following Nicer API
Rails.cache.read('key')
Rails.cache.write('key', value)
Rails.cache.fetch('key') { value }
Now, rails for actually caching. you can use Declarative Etags or Fragment Caching to cache the notifications. here is an example using Declarative Etags
def get_notification
if current_user
#new_notification = Notification.getNew(current_user.id)
end
refresh_when #new_notification
end
Now the way declarative E-tags works is Template is not rendered when request
sends a matching ETag & cache copy is sent. However, when #new_notification changes the E-tag value will change too. Thus causing the cache to expire. Now, Caching is a vast topic to cover & there are variously techniques to do it. so probally I won't give you a full answers but I would point to the following resources so you can learn more:
Caching with Rail
Rails 4: Zombie Outlaws Course
Rails Cache for dummies
Caching Strategies for Rails
Happy Caching ;-)