My questions is very simple
Assuming I'm not specifying expires_in key for my generated cache key
Let's says i generate a cache key for posts with key "posts/#{maximum_record_updated at}" with no expires_in key
Now my content changed and new key has been set and is getting used with new "posts/#{maximum_record_updated_at}"
The cache is now calling the latest key only
Now the question is... what happens to the first key which is not going to be used anymore and has no expires_in specified ?
will it live forever or Redis will manage deleting it if it's not going to be used anymore ?
I know i would just specify the expires_in simply, but posts (in my case) could stay 1 week without any changes, maybe months, years, so I'm generating new cache key only when something changes
I'm just worried about the old keys and any unexpected memory issue
The old unused key will stay there until Redis reaches maxmemory usage.
Then, Redis will stop accepting write commands or will start evicting keys, depending on the config value of maxmemory-policy. See https://redis.io/topics/lru-cache
Related
ok, first time making an API!
My assumption is that if data needs to be stored on the back end such that it persists across multiple API calls, it needs to be 1) in cache or 2) in a Database. is that right?
I was looking at the code for the gem "google-id-token". it seems to do just what i need for my google login application. My front end app will send the google tokens to the API with requests.
the gem appears to cache the public (PEM) certificates from Google (for an hour by default) and then uses them to validate the Google JWT you provide.
but when i look at the code (https://github.com/google/google-id-token/blob/master/lib/google-id-token.rb) it just seems to fetch the google certificates and put them into an instance variable.
am i right in thinking that the next time someone calls the API, it will have no memory of that stored data and just fetch it again?
i guess its a 2 part question:
if i put something in an #instance_variable in my API, will that data exist when the next API call comes in?
if not, is there any way that "google-id-token" is caching its data correctly? maybe HTTP requests are somehow cached on the backend and therefore the network request doesnt actually happen over and over? can i test this?
my impulse is to write "google-id-token" functionality in a way that caches the google certs using MemCachier. but since i dont know what I'm doing i thought i would ask.? Maybe the gem works fine as is, i dont know how to test it.
Not sure about google-id-token, but Rails instance variables are not available beyond single requests and views (and definitely not from one user's session to another).
You can low-level cache anything you want with Rails.cache.fetch this is put in a block, takes a key name, and an expiration. So it looks like this:
Rails.cache.fetch("google-id-token", expires_in: 24.hours) do
#instance_variable = something
end
If the cache exists and it is not past its expiration date/time, Rails grabs it from the cache; otherwise, it would make your API request.
It's important to note that low-level caching doesn't work with mem_store (the default for development) and so you need to implement a solution with redis or memcached or something like that for development, too. Also, make sure the file tmp/cache.txt exists. You can run rails dev:cache or just touch it to create it.
More on Rails caching
Anyone know how long Azure Keyvault retains old keys? The scenario would be an on-prem SQL server VM using keyvault for TDE with keys being rotated regularly.
If we were to restore an old snapshot of the SQL server, will it be able to access the old, retired key?
Soft deleted resources are retained for a set period of time, 90 days.
Upon deleting a key vault object, such as a key, the service will place the object in a deleted state, making it inaccessible to any retrieval operations. While in this state, the key vault object can only be listed, recovered, or forcefully/permanently deleted.
At the same time, Key Vault will schedule the deletion of the underlying data corresponding to the deleted key vault or key vault object for execution after a predetermined retention interval. The DNS record corresponding to the vault is also retained for the duration of the retention interval.
Use a key without an expiration date – and don't set an expiration date on a key already in use: once the key expires, the encrypted databases lose access to their TDE Protector and are inaccessible within 24 hours.
For more details, you could refer to this article and this one.
Ruby on Rails 4.1.4
I made an interface to a Twitch gem, to fetch information of the current stream, mainly whether it is online or not, but also stuff like the current title and game being played.
Since the website has a lot of traffic, I can't make a request every time a user walks in, so instead I need to cache this information.
Cached information is stored as a class variable ##stream_data inside class: Twitcher.
I've made a rake task to update this using cronjobs, calling Twitcher.refresh_stream, but naturally that is not running within my active process (to which every visitor is connecting to) but instead a separate process. So the ##stream_data on the actual app is always empty.
Is there a way to run code, within my currently running rails app, every X minutes? Or a better approach, for that matter.
Thank you for your time!
This sounds like a good call for caching
Rails.cache.fetch("stream_data", expires_in: 5.minutes) do
fetch_new_data
end
If the data is in the cache and is not old then it will be returned without executing the block, if not the block is used to populate the cache.
The default cache store just keeps things in memory so doesn't fix your problem: you'll need to pick a cache store that is shared across your processes. Both redis and memcached (via the dalli gem) are popular choices.
Check out Whenever (basically a ruby interface to cron) to invoke something on a regular schedule.
I actually had a similar problem with using google analytics. Google analytics requires that you have an API key for each request. However the api key would expire every hour. If you requested a new api key for every google analytics request, it'd be very slow per request.
So what I did was make another class variable ##expires_at. Now in every method that made a request to google analytics, I would check ##expires_at.past?. If it was true, then I would refresh the api key and set ##expires_at = 45.minutes.from_now.
You can do something like this.
def method_that_needs_stream_data
renew_data if ##expires_at.past?
# use ##stream_data
end
def renew_data
# renew ##stream_data here
##expires_at = 5.minutes.from_now
end
Tell me how it goes.
I have an application with user 'Notifications' think SO or facebook or twitter. However, as notifications won't necessarily change on every page view I decided to save them in memcached.
def get_notification
if current_user
mc = Dalli::Client.new('localhost:11211')
require_dependency 'notification.rb'
#new_notification = mc.get(current_user.id.to_s+'new_notification')
if #new_notification == nil
#new_notification = Notification.getNew(current_user.id)
mc.set(current_user.id.to_s+'notification',#new_notification)
end
end
end
I overlooked the obvious flaw in this implementation. Once the notifications are loaded they would never be refreshed until the user logs out or the cache entry expires. One way to do this is to negate the user's cache entry when a event for a new notification occurs. This would force a new request to the db. Is there any other way to implement this?
Currently you are manually connecting to Memchaced, check if key exists, store content, expire it. But as you may notice this gets tedious & repetitive very soon.
However, Rails Provides you with few patterns that you can use to accomplish this but more easily.
First using Cache Stores option you can instruct rails to use Memchached
config.cache_store = :mem_cache_store, "example.com"
This cache store uses memcached server to provide a
centralized cache for your application. Rails uses the bundled dalli
gem by default. This is currently the most popular cache store for
production websites. It can be used to provide a single, shared cache
cluster with very a high performance and redundancy.
When initializing the cache, you need to specify the addresses for all
memcached servers in your cluster. If none is specified, it will
assume memcached is running on the local host on the default port, but
this is not an ideal set up for larger sites.
The write and fetch methods on this cache accept two additional
options that take advantage of features specific to memcached. You can
specify :raw to send a value directly to the server with no
serialization. The value must be a string or number. You can use
memcached direct operation like increment and decrement only on raw
values. You can also specify :unless_exist if you don't want memcached
to overwrite an existing entry.
Using rails Cache store instead of directly using Dalli allows you to use the following Nicer API
Rails.cache.read('key')
Rails.cache.write('key', value)
Rails.cache.fetch('key') { value }
Now, rails for actually caching. you can use Declarative Etags or Fragment Caching to cache the notifications. here is an example using Declarative Etags
def get_notification
if current_user
#new_notification = Notification.getNew(current_user.id)
end
refresh_when #new_notification
end
Now the way declarative E-tags works is Template is not rendered when request
sends a matching ETag & cache copy is sent. However, when #new_notification changes the E-tag value will change too. Thus causing the cache to expire. Now, Caching is a vast topic to cover & there are variously techniques to do it. so probally I won't give you a full answers but I would point to the following resources so you can learn more:
Caching with Rail
Rails 4: Zombie Outlaws Course
Rails Cache for dummies
Caching Strategies for Rails
Happy Caching ;-)
Consider this scenario. We have an internal Rails 2 app that connects to a vendor site using Savon ( Ruby SOAP library). The connection relies on authentication using a username and password. Password changes every 30 days. Right now - every 30 days on of the developers has to go to vendor's site, update the password, and the go back to our internal rails app, update a password, push the app to the webserver and restart passenger.
We store the password as a CONSTANT in the environment file along with a couple of more constants, specific to the web. We kind of want to offload this responsibility to a customer service manager, so that she updates the password every 30 days, the go to a little form in the app and updates the password.
What is the best way to store that password? We obviously can't do the CONSTANT in an environment file and keep reloading the app. But at the same time creating a model to store one password in a separate table seems like an overkill, also since the part of the app that make a request to vendor site - does about 1000 requests per day, and needs to be pretty fast, that means that storing that password in a table requires an extra query every time the request is made. Of course for our volume it's not a big deal, but theoretically what is the best\efficient solution for this problem, other than doing what we do now, loading it into memory on app start?
Store the (encrypted) password in a flat file. Reload the file by a signal, or on auth failure, or if more than 10 seconds passed since last reload and file's timestamp has changed. Wrap this logic into a single method like .getCredentials(). Most of the time it will just return cached login and password values.
Storing this in a proper database does look like an overkill to me.