I am attempting to generate a fragment cache (using a Dalli/Memcached store) however the key is being generated with "#" as part of the key, so Rails doesn't seem to be recognizing that there is a cache value and is hitting the database.
My cache key in the view looks like this:
cache([#jobs, "index"]) do
The controller has:
#jobs = #current_tenant.active_jobs
With the actual Active Record query like this:
def active_jobs
self.jobs.where("published = ? and expiration_date >= ?", true, Date.today).order("(featured and created_at > now() - interval '" + self.pinned_time_limit.to_s + " days') desc nulls last, created_at desc")
end
Looking at the rails server, I see the cache read, but the SQL Query still runs:
Cache read: views/#<ActiveRecord::Relation:0x007fbabef9cd58>/1-index
Read fragment views/#<ActiveRecord::Relation:0x007fbabef9cd58>/1-index (1.0ms)
(0.6ms) SELECT COUNT(*) FROM "jobs" WHERE "jobs"."tenant_id" = 1 AND (published = 't' and expiration_date >= '2013-03-03')
Job Load (1.2ms) SELECT "jobs".* FROM "jobs" WHERE "jobs"."tenant_id" = 1 AND (published = 't' and expiration_date >= '2013-03-03') ORDER BY (featured and created_at > now() - interval '7 days') desc nulls last, created_at desc
Any ideas as to what I might be doing wrong? I'm sure it has to do w/ the key generation and ActiveRecord::Relation, but i'm not sure how.
Background:
The problem is that the string representation of the relation is different each time your code is run:
|This changes|
views/#<ActiveRecord::Relation:0x007fbabef9cd58>/...
So you get a different cache key each time.
Besides that it is not possible to get rid of database queries completely. (Your own answer is the best one can do)
Solution:
To generate a valid key, instead of this
cache([#jobs, "index"])
do this:
cache([#jobs.to_a, "index"])
This queries the database and builds an array of the models, from which the cache_key is retrieved.
PS: I could swear using relations worked in previous versions of Rails...
We've been doing exactly what you're mentioning in production for about a year. I extracted it into a gem a few months ago:
https://github.com/cmer/scope_cache_key
Basically, it allows you to use a scope as part of your cache key. There are significant performance benefits to doing so since you can now cache a page containing multiple records in a single cache element rather than looping each element in the scope and retrieving caches individually. I feel that combining this with with the standard "Russian Doll Caching" principles is optimal.
I have had similar problems, I have not been able to successfully pass relations to the cache function and your #jobs variable is a relation.
I coded up a solution for cache keys that deals with this issue along with some others that I was having. It basically involves generating a cache key by iterating through the relation.
A full write up is on my site here.
http://mark.stratmann.me/content_items/rails-caching-strategy-using-key-based-approach
In summary I added a get_cache_keys function to ActiveRecord::Base
module CacheKeys
extend ActiveSupport::Concern
# Instance Methods
def get_cache_key(prefix=nil)
cache_key = []
cache_key << prefix if prefix
cache_key << self
self.class.get_cache_key_children.each do |child|
if child.macro == :has_many
self.send(child.name).all.each do |child_record|
cache_key << child_record.get_cache_key
end
end
if child.macro == :belongs_to
cache_key << self.send(child.name).get_cache_key
end
end
return cache_key.flatten
end
# Class Methods
module ClassMethods
def cache_key_children(*args)
#v_cache_key_children = []
# validate the children
args.each do |child|
#is it an association
association = reflect_on_association(child)
if association == nil
raise "#{child} is not an association!"
end
#v_cache_key_children << association
end
end
def get_cache_key_children
return #v_cache_key_children ||= []
end
end
end
# include the extension
ActiveRecord::Base.send(:include, CacheKeys)
I can now create cache fragments by doing
cache(#model.get_cache_key(['textlabel'])) do
I've done something like Hopsoft, but it uses the method in the Rails Guide as a template. I've used the MD5 digest to distinguish between relations (so User.active.cache_key can be differentiated from User.deactivated.cache_key), and used the count and max updated_at to auto-expire the cache on updates to the relation.
require "digest/md5"
module RelationCacheKey
def cache_key
model_identifier = name.underscore.pluralize
relation_identifier = Digest::MD5.hexdigest(to_sql.downcase)
max_updated_at = maximum(:updated_at).try(:utc).try(:to_s, :number)
"#{model_identifier}/#{relation_identifier}-#{count}-#{max_updated_at}"
end
end
ActiveRecord::Relation.send :include, RelationCacheKey
While I marked #mark-stratmann 's response as correct I actually resolved this by simplifying the implementation. I added touch: true to my model relationship declaration:
belongs_to :tenant, touch: true
and then set the cache key based on the tenant (with a required query param as well):
<% cache([#current_tenant, params[:query], "#{#current_tenant.id}-index"]) do %>
That way if a new Job is added, it touches the Tenant cache as well. Not sure if this is the best route, but it works and seems pretty simple.
Im using this code:
class ActiveRecord::Base
def self.cache_key
pluck("concat_ws('/', '#{table_name}', group_concat(#{table_name}.id), date_format(max(#{table_name}.updated_at), '%Y%m%d%H%i%s'))").first
end
def self.updated_at
maximum(:updated_at)
end
end
maybe this can help you out
https://github.com/casiodk/class_cacher , it generates a cache_key from the Model itself, but maybe you can use some of the principles in the codebase
As a starting point you could try something like this:
def self.cache_key
["#{model_name.cache_key}-all",
"#{count}-#{updated_at.utc.to_s(cache_timestamp_format) rescue 'empty'}"
] * '/'
end
def self.updated_at
maximum :updated_at
end
I'm having normalized database where multiple models relate to the same other model, think of clients, locations, etc. all having addresses by means of a street_id.
With this solution you can generate cache_keys based on scope, e.g.
cache [#client, #client.locations] do
# ...
end
cache [#client, #client.locations.active, 'active'] do
# ...
end
and I could simply modify self.updated from above to also include associated objects (because has_many does not support "touch", so if I updated the street, it won't be seen by the cache otherwise):
belongs_to :street
def cache_key
[street.cache_key, super] * '/'
end
# ...
def self.updated_at
[maximum(:updated_at),
joins(:street).maximum('streets.updated_at')
].max
end
As long as you don't "undelete" records and use touch in belongs_to, you should be alright with the assumption that a cache key made of count and max updated_at is sufficient.
I'm using a simple patch on ActiveRecord::Relation to generate cache keys for relations.
require "digest/md5"
module RelationCacheKey
def cache_key
Digest::MD5.hexdigest to_sql.downcase
end
end
ActiveRecord::Relation.send :include, RelationCacheKey
Related
I use Arel to build reusable and structured queries, but reading around, I didn't find a clear and efficient way to alter extracted date to actually retrieve the calendar week.
Using cweek Ruby method, I am trying to build the following query (on Postgres):
week_series = Skill.joins(concepts_list).select(skills[:created_at].cweek, skills[:id].count.as("count")).group(skills[:created_at].cweek)
Here is my base query upon skills:
### Base object
def skills
Skill.arel_table
end
# Additional tables
def users
User.arel_table
end
def organisations
Organisation.arel_table
end
def themes
Playground.arel_table.alias('themes')
end
def domains
BusinessArea.arel_table.alias('domains')
end
def collections
BusinessObject.arel_table.alias('collections')
end
# Queries
def concepts_list
skills.
join(users).on(skills[:owner_id].eq(users[:id])).
join(organisations).on(skills[:organisation_id].eq(organisations[:id])).
join(collections).on(skills[:business_object_id].eq(collections[:id])).
join(domains).on(collections[:parent_id].eq(domains[:id]).and(collections[:parent_type].eq('BusinessArea'))).
join(themes).on(domains[:playground_id].eq(themes[:id])).
join_sources
end
def concepts_list_output
[skills[:id], skills[:created_at], users[:user_name], users[:name],
organisations[:code], themes[:code], domains[:code]]
end
The equivalent for ruby cweek in postgresql is EXTRACT/DATE_PART. Might differ a bit if you're using a different database.
The sql you're after is
DATE_PART('week', "skills"."created_at")
That is just a NamedFunction
Arel::Nodes::NamedFunction.new(
'DATE_PART',
[Arel::Nodes.build_quoted('week'), skills[:created_at]]
)
I have an API endpoint that accounts for a little less than half of the average response time (on averaging taking about 514 ms, yikes). The endpoint simply returns some statistics about stored data scoped to particular time periods, such as this week, last week, this month, and so on...
There are a number of ways that we could reduce it's impact, like getting the clients to hit it less and with more particular queries such as only querying for "this week" when only that data is used. Here we focus on what can be done at the database-level first. In our current implementation we generate this data for all "time scopes" on-the-fly and the number of queries is enormous and made multiple times per second. No caching is used, but maybe there is a way to use Rails's cache_key, or the low-level Rails.cache?
The current implementation look something like this:
class FooSummaries
include SummaryStructs
def self.generate_for(user)
#user = user
summaries = Struct::Summaries.new
TimeScope::TIME_SCOPES.each do |scope|
foos = user.foos.by_scope(scope.to_sym)
summary = Struct::Summary.new
# e.g: summaries.last_week = build_summary(foos)
summaries.send("#{scope}=", build_summary(summary, foos))
end
summaries
end
private_class_method
def self.build_summary(summary, foos)
summary.all_quuz = #user.foos_count
summary.all_quux = all_quux(foos)
summary.quuw = quuw(foos).to_f
%w[foo bar baz qux].product(
%w[quux quuz corge]
).each do |a, b|
# e.g: summary.foo_quux = quux(foos, "foo")
summary.send("#{a.downcase}_#{b}=", send(b, foos, a) || 0)
end
summary
end
def self.all_quuz(foos)
foos.count
end
def self.all_quux(foos)
foos.sum(:quux)
end
def self.quuw(foos)
foos.quuwable.total_quuw
end
def self.corge(foos, foo_type)
return if foos.count.zero?
count = self.quuz(foos, foo_type) || 0
count.to_f / foos.count
end
def self.quux(foos, foo_type)
case foo_type
when "foo"
foos.where(foo: true).sum(:quux)
when "bar"
foos.bar.where(foo: false).sum(:quux)
when "baz"
foos.baz.where(foo: false).sum(:quux)
when "qux"
foos.qux.sum(:quux)
end
end
def self.quuz(foos, foo_type)
case trip_type
when "foo"
foos.where(foo: true).count
when "bar"
foos.bar.where(foo: false).count
when "baz"
foos.baz.where(foo: false).count
when "qux"
foos.qux.count
end
end
end
To avoid making changes to the model, or creating migrations to create a table to store this data (both of which may be valid and better solutions) I decided maybe it would be easier to construct one large sql query that will be executed at once in the hopes that it will be faster to build the query string and execute it without the overhead of active record set up and tear down of SQL queries.
The new approach looks something like this, it is horrifying to me and I know there must be a more elegant way:
class FooSummaries
include SummaryStructs
def self.generate_for(user)
results = ActiveRecord::Base.connection.execute(build_query_for(user))
results.each do |result|
# build up summary struct from query results
end
end
def self.build_query_for(user)
TimeScope::TIME_SCOPES.map do |scope|
time_scope = TimeScope.new(scope)
%w[foo bar baz qux].map do |foo_type|
%[
select
'#{scope}_#{foo_type}',
sum(quux) as quux,
count(*), as quuz,
round(100.0 * (count(*) / #{user.foos_count.to_f}), 3) as corge
from
"foos"
where
"foo"."user_id" = #{user.id}
and "foos"."foo_type" = '#{foo_type.humanize}'
and "foos"."end_time" between '#{time_scope.from}' AND '#{time_scope.to}'
and "foos"."foo" = '#{foo_type == 'foo' ? 't' : 'f'}'
union
]
end
end.join.reverse.sub("union".reverse, "").reverse
end
end
The funny way of replacing the last occurance of union also horrifies but it seems to work. There must be a beter way as there are probably many things that are wrong with the above implementation(s). It may be helpful to note that I use Postgresql and have no problem with writing queries that are not portable to other DB's. Any advice is truly appreciated!
Thanks for reading!
Update: I found a solution that works for me and sped up the endpoint that uses this service object by 500% ! Essentially the idea is, instead of building a query string and then executing it for each set of parameters, we create a prepared statement using prepare followed by an exec_prepared passing in parameters to the query. Since this query is made many times over this is a useful optmization because, as per the documentation:
A prepared statement is a server-side object that can be used to optimize performance. When the PREPARE statement is executed, the specified statement is parsed, analyzed, and rewritten. When an EXECUTE command is subsequently issued, the prepared statement is planned and executed. This division of labor avoids repetitive parse analysis work, while allowing the execution plan to depend on the specific parameter values supplied.
We prepare the query like so:
def prepare_query!
ActiveRecord::Base.transaction do
connection.prepare("foos_summary",
%[with scoped_foos as (
select
*
from
"foos"
where
"foos"."user_id" = $3
and ("foos"."end_time" between $4 and $5)
)
select
$1::text as scope,
$2::text as foo_type,
sum(quux)::float as quux,
sum(eggs + bacon + ham)::float as food,
count(*) as count,
round((sum(quux) / nullif(
(select
sum(quux)
from
scoped_foos), 0))::numeric,
5)::float as quuz
from
scoped_foos
where
(case $6
when 'Baz'
then (baz = 't')
else
(baz = 'f' and foo_type = $6)
end
)
])
end
You can see in this query we use a common table expression for more readability and to avoid writing the same select query twice over.
Then we execute the query, passing in the parameters we need:
def connection
#connection ||= ActiveRecord::Base.connection.raw_connection
end
def query_results
prepare_query! unless query_already_prepared?
#results ||= TimeScope::TIME_SCOPES.map do |scope|
time_scope = TimeScope.new(scope)
%w[bacon eggs ham spam].map do |foo_type|
connection.exec_prepared("foos_summary",
[scope,
foo_type,
#user.id,
time_scope.from,
time_scope.to,
foo_type.humanize])
end
end
end
Where query_already_prepared? is a simple check in the prepared statements table maintained by postgres:
def query_already_prepared?
connection.exec(%(select
name
from
pg_prepared_statements
where name = 'foos_summary')).count.positive?
end
A nice solution, I thought! Hopefully the technique illustrated here will help others with a similar problems.
I'm trying to build a simple search form in Ruby on Rails, my form is simple enough basically you select fields from a series of options and then all the events matching the fields are shown. The problem comes when I leave any field blank.
Here is the code responsible for filtering the parameters
Event.joins(:eventdates).joins(:categories).where
("eventdates.start_date = ? AND city = ? AND categories.name = ?",
params[:event][:date], params[:event][:city], params[:event][:category]).all
From what I get it's that it looks for events with any empty field, but since all of them have them not empty, it wont match unless all 3 are filled, another problem arises when I try to say, look events inside a range or array of dates, I'm clueless on how to pass multiple days into the search.
I'm pretty new to making search forms in general, so I don't even know if this is the best approach, also I'm trying to keep the searches without the need of a secialized model.
Below is probably what you are looking for. (Note: If all fields all blank, it shows all data in the events table linkable with eventdates and categories.)
events = Event.joins(:eventdates).joins(:categories)
if params[:event]
# includes below where condition to query only if params[:event][:date] has a value
events = events.where("eventdates.start_date = ?", params[:event][:date]) if params[:event][:date].present?
# includes below where condition to query only if params[:event][:city] has a value
events = events.where("city = ?", params[:event][:city]) if params[:event][:city].present?
# includes below where condition to query only if params[:event][:city] has a value
events = events.where("categories.name = ?", params[:event][:category]) if params[:event][:category].present?
end
To search using multiple days:
# params[:event][:dates] is expected to be array of dates.
# Below query gets converted into an 'IN' operation in SQL, something like "where eventdates.start_date IN ['date1', 'date2']"
events = events.where("eventdates.start_date = ?", params[:event][:dates]) if params[:event][:dates].present?
It will be more easy and optimised . If you use concern for filter data.
Make one concern in Model.
filterable.rb
module Filterable
extend ActiveSupport::Concern
module ClassMethods
def filter(filtering_params)
results = self.where(nil)
filtering_params.each do |key, value|
if column_type(key) == :date || column_type(key) ==
:datetime
results = results.where("DATE(#{column(key)}) = ?",
Date.strptime(value, "%m/%d/%Y")) if
value.present?
else
results = results.where("#{column(key)} Like ? ", "%#{value}%") if
value.present?
end
end
results
end
def resource_name
self.table_name
end
def column(key)
return key if key.split(".").count > 1
return "#{resource_name}.#{key}"
end
def column_type(key)
self.columns_hash[key].type
end
end
end
Include this concern in model file that you want to filter.
Model.rb
include Filterable
In your controller Add this methods
def search
#resources = Model.filter(class_search_params)
render 'index'
end
def class_search_params
params.slice(:id,:name) #Your field names
end
So, It is global solution. You dont need to use query for filter. just add this concern in your model file.
That's it.
I created a simple web form where users can enter some search criteria to look for venues e.g. a price range. When a user clicks "find" I use active record to query the database. This all works very well if all fields are filled in. Problems occur when one or more fields are left open and therefore have a value of null.
How can I work around this in my controller? Should I first check whether a value is null and create a query based on that? I can imagine I end up with many different queries and a lot of code. There must be a quicker way to achieve this?
Controller:
def search
#venues = Venue.where("price >= ? AND price <= ? AND romance = ? AND firstdate = ?", params[:minPrice], params[:maxPrice], params[:romance], params[:firstdate])
end
You may want to filter out all of the blank parameters that were sent with the request.
Here is a quick and DRY solution for filtering out blank values, triggers only one query of the database, and builds the where clause with Rails' ActiveRecord ORM.
This approach safeguards against SQL-injection, as pointed out by #DanBrooking. Rails 4.0+ provides "strong parameters." You should use the feature.
class VenuesController < ActiveRecord::Base
def search
# Pass a hash to your query
#venues = Venue.where(search_params)
end
private
def search_params
params.
# Optionally, whitelist your search parameters with permit
permit(:min_price, :max_price, :romance, :first_date).
# Delete any passed params that are nil or empty string
delete_if {|key, value| value.blank? }
end
end
I would recommend to make method in Venue
def self.find_by_price(min_price, max_price)
if min_price && max_price
where("price between ? and ?", min_price, max_price)
else
all
end
end
def self.find_by_romance(romance)
if romance
where("romance = ?", romance)
else
all
end
end
def self.find_by_firstdate(firstdate)
if firstdate
where("firstdate = ?", firstdate)
else
all
end
end
And use it in your controller
Venue
.find_by_price(params[:minPrice], params[:maxPrice])
.find_by_romance(params[:romance])
.find_by_firstdate(params[:firstdate])
Another solution to this problem, and I think a more elegant one, is using scopes with conditions.
You could do something like
class Venue < ActiveRecord::Base
scope :romance, ->(genre) { where("romance = ?", genre) if genre.present? }
end
You can then chain those, which would work as an AND if there is no argument present, then it is not part of the chain.
http://guides.rubyonrails.org/active_record_querying.html#scopes
Try below code, it will ignore parameters those are not present
conditions = []
conditions << "price >= '#{params[:minPrice]}'" if params[:minPrice].present?
conditions << "price <= '#{params[:maxPrice]}'" if params[:maxPrice].present?
conditions << "romance = '#{params[:romance]}'" if params[:romance].present?
conditions << "firstdate = '#{params[:firstdate]}'" if params[:firstdate].present?
#venues = Venue.where(conditions.join(" AND "))
I'm implementing a search system that uses name, tags, and location. There is a has_and_belongs_to_many relationship between Server and Tag. Here's what my search method currently looks like:
def self.search(params)
#servers = Server.all
if params[:name]
#servers = #servers.where "name ILIKE ?", "%#{params[:name]}%"
end
if params[:tags]
#tags = Tag.find params[:tags].split(",")
# How do I eliminate servers that do not have these tags?
end
# TODO: Eliminate those that do not have the location specified in params.
end
The tags parameter is just a comma-separated list of IDs. My question is stated in a comment in the if params[:tags] conditional block. How can I eliminate servers that do not have the tags specified?
Bonus question: any way to speed this up? All fields are optional, and I am using Postgres exclusively.
EDIT
I found a way to do this, but I have reason to believe it will be extremely slow to run. Is there any way that's faster than what I've done? Perhaps a way to make the database do the work?
tags = Tag.find tokens
servers = servers.reject do |server|
missing_a_tag = false
tags.each do |tag|
if server.tags.find_by_id(tag.id).nil?
missing_a_tag = true
end
end
missing_a_tag
end
Retrieve the servers with all the given tags with
if params[:tags]
tags_ids = params[:tags].split(',')
#tags = Tag.find(tags_ids)
#servers = #servers.joins(:tags).where(tags: {id: tags_ids}).group('servers.id').having("count(*) = #{tags_ids.count}")
end
The group(...).having(...) part selects the servers with all requested tags. If you're looking for servers which have at least one of the tags, remove it.
With this solution, the search is done in a single SQL request, so it will be better than your solution.