So I am developing a rails app, and I am working on paginating the feed. While I was doing it I wondered if I was doing it the right way because my load times were over 1500ms. My code was:
stories = Story.feed
#stories = Kaminari.paginate_array(stories).page(params[:page]).per(params[:pageSize])
I have a few questions about this:
Should I be paginating Story.feed, or is there some sort of method
that only returns some the stories I need?
Is this load time normal?
What are other things I can be doing to optimize this
(Also, Story.feed returns an array of story objects. The code for that is here:
def self.feed
rawStories = Story.includes([:likes, :viewers, :user, :storyblocks]).all
newFeaturedStories = rawStories.where(:featured => true).where(:updated_at.gte => (Date.today - 3)).desc(:created_at).entries
normalStories = rawStories.not_in(:featured => true, :or => [:updated_at.gte => (Date.today - 3)]).desc(:created_at).entries
newFeaturedStories.entries.concat(normalStories.entries)
end
I am using mongoid and mongodb
The issue is that you get all feeds from db in an array and this takes long time.
I suggest you use the any_of query from this great gem.
From there, do:
def self.feed_stories
newFeaturedStories = Story.where(:featured => true).where(:updated_at.gte => (Date.today - 3.days))
normalStories = Story.not_in(:featured => true, :or => [:updated_at.gte => (Date.today - 3.days)])
Story.includes([:likes, :viewers, :user, :storyblocks]).any_of(newFeaturedStories, normalStories).desc(:created_at)
end
Then paginate this:
selected_stories = Story.feed_stories.per(page_size).page(page)
Dont really understand what are your entries but get them at this moment.
To sum up: the idea s to make a unique paginated db query.
I suspect that when you call Kaminari.paginate_array on an ActiveRecord::Relation, it causes the whole result set to be fetched from DB and loaded in memory similar to calling Model.all.to_a.
To avoid this, I'd first find a way to turn Story.feed into a scope, rather than a class method. Superficially they'll seem the same—the differences are subtle but deep. See Active Record scopes vs class methods.
Next, ditch paginate_array in favor of chain Kaminari's page() and per() scopes.
For example (simplified version of yours):
class Article < ActiveRecord::Base
scope :featured, -> { where(featured: true) }
scope :last_3_days, -> { where(:updated_at.gte => (Date.today - 3)).desc(:created_at) }
scope :feed, -> { featured.last_3_days }
And then paginate simply by going:
Article.feed.per(page_size).page(page)
The biggest advantage of this is that Kaminari can chain into the generated SQL inserting the proper LIMIT and OFFSET clauses thereby reducing the size of the result set returned to only what needs to be displayed, as opposed to returning every matching record.
I think Will Paginate will help you out here -> mislav/will_paginate.
From there you can simply give your controller action .per_page(20) for example and after 20 objects (you can define the objects, see the wiki) there will be pagination
Related
This is an existing code written by someone else and am trying to enhance it. I am a java developer working on Ruby on Rails, so kindly be considerate.
I have entities like this
User
Delivery entity,
Delivery
belongs_to :user
named_scope :for_abcs, :conditions => {'deliveries.xyz_type' => ['Xyz1', 'Xyz2']},
many such named-scopes are defined.
Now to fetch the deliveries its written like this
#deliveries = current_user.deliveries.send("for_abcs").with(:xyz, :sender, :receiver)
...
...
...
# few other conditions added to #deliveries
finally
#deliveries.sort(...)
This sort is taking huge sql and giving performance issues. I want to use find_each, but find_each is only for Active Entity in Ruby on Rails, How can I achieve this (if possible) without much code change)
Earlier I used to do
Delevery.find_each
wherever it is
Delivery.find
Now I cant do as it is an array, what is the workaround or right procedure to do that in Ruby on Rails.
EDIT :
What I tried :
deliveries_temp = []
#deliveries.find_each(:batch_size=>999) do |delivery_temp|
deliveries_temp.push(delivery_temp)
end
This gave me error
undefined method `find_each' for []:Array
type(#deliveries) returned ActiveRecord::NamedScope::Scope , rails version 2.3.18
find_each should work on anything that returns a Relation (which includes scopes).
#deliveries = current_user.deliveries.for_abcs(:xyz, :sender, :receiver).find_each
Update
It sounds like you're using Rails 2.3. find_each is a class method in 2.3, so you'll need a way to extract the conditions from your scope and pass them to find_each. I found an article that looks promising, so give this a try:
Delivery.find_each(current_user.deliveries.for_abcs.scope(:find))
Also, I'm still not sure what that #with is doing. Maybe it's supposed to be #includes?
After lot of research for a week and learning about named_scopes by checking its source code. I understood what the problem was. The #deliveries is an object of class ActiveRecord::NamedScope::Scope . This class do not have find_each method. So I wrote a new named_scope for limit and offset in Delivery model file as follows :
named_scope :limit_and_offset, lambda { |lim,off| { :limit => lim, :offset=>off } }
After this , I called it in a loop passing offset and limit , for ex. first loop has offset=0, limit=999 , second loop has offset=999, limit=999 . I will add all the results into an emptry array. This loop continues till the result size is less than the limit value . This is working exactly the way I wanted , in batches.
set = 1
total_deliveries = []
set_limit=999
original_condition = #deliveries
loop do
offset = (set-1) * set_limit
temp_condition = original_condition.limit_and_offset(set_limit,offset)
temp_deliveries = temp_condition.find(:all)
total_deliveries+= temp_deliveries
set += 1
break if temp_deliveries.size < set_limit
end
#deliveries = total_deliveries.sort do |a, b|
I am pretty new to Rails and I have a feeling I'm approaching this from the wrong angle but here it goes... I have a list page that displays vehicles and i am trying to add filter functionality where the user can filter the results by vehicle_size, manufacturer and/or payment_options.
Using three select form fields the user can set the values of :vehicle_size, :manufacturer and/or :payment_options parameters and submit these values to the controller where i'm using a
#vehicles = Vehicle.order("vehicles.id ASC").where(:visible => true, :vehicle_size => params[:vehicle_size] )
kind of query. this works fine for individual params (the above returns results for the correct vehicle size) but I want to be able to pass in all 3 params without getting no results if one of the parameters is left blank..
Is there a way of doing this without going through the process of writing if statements that define different where statements depending on what params are set? This could become very tedious if I add more filter options.. perhaps some sort of inline if has_key solution to the effect of:
#vehicles = Vehicle.order("vehicles.id ASC").where(:visible => true, if(params.has_key?(:vehicle_size):vehicle_size => params[:vehicle_size], end if(params.has_key?(:manufacturer):manufacturer => params[:manufacturer] end )
You can do:
#vehicles = Vehicle.order('vehicles.id ASC')
if params[:vehicle_size].present?
#vehicles = #vehicles.where(vehicle_size: params[:vehicle_size])
end
Or, you can create scope in your model:
scope :vehicle_size, ->(vehicle_size) { where(vehicle_size: vehicle_size) if vehicle_size.present? }
Or, according to this answer, you can create class method:
def self.vehicle_size(vehicle_size)
if vehicle_size.present?
where(vehicle_size: vehicle_size)
else
scoped # `all` if you use Rails 4
end
end
You call both scope and class method in your controller with, for example:
#vehicles = Vehicle.order('vehicles.id ASC').vehicle_size(params[:vehicle_size])
You can do same thing with remaining parameters respectively.
The has_scope gem applies scope methods to your search queries, and by default it ignores when parameters are empty, it might be worth checking
Can somebody explain what this method does and what I can pass to it?
scoped(options = nil)
Returns an anonymous scope.
And also what the scope method does? I don't understand after reading the documentation.
In ActiveRecord, all query building methods (like where, order, joins, limit and so forth) return a so called scope. Only when you call a kicker method like all or first the built-up query is executed and the results from the database are returned.
The scoped class method also returns a scope. The scope returned is by default empty meaning the result set would not be restricted in any way meaning all records would be returned if the query was executed.
You can use it to provide an "empty" alternative like in the query_by_date example by MurifoX.
Or you can use it to combine multiple conditions into one method call, like for example:
Model.scoped(:conditions => 'id < 100', :limit => 10, :order => 'title ASC')
# which would be equivalent to
Model.where('id < 100').limit(10).order('title ASC')
The scope class method allows you to define a class method that also returns a scope, like for example:
class Model
scope :colored, lambda {|col|
where(:color => col)
}
end
which can be used like this:
Model.colored
The nice thing with scopes is that you can combine them (almost) as you wish, so the following is absolutely possible:
Model.red.where('id < 100').order('title ASC').scoped(:limit => 10)
I also strongly suggest reading through http://guides.rubyonrails.org/active_record_querying.html
I have used it in the past.When you make chained calls to the ActiveRecord query interface like this:
Model.where(:conditions).where(:more_conditions).where(:final_conditions)
Each one of them is already scoped, making the chain work without any problems. But let's say you have something like this:
Model.query_by_date(date).query_by_user(user).query_by_status(status)
scope :query_by_date, lambda { |date|
case date
when "today"
where(:date => Date.today)
when "tomorrow"
where(:date => Date.tomorrow)
else
# Any value like '' or 0 or Date.whatever
end
}
This would cause an error if the date param is not today or tomorrow. It would pick the last value and try to chain this query with the next one query_by_user, resulting in a undefined method default_scoped? for ''. But if you put a scoped method in the else condition, it would work without any flaws, because you are saying to activerecord that you pass through this method/named scope and didn't make any calls to where/find/other activerecord methods, but returned a scoped object, so you can continue chaining queries and stuff.
It would be this way in the end.
else
scoped
end
Hope you understand this simple example.
The problem essence as I see it
One day, if I'm not mistaken, I have seen an example of reusing a named_scope to define another named_scope. Something like this (can't remember the exact syntax, but that's exactly my question):
named_scope :billable, :conditions => ...
named_scope :billable_by_tom, :conditions => {
:billable => true,
:user => User.find_by_name('Tom')
}
The question is: what is the exact syntax, if it's possible at all? I can't find it back, and Google was of no help either.
Some explanations
Why I actually want it, is that I'm using Searchlogic to define a complex search, which can result in an expression like this:
Card.user_group_managers_salary_greater_than(100)
But it's too long to be put everywhere. Because, as far as I know, Searchlogic simply defines named_scopes on the fly, I would like to set a named_scope on the Card class like this:
named_scope from_big_guys, { user_group_managers_salary_greater_than(100) }
- this is where I would use that long Searchlogic method inside my named_scope. But, again, what would be the syntax? Can't figure it out.
Resume
So, is named_scope nesting (and I do not mean chaining) actually possible?
You can use proxy_options to recycle one named_scope into another:
class Thing
#...
named_scope :billable_by, lambda{|user| {:conditions => {:billable_id => user.id } } }
named_scope :billable_by_tom, lambda{ self.billable_by(User.find_by_name('Tom').id).proxy_options }
#...
end
This way it can be chained with other named_scopes.
I use this in my code and it works perfectly.
I hope it helps.
Refer to this question raised time ago here at SO.
There is a patch at lighthouse to achieve your requirement.
Rails 3+
I had this same question and the good news is that over the last five years the Rails core team has made some good strides in the scopes department.
In Rails 3+ you can now do this, as you'd expect:
scope :billable, where( due: true )
scope :billable_by_tom, -> { billable.where( user: User.find_by_name('Tom') ) }
Invoice.billable.to_sql #=> "... WHERE due = 1 ..."
Invoice.billiable_by_tom.to_sql #=> "... WHERE due = 1 AND user_id = 5 ..."
FYI, Rails 3+ they've renamed named_scope to just scope. I'm also using Ruby 1.9 syntax.
Bonus Round: Generic Scope.
If there are multiple people that are "billable" besides just "Tom" then it might be useful to make a generic scope that accepts a name param that gets passed into the block:
scope :billable_by, lambda { |name| billable.where( user: User.find_by_name( name ) ) }
Then you can just call it with:
Invoice.billable_by( "Tom" ).to_sql #=> "... WHERE due = 1 AND user_id = 5 ..."
Btw, you can see I used the older lambda syntax in the bonus round. That's because I think the new syntax looks atrocious when you're passing a param to it: ->( name ) { ... }.
Chain Scopes.
Why not have a scope for stuff just by Tom in general, like:
scope :by_tom, where( user: User.find_by_name('Tom') )
And then you can get those records that are "billable by Tom" with:
Record.billable.by_tom
You can use a method to combine some named_scope like :
def self.from_big_guys
self.class.user_group_managers_salary_greater_than(100)
end
This feature is add on Rails 3 with new syntax (http://m.onkey.org/2010/1/22/active-record-query-interface)
named_scope :with_country, lambad { |country_id| ...}
named_scope :with_language, lambad { |language_id| ...}
named_scope :with_gender, lambad { |gender_id| ...}
if params[:country_id]
Event.with_country(params[:country_id])
elsif params[:langauge_id]
Event.with_state(params[:language_id])
else
......
#so many combinations
end
If I get both country and language then I need to apply both of them. In my real application I have 8 different named_scopes that could be applied depending on the case. How to apply named_scopes incrementally or hold on to named_scopes somewhere and then later apply in one shot.
I tried holding on to values like this
tmp = Event.with_country(1)
but that fires the sql instantly.
I guess I can write something like
if !params[:country_id].blank? && !params[:language_id].blank? && !params[:gender_id].blank?
Event.with_country(params[:country_id]).with_language(..).with_gender
elsif country && language
elsif country && gender
elsif country && gender
.. you see the problem
Actually, the SQL does not fire instantly. Though I haven't bothered to look up how Rails pulls off this magic (though now I'm curious), the query isn't fired until you actually inspect the result set's contents.
So if you run the following in the console:
wc = Event.with_country(Country.first.id);nil # line returns nil, so wc remains uninspected
wc.with_state(State.first.id)
you'll note that no Event query is fired for the first line, whereas one large Event query is fired for the second. As such, you can safely store Event.with_country(params[:country_id]) as a variable and add more scopes to it later, since the query will only be fired at the end.
To confirm that this is true, try the approach I'm describing, and check your server logs to confirm that only one query is being fired on the page itself for events.
Check Anonymous Scopes.
I had to do something similar, having many filters applied in a view. What I did was create named_scopes with conditions:
named_scope :with_filter, lambda{|filter| { :conditions => {:field => filter}} unless filter.blank?}
In the same class there is a method which receives the params from the action and returns the filtered records:
def self.filter(params)
ClassObject
.with_filter(params[:filter1])
.with_filter2(params[:filter2])
end
Like that you can add all the filters using named_scopes and they are used depending on the params that are sent.
I took the idea from here: http://www.idolhands.com/ruby-on-rails/guides-tips-and-tutorials/add-filters-to-views-using-named-scopes-in-rails
Event.with_country(params[:country_id]).with_state(params[:language_id])
will work and won't fire the SQL until the end (if you try it in the console, it'll happen right away because the console will call to_s on the results. IRL the SQL won't fire until the end).
I suspect you also need to be sure each named_scope tests the existence of what is passed in:
named_scope :with_country, lambda { |country_id| country_id.nil? ? {} : {:conditions=>...} }
This will be easy with Rails 3:
products = Product.where("price = 100").limit(5) # No query executed yet
products = products.order("created_at DESC") # Adding to the query, still no execution
products.each { |product| puts product.price } # That's when the SQL query is actually fired
class Product < ActiveRecord::Base
named_scope :pricey, where("price > 100")
named_scope :latest, order("created_at DESC").limit(10)
end
The short answer is to simply shift the scope as required, narrowing it down depending on what parameters are present:
scope = Example
# Only apply to parameters that are present and not empty
if (!params[:foo].blank?)
scope = scope.with_foo(params[:foo])
end
if (!params[:bar].blank?)
scope = scope.with_bar(params[:bar])
end
results = scope.all
A better approach would be to use something like Searchlogic (http://github.com/binarylogic/searchlogic) which encapsulates all of this for you.