My Rails app has the following conditions:
Each Style has many Bookings
Each Booking has a single warehouse value, and a single netbooked value
I need to update the warehouse_netbooked column of every Style with a hash containing the total netbooked sum for each warehouse across all of the style's bookings.
My current code works, but is way too slow (each iteration is taking ~0.5s, and there are thousands of styles):
def assign_warehouse_bookings
warehouses = ["WH1","WH2","WH3"]
Style.all.each do |s|
style_warehouse_bookings = Hash.new
warehouses.each do |wh|
total_netbooked = s.bookings.where(warehouse: wh).sum(:netbooked)
style_warehouse_bookings[wh] = total_netbooked
end
s.update(warehouse_netbooked: "#{style_warehouse_bookings}")
end
end
Here a small change to your code to avoid to do many queries following #eric-duminil advise
#styles = Style.includes(:bookings)
#styles.each do |s|
style_warehouse_bookings = Hash.new
warehouses.each do |wh|
total_netbooked = s.bookings.map {|book| book.warehouse.eql?(wh) ? book.netbooked : 0}.sum
style_warehouse_bookings[wh] = total_netbooked
end
s.update(warehouse_netbooked: "#{style_warehouse_bookings}")
end
end
I hope it help you.
In your case, you have for a single style, you are fetching booking with 3 type of warehouses finding sum of netbooked for each type of warehouse. This is highly inefficient.
One good rule is, first fetch required data from database, and after fetching data use ruby to handle data. It's important to fetch only required data. So, in your case, you can fetch styles and all related bookings. Then you can iterate through collection of bookings and prepare style_warehouse_bookings hash.
use find_each instead of each
use includes or preload to preload data.
Here is simple example which will definitely improve performance,
warehouses = ["WH1","WH2","WH3"]
# preload bookings with style, `preload` used explicitely instead of `includes` to prevent cross join queries.
styles = Style.joins(:bookings).where('bookings.warehouse' => warehouses).preload(:bookings)
# find_each fetches data in batches of 1000 records
styles.find_each do |s|
style_warehouse_bookings = Hash.new
warehouses.each do |wh|
# select and sum methods of ruby are used instead of where and sum of active-record
total_netbooked = s.bookings.select{ |booking| booking.warehouse = wh }.sum(&:netbooked)
style_warehouse_bookings[wh] = total_netbooked
end
s.update(warehouse_netbooked: "#{style_warehouse_bookings}")
end
Read in depth about preload, includes and joins at eager loading associations documentation. Apart from that I wrote an article on when to use preload, includes and joins here which can help.
I think you want to do batch update. If I am correct check the following
link
Is there anything like batch update in Rails?
Also you can introduce transaction to avoid too many commits
For example
def assign_warehouse_bookings
Style.transaction do
<your remaining code goes here>
end
end
Related
So I have a table trips and a table conditions. Trips has a start_date and end_date, and neither last more than a few hours on the same day. Conditions has a date column. I am trying to compare them to find the trip.id of a date in order to seed with a :trip_id foreign key located in the conditions table with a very large csv. I understand that it is a many_to_many relationship, but I am wondering how to convert what I wrote in Ruby (transitioning to rails currently) to the "ActiveRecord" way without hitting the database so many times.
Here is my code:
Helper method
def self.id_by_date(date)
find_by(date: date).id
end
Main method
def self.sort_temp(range)
array = where(max_temperature: range).all.map {|condition| condition.date}
trip_nums = array.map do |date|
Trip.where(start_date: date.beginning_of_day...date.end_of_day).count
end
output = {}
output[:max] = trip_nums.sort.last
output[:min] = trip_nums.sort.reverse.last
output[:avg] = trip_nums.inject(:+) / trip_nums.length unless trip_nums.length == 0
output
end
EDIT: To be clear, I am trying to find all conditions associated with a day
I have the following models
App has_many features
Feature has_many translations
Now I want to lookup a translation in order see to if I need to create or update.
App.joins(features: :translations).first.features.each do |feature|
languages.each do |lang|
translation = feature.translations.where(language: lang).first_or_initialize
end
end
Now the problem is that I have a db lookup for each language which is quite a problem since I have a lot of features.
How can I benefit from eager loading in order to just lookup the translation in the joined table?
Lets partition the languages into two two arrays, found and not_found:
found, not_found = languages.partition do |lang|
App.joins(:features => :translations).where("translations.language" => lang).present?
end
found_language_translations = Translation.where("language in (?)", found)
new_language_translations = not_found.map{|lang| Translation.new(language: lang) } #This will fire up too many queries though :/
I have a one-to-many relationship where one Thing :has_many Elements
I'm looking for a way to create a Thing and all its N Elements without doing N+1 queries. I tried:
[loop in Thing model]
self.elements.build({...})
...
self.save
But it does a separate insert for each Element.
This capability is not built in.
One option is to use a transaction, which will not eliminate the multiple INSERTs but will send all of them in one request, which will help with performance some. For example:
ActiveRecord::Base.transaction do
1000.times { MyModel.create(options) }
end
To do a true bulk INSERT, though, you'll either have to write and execute a raw query, or use a gem such as activerecord-import (formerly part of ar-extensions). An example from the documentation:
books = []
10.times do |i|
books << Book.new(:name => "book #{i}")
end
Book.import books
I think this may be the best option for you.
I'd like to make a newsfeed for the homepage of a site i'm playing around with. There are two models: Articles, and Posts. If I wanted just one in the newsfeed it would be easy:
#newsfeed_items = Article.paginate(:page => params[:page])
But I would like for the two to be both paginated into the same feed, in reverse chronological order. The default scope for the article and post model are already in that order.
How do I get the articles and posts to be combined in to the newsfeed as such?
Thanks!
EDIT: What about using SQL in the users model?
Just wondering: maybe would it be possible define in User.rb:
def feed
#some sql like (SELECT * FROM articles....)
end
Would this work at all?
in my last project i stuck into a problem, i had to paginate multiple models with single pagination in my search functionality. it should work in a way that the first model should appear first when the results of the first model a second model should continue the results and the third and so on as one single search feed, just like facebook feeds. this is the function i created to do this functionality
def multi_paginate(models, page, per_page)
WillPaginate::Collection.create(page, per_page) do |pager|
# set total entries
pager.total_entries = 0
counts = [0]
offsets = []
for model in models
pager.total_entries += model.count
counts << model.count
offset = pager.offset-(offsets[-1] || 0)
offset = offset>model.count ? model.count : offset
offsets << (offset<0 ? 0 : offset)
end
result = []
for i in 0...models.count
result += models[i].limit(pager.per_page-result.length).offset(offsets[i]).to_a
end
pager.replace(result)
end
end
try it and let me know if you have any problem with it, i also posted it as an issue to will_paginate repository, if everyone confirmed that it works correctly i'll fork and commit it to the library. https://github.com/mislav/will_paginate/issues/351
for those interested, please check this question: Creating a "feed" from multiple rails models, efficiently?
Here, Victor Piousbox provides a good, efficient solution.
Look at paginate_by_sql method. You can write unione query to fetch both articles and posts:
select 'article' as type, id from articles
union
select 'post' as type, id from posts
You can paginate both if you use AJAX. Here is well explained how to paginate using AJAX with WillPaginate.
You can paginate an array using WillPaginate::Collection.create. So you'd need to use ActiveRecord to find both sets of data and then combine them in a single array.
Then take a look at https://github.com/mislav/will_paginate/blob/master/lib/will_paginate/collection.rb for documentation on how to use the Collection to paginate any array.
Pretty sure that I'm missing something really simple here:
I'm trying to display a series of pages that contain instances of two different models - Profiles and Groups. I need them ordering by their name attribute. I could select all of the instances for each model, then sort and paginate them, but this feels sloppy and inefficient.
I'm using mislav-will_paginate, and was wondering if there is any better way of achieving this? Something like:
[Profile, Group].paginate(...)
would be ideal!
Good question, I ran into the same problem a couple of times. Each time, I ended it up by writing my own sql query based on sql unions (it works fine with sqlite and mysql). Then, you may use will paginate by passing the results (http://www.pathf.com/blogs/2008/06/how-to-use-will_paginate-with-non-activerecord-collectionarray/). Do not forget to perform the query to count all the rows.
Some lines of code (not tested)
my_query = "(select posts.title from posts) UNIONS (select profiles.name from profiles)"
total_entries = ActiveRecord::Base.connection.execute("select count(*) as count from (#{my_query})").first['count'].to_i
results = ActiveRecord::Base.connection.select_rows("select * from (#{my_query}) limit #{limit} offset #{offset}")
Is it overkilled ? Maybe but you've got the minimal number of queries and results are consistent.
Hope it helps.
Note: If you get the offset value from a http param, you should use sanitize_sql_for_conditions (ie: sql injection ....)
You can get close doing something like:
#profiles, #groups = [Profile, Group].map do |clazz|
clazz.paginate(:page => params[clazz.to_s.downcase + "_page"], :order => 'name')
end
That will then paginate using page parameters profile_page and group_page. You can get the will_paginate call in the view to use the correct page using:
<%= will_paginate #profiles, :page_param => 'profile_page' %>
....
<%= will_paginate #groups, :page_param => 'group_page' %>
Still, I'm not sure there's a huge benefit over setting up #groups and #profiles individually.
in my last project i stuck into a problem, i had to paginate multiple models with single pagination in my search functionality.
it should work in a way that the first model should appear first when the results of the first model a second model should continue the results and the third and so on as one single search feed, just like facebook feeds.
this is the function i created to do this functionality
def multi_paginate(models, page, per_page)
WillPaginate::Collection.create(page, per_page) do |pager|
# set total entries
pager.total_entries = 0
counts = [0]
offsets = []
for model in models
pager.total_entries += model.count
counts << model.count
offset = pager.offset-(offsets[-1] || 0)
offset = offset>model.count ? model.count : offset
offsets << (offset<0 ? 0 : offset)
end
result = []
for i in 0...models.count
result += models[i].limit(pager.per_page-result.length).offset(offsets[i]).to_a
end
pager.replace(result)
end
end
try it and let me know if you have any problem with it, i also posted it as an issue to will_paginate repository, if everyone confirmed that it works correctly i'll fork and commit it to the library.
https://github.com/mislav/will_paginate/issues/351
Have you tried displaying two different sets of results with their own paginators and update them via AJAX? It is not exactly what you want, but the result is similar.