I've got something like the following:
module Bar < ActiveRecord::Base
belongs_to :foo
...
module Foo < ActiveRecord::Base
has_many :bars, dependent: :destroy
def build_bars
1000.times do |i|
bars.build(num: i)
end
end
def create_default_bars!
build_bars
save
end
Notice that Foo#build_bars is cheap. Even though it loops 1000 times, it takes very little time. But then, once you hit save, suddenly ActiveRecord decides to perform 1000 inserts, which is incredibly slow.
How can I write a custom save_the_bars method such that it performs a single bulk INSERT query for all of the bars I have built up (I remind you, the 1000 builds are apparently very cheap), but which functions as a drop-in replacement for save in this example?
I'm expecting an answer along the lines of recommendation #3 of this blog post:
https://www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-without-killing-your-performance/
But since my example uses build and relies on some slightly nontrivial rails magic, it isn't immediately obvious how to translate it over. Bonus points for benchmarks!
I would try the activerecord-import gem.
def save_the_bars(bars)
Bars.import bars
end
This call to import does whatever is most efficient for the underlying database adapter. Pretty slick, eh?
For bonus points: Benchmarks
Question-asker here, hijacking this answer with details on what I did following the above suggestion:
def build_bars
built_bars = []
1000.times do |i|
built_bars << bars.build(num: i)
end
built_bars
end
def create_default_bars
save
Bar.insert built_bars, validate: false
reload
end
This gave a nice speedup for relatively little effort. I still suspect that more speedup could be had (by leveraging the nuances of insert) but I'm satisfied with this for now. In my use case, it was safe to turn off validation, since the method generating all the Bars is guaranteed to generate valid ones.
Related
Removed N+1 queries but it didn't help me. There are only 40 objects and it takes 15 seconds.
I quess there are so many Stock.with_translations(I18n.locale) and Distributor.with_translations(I18n.locale) db calls that serializing works so slowly. How could I refactor that db calls?
class ShopsSerializer < ActiveModel::Serializer
include ActionView::Helpers::SanitizeHelper
attributes :id, :title, :description, :audio_sizes, :stocks_count, :image_sizes, :audio_count, :country
has_many :images, serializer: ShopImageSerializer
has_many :products, serializer: ProductSerializer
def image_sizes
total = 0.0
stocks = Stock.with_translations(I18n.locale).includes(:images).where(city_id: object.id)
stocks.each do |stock|
sum = stock.images.inject(0){|sum, item| sum + item.image_size if item.image.present?} || 0
total += sum
end
total.round(2)
end
def audio_sizes
size = 0.0
Stock.with_translations(I18n.locale).where(city_id: object.id).map{|s| size += s.audio.size if s.audio.present?}
Distributor.with_translations(I18n.locale).where(city_id: object.id).map{|d| size += d.audio.size if d.audio.present?}
size
end
def stocks_count
Stock.with_translations(I18n.locale).where(city_id: object.id).count + Distributor.with_translations(I18n.locale).where(city_id: object.id).count
end
def audio_count
count = 0
Stock.with_translations(I18n.locale).where(city_id: object.id).map do |s|
if s.audio.present?
count += 1
end
end
Distributor.with_translations(I18n.locale).where(city_id: object.id).map do |d|
if d.audio.present?
count += 1
end
end
count
end
end
Ideally you should move your calculations at the db level but I dont have time to write this for you. Otherwise you still have the N + 1 problem because for each object to serialize, you query stuff.
Anyway a win in your situation would be to at least do queries once, memoizing them like:
class ShopsSerializer < ActiveModel::Serializer
include ActionView::Helpers::SanitizeHelper
attributes :id, :title, :description, :audio_sizes, :stocks_count, :image_sizes, :audio_count, :country
has_many :images, serializer: ShopImageSerializer
has_many :products, serializer: ProductSerializer
def image_sizes
total = 0.0
stocks.each do |stock|
sum = stock.images.inject(0){|sum, item| sum + item.image_size if item.image.present?} || 0
total += sum
end
total.round(2)
end
def audio_sizes
size = 0.0
stocks.map{|s| size += s.audio.size if s.audio.present?}
distributors.map{|d| size += d.audio.size if d.audio.present?}
size
end
def stocks_count
stocks.count + distributors.count
end
def audio_count
count = 0
stocks.map do |s|
if s.audio.present?
count += 1
end
end
distributors.map do |d|
if d.audio.present?
count += 1
end
end
count
end
private
def stocks
#stocks ||= Stock.with_translations(I18n.locale).includes(:images).where(city_id: object.id)
end
def distributors
#distributors||= Distributor.with_translations(I18n.locale).where(city_id: object.id)
end
end
You don't say what version of AMS you're on, how you're using the serializer, or much about your ar models or association serializers..
You also don't say what you've tried, or what docs you've read, so it's hard to know your investment in solving it yourself vs. asking the internet to do the work for you. If that sounds harsh, sorry, it's just comes from experience handling open source issues.
That said, AMS itself won't do any db operations for you. If you want to eager load anything, that's something you need to do in your app, which means reading rails docs on associations and querying
It is a common problem in asking tech questions to not give enough info. I recommend you take a look at https://www.chiark.greenend.org.uk/~sgtatham/bugs.html or https://github.com/rails-api/active_model_serializers/blob/f5ec8ed9d4624afa6ede9b39d51d145b53b1f344/CONTRIBUTING.md#filing-an-issue or https://github.com/norman/yourbugreportneedsmore.info/blob/master/index.html
Quoting the last one:
Hello there!
You've been directed to this website because you submitted a bug report to
an open source project, but you provided too little information for the
developers to be able to help you. Does this look familiar?
Hi, I'm getting a weird error when I use <program>, do you know
what might be wrong?
Debugging software is hard, even when you have the code in front of you.
Now imagine, trying to debug software on somebody else's computer, without
any access to the code, without knowing what operating system is on the
computer, or even what version of your software is being used. Your only
hint is that "there's a weird error" and you have 1 line out of a 50 line
stack trace to work with. Sound impossible? That's because it is!
So you want help?
If you want to actually get your problem solved, here is how you can
submit a good bug report that a developer will actually respond to:
Got a stack trace? Send the whole thing - or better yet, send a link to
it pasted on Gist or Pastie.
Provide context, for example what version of Ruby or Python or COBOL or
whatever you're using, as well as the code that causes the problem.
Again, Gist and Pastie are your friends.
Better yet, create a small program that reproduces the problem, and put
it on Github, or zip it and send it in
an email.
Even better yet, if you are able to, add a failing test case that
demonstrates the problem you are having, and send it as a pull request
or patch.
Too much info?
If you only remember one thing, remember this: reproducibility is
key. If I can't reproduce your problem, I can't fix it.
Not enough info?
For a longer guide on proper bug reporting, please check
Simon Tatham's excellent article.
Do you remember DRY? I would refactor
Stock.with_translations(I18n.locale).where(city_id: object.id)
to Stock model and make it a scope or scopes. It could be useful for the future too.
It may be that Rails will cache it better then.
One point to speed up your query could be by adding index to stocks.city_id unless you have it already.
You could also check the performance by joining tables
joins(:images) instead of includes(:images)
Using rails caching instead of activemodel serializers caching solved my problem. link
For each of those 40 items you are querying Stock.with_translations(I18n.locale).includes(:images) once, Stock.with_translations(I18n.locale) twice and Distributor.with_translations(I18n.locale) one time.
This leads to at least 40*2 + 40*2 + 40 queries.
You clearly need a way to create association between Shop and Stock, Distributor, probably with has_many :through. But as you are trying to access count of items from Stock and Distributor across city you can query them and pass them along with options in serializer, or you can memoize and run those queries once as suggested by #apneadiving.
I'm a bit of a noob programmer so apologies if the question isn't clear enough.
I'm trying to create a basic rails app where I have 3 different tables: usages(month, usage), prices(month, price) and spends(month, spend).
I'm trying to get it so that spend = usages.usage * prices.price. I've put the following code into my Spend model:
class Spend < ActiveRecord::Base
c = Usage.all.count
i = 1
while i <= c
u = Usage.find(i)
p = Price.find(i)
Spend.create(month:u.month, spend:u.usage*p.price)
i += 1
end
end
This works great initially, but as soon as I start adding and removing usages and prices, their id's change so it isn't as clear cut. How can I do this in a much better way?
Thanks,
Kev
In this case, I would lean against making a separate Spend model, since all it does is calculate data that is already present in the database. Unless you have severe caching requirements (and I doubt it in your case), you can use simple instance methods to retrieve the data you want.
First figure out how your Usage and Price models are related. Since you seem to be associating them by id, it appears to be a one-to-one relationship (correct me if I'm wrong on this). However, associating by assuming they have the same primary key is a dangerous approach - rather have one model point to the other using a foreign key. We'll pick the Price model to hold a primary key for Usage, but the reverse can also work. You'll need to add a column using a migration like this:
def change
add_column :prices, :usage_id, :integer
end
Your models should then look like this:
class Usage < ActiveRecord::Base
has_one :price
def spend
usage * price.price
end
end
class Price < ActiveRecord::Base
belongs_to :usage
end
And you can find your spend value for an individual usage item like this:
usage = Usage.find(some_id)
puts usage.spend
Or you can get multiple 'spends' like this:
Usage.include(:price).each do |usage|
puts usage.spend
end
I've left out any reference to month, as I'm not sure how you are using it or if it's needed at all for calculating spend.
Have a look at the Active Record association guide: http://guides.rubyonrails.org/association_basics.html
I have a model Foo with several belongs_to associations; I'll refer to them here as Bar and Baz. So the model would look like this:
class Foo
belongs_to :bar
belongs_to :baz
def do_stuff_with_bar_and_baz
bar.do_stuff(baz)
end
end
We noticed that do_stuff_with_bar_and_baz was unusually slow (~4 seconds), even though the underlying MySQL statements were very fast (~0.5ms). I benchmarked the bar and baz calls, and discovered that they took ~2.3s and ~221ms respectively... just to go through the Rails association code.
I then put in the following methods:
class Foo
belongs_to :bar
belongs_to :baz
def bar
Bar.find(self.bar_id)
end
def baz
Baz.find(self.baz_id)
end
def do_stuff_with_bar_and_baz
bar.do_stuff(baz)
end
end
This bypasses the ActiveRecord association code and loads the associated records directly. With this code, the time to load the Bar and Baz in do_stuff_with_bar_and_baz dropped to 754ms and 5ms respectively.
This is disheartening. The standard Rails associations appear to be horrendously inefficient, but I really don't want to have to replace all of them (that defeats a the purpose of a significant amount of ActiveRecord).
So, I'm looking for alternatives:
Is there something that I'm potentially doing wrong that's slowing things down? (The real code is obviously more complicated that this. However, the belongs_to is accurate; there's no additional options on the real code).
Have other people encountered this?
How have they dealt with it?
Seems like you have different queries and problem is not in rails, but in DB.
Maybe you have other conditions and you haven't right indexes for they.
I'm creating an online bookmaker odds comparison site for soccer and I'm wondering how to calculate the best odds in Ruby/Rails.
I have two models: fixture and odds
Fixture has home and away teams, and odds model has bookmaker ID, home odds, draw odds and away odds.
I have selections which just stores the selected fixtures/teams in the DB.
I'm thinking of doing it this way where I create an multi-dimensional array of the different bookmakers and then add the fixture_id and 1,2 or 3 for home/draw/away and then use that as the key to add the odds
Something like odds[bookmaker][fixture][1/2/3] = price then add up the odds = count(odds[bookmaker][fixture][1/2/3])?
Is there an easier way? Maybe do it in the DB?
Without taking performance into account - it's probably not an issue and anyway, we shouldn't optimise for performance until we know we have a problem - I'd say you might introduce a Bookmaker model (if only to store the name) and start making use of ActiveRecord associations. I'd also consider splitting Odds into the three individual result types, which could be more flexible, especially if you want to add more bets later. You might get something like:
class Bookmaker < ActiveRecord::Base
has_many :odds
end
class Odd < ActiveRecord::Base # good name? Price is almost as common and less likely to be misinterpreted
belongs_to :fixture
belongs_to :bookmaker
# let's assume we use result type = 1/2/3 or maybe :home/:draw/:away
end
class Fixture < ActiveRecord::Base
has_many :odds
end
What you look to be trying to do is calculate the best price for each result across all bookies making a price on that fixture, or the "overround". If it's less than 100% then a potential arbitrage exists.
class Odd
named_scope :for_result, lambda { |res_tp| {:conditions => ['type = ?', res_tp]}}
end
class Fixture
def best_price(res_type)
# assumes you have odds stored as a percentage
odds.for_result(res_type).minimum(:pctage)
end
def overround
[:home, :away, :draw].inject(0.0){|sum, res_tp| sum + best_price(res_tp)}
end
end
I'm sure the above doesn't exactly fit your data, but it might give an idea of how you might go about it.
I need to cache (and expire) all the models in a table.
For example, if i have a model named Currency, i only have less than 10 possible currencies. Therefore, it would be nice to have:
class Currency < ActiveRecord::Base
cache_all(:expire_in => 10.minutes)
end
so that
Currency.all
Currency.find_by_name("USD")
should not hit the DB.
What do you think it could be a good approach?
Also, if you believe it would be better to use a model that is not backed up by a DB, please comment on this. Please notice that i would like to have a AR-style association.
Since the data set is so small, probably the best thing is to cache it in local memory. There are a couple ways to do this, one is to use Memoization like I show here. However that isn't the most efficient because it will store the all method and find_by_name method in separate caches even though they are the same object.
An alternative is to redefine the methods to cache the objects manually. Something like this.
class Currency < ActiveRecord::Base
def self.all
#all_cache ||= super.map(&:freeze) # freeze so you don't modify the cached objects
end
def self.find_by_name(name)
all.detect { |c| c.name.to_s.downcase == name.to_s.downcase }
end
def self.flush_all_cache
#all_cache = nil
end
end
There may be a plugin to handle this for you, but I haven't looked into that at all.