Removed N+1 queries but it didn't help me. There are only 40 objects and it takes 15 seconds.
I quess there are so many Stock.with_translations(I18n.locale) and Distributor.with_translations(I18n.locale) db calls that serializing works so slowly. How could I refactor that db calls?
class ShopsSerializer < ActiveModel::Serializer
include ActionView::Helpers::SanitizeHelper
attributes :id, :title, :description, :audio_sizes, :stocks_count, :image_sizes, :audio_count, :country
has_many :images, serializer: ShopImageSerializer
has_many :products, serializer: ProductSerializer
def image_sizes
total = 0.0
stocks = Stock.with_translations(I18n.locale).includes(:images).where(city_id: object.id)
stocks.each do |stock|
sum = stock.images.inject(0){|sum, item| sum + item.image_size if item.image.present?} || 0
total += sum
end
total.round(2)
end
def audio_sizes
size = 0.0
Stock.with_translations(I18n.locale).where(city_id: object.id).map{|s| size += s.audio.size if s.audio.present?}
Distributor.with_translations(I18n.locale).where(city_id: object.id).map{|d| size += d.audio.size if d.audio.present?}
size
end
def stocks_count
Stock.with_translations(I18n.locale).where(city_id: object.id).count + Distributor.with_translations(I18n.locale).where(city_id: object.id).count
end
def audio_count
count = 0
Stock.with_translations(I18n.locale).where(city_id: object.id).map do |s|
if s.audio.present?
count += 1
end
end
Distributor.with_translations(I18n.locale).where(city_id: object.id).map do |d|
if d.audio.present?
count += 1
end
end
count
end
end
Ideally you should move your calculations at the db level but I dont have time to write this for you. Otherwise you still have the N + 1 problem because for each object to serialize, you query stuff.
Anyway a win in your situation would be to at least do queries once, memoizing them like:
class ShopsSerializer < ActiveModel::Serializer
include ActionView::Helpers::SanitizeHelper
attributes :id, :title, :description, :audio_sizes, :stocks_count, :image_sizes, :audio_count, :country
has_many :images, serializer: ShopImageSerializer
has_many :products, serializer: ProductSerializer
def image_sizes
total = 0.0
stocks.each do |stock|
sum = stock.images.inject(0){|sum, item| sum + item.image_size if item.image.present?} || 0
total += sum
end
total.round(2)
end
def audio_sizes
size = 0.0
stocks.map{|s| size += s.audio.size if s.audio.present?}
distributors.map{|d| size += d.audio.size if d.audio.present?}
size
end
def stocks_count
stocks.count + distributors.count
end
def audio_count
count = 0
stocks.map do |s|
if s.audio.present?
count += 1
end
end
distributors.map do |d|
if d.audio.present?
count += 1
end
end
count
end
private
def stocks
#stocks ||= Stock.with_translations(I18n.locale).includes(:images).where(city_id: object.id)
end
def distributors
#distributors||= Distributor.with_translations(I18n.locale).where(city_id: object.id)
end
end
You don't say what version of AMS you're on, how you're using the serializer, or much about your ar models or association serializers..
You also don't say what you've tried, or what docs you've read, so it's hard to know your investment in solving it yourself vs. asking the internet to do the work for you. If that sounds harsh, sorry, it's just comes from experience handling open source issues.
That said, AMS itself won't do any db operations for you. If you want to eager load anything, that's something you need to do in your app, which means reading rails docs on associations and querying
It is a common problem in asking tech questions to not give enough info. I recommend you take a look at https://www.chiark.greenend.org.uk/~sgtatham/bugs.html or https://github.com/rails-api/active_model_serializers/blob/f5ec8ed9d4624afa6ede9b39d51d145b53b1f344/CONTRIBUTING.md#filing-an-issue or https://github.com/norman/yourbugreportneedsmore.info/blob/master/index.html
Quoting the last one:
Hello there!
You've been directed to this website because you submitted a bug report to
an open source project, but you provided too little information for the
developers to be able to help you. Does this look familiar?
Hi, I'm getting a weird error when I use <program>, do you know
what might be wrong?
Debugging software is hard, even when you have the code in front of you.
Now imagine, trying to debug software on somebody else's computer, without
any access to the code, without knowing what operating system is on the
computer, or even what version of your software is being used. Your only
hint is that "there's a weird error" and you have 1 line out of a 50 line
stack trace to work with. Sound impossible? That's because it is!
So you want help?
If you want to actually get your problem solved, here is how you can
submit a good bug report that a developer will actually respond to:
Got a stack trace? Send the whole thing - or better yet, send a link to
it pasted on Gist or Pastie.
Provide context, for example what version of Ruby or Python or COBOL or
whatever you're using, as well as the code that causes the problem.
Again, Gist and Pastie are your friends.
Better yet, create a small program that reproduces the problem, and put
it on Github, or zip it and send it in
an email.
Even better yet, if you are able to, add a failing test case that
demonstrates the problem you are having, and send it as a pull request
or patch.
Too much info?
If you only remember one thing, remember this: reproducibility is
key. If I can't reproduce your problem, I can't fix it.
Not enough info?
For a longer guide on proper bug reporting, please check
Simon Tatham's excellent article.
Do you remember DRY? I would refactor
Stock.with_translations(I18n.locale).where(city_id: object.id)
to Stock model and make it a scope or scopes. It could be useful for the future too.
It may be that Rails will cache it better then.
One point to speed up your query could be by adding index to stocks.city_id unless you have it already.
You could also check the performance by joining tables
joins(:images) instead of includes(:images)
Using rails caching instead of activemodel serializers caching solved my problem. link
For each of those 40 items you are querying Stock.with_translations(I18n.locale).includes(:images) once, Stock.with_translations(I18n.locale) twice and Distributor.with_translations(I18n.locale) one time.
This leads to at least 40*2 + 40*2 + 40 queries.
You clearly need a way to create association between Shop and Stock, Distributor, probably with has_many :through. But as you are trying to access count of items from Stock and Distributor across city you can query them and pass them along with options in serializer, or you can memoize and run those queries once as suggested by #apneadiving.
Related
I've got something like the following:
module Bar < ActiveRecord::Base
belongs_to :foo
...
module Foo < ActiveRecord::Base
has_many :bars, dependent: :destroy
def build_bars
1000.times do |i|
bars.build(num: i)
end
end
def create_default_bars!
build_bars
save
end
Notice that Foo#build_bars is cheap. Even though it loops 1000 times, it takes very little time. But then, once you hit save, suddenly ActiveRecord decides to perform 1000 inserts, which is incredibly slow.
How can I write a custom save_the_bars method such that it performs a single bulk INSERT query for all of the bars I have built up (I remind you, the 1000 builds are apparently very cheap), but which functions as a drop-in replacement for save in this example?
I'm expecting an answer along the lines of recommendation #3 of this blog post:
https://www.coffeepowered.net/2009/01/23/mass-inserting-data-in-rails-without-killing-your-performance/
But since my example uses build and relies on some slightly nontrivial rails magic, it isn't immediately obvious how to translate it over. Bonus points for benchmarks!
I would try the activerecord-import gem.
def save_the_bars(bars)
Bars.import bars
end
This call to import does whatever is most efficient for the underlying database adapter. Pretty slick, eh?
For bonus points: Benchmarks
Question-asker here, hijacking this answer with details on what I did following the above suggestion:
def build_bars
built_bars = []
1000.times do |i|
built_bars << bars.build(num: i)
end
built_bars
end
def create_default_bars
save
Bar.insert built_bars, validate: false
reload
end
This gave a nice speedup for relatively little effort. I still suspect that more speedup could be had (by leveraging the nuances of insert) but I'm satisfied with this for now. In my use case, it was safe to turn off validation, since the method generating all the Bars is guaranteed to generate valid ones.
I'm a bit of a noob programmer so apologies if the question isn't clear enough.
I'm trying to create a basic rails app where I have 3 different tables: usages(month, usage), prices(month, price) and spends(month, spend).
I'm trying to get it so that spend = usages.usage * prices.price. I've put the following code into my Spend model:
class Spend < ActiveRecord::Base
c = Usage.all.count
i = 1
while i <= c
u = Usage.find(i)
p = Price.find(i)
Spend.create(month:u.month, spend:u.usage*p.price)
i += 1
end
end
This works great initially, but as soon as I start adding and removing usages and prices, their id's change so it isn't as clear cut. How can I do this in a much better way?
Thanks,
Kev
In this case, I would lean against making a separate Spend model, since all it does is calculate data that is already present in the database. Unless you have severe caching requirements (and I doubt it in your case), you can use simple instance methods to retrieve the data you want.
First figure out how your Usage and Price models are related. Since you seem to be associating them by id, it appears to be a one-to-one relationship (correct me if I'm wrong on this). However, associating by assuming they have the same primary key is a dangerous approach - rather have one model point to the other using a foreign key. We'll pick the Price model to hold a primary key for Usage, but the reverse can also work. You'll need to add a column using a migration like this:
def change
add_column :prices, :usage_id, :integer
end
Your models should then look like this:
class Usage < ActiveRecord::Base
has_one :price
def spend
usage * price.price
end
end
class Price < ActiveRecord::Base
belongs_to :usage
end
And you can find your spend value for an individual usage item like this:
usage = Usage.find(some_id)
puts usage.spend
Or you can get multiple 'spends' like this:
Usage.include(:price).each do |usage|
puts usage.spend
end
I've left out any reference to month, as I'm not sure how you are using it or if it's needed at all for calculating spend.
Have a look at the Active Record association guide: http://guides.rubyonrails.org/association_basics.html
I have read a lot about soft deletes and archive and saw all the pros and cons. I'm still confused on which approach would work best for my situation. I'll use the concept of posts and comments to see if I can explain it a bit easier
Post -> Comments
Post.all
Outside RSS Feeds -> Post -> Comments
RSSFeed.posts (Return the ones that are deleted or not)
Post gets "deleted" but I need the posts still accessible from say an RSS Feed but not the admin of the application.
I hear a lot of headaches with soft deletes but think it might make the most sense for my application and feel if I use Archive then I would have to run multiple queries
RSSFeed.posts || RSSFeed.archived_posts
not sure which would be more efficient or more a pain in the #$$. Thoughts or examples? I know this example sounds stupid but trying to think of multiple situations that could be used to figure out which way to go.
Just add another column in your database and call it archivated.
Use link_to_if for the links:
<%= link_to_unless #post.archivated?, #post.name, post_path(#path) %>
Some more rails goodness:
app/models/post.rb
class Post < ActiveRecord::Base
default_scope where( active: true )
def archivate
unless self.archivated?
self.archivated = true
self.save
end
end
def dectivate
if self.archivated?
self.archivated = false
self.save
end
end
end
app/models/archive.rb
class Archive < Post
set_table_name :posts # make this model use the posts table
default_scope where( active: false )
end
Now you can do stuff like this:
#post = Post.find(some_id)
#post.archivate
Archive.find(some_id) # should return the post you just archivated
Definitely you will get idea , take a look :
http://railspikes.com/2010/2/26/acts-as-archive
Live site: http://iatidata.heroku.com
Github: https://github.com/markbrough/IATI-Data
Based on aid information released through the IATI Registry: iatiregistry.org
I'm a bit of a Rails n00b so sorry if this is a really stupid question.
There are two key Models in this app:
Activity - which contains details
such as recipient country, funding
organisation
Transaction - which contains details such as how much money (value) was committed or disbursed (transaction_type), when, to whom, etc.
All Transactions nest under an Activity. Each Activity has multiple Transactions. They are connected together by activity_id. has_many :transactions and belongs_to :activity are defined in the Activity and Transaction Models respectively.
So: all of this works great when I'm trying to get details of transactions for a single activity - either when looking at a single activity (activity->show) or looping through activities on the all activities page (activity->index). I just call
#activities.each do |activity|
activity.transactions.each do |transaction|
transaction.value # do something like display it
end
end
But what I now really want to do is to get the sum of all transactions for all activities (subject to :conditions for the activity).
What's the best way to do this? I guess I could do something like:
#totalvalue = 0
#activities.each do |activity|
activity.transactions.each do |transaction|
#totalvalue = #totalvalue + transaction.value
end
end
... but that doesn't seem very clean and making the server do unnecessary work. I figure it might be something to do with the model...?! sum() is another option maybe?
This has partly come about because I want to show the total amount going to each country for the nice bubbles on the front page :)
Thanks very much for any help!
Update:
Thanks for all the responses! So, this works now:
#thiscountry_activities.each do |a|
#thiscountry_value = #thiscountry_value + a.transactions.sum(:value)
end
But this doesn't work:
#thiscountry_value = #thiscountry_activities.transactions.sum(:value)
It gives this error:
undefined method `transactions' for #<Array:0xb5670038>
Looks like I have some sort of association problem. This is how the models are set up:
class Transaction < ActiveRecord::Base
belongs_to :activity
end
class Activity < ActiveRecord::Base
has_and_belongs_to_many :policy_markers
has_and_belongs_to_many :sectors
has_many :transactions
end
I think this is probably quite a simple problem, but I can't work out what's going on. The two models are connected together via id (in Activity) and activity_id (in Transactions).
Thanks again!
Use Active Record's awesome sum method, available for classes:
Transaction.sum(:value)
Or, like you want, associations:
activity.transactions.sum(:value)
Let the database do the work:
#total_value = Transaction.sum(:value)
This gives the total for all transactions. If you have some activities already loaded, you can filter them this way:
#total_value = Transaction.where(:activity_id => #activities.map(&:id)).sum(:value)
You can do it with one query:
#total_value = Transaction.joins(:activity).where("activities.name" => 'foo').sum(:value)
My code was getting pretty messy summing up virtual attributes. So I wrote this little method to do it for me. You just pass in a collection and a method name as a string or symbol and you get back a total. I hope someone finds this useful.
def vsum collection, v_attr # Totals the virtual attributes of a collection
total = 0
collection.each { |collect| total += collect.method(v_attr).call }
return total
end
# Example use
total_credits = vsum(Account.transactions, :credit)
Of course you don't need this if :credit is a table column. You are better off using the built in ActiveRecord method above. In my case i have a :quantity column that when positive is a :credit and negative is a :debit. Since :debit and :credit are not table columns they can't be summed using ActiveRecord.
As I understood, you would like to have the sum of all values of the transaction table. You can use SQL for that. I think it will be faster than doing it the Ruby way.
select sum(value) as transaction_value_sum from transaction;
You could do
#total_value = activity.transactions.sum(:value)
http://ar.rubyonrails.org/classes/ActiveRecord/Calculations/ClassMethods.html
I'm creating an online bookmaker odds comparison site for soccer and I'm wondering how to calculate the best odds in Ruby/Rails.
I have two models: fixture and odds
Fixture has home and away teams, and odds model has bookmaker ID, home odds, draw odds and away odds.
I have selections which just stores the selected fixtures/teams in the DB.
I'm thinking of doing it this way where I create an multi-dimensional array of the different bookmakers and then add the fixture_id and 1,2 or 3 for home/draw/away and then use that as the key to add the odds
Something like odds[bookmaker][fixture][1/2/3] = price then add up the odds = count(odds[bookmaker][fixture][1/2/3])?
Is there an easier way? Maybe do it in the DB?
Without taking performance into account - it's probably not an issue and anyway, we shouldn't optimise for performance until we know we have a problem - I'd say you might introduce a Bookmaker model (if only to store the name) and start making use of ActiveRecord associations. I'd also consider splitting Odds into the three individual result types, which could be more flexible, especially if you want to add more bets later. You might get something like:
class Bookmaker < ActiveRecord::Base
has_many :odds
end
class Odd < ActiveRecord::Base # good name? Price is almost as common and less likely to be misinterpreted
belongs_to :fixture
belongs_to :bookmaker
# let's assume we use result type = 1/2/3 or maybe :home/:draw/:away
end
class Fixture < ActiveRecord::Base
has_many :odds
end
What you look to be trying to do is calculate the best price for each result across all bookies making a price on that fixture, or the "overround". If it's less than 100% then a potential arbitrage exists.
class Odd
named_scope :for_result, lambda { |res_tp| {:conditions => ['type = ?', res_tp]}}
end
class Fixture
def best_price(res_type)
# assumes you have odds stored as a percentage
odds.for_result(res_type).minimum(:pctage)
end
def overround
[:home, :away, :draw].inject(0.0){|sum, res_tp| sum + best_price(res_tp)}
end
end
I'm sure the above doesn't exactly fit your data, but it might give an idea of how you might go about it.