How to set a counter cache value to a given one? - ruby-on-rails

I am using Ruby on Rails 3.2.2 and I would like to set a counter cache value to a "custom" one. That is, at this time (in my migration file) I am trying to use the following code:
def up
add_column :articles, :comments_count, :integer, :default => 0
Article.reset_column_information
Article.find_each do |article|
# Note: The following code doesn't work (when I migrate the database it
# raises the error "comments_count is marked as readonly").
Article.update_column(:comments_count, article.custom_comments.count)
end
end
In other words, I would like to set the :comments_count value (a counter cache database table column) to a custom value (in my case that value is article.custom_comments.count - note: the custom_comments is not an ActiveRecord Association but a method stated in the Article model class; it returns an integer value as well) that is not related to a has_many associations.
Maybe, I could / should use something like
Article.reset_column_information
Article.find_each do |article|
Article.reset_counters(article.id, ...)
end
but it seems that the reset_counters method cannot work without has_many associations.
How can I set the :comments_count counter cache value to a given value that is related to a "custom association"?

The accept answer includes the iterating method, which is wrong for existing values of comment_count other than 0: update_counter sets the counter relative to it's current values. To set an absolute value, do:
Article.update_counters(article.id, comments_count: comments.count - article.comments_count)
If you have to fetch each row's correct count anyway, you can also more easily use Article.reset_counters(article.id, :comments)
To do it with far fewer queries, use this:
Author
.joins(:books)
.select("authors.id, authors.books_count, count(books.id) as count")
.group("authors.id")
.having("authors.books_count != count(books.id)")
.pluck(:id, :books_count, "count(books.id)")
.each_with_index do |(author_id, old_count, fixed_count), index|
puts "at index %7i: fixed author id %7i, new books_count %4i, previous count %4i" % [index, author_id, fixed_count, old_count] if index % 1000 == 0
Author.update_counters(author_id, books_count: fixed_count - old_count)
end

You describe comments_count as a counter cache, yet a counter cache is strictly defined as the number of associated records in a has_many relationship, which you say this isn't.
If the only way to get the value you want is via method on Article, then you're going to have to iterate over all your Article objects and update each one.
Article.find_each do |article|
article.update_attribute(:comments_count, article.custom_comments.count)
end
This is pretty inefficient, since it's loading and saving every object.
If the definition of custom_comments (which you don't actually explain) is something you can express in SQL, it would undoubtedly be faster to do this update in the database. Which might look something like this:
CREATE TEMP TABLE custom_comment_counts_temp AS
SELECT articles.id as id, count(comments.id) as custom_comments
FROM articles
LEFT JOIN comments ON articles.id = comments.article_id
WHERE <whatever condition indicates custom comments>
GROUP BY articles.id;
CREATE INDEX ON custom_comments_counts_temp(id);
UPDATE articles SET comments_count = (SELECT custom_comments FROM custom_comment_counts_temp WHERE custom_comment_counts_temp.id = articles.id);
DROP TABLE custom_comment_counts_temp;
(this assumes postgresql - if you're using mySQL or some other database, it may look different. If you're not using a relational database at all, it may not be possible)
Additionally, since it's not a counter cache according to Rails' fairly narrow definition, you'll need to write some callbacks that keep these values updated - probably an after_save callback on comment, something like this:
comment.rb:
after_save :set_article_custom_comments
def set_article_custom_comments
a = self.article
a.update_attribute(:comments_count, a.custom_comments.count)
end

Related

How to efficiently update associated collection in rails (eager loading)

I have a simple association like
class Slot < ActiveRecord::Base
has_many :media_items, dependent: :destroy
end
class MediaItem < ActiveRecord::Base
belongs_to :slot
end
The MediaItems are ordered per Slot and have a field called ordering.
And want to avoid n+1 querying but nothing I tried works. I had read several blogposts, railscasts etc but hmm.. they never operate on a single model and so on...
What I do is:
def update
#slot = Slot.find(params.require(:id))
media_items = #slot.media_items
par = params[:ordering_media]
# TODO: IMP remove n+1 query
par.each do |item|
item_id = item[:media_item_id]
item_order = item[:ordering]
media_items.find(item_id).update(ordering: item_order)
end
#slot.save
end
params[:ordering_media] is a json array with media_item_id and an integer for ordering
I tried things like
#slot = Slot.includes(:media_items).find(params.require(:id)) # still n+1
#slot = Slot.find(params.require(:id)).includes(:media_items) # not working at all b/c is a Slot already
media_items = #slot.media_items.to_a # looks good but then in the array of MediaItems it is difficult to retrieve the right instance in my loop
This seems like a common thing to do, so I think there is a simple approach to solve this. Would be great to learn about it.
First at all, at this line media_items.find(item_id).update(ordering: item_order) you don't have an n + 1 issue, you have a 2 * n issue. Because for each media_item you make 2 queries: one for find, one for update. To fix you can do this:
params[:ordering_media].each do |item|
MediaItem.update_all({ordering: item[:ordering]}, {id: item[:media_item_id]})
end
Here you have n queries. That is the best we can do, there's no way to update a column on n records with n distinct values, with less than n queries.
Now you can remove the lines #slot = Slot.find(params.require(:id)) and #slot.save, because #slot was not modified or used at the update action.
With this refactor, we see a problem: the action SlotsController#update don't update slot at all. A better place for this code could be MediaItemsController#sort or SortMediaItemsController#update (more RESTful).
At the last #slot = Slot.includes(:media_items).find(params.require(:id)) this is not n + 1 query, this is 2 SQL statements query, because you retrieve n media_items and 1 slot with only 2 db calls. Also it's the best option.
I hope it helps.

Rails best way to get previous and next active record object

I need to get the previous and next active record objects with Rails. I did it, but don't know if it's the right way to do that.
What I've got:
Controller:
#product = Product.friendly.find(params[:id])
order_list = Product.select(:id).all.map(&:id)
current_position = order_list.index(#product.id)
#previous_product = #collection.products.find(order_list[current_position - 1]) if order_list[current_position - 1]
#next_product = #collection.products.find(order_list[current_position + 1]) if order_list[current_position + 1]
#previous_product ||= Product.last
#next_product ||= Product.first
product_model.rb
default_scope -> {order(:product_sub_group_id => :asc, :id => :asc)}
So, the problem here is that I need to go to my database and get all this ids to know who is the previous and the next.
Tried to use the gem order_query, but it did not work for me and I noted that it goes to the database and fetch all the records in that order, so, that's why I did the same but getting only the ids.
All the solutions that I found was with simple order querys. Order by id or something like a priority field.
Write these methods in your Product model:
class Product
def next
self.class.where("id > ?", id).first
end
def previous
self.class.where("id < ?", id).last
end
end
Now you can do in your controller:
#product = Product.friendly.find(params[:id])
#previous_product = #product.next
#next_product = #product.previous
Please try it, but its not tested.
Thanks
I think it would be faster to do it with only two SQL requests, that only select two rows (and not the entire table). Considering that your default order is sorted by id (otherwise, force the sorting by id) :
#previous_product = Product.where('id < ?', params[:id]).last
#next_product = Product.where('id > ?', params[:id]).first
If the product is the last, then #next_product will be nil, and if it is the first, then, #previous_product will be nil.
There's no easy out-of-the-box solution.
A little dirty, but working way is carefully sorting out what conditions are there for finding next and previous items. With id it's quite easy, since all ids are different, and Rails Guy's answer describes just that: in next for a known id pick a first entry with a larger id (if results are ordered by id, as per defaults). More than that - his answer hints to place next and previous into the model class. Do so.
If there are multiple order criteria, things get complicated. Say, we have a set of rows sorted by group parameter first (which can possibly have equal values on different rows) and then by id (which id different everywhere, guaranteed). Results are ordered by group and then by id (both ascending), so we can possibly encounter two situations of getting the next element, it's the first from the list that has elements, that (so many that):
have the same group and a larger id
have a larger group
Same with previous element: you need the last one from the list
have the same group and a smaller id
have a smaller group
Those fetch all next and previous entries respectively. If you need only one, use Rails' first and last (as suggested by Rails Guy) or limit(1) (and be wary of the asc/desc ordering).
This is what order_query does. Please try the latest version, I can help if it doesn't work for you:
class Product < ActiveRecord::Base
order_query :my_order,
[:product_sub_group_id, :asc],
[:id, :asc]
default_scope -> { my_order }
end
#product.my_order(#collection.products).next
#collection.products.my_order_at(#product).next
This runs one query loading only the next record. Read more on Github.

Texticle and ActsAsTaggableOn

I'm trying to implement search over tags as part of a Texticle search. Since texticle doesn't search over multiple tables from the same model, I ended up creating a new model called PostSearch, following Texticle's suggestion about System-Wide Searching
class PostSearch < ActiveRecord::Base
# We want to reference various models
belongs_to :searchable, :polymorphic => true
# Wish we could eliminate n + 1 query problems,
# but we can't include polymorphic models when
# using scopes to search in Rails 3
# default_scope :include => :searchable
# Search.new('query') to search for 'query'
# across searchable models
def self.new(query)
debugger
query = query.to_s
return [] if query.empty?
self.search(query).map!(&:searchable)
#self.search(query) <-- this works, not sure why I shouldn't use it.
end
# Search records are never modified
def readonly?; true; end
# Our view doesn't have primary keys, so we need
# to be explicit about how to tell different search
# results apart; without this, we can't use :include
# to avoid n + 1 query problems
def hash
id.hash
end
def eql?(result)
id == result.id
end
end
In my Postgres DB I created a view like this:
CREATE VIEW post_searches AS
SELECT posts.id, posts.name, string_agg(tags.name, ', ') AS tags
FROM posts
LEFT JOIN taggings ON taggings.taggable_id = posts.id
LEFT JOIN tags ON taggings.tag_id = tags.id
GROUP BY posts.id;
This allows me to get posts like this:
SELECT * FROM post_searches
id | name | tags
1 Intro introduction, funny, nice
So it seems like that should all be fine. Unfortunately calling
PostSearch.new("funny") returns [nil] (NOT []). Looking through the Texticle source code, it seems like this line in the PostSearch.new
self.search(query).map!(&:searchable)
maps the fields using some sort of searchable_columns method and does it ?incorrectly? and results in a nil.
On a different note, the tags field doesn't get searched in the texticle SQL query unless I cast it from a text type to a varchar type.
So, in summary:
Why does the object get mapped to nil when it is found?
AND
Why does texticle ignore my tags field unless it is varchar?
Texticle maps objects to nil instead of nothing so that you can check for nil? - it's a safeguard against erroring out checking against non-existent items. It might be worth asking tenderlove himself as to exactly why he did it that way.
I'm not completely positive as to why Texticle ignores non-varchars, but it looks like it's a performance safeguard so that Postgres does not do full table scans (under the section Creating Indexes for Super Speed):
You will need to add an index for every text/string column you query against, or else Postgresql will revert to a full table scan instead of using the indexes.

Searching serialized data, using active record

I'm trying to do a simple query of a serialized column, how do you do this?
serialize :mycode, Array
1.9.3p125 :026 > MyModel.find(104).mycode
MyModel Load (0.6ms) SELECT `mymodels`.* FROM `mymodels` WHERE `mymodels`.`id` = 104 LIMIT 1
=> [43565, 43402]
1.9.3p125 :027 > MyModel.find_all_by_mycode("[43402]")
MyModel Load (0.7ms) SELECT `mymodels`.* FROM `mymodels` WHERE `mymodels`.`mycode` = '[43402]'
=> []
1.9.3p125 :028 > MyModel.find_all_by_mycode(43402)
MyModel Load (1.2ms) SELECT `mymodels`.* FROM `mymodels` WHERE `mymodels`.`mycode` = 43402
=> []
1.9.3p125 :029 > MyModel.find_all_by_mycode([43565, 43402])
MyModel Load (1.1ms) SELECT `mymodels`.* FROM `mymodels` WHERE `mymodels`.`mycode` IN (43565, 43402)
=> []
It's just a trick to not slow your application. You have to use .to_yaml.
exact result:
MyModel.where("mycode = ?", [43565, 43402].to_yaml)
#=> [#<MyModel id:...]
Tested only for MySQL.
Basically, you can't. The downside of #serialize is that you're bypassing your database's native abstractions. You're pretty much limited to loading and saving the data.
That said, one very good way to slow your application to a crawl could be:
MyModel.all.select { |m| m.mycode.include? 43402 }
Moral of the story: don't use #serialize for any data you need to query on.
Serialized array is stored in database in particular fashion eg:
[1, 2, 3, 4]
in
1\n 2\n 3\n etc
hence the query would be
MyModel.where("mycode like ?", "% 2\n%")
put space between % and 2.
Noodl's answer is right, but not entirely correct.
It really depends on the database/ORM adapter you are using: for instance PostgreSQL can now store and search hashes/json - check out hstore. I remember reading that ActiveRecord adapter for PostgreSQl now handles it properly. And if you are using mongoid or something like that - then you are using unstructured data (i.e. json) on a database level everywhere.
However if you are using a db that can't really handle hashes - like MySQL / ActiveRecord combination - then the only reason you would use serialized field is for somet data that you can create / write in some background process and display / output on demand - the only two uses that I found in my experience are some reports ( like a stat field on a Product model - where I need to store some averages and medians for a product), and user options ( like their preferred template color -I really don't need to query on that) - however user information - like their subscription for a mailing list - needs to be searchable for email blasts.
PostgreSQL hstore ActiveRecord Example:
MyModel.where("mycode #> 'KEY=>\"#{VALUE}\"'")
UPDATE
As of 2017 both MariaDB and MySQL support JSON field types.
You can query the serialized column with a sql LIKE statement.
MyModel.where("mycode LIKE '%?%'", 43402)
This is quicker than using include?, however, you cannot use an array as the parameter.
Good news! If you're using PostgreSQL with hstore (which is super easy with Rails 4), you can now totally search serialized data. This is a handy guide, and here's the syntax documentation from PG.
In my case I have a dictionary stored as a hash in an hstore column called amenities. I want to check for a couple queried amenities that have a value of 1 in the hash, I just do
House.where("amenities #> 'wifi => 1' AND amenities #> 'pool => 1'")
Hooray for improvements!
There's a blog post from 2009 from FriendFeed that describes how to use serialized data within MySQL.
What you can do is create tables that function as indexes for any data that you want to search.
Create a model that contains the searchable values/fields
In your example, the models would look something like this:
class MyModel < ApplicationRecord
# id, name, other fields...
serialize :mycode, Array
end
class Item < ApplicationRecord
# id, value...
belongs_to :my_model
end
Creating an "index" table for searchable fields
When you save MyModel, you can do something like this to create the index:
Item.where(my_model: self).destroy
self.mycode.each do |mycode_item|
Item.create(my_model: self, value: mycode_item)
end
Querying and Searching
Then when you want to query and search just do:
Item.where(value: [43565, 43402]).all.map(&:my_model)
Item.where(value: 43402).all.map(&:my_model)
You can add a method to MyModel to make that simpler:
def find_by_mycode(value_or_values)
Item.where(value: value_or_values).all.map(&my_model)
end
MyModel.find_by_mycode([43565, 43402])
MyModel.find_by_mycode(43402)
To speed things up, you will want to create a SQL index for that table.
Using the following comments in this post
https://stackoverflow.com/a/14555151/936494
https://stackoverflow.com/a/15287674/936494
I was successfully able to query a serialized Hash in my model
class Model < ApplicationRecord
serialize :column_name, Hash
end
When column_name holds a Hash like
{ my_data: [ { data_type: 'MyType', data_id: 113 } ] }
we can query it in following manner
Model.where("column_name = ?", hash.to_yaml)
That generates a SQL query like
Model Load (0.3ms) SELECT "models".* FROM "models" WHERE (column_name = '---
:my_data:
- :data_type: MyType
:data_id: 113
')
In case anybody is interested in executing the generated query in SQL terminal it should work, however care should be taken that value is in exact format stored in DB. However there is another easy way I found at PostgreSQL newline character to use a raw string containing newline characters
select * from table_name where column_name = E'---\n:my_data:\n- :data_type: MyType\n :data_id: 113\n'
The most important part in above query is E.
Note: The database on which I executed above is PostgreSQL.
To search serialized list you need to prefix and postfix the data with unique characters.
Example:
Rather than something like:
2345,12345,1234567 which would cause issues you tried to search for 2345 instead, you do something like <2345>,<12345>,<1234567> and search for <2345> (the search query get's transformed) instead. Of course choice of prefix/postfix characters depends on the valid data that will be stored. You might instead use something like ||| if you expect < to be used and potentially| to be used. Of course that increases the data the field uses and could cause performance issues.
Using a trigrams index or something would avoid potential performance issues.
You can serialize it like data.map { |d| "<#{d}>" }.join(',') and deserialize it via data.gsub('<').gsub('>','').split(','). A serializer class would do the job quite well to load/extract tha data.
The way you do this is by setting the database field to text and using rail's serialize model method with a custom lib class. The lib class needs to implement two methods:
def self.dump(obj) # (returns string to be saved to database)
def self.load(text) # (returns object)
Example with duration. Extracted from the article so link rot wouldn't get it, please visit the article for more information. The example uses a single value, but it's fairly straightforward to serialize a list of values and deserialize the list using the methods mentioned above.
class Duration
# Used for `serialize` method in ActiveRecord
class << self
def load(duration)
self.new(duration || 0)
end
def dump(obj)
unless obj.is_a?(self)
raise ::ActiveRecord::SerializationTypeMismatch,
"Attribute was supposed to be a #{self}, but was a #{obj.class}. -- #{obj.inspect}"
end
obj.length
end
end
attr_accessor :minutes, :seconds
def initialize(duration)
#minutes = duration / 60
#seconds = duration % 60
end
def length
(minutes.to_i * 60) + seconds.to_i
end
end
If you have serialized json column and you want to apply like query on that. do it like that
YourModel.where("hashcolumn like ?", "%#{search}%")

How do I calculate the most popular combination of a order lines? (or any similar order/order lines db arrangement)

I'm using Ruby on Rails. I have a couple of models which fit the normal order/order lines arrangement, i.e.
class Order
has_many :order_lines
end
class OrderLines
belongs_to :order
belongs_to :product
end
class Product
has_many :order_lines
end
(greatly simplified from my real model!)
It's fairly straightforward to work out the most popular individual products via order line, but what magical ruby-fu could I use to calculate the most popular combination(s) of products ordered.
Cheers,
Graeme
My suggestion is to create an array a of Product.id numbers for each order and then do the equivalent of
h = Hash.new(0)
# for each a
h[a.sort.hash] += 1
You will naturally need to consider the scale of your operation and how much you are willing to approximate the results.
External Solution
Create a "Combination" model and index the table by the hash, then each order could increment a counter field. Another field would record exactly which combination that hash value referred to.
In-memory Solution
Look at the last 100 orders and recompute the order popularity in memory when you need it. Hash#sort will give you a sorted list of popularity hashes. You could either make a composite object that remembered what order combination was being counted, or just scan the original data looking for the hash value.
Thanks for the tip digitalross. I followed the external solution idea and did the following. It varies slightly from the suggestion as it keeps a record of individual order_combos, rather than storing a counter so it's possible to query by date as well e.g. most popular top 10 orders in the last week.
I created a method in my order which converts the list of order items to a comma separated string.
def to_s
order_lines.sort.map { |ol| ol.id }.join(",")
end
I then added a filter so the combo is created every time an order is placed.
after_save :create_order_combo
def create_order_combo
oc = OrderCombo.create(:user => user, :combo => self.to_s)
end
And finally my OrderCombo class looks something like below. I've also included a cached version of the method.
class OrderCombo
belongs_to :user
scope :by_user, lambda{ |user| where(:user_id => user.id) }
def self.top_n_orders_by_user(user,count=10)
OrderCombo.by_user(user).count(:group => :combo).sort { |a,b| a[1] <=> b[1] }.reverse[0..count-1]
end
def self.cached_top_orders_by_user(user,count=10)
Rails.cache.fetch("order_combo_#{user.id.to_s}_#{count.to_s}", :expiry => 10.minutes) { OrderCombo.top_n_orders_by_user(user, count) }
end
end
It's not perfect as it doesn't take into account increased popularity when someone orders more of one item in an order.

Resources