I'm using Ruby on Rails 2.3.8 and I would like to get all the #products that haven't been already added to the listed_products array.
For example, let's say I've got the following code:
listed_products = ['1', '2', '3', '4', '5']
#Then, I would like to do something like SELECT * FROM products where id not in
#(listed_products), and save the result in #products
I know the above SQL syntax won't work, but I just wanted to give you guys the idea of what I want to achieve.
Is there a "rails way" for doing this?
Thanks in advance!
Yes, you can do the following (Rails 2.3.x):
listed_products = [1,2,3,4,5]
Product.find(:all, :conditions => ["id NOT IN (?)", listed_products])
Or this in Rails 3.0.x:
listed_products = [1,2,3,4,5]
Product.where("id NOT IN (?)", listed_products)
Pan's answer is correct, but you can also use scopes in 2.3.8, which lets you chain with other scopes for the class:
class Product
...
named_scope :excluding_ids, lambda { |*ids|
if ids.count==0 then
{}
else
{:conditions => ["id NOT IN (?)",ids]}
end
}
...
end
Then you can chain with other scopes in your class. Say you have a scope named :active. Then you can do:
Products.active.excluding_ids(*listed_products).find :all, ... more conditions ...
and it would be independent of order of the scopes:
Products.excluding_ids(*listed_products).active.find :all, ..
The best I can think of is:
SELECT * FROM products where id not = '1' and id not = '2' (etc...)
Not what you're after I'm sure, but it may be the only way!
Related
I'm hoping there is a easy solution that doesn't involve find_by_sql, if not then I guess that will have to work.
I found this article which references this:
Topic.find(:all, :conditions => { :forum_id => #forums.map(&:id) })
which is the same as
SELECT * FROM topics WHERE forum_id IN (<#forum ids>)
I am wondering if there is a way to do NOT IN with that, like:
SELECT * FROM topics WHERE forum_id NOT IN (<#forum ids>)
Rails 4+:
Article.where.not(title: ['Rails 3', 'Rails 5'])
Rails 3:
Topic.where('id NOT IN (?)', Array.wrap(actions))
Where actions is an array with: [1,2,3,4,5]
FYI, In Rails 4, you can use not syntax:
Article.where.not(title: ['Rails 3', 'Rails 5'])
Using Arel:
topics=Topic.arel_table
Topic.where(topics[:forum_id].not_in(#forum_ids))
or, if preferred:
topics=Topic.arel_table
Topic.where(topics[:forum_id].in(#forum_ids).not)
and since rails 4 on:
topics=Topic.arel_table
Topic.where.not(topics[:forum_id].in(#forum_ids))
Please notice that eventually you do not want the forum_ids to be the ids list, but rather a subquery, if so then you should do something like this before getting the topics:
#forum_ids = Forum.where(/*whatever conditions are desirable*/).select(:id)
in this way you get everything in a single query: something like:
select * from topic
where forum_id in (select id
from forum
where /*whatever conditions are desirable*/)
Also notice that eventually you do not want to do this, but rather a join - what might be more efficient.
You can try something like:
Topic.find(:all, :conditions => ['forum_id not in (?)', #forums.map(&:id)])
You might need to do #forums.map(&:id).join(','). I can't remember if Rails will the argument into a CSV list if it is enumerable.
You could also do this:
# in topic.rb
named_scope :not_in_forums, lambda { |forums| { :conditions => ['forum_id not in (?)', forums.select(&:id).join(',')] }
# in your controller
Topic.not_in_forums(#forums)
To expand on #Trung Lê answer, in Rails 4 you can do the following:
Topic.where.not(forum_id:#forums.map(&:id))
And you could take it a step further.
If you need to first filter for only published Topics and then filter out the ids you don't want, you could do this:
Topic.where(published:true).where.not(forum_id:#forums.map(&:id))
Rails 4 makes it so much easier!
The accepted solution fails if #forums is empty. To workaround this I had to do
Topic.find(:all, :conditions => ['forum_id not in (?)', (#forums.empty? ? '' : #forums.map(&:id))])
Or, if using Rails 3+:
Topic.where( 'forum_id not in (?)', (#forums.empty? ? '' : #forums.map(&:id)) ).all
Most of the answers above should suffice you but if you are doing a lot more of such predicate and complex combinations check out Squeel. You will be able to doing something like:
Topic.where{{forum_id.not_in => #forums.map(&:id)}}
Topic.where{forum_id.not_in #forums.map(&:id)}
Topic.where{forum_id << #forums.map(&:id)}
You may want to have a look at the meta_where plugin by Ernie Miller. Your SQL statement:
SELECT * FROM topics WHERE forum_id NOT IN (<#forum ids>)
...could be expressed like this:
Topic.where(:forum_id.nin => #forum_ids)
Ryan Bates of Railscasts created a nice screencast explaining MetaWhere.
Not sure if this is what you're looking for but to my eyes it certainly looks better than an embedded SQL query.
The original post specifically mentions using numeric IDs, but I came here looking for the syntax for doing a NOT IN with an array of strings.
ActiveRecord will handle that nicely for you too:
Thing.where(['state NOT IN (?)', %w{state1 state2}])
Can these forum ids be worked out in a pragmatic way? e.g. can you find these forums somehow - if that is the case you should do something like
Topic.all(:joins => "left join forums on (forums.id = topics.forum_id and some_condition)", :conditions => "forums.id is null")
Which would be more efficient than doing an SQL not in
This way optimizes for readability, but it's not as efficient in terms of database queries:
# Retrieve all topics, then use array subtraction to
# find the ones not in our list
Topic.all - #forums.map(&:id)
You can use sql in your conditions:
Topic.find(:all, :conditions => [ "forum_id NOT IN (?)", #forums.map(&:id)])
Piggybacking off of jonnii:
Topic.find(:all, :conditions => ['forum_id not in (?)', #forums.pluck(:id)])
using pluck rather than mapping over the elements
found via railsconf 2012 10 things you did not know rails could do
When you query a blank array add "<< 0" to the array in the where block so it doesn't return "NULL" and break the query.
Topic.where('id not in (?)',actions << 0)
If actions could be an empty or blank array.
Here is a more complex "not in" query, using a subquery in rails 4 using squeel. Of course very slow compared to the equivalent sql, but hey, it works.
scope :translations_not_in_english, ->(calmapp_version_id, language_iso_code){
join_to_cavs_tls_arr(calmapp_version_id).
joins_to_tl_arr.
where{ tl1.iso_code == 'en' }.
where{ cavtl1.calmapp_version_id == my{calmapp_version_id}}.
where{ dot_key_code << (Translation.
join_to_cavs_tls_arr(calmapp_version_id).
joins_to_tl_arr.
where{ tl1.iso_code == my{language_iso_code} }.
select{ "dot_key_code" }.all)}
}
The first 2 methods in the scope are other scopes which declare the aliases cavtl1 and tl1. << is the not in operator in squeel.
Hope this helps someone.
If someone want to use two or more conditions, you can do that:
your_array = [1,2,3,4]
your_string = "SOMETHING"
YourModel.where('variable1 NOT IN (?) AND variable2=(?)',Array.wrap(your_array),your_string)
This is what I'd like to do:
I have this piece of code:
customer = Customer.find(:first, :conditions => {:siteId => params[:siteId], :customerCode => params[:id]})
If :customerCode is null, I'd like to use :temporaryCode instead. But I don't know how.
Thanks in advance.
customer = Customer.find_by_siteid_and_customercode params[:siteId], params[:id]
customer ||= Customer.find_by_siteid_and_temporarycode params[:siteId], params[:id]
making use of finders is most safer, cleaner
I'm pretty sure you want to use COALESCE inside the database:
customer = Customer.where(:siteId => params[:siteId])
.where('coalesce(customercode, temporarycode) = ?', params[:id])
.first
The SQL COALESCE function returns the first of its arguments that isn't NULL. For example:
coalesce(column1, column2)
gives you column1 if column1 isn't NULL and column2 if column1 is NULL.
If you're using Rails2 then something like this should work:
Customer.find(:first, :conditions => [
'siteid = ? and coalesce(customercode, temporarycode) = ?',
params[:siteId], params[:id]
])
I've written a couple of complex queries (at least to me) with Ruby on Rail's query interface:
watched_news_posts = Post.joins(:news => :watched).where(:watched => {:user_id => id})
watched_topic_posts = Post.joins(:post_topic_relationships => {:topic => :watched}).where(:watched => {:user_id => id})
Both of these queries work fine by themselves. Both return Post objects. I would like to combine these posts into a single ActiveRelation. Since there could be hundreds of thousands of posts at some point, this needs to be done at the database level. If it were a MySQL query, I could simply user the UNION operator. Does anybody know if I can do something similar with RoR's query interface?
Here's a quick little module I wrote that allows you to UNION multiple scopes. It also returns the results as an instance of ActiveRecord::Relation.
module ActiveRecord::UnionScope
def self.included(base)
base.send :extend, ClassMethods
end
module ClassMethods
def union_scope(*scopes)
id_column = "#{table_name}.id"
sub_query = scopes.map { |s| s.select(id_column).to_sql }.join(" UNION ")
where "#{id_column} IN (#{sub_query})"
end
end
end
Here's the gist: https://gist.github.com/tlowrimore/5162327
Edit:
As requested, here's an example of how UnionScope works:
class Property < ActiveRecord::Base
include ActiveRecord::UnionScope
# some silly, contrived scopes
scope :active_nearby, -> { where(active: true).where('distance <= 25') }
scope :inactive_distant, -> { where(active: false).where('distance >= 200') }
# A union of the aforementioned scopes
scope :active_near_and_inactive_distant, -> { union_scope(active_nearby, inactive_distant) }
end
I also have encountered this problem, and now my go-to strategy is to generate SQL (by hand or using to_sql on an existing scope) and then stick it in the from clause. I can't guarantee it's any more efficient than your accepted method, but it's relatively easy on the eyes and gives you a normal ARel object back.
watched_news_posts = Post.joins(:news => :watched).where(:watched => {:user_id => id})
watched_topic_posts = Post.joins(:post_topic_relationships => {:topic => :watched}).where(:watched => {:user_id => id})
Post.from("(#{watched_news_posts.to_sql} UNION #{watched_topic_posts.to_sql}) AS posts")
You can do this with two different models as well, but you need to make sure they both "look the same" inside the UNION -- you can use select on both queries to make sure they will produce the same columns.
topics = Topic.select('user_id AS author_id, description AS body, created_at')
comments = Comment.select('author_id, body, created_at')
Comment.from("(#{comments.to_sql} UNION #{topics.to_sql}) AS comments")
Based on Olives' answer, I did come up with another solution to this problem. It feels a little bit like a hack, but it returns an instance of ActiveRelation, which is what I was after in the first place.
Post.where('posts.id IN
(
SELECT post_topic_relationships.post_id FROM post_topic_relationships
INNER JOIN "watched" ON "watched"."watched_item_id" = "post_topic_relationships"."topic_id" AND "watched"."watched_item_type" = "Topic" WHERE "watched"."user_id" = ?
)
OR posts.id IN
(
SELECT "posts"."id" FROM "posts" INNER JOIN "news" ON "news"."id" = "posts"."news_id"
INNER JOIN "watched" ON "watched"."watched_item_id" = "news"."id" AND "watched"."watched_item_type" = "News" WHERE "watched"."user_id" = ?
)', id, id)
I'd still appreciate it if anybody has any suggestions to optimize this or improve the performance, because it's essentially executing three queries and feels a little redundant.
You could also use Brian Hempel's active_record_union gem that extends ActiveRecord with an union method for scopes.
Your query would be like this:
Post.joins(:news => :watched).
where(:watched => {:user_id => id}).
union(Post.joins(:post_topic_relationships => {:topic => :watched}
.where(:watched => {:user_id => id}))
Hopefully this will be eventually merged into ActiveRecord some day.
Could you use an OR instead of a UNION?
Then you could do something like:
Post.joins(:news => :watched, :post_topic_relationships => {:topic => :watched})
.where("watched.user_id = :id OR topic_watched.user_id = :id", :id => id)
(Since you are joins the watched table twice I'm not too sure what the names of the tables will be for the query)
Since there are a lot of joins, it might also be quite heavy on the database, but it might be able to be optimized.
How about...
def union(scope1, scope2)
ids = scope1.pluck(:id) + scope2.pluck(:id)
where(id: ids.uniq)
end
Arguably, this improves readability, but not necessarily performance:
def my_posts
Post.where <<-SQL, self.id, self.id
posts.id IN
(SELECT post_topic_relationships.post_id FROM post_topic_relationships
INNER JOIN watched ON watched.watched_item_id = post_topic_relationships.topic_id
AND watched.watched_item_type = "Topic"
AND watched.user_id = ?
UNION
SELECT posts.id FROM posts
INNER JOIN news ON news.id = posts.news_id
INNER JOIN watched ON watched.watched_item_id = news.id
AND watched.watched_item_type = "News"
AND watched.user_id = ?)
SQL
end
This method returns an ActiveRecord::Relation, so you could call it like this:
my_posts.order("watched_item_type, post.id DESC")
There is an active_record_union gem.
Might be helpful
https://github.com/brianhempel/active_record_union
With ActiveRecordUnion, we can do:
the current user's (draft) posts and all published posts from anyone
current_user.posts.union(Post.published)
Which is equivalent to the following SQL:
SELECT "posts".* FROM (
SELECT "posts".* FROM "posts" WHERE "posts"."user_id" = 1
UNION
SELECT "posts".* FROM "posts" WHERE (published_at < '2014-07-19 16:04:21.918366')
) posts
In a similar case I summed two arrays and used Kaminari:paginate_array(). Very nice and working solution. I was unable to use where(), because I need to sum two results with different order() on the same table.
Heres how I joined SQL queries using UNION on my own ruby on rails application.
You can use the below as inspiration on your own code.
class Preference < ApplicationRecord
scope :for, ->(object) { where(preferenceable: object) }
end
Below is the UNION where i joined the scopes together.
def zone_preferences
zone = Zone.find params[:zone_id]
zone_sql = Preference.for(zone).to_sql
region_sql = Preference.for(zone.region).to_sql
operator_sql = Preference.for(Operator.current).to_sql
Preference.from("(#{zone_sql} UNION #{region_sql} UNION #{operator_sql}) AS preferences")
end
Less problems and easier to follow:
def union_scope(*scopes)
scopes[1..-1].inject(where(id: scopes.first)) { |all, scope| all.or(where(id: scope)) }
end
So in the end:
union_scope(watched_news_posts, watched_topic_posts)
gem 'active_record_extended'
Also has a set of union helpers among many others.
I would just run the two queries you need and combine the arrays of records that are returned:
#posts = watched_news_posts + watched_topics_posts
Or, at the least test it out. Do you think the array combination in ruby will be far too slow? Looking at the suggested queries to get around the problem, I'm not convinced that there will be that significant of a performance difference.
Elliot Nelson answered good, except the case where some of the relations are empty. I would do something like that:
def union_2_relations(relation1,relation2)
sql = ""
if relation1.any? && relation2.any?
sql = "(#{relation1.to_sql}) UNION (#{relation2.to_sql}) as #{relation1.klass.table_name}"
elsif relation1.any?
sql = relation1.to_sql
elsif relation2.any?
sql = relation2.to_sql
end
relation1.klass.from(sql)
end
When we add UNION to the scopes, it breaks at time due to order_by clause added before the UNION.
So I changed it in a way to give it a UNION effect.
module UnionScope
def self.included(base)
base.send(:extend, ClassMethods)
end
module ClassMethods
def union_scope(*scopes)
id_column = "#{table_name}.id"
sub_query = scopes.map { |s| s.pluck(:id) }.flatten
where("#{id_column} IN (?)", sub_query)
end
end
end
And then use it like this in any model
class Model
include UnionScope
scope :union_of_scopeA_scopeB, -> { union_scope(scopeA, scopeB) }
end
Tim's answer is great. It uses the ids of the scopes in the WHERE clause. As shosti reports, this method is problematic in terms of performance because all ids need to be generated during query execution. This is why, I prefer joeyk16 answer. Here a generalized module:
module ActiveRecord::UnionScope
def self.included(base)
base.send :extend, ClassMethods
end
module ClassMethods
def self.union(*scopes)
self.from("(#{scopes.map(&:to_sql).join(' UNION ')}) AS #{self.table_name}")
end
end
end
If you don't want to use SQL syntax inside your code, here's solution with arel
watched_news_posts = Post.joins(:news => :watched).where(:watched => {:user_id => id}).arel
watched_topic_posts = Post.joins(:post_topic_relationships => {:topic => :watched}).where(:watched => {:user_id => id}).arel
results = Arel::Nodes::Union.new(watched_news_posts, watched_topic_posts)
from(Post.arel_table.create_table_alias(results, :posts))
I'm hoping there is a easy solution that doesn't involve find_by_sql, if not then I guess that will have to work.
I found this article which references this:
Topic.find(:all, :conditions => { :forum_id => #forums.map(&:id) })
which is the same as
SELECT * FROM topics WHERE forum_id IN (<#forum ids>)
I am wondering if there is a way to do NOT IN with that, like:
SELECT * FROM topics WHERE forum_id NOT IN (<#forum ids>)
Rails 4+:
Article.where.not(title: ['Rails 3', 'Rails 5'])
Rails 3:
Topic.where('id NOT IN (?)', Array.wrap(actions))
Where actions is an array with: [1,2,3,4,5]
FYI, In Rails 4, you can use not syntax:
Article.where.not(title: ['Rails 3', 'Rails 5'])
Using Arel:
topics=Topic.arel_table
Topic.where(topics[:forum_id].not_in(#forum_ids))
or, if preferred:
topics=Topic.arel_table
Topic.where(topics[:forum_id].in(#forum_ids).not)
and since rails 4 on:
topics=Topic.arel_table
Topic.where.not(topics[:forum_id].in(#forum_ids))
Please notice that eventually you do not want the forum_ids to be the ids list, but rather a subquery, if so then you should do something like this before getting the topics:
#forum_ids = Forum.where(/*whatever conditions are desirable*/).select(:id)
in this way you get everything in a single query: something like:
select * from topic
where forum_id in (select id
from forum
where /*whatever conditions are desirable*/)
Also notice that eventually you do not want to do this, but rather a join - what might be more efficient.
You can try something like:
Topic.find(:all, :conditions => ['forum_id not in (?)', #forums.map(&:id)])
You might need to do #forums.map(&:id).join(','). I can't remember if Rails will the argument into a CSV list if it is enumerable.
You could also do this:
# in topic.rb
named_scope :not_in_forums, lambda { |forums| { :conditions => ['forum_id not in (?)', forums.select(&:id).join(',')] }
# in your controller
Topic.not_in_forums(#forums)
To expand on #Trung Lê answer, in Rails 4 you can do the following:
Topic.where.not(forum_id:#forums.map(&:id))
And you could take it a step further.
If you need to first filter for only published Topics and then filter out the ids you don't want, you could do this:
Topic.where(published:true).where.not(forum_id:#forums.map(&:id))
Rails 4 makes it so much easier!
The accepted solution fails if #forums is empty. To workaround this I had to do
Topic.find(:all, :conditions => ['forum_id not in (?)', (#forums.empty? ? '' : #forums.map(&:id))])
Or, if using Rails 3+:
Topic.where( 'forum_id not in (?)', (#forums.empty? ? '' : #forums.map(&:id)) ).all
Most of the answers above should suffice you but if you are doing a lot more of such predicate and complex combinations check out Squeel. You will be able to doing something like:
Topic.where{{forum_id.not_in => #forums.map(&:id)}}
Topic.where{forum_id.not_in #forums.map(&:id)}
Topic.where{forum_id << #forums.map(&:id)}
You may want to have a look at the meta_where plugin by Ernie Miller. Your SQL statement:
SELECT * FROM topics WHERE forum_id NOT IN (<#forum ids>)
...could be expressed like this:
Topic.where(:forum_id.nin => #forum_ids)
Ryan Bates of Railscasts created a nice screencast explaining MetaWhere.
Not sure if this is what you're looking for but to my eyes it certainly looks better than an embedded SQL query.
The original post specifically mentions using numeric IDs, but I came here looking for the syntax for doing a NOT IN with an array of strings.
ActiveRecord will handle that nicely for you too:
Thing.where(['state NOT IN (?)', %w{state1 state2}])
Can these forum ids be worked out in a pragmatic way? e.g. can you find these forums somehow - if that is the case you should do something like
Topic.all(:joins => "left join forums on (forums.id = topics.forum_id and some_condition)", :conditions => "forums.id is null")
Which would be more efficient than doing an SQL not in
This way optimizes for readability, but it's not as efficient in terms of database queries:
# Retrieve all topics, then use array subtraction to
# find the ones not in our list
Topic.all - #forums.map(&:id)
You can use sql in your conditions:
Topic.find(:all, :conditions => [ "forum_id NOT IN (?)", #forums.map(&:id)])
Piggybacking off of jonnii:
Topic.find(:all, :conditions => ['forum_id not in (?)', #forums.pluck(:id)])
using pluck rather than mapping over the elements
found via railsconf 2012 10 things you did not know rails could do
When you query a blank array add "<< 0" to the array in the where block so it doesn't return "NULL" and break the query.
Topic.where('id not in (?)',actions << 0)
If actions could be an empty or blank array.
Here is a more complex "not in" query, using a subquery in rails 4 using squeel. Of course very slow compared to the equivalent sql, but hey, it works.
scope :translations_not_in_english, ->(calmapp_version_id, language_iso_code){
join_to_cavs_tls_arr(calmapp_version_id).
joins_to_tl_arr.
where{ tl1.iso_code == 'en' }.
where{ cavtl1.calmapp_version_id == my{calmapp_version_id}}.
where{ dot_key_code << (Translation.
join_to_cavs_tls_arr(calmapp_version_id).
joins_to_tl_arr.
where{ tl1.iso_code == my{language_iso_code} }.
select{ "dot_key_code" }.all)}
}
The first 2 methods in the scope are other scopes which declare the aliases cavtl1 and tl1. << is the not in operator in squeel.
Hope this helps someone.
If someone want to use two or more conditions, you can do that:
your_array = [1,2,3,4]
your_string = "SOMETHING"
YourModel.where('variable1 NOT IN (?) AND variable2=(?)',Array.wrap(your_array),your_string)
I am stuck and I guess it is a syntax thing I just don't get:
It's a one(category)_to_many(product) relationship.
In my category model I have the columns pick1, pick2, pick3, pick4, pick5.
Each of those holds the id of a product.
In my category_controller I want to retrieve those in a find:
#productpicks = Product.find(:all, :conditions =>
['online = ? and category_id IN (?)', true,
[#category.pick1, #category.pick2, #category.pick3, #category.pick4, #category.pick5]])
... and iterate over them in the view like this:
do something
But there is nothing to be found in that array ...
Does anyone have an idea what I am doing wrong?
Thanks for helping!
Val
Shouldn't it be:
#productpicks = Product.find(
:all,
:conditions => [
'online = ? and id IN (?)',
true, [
#category.pick1,
#category.pick2,
#category.pick3,
#category.pick4,
#category.pick5
]
]
)
Replacing category_id with id in the where clause?
As pick1-5 hold product ids, and you are trying to find those specific products.