This is a repost from https://github.com/rails/rails/issues/22491
Given this statement:
#items = DailyList.joins(:bar)
.select("daily_lists.*, (point(12, 34) <#> bars.coords) as bar_distance")
.order("bar_distance ASC")
.group('bars.id, daily_lists.id')
.limit(3)
it produces the following SQL:
SELECT daily_lists.*, (point(12, 34) <#> bars.coords) bar_distance
FROM "daily_lists"
INNER JOIN "bars" ON "bars"."id" = "daily_lists"."bar_id"
GROUP BY bars.id, daily_lists.id
ORDER BY bar_distance ASC
LIMIT 3
Now we want to check how much records we've got, #items.count:
SELECT COUNT(daily_lists.*, (point(12, 34) <#> bars.coords) as bar_distance) AS count_daily_lists_all_point_12_34_bars_coords_as_bar_distance, bars.id, daily_lists.id AS bars_id_daily_lists_id
FROM "daily_lists"
INNER JOIN "bars" ON "bars"."id" = "daily_lists"."bar_id"
GROUP BY bars.id, daily_lists.id
ORDER BY bar_distance ASC
LIMIT 3
ActiveRecord::StatementInvalid: PG::SyntaxError: ERROR: syntax error at or near "as"
LINE 1: ...NT(daily_lists.*, (point(12, 34) <#> bars.coords) as bar_dis...
Now I'm trying to use :all as a workaround, #items.count(:all):
SELECT COUNT(*) AS count_all, bars.id, daily_lists.id AS bars_id_daily_lists_id
FROM "daily_lists"
INNER JOIN "bars" ON "bars"."id" = "daily_lists"."bar_id"
GROUP BY bars.id, daily_lists.id
ORDER BY bar_distance ASC
LIMIT 3
PG::UndefinedColumn: ERROR: column "bar_distance" does not exist
Guys... what to do?
The "as bar_distance" is the problem, I think. You cannot alias within a count.
#items = DailyList.joins(:bar)
.select("daily_lists.*, (point(12, 34) <#> bars.coords) as bar_distance")
.order("bar_distance ASC")
.group('bars.id, daily_lists.id')
.limit(3)
#item_count = DailyList.joins(:bar)
.select("daily_lists.*, (point(12, 34) <#> bars.coords)")
.group('bars.id, daily_lists.id')
.limit(3)
.count
Related
Context
We have a Rails app that is retrieving conversations with the following raw SQL query:
SELECT sub.*,
profiles.status AS interlocutor_status
FROM (
SELECT DISTINCT ON (conversations.id) conversations.id,
conversation_preferences.unread_counter,
left(messages.content, 50) AS last_message,
posts.id AS post_id,
messages.created_at AS last_activity_on,
categories.root_name AS site_name,
conversation_preferences.state,
COALESCE(NULLIF(post_owner, 1234567), NULLIF(post_applicant, 1234567)) AS interlocutor_id
FROM "conversations"
LEFT OUTER JOIN "conversation_preferences" ON "conversation_preferences"."conversation_id" = "conversations"."id"
LEFT OUTER JOIN "posts" ON "posts"."id" = "conversations"."post_id"
LEFT OUTER JOIN "categories" ON "categories"."id" = "posts"."category_id"
LEFT OUTER JOIN "messages" ON "messages"."conversation_id" = "conversations"."id"
WHERE (post_applicant = 1234567 OR post_owner = 1234567)
AND "conversation_preferences"."user_id" = 1234567
ORDER BY "conversations"."id" ASC, messages.created_at DESC
) sub
LEFT OUTER JOIN users ON interlocutor_id = users.id
LEFT OUTER JOIN profiles ON interlocutor_id = profiles.user_id
WHERE ("profiles"."status" != 'pending')
AND (last_activity_on >= '2021-01-19 04:40:22.881985')
AND (state = 'active')
ORDER BY profiles.status, sub.unread_counter DESC, sub.last_activity_on DESC
LIMIT 25
We generate this query using the following ActiveRecord code:
def fetch
distinct = Conversation.left_outer_joins(:preferences)
.left_outer_joins(post: :category)
.left_outer_joins(:messages)
.where('post_applicant = :id OR post_owner = :id', id: current_user.id)
.where(conversation_preferences: { user_id: current_user.id })
.select(
<<-SQL.squish
DISTINCT ON (conversations.id) conversations.id,
conversation_preferences.unread_counter,
left(messages.content, 50) AS last_message,
posts.id AS post_id,
messages.created_at AS last_activity_on,
categories.root_name AS site_name,
conversation_preferences.state,
COALESCE(NULLIF(post_owner, #{current_user.id}), NULLIF(post_applicant, #{current_user.id})) AS interlocutor_id
SQL
)
.order(:id, 'messages.created_at DESC')
Conversation.includes(post: :category)
.from(distinct, :sub)
.select('sub.*, profiles.status AS interlocutor_status')
.joins('LEFT OUTER JOIN users ON interlocutor_id = users.id')
.joins('LEFT OUTER JOIN profiles ON interlocutor_id = profiles.user_id')
.where.not('profiles.status' => :pending)
.order('profiles.status, sub.unread_counter DESC, sub.last_activity_on DESC')
end
Problem
We want to stop ordering by profiles.status. To do this, we naturally removed it from the last order statement:
order('sub.unread_counter DESC, sub.last_activity_on DESC')
That's the problem. Doing that is entirely breaking the generated query, that generate an error which is irrelevant here because we don't want the modified query (note how it is different from the 1st one):
SELECT sub.*,
profiles.status AS interlocutor_status,
"conversations"."id" AS t0_r0,
"conversations"."post_id" AS t0_r1,
"conversations"."post_owner" AS t0_r2,
"conversations"."post_applicant" AS t0_r3,
"conversations"."created_at" AS t0_r4,
"conversations"."updated_at" AS t0_r5,
"posts"."id" AS t1_r0,
"posts"."title" AS t1_r1,
"posts"."description" AS t1_r2,
"categories"."id" AS t2_r0,
"categories"."name" AS t2_r1
FROM (
SELECT DISTINCT ON (conversations.id) conversations.id,
conversation_preferences.unread_counter,
left(messages.content, 50) AS last_message,
posts.id AS post_id,
messages.created_at AS last_activity_on,
categories.root_name AS site_name,
conversation_preferences.state,
COALESCE(NULLIF(post_owner, 1234567), NULLIF(post_applicant, 1234567)) AS interlocutor_id
FROM "conversations"
LEFT OUTER JOIN "conversation_preferences" ON "conversation_preferences"."conversation_id" = "conversations"."id"
LEFT OUTER JOIN "posts" ON "posts"."id" = "conversations"."post_id"
LEFT OUTER JOIN "categories" ON "categories"."id" = "posts"."category_id"
LEFT OUTER JOIN "messages" ON "messages"."conversation_id" = "conversations"."id"
WHERE (post_applicant = 1234567 OR post_owner = 1234567)
AND "conversation_preferences"."user_id" = 1234567
ORDER BY "conversations"."id" ASC, messages.created_at DESC
) sub
LEFT OUTER JOIN "posts" ON "posts"."id" = "conversations"."post_id"
LEFT OUTER JOIN "categories" ON "categories"."id" = "posts"."category_id"
LEFT OUTER JOIN users ON interlocutor_id = users.id
LEFT OUTER JOIN profiles ON interlocutor_id = profiles.user_id
WHERE ("profiles"."status" != 'pending')
AND (last_activity_on >= '2021-01-19 05:04:06.084499')
AND (state = 'active')
ORDER BY sub.unread_counter DESC, sub.last_activity_on DESC
LIMIT 25
I know without a bit of context it'll be hard to help us but if someone knows why ActiveRecord is changing the query after trying to just remove profiles.status from the order statement, that would be awesome. Thanks in advance
NOTE: modifying the 1st raw SQL directly (from our postgres client) does works. The issue is not the first query, but maybe how ActiveRecord is handling it
Finally found a way to make it work using preload instead of includes. We wanted to avoid having seperate queries to load posts and categories but since performance is not affected by it, we don't mind it.
Here is how it look like:
Conversation.preload(post: :category)
.from(distinct, :sub)
.select('sub.*, profiles.status AS interlocutor_status')
.joins('LEFT OUTER JOIN users ON interlocutor_id = users.id')
.joins('LEFT OUTER JOIN profiles ON interlocutor_id = profiles.user_id')
.where.not('profiles.status' => :pending)
.order('sub.unread_counter DESC, sub.last_activity_on DESC')
Which generates 3 queries:
-- Query 1
SELECT sub.*, profiles.status AS interlocutor_status
FROM (
SELECT DISTINCT
....
-- Query 2
SELECT "posts".* FROM "posts" WHERE "posts"."id" IN (............)
-- Query 3
SELECT "categories".* FROM "categories" WHERE "categories"."id" IN (..........)
Thanks to everyone for the help in comments (max and Sebastian Palma)
user has_many tasks
I'm trying to create a 3 select UNION:
Task.from("(#{select1.to_sql} UNION #{select2.to_sql} UNION #{select3.to_sql}) AS tasks")
But with Arel.
I can easily use Arel for UNION query for 2 selects:
tasks_t = Task.arel_table
select1 = tasks_t.project('id').where(tasks_t[:id].eq(1)) # SELECT id FROM tasks WHERE tasks.id = 1
select2 = Task.joins(:user).select(:id).where(users: {id: 15}).arel # SELECT "tasks".id FROM "tasks" INNER JOIN "users" ON "users"."id" = "tasks"."user_id" WHERE "users"."id" = 15
union = select1.union(select2) # SELECT * FROM users WHERE users.id = 1 UNION SELECT * FROM users WHERE users.id = 2
union_with_table_alias = tasks_t.create_table_alias(union, tasks_t.name)
Task.from(union_with_table_alias)
# SELECT "tasks".* FROM ( SELECT id FROM "tasks" WHERE "tasks"."id" = 1 UNION SELECT "tasks"."id" FROM "tasks" INNER JOIN "users" ON "users"."id" = "tasks"."user_id" WHERE "users"."id" = $1 ) "tasks" [["id", 15]]
#=> Task::ActiveRecord_Relation [#<Task: id: 1>, #<Task: id: 2>]
How can I do it with triple union select?
select3 = tasks_t.project('id').where(tasks_t[:id].eq(3)) # SELECT id FROM tasks WHERE tasks.id = 3
Something like:
triple_union = select1.union(select2).union(select3)
triple_union_with_table_alias = tasks_t.create_table_alias(triple_union, tasks_t.name)
Task.from(triple_union_with_table_alias)
Which should roughly translate to
Task.from("(#{select1.to_sql} UNION #{select2.to_sql} UNION #{select3.to_sql}) AS tasks")
Note the above line also fails:
Caused by PG::ProtocolViolation: ERROR: bind message supplies 0 parameters, but prepared statement "" requires 1
To summarize:
How to use Arel to build a tripple UNION query? select 1 UNION select 2 UNION select 3
Thank you
Ruby 2.5.3 Rails 5.2.4 Arel 9.0.0
You can not invoke union in an Arel::Nodes::Union, because in the method definition the ast of both objects is called. And the result of a union operation doesn't respond to ast:
def union(operation, other = nil)
if other
node_class = Nodes.const_get("Union#{operation.to_s.capitalize}")
else
other = operation
node_class = Nodes::Union
end
node_class.new(self.ast, other.ast)
end
What you can do is to manually call Arel::Nodes::Union, passing as any of those arguments the result of a union:
Arel::Nodes::Union.new(Arel::Nodes::Union.new(select1, select2), select3)
How can I use order with uniq?
auction.invoices.get_auction_invoices.item_invoices.
joins("INNER JOIN users ON users.id = invoices.usable_id").order("users.first_name").uniq
The above query gives me following error:
This is my scopes
scope :item_invoices, ->{ joins(:invoice_details).where("invoice_details.invoiceable_type = ?", "Item")}
scope :get_auction_invoices, ->{where(:id => (item_invoices.skip_cancelled_invoice + donators.skip_cancelled_invoice))}
PG::InvalidColumnReference: ERROR: for SELECT DISTINCT, ORDER BY expressions must appear in select list
LINE 1: ...oice_details.invoiceable_type = 'Item') ORDER BY users.firs...
: SELECT DISTINCT "invoices".* FROM "invoices" INNER JOIN "invoice_details" ON "invoice_details"."invoice_id" = "invoices"."id" INNER JOIN users ON users.id = invoices.usable_id WHERE "invoices"."eventable_id" = $1 AND "invoices"."eventable_type" = $2 AND "invoices"."id" IN (1132, 1131, 777, 777, 777, 3013, 3024, 3024, 3024, 3024, 3041, 3041, 3013) AND (invoice_details.invoiceable_type = 'Item') ORDER BY users.first_name
Try this:
auction.invoices.get_auction_invoices.item_invoices.\
select("invoices.*, users.*").\
joins("INNER JOIN users ON users.id = invoices.usable_id").\
order("users.first_name").\
uniq
i am trying to loop through a list of products which a supplier sells via its different variants. i can get the list of products to display, but i wish to group these by the product id as to only display it once.
in my controller i have
#supplier = Supplier.joins(products: :variants).find(params[:id])
in my view i have
- #supplier.variants.group_by(&:product_id).each do |product_id, item|
= render :partial => 'product', :locals => {:item => item }
and my partial
= link_to shopping_supplier_path(item) do
%li.mdl-list__item.mdl-list__item--three-line
%span.mdl-list__item-primary-content
%span= item.product.name
%span.mdl-list__item-text-body
= item.product.description.downcase
%span.mdl-list__item-secondary-content
%i.material-icons
chevron_right
%hr
which when the sql executes returns the following query
Started GET "/shopping/suppliers/latte-cartelle-drive-thru-coffee-241---245-princes-hwy--ha-1" for 127.0.0.1 at 2016-04-19 23:22:08 +1000
Processing by Shopping::SuppliersController#show as HTML
Parameters: {"id"=>"latte-cartelle-drive-thru-coffee-241---245-princes-hwy--ha-1"}
User Load (0.6ms) SELECT "users".* FROM "users" WHERE "users"."id" = $1 ORDER BY "users"."id" ASC LIMIT 1 [["id", 1]]
Supplier Load (32.7ms) SELECT "suppliers".* FROM "suppliers" WHERE "suppliers"."permalink" = $1 ORDER BY "suppliers"."id" ASC LIMIT 1 [["permalink", "latte-cartelle-drive-thru-coffee-241---245-princes-hwy--ha-1"]]
Supplier Load (41.9ms) SELECT "suppliers".* FROM "suppliers" INNER JOIN "variant_suppliers" ON "variant_suppliers"."supplier_id" = "suppliers"."id" INNER JOIN "variants" ON "variants"."id" = "variant_suppliers"."variant_id" INNER JOIN "products" ON "products"."id" = "variants"."product_id" INNER JOIN "variants" "variants_products" ON "variants_products"."product_id" = "products"."id" WHERE "suppliers"."permalink" = $1 ORDER BY "suppliers"."id" ASC LIMIT 1 [["permalink", "latte-cartelle-drive-thru-coffee-241---245-princes-hwy--ha-1"]]
Variant Load (0.9ms) SELECT "variants".* FROM "variants" INNER JOIN "variant_suppliers" ON "variants"."id" = "variant_suppliers"."variant_id" WHERE "variant_suppliers"."supplier_id" = $1 [["supplier_id", 1]]
Rendered shopping/suppliers/_product.html.haml (53.5ms)
error
NoMethodError at /shopping/suppliers/latte-cartelle-drive-thru-coffee-241---245-princes-hwy--ha-1
undefined method `name' for #<Array:0x007faa5302d5a0>
Use SQL joins. When you create a query joining the tables that you are going to use, they will be previously loaded in memory, so the famous n+1 queries will not occur.
#supplier = Supplier.joins(variants: :products).find(params[:id])
# This will translate to something like this
SELECT * FROM suppliers
INNER JOIN variants ON variants.supplier_id = suppliers.id
INNER JOIN products ON products.variant_id = variants.id
WHERE suppliers.id = ?
Remember to always avoid the lazy loading of your associations.
In my controller, my filter_with_params() method is causing a syntax error in postgres when I try and stack will_paginate on top of it.
In my controller I call:
#styles = Style.filter_with_params(params).paginate(:page => params[:page], :per_page => 6)
Model method:
class Style < ActiveRecord::Base
def self.filter_with_params(params)
scoped = where("styles.item != ''")
if params[:t]
scoped = scoped.joins(:tags)
scoped = scoped.select("distinct styles.*, count(*) AS count")
scoped = scoped.where(tags: { name: params[:t] })
scoped = scoped.group('styles.id')
scoped = scoped.having("count(*) = #{params[:t].size}")
end
scoped
end
basically my filter method is looking for tags, and then i need to paginate on top of those results. Anyone with similar experience?
I'm using Rails 3 on this app
here is the postgres error
PG::Error: ERROR: syntax error at or near "distinct" LINE 1: SELECT COUNT(*) AS count_all, distinct styles.*, count(*) AS...
^
: SELECT COUNT(*) AS count_all, distinct styles.*, count(*) AS count, styles.id AS styles_id FROM "styles" INNER JOIN "tagizations" ON "tagizations"."style_id" = "styles"."id" INNER JOIN "tags" ON "tags"."id" = "tagizations"."tag_id" WHERE "tags"."name" IN ('engagement') AND (styles.polenza_item != '') GROUP BY styles.id HAVING count(*) = 1
Your SQL has a problem. You need to say distinct clause before the count ('count(*) as count_all'). That is, once you remove the first call to the count function, it should work.
SELECT distinct styles.*, count(*) AS count, styles.id AS styles_id FROM "styles" INNER JOIN "tagizations" ON "tagizations"."style_id" = "styles"."id" ...
You can test your query in your rails console:
>> sql = "SELECT distinct styles.*, count(*) AS count, styles.id AS styles_id..."
>> a = ActiveRecord::Base.connection.execute(sql)
>> a[0]
Hope this helps.