I have a couple really hard DB queries I need help to re-write in the correct way for Rails 6 Active Record. These are currently working in an app I an re-writing to the new version of Ruby on Rails (6.1.4.2).
It was originally written on Rails v3.2 with a Hell gem called squeel which uses its own DSL Language.
https://github.com/activerecord-hackery/squeel
I have been trying for days now and haven't been able to get it figured out. The first time I asked it I probably wasn't as clear as I needed to be. So this time I am going to put the query as it was written in squeel, and the SQL that the console from Heroku is spitting out and that's all. If anyone wants any additional information ask and I will HAPPILY post it. I want to keep it simple to start with as they are confusing enough.
WARNING: These seem to be EXTREMLY COMPLICATED.
ANY HELP would be VERY Appreciated! :)
Here is squeel DB Query 1:
Project.joins{vendor}.joins{certifications.outer}.where{
(projects.vendor_id.eq my{ vendor_id }) |
(vendors.parent_vendor_id.eq my{ vendor_id }) |
((certifications.cdti == true) & (certifications.published == true))
}.uniq
Here is the strait SQL from query 1 out of Rails v3.2:
SELECT DISTINCT "vendors".* FROM "vendors" INNER JOIN "projects" ON "projects"."vendor_id" = "vendors"."id"
INNER JOIN "certifications" ON "certifications"."project_id" = "projects"."id"
WHERE (("certifications"."cdti" = 't' AND "certifications"."published" = 't'))
ORDER BY "vendors"."parent_vendor_id", "vendors"."name"
Here is the squeel DB query 2:
Fleet.joins{vendor.projects.certifications}.
where{(certifications.cdti.eq true) & (certifications.published.eq true)}.
uniq.includes(:vendor).
order(:vendor_id, :name)
Here is the strait SQL from query 2 out of Rails v3.2:
(I hit enter in a few places so you could see the entire statement without having to scroll to the right
SELECT DISTINCT "fleets".* FROM "fleets" INNER JOIN "vendors" ON "vendors"."id" = "fleets"."vendor_id"
INNER JOIN "projects" ON "projects"."vendor_id" = "vendors"."id"
INNER JOIN "certifications" ON "certifications"."project_id" = "projects"."id"
WHERE (("certifications"."cdti" = 't' AND "certifications"."published" = 't'))
ORDER BY "fleets"."vendor_id", "fleets"."name"
Again if anyone wants to see or know anything else just let me know as I am trying my best to figure this out, but these seem so advanced I just don't think I know the correct syntax.
Thank You for your time,
Scott
Query 1 equivalent is:
Vendor.joins(projects: :certifications).where(certifications: { cdti: 't', published: 't' }).order(:parent_vendor_id, :name).distinct
Query 2:
Fleet.joins(vendor: { projects: :certifications }).where(certifications: { cdti: 't', published: 't' }).order(:vendor_id, :name).distinct
I find my query is taking too long to load so I'm wondering if the position of the includes matters.
Example A:
people = Person.where(name: 'guillaume').includes(:jobs)
Example B:
people = Person.includes(:jobs).where(name: 'guillaume')
Is example A faster because I should have fewer people's jobs to load?
Short answer: no.
ActiveRecord builds your query and as long as you don't need the records, it won't send the final SQL query to the database to fetch them. The 2 queries you pasted are identical.
Whenever in doubt, you can always open up rails console, write your queries there and observe the queries printed out. In your example it would be something like:
SELECT "people".* FROM "people" WHERE "people"."name" = $1 LIMIT $2 [["name", "guillaume"], ["LIMIT", 11]]
SELECT "jobs".* FROM "jobs" WHERE "jobs"."person_id" = 1
in both of the cases.
I know that when you utilize includes and you specify a where clause on the joined table, you should use .references
example:
# will error out or throw deprecation warning in logs
users = User.includes(:orders).where("Orders.cost < ?", 20)
In rails 4 or later, you will get an error like the following:
Mysql2::Error: Unknown column 'Orders.cost' in 'where clause': SELECT
customers.* FROM customers WHERE (Orders.cost < 100)
Or you will get a deprecation warning:
DEPRECATION WARNING: It looks like you are eager loading table(s) (one
of: users, addresses) that are referenced in a string SQL snippet. For
example:
Post.includes(:comments).where("comments.title = 'foo'") Currently,
Active Record recognizes the table in the string, and knows to JOIN
the comments table to the query, rather than loading comments in a
separate query. However, doing this without writing a full-blown SQL
parser is inherently flawed. Since we don't want to write an SQL
parser, we are removing this functionality. From now on, you must
explicitly tell Active Record when you are referencing a table from a
string:
Post.includes(:comments).where("comments.title =
'foo'").references(:comments)
If you don't rely on implicit join references you can disable the
feature entirely by setting
config.active_record.disable_implicit_join_references = true. (
SELECT "users"."id" AS t0_r0, "users"."name" AS t0_r1, "users"."email"
AS t0_r2, "users"."created_at" AS t0_r3, "users"."updated_at" AS
t0_r4, "addresses"."id" AS t1_r0, "addresses"."user_id" AS t1_r1,
"addresses"."country" AS t1_r2, "addresses"."street" AS t1_r3,
"addresses"."postal_code" AS t1_r4, "addresses"."city" AS t1_r5,
"addresses"."created_at" AS t1_r6, "addresses"."updated_at" AS t1_r7
FROM "users" LEFT OUTER JOIN "addresses" ON "addresses"."user_id" =
"users"."id" WHERE (addresses.country = 'Poland')
so we do this:
# added .references(:orders)
users = User.includes(:orders).where("Orders.cost < ?", 20).references(:orders)
And it executes just fine:
SELECT "users"."id" AS t0_r0,
"users"."name" AS t0_r1,
"users"."created_at" AS t0_r2,
"users"."updated_at" AS t0_r3,
"orders"."id" AS t1_r0,
"orders"."cost" AS t1_r1,
"orders"."user_id" AS t1_r2,
"orders"."created_at" AS t1_r3,
"orders"."updated_at" AS t1_r4
FROM "users"
LEFT OUTER JOIN "orders"
ON "orders"."user_id" = "users"."id"
WHERE ( orders.cost < 20 )
I know that .includes is just a wrapper for two methods: eager_load and preload. I know that since my query above is doing a filter on a joined table (orders in this example), includes is smart and knows to pick the eager_load implementation over preload because preload cannot handle doing this query since preload does not join tables.
Here is where I am confused. Ok: So on that query above: under the hood includes will utilize the eager_load implementation. But notice how when I explicitly use eager_load for this same query (which is what includes is essentially doing): I do not need to use .references! It runs the query and loads the data just fine. No error and no deprecation warning:
# did not specify .references(:orders), and yet no error and no deprecation warning
users = User.eager_load(:orders).where("Orders.cost < ?", 20)
And it executes the exact same process with no problem:
SELECT "users"."id" AS t0_r0,
"users"."name" AS t0_r1,
"users"."created_at" AS t0_r2,
"users"."updated_at" AS t0_r3,
"orders"."id" AS t1_r0,
"orders"."cost" AS t1_r1,
"orders"."user_id" AS t1_r2,
"orders"."created_at" AS t1_r3,
"orders"."updated_at" AS t1_r4
FROM "users"
LEFT OUTER JOIN "orders"
ON "orders"."user_id" = "users"."id"
WHERE ( orders.cost < 20 )
That seems odd. Why does .references need to be specified for the includes version of the query, whereas .references does not need to be specified for the eager_load version of the query? What am I missing here?
It comes down to the problem they mention in the deprecation warning:
Currently, Active Record recognizes the table in the string, and knows to JOIN the comments table to the query, rather than loading comments in a separate query. However, doing this without writing a full-blown SQL parser is inherently flawed. Since we don't want to write an SQL parser, we are removing this functionality.
In older versions, Rails tried to be helpful about selecting the query pattern to use, and includes would use the preload strategy if it could, but switch to the eager_load strategy when it looks like you're referencing something in a joined table. But without a full SQL parser figuring out what tables are actually referenced, it's like parsing XHTML with a Regex - you can get some things done, but Rails can't decide correctly in every case. Consider:
User.includes(:orders).where("Orders.cost < 20")
This is a nice, simple example, and Rails could tell that you need Orders joined. Now try this one:
User.includes(:orders).where("id IN (select user_id from Orders where Orders.cost < 20)")
This gives the same result, but the subquery rendered joining Orders unnecessary. It's a contrived example, and I don't know whether Rails would decide the second query needed to join or not, but the point is there are cases when the heuristic could make the wrong decision. In those cases, either Rails would perform an unnecessary join, burning memory and slowing the query down, or not perform a necessary join, causing an error.
Rather than maintain a heuristic with a pretty bad failure case, the developers decided to just ask the programmer whether the join is needed. You're able to get it right more often than Rails can (hopefully), and when you get it wrong, it's clear what to change.
Instead of adding references you could switch to eager_load, but keeping includes and references separate allows the implementation flexibility in its query pattern. You could conceivably .includes(:orders, :addresses).references(:orders) and have addresses loaded in a second preload-style query because it's not needed during the join (though Rails actually just includes addresses in the join anyway). You don't need to specify references when you're using eager_load because eager_load always joins, where preload always does multiple queries. All references does is instruct includes to use the necessary eager_load strategy and specify which tables are needed.
I have a rails app that uses tagging and sqlite i am trying to deploy as a school assignment. when i deploy to Heroku (it converts to Postgresql) I get an error when trying to run it. It has to do with some differences between SQLITE and POSTGRESQL(from the logs it seems)
There are two tables being joined, tags and tagging which store tags related to posts.
Rails code:
self.select("name, count(taggings.tag_id) as count")
.joins(:taggings).group("taggings.tag_id")
Here is SQLite generated sql captured from console:
Tag Load (0.2ms) SELECT name, count(taggings.tag_id) as count
FROM "tags" INNER JOIN "taggings" ON "taggings"."tag_id" = "tags"."id"
GROUP BY taggings.tag_id
Here is the Heroku log error based on that statement running under Postgresql:
2015-06-20T03:25:00.360433+00:00 app[web.1]: **PG::GroupingError**: ERROR: column "tags.name" must appear in the GROUP BY clause or be used in an aggregate function
2015-06-20T03:25:00.360435+00:00 app[web.1]: LINE 1: SELECT name, count(taggings.tag_id) as count FROM "tags" INN...
2015-06-20T03:25:00.360417+00:00 app[web.1]: Tag Load (2.5ms) SELECT name, count(taggings.tag_id) as count FROM "tags" INNER JOIN "taggings" ON "taggings"."tag_id" = "tags"."id" GROUP BY taggings.tag_id
I am currently experimenting trying to find a solution but could use some help thanks!
I am using postgres 9.3 and rails 4.1
Postgres requires that non-aggregate fields you your SELECT clause appear in your GROUP clause, so it's unhappy that you're selecting name but grouping by taggings.tag_id. Try grouping by name instead.
Actually I found the exact same issue and solution here after i posted. The error pointed to exactly what needed to be done. Include name somewhere in the syntax...I just needed an example. Here was the final code.
Tag.select("tags.id, tags.name,count(taggings.tag_id) as count").
joins(:taggings).group("taggings.tag_id, tags.id, tags.name")
self.select("name, count(taggings.tag_id) as count").joins(:taggings).group("name, taggings.tag_id")
easy
I'm attempting a transition from MySQL to what seems to be the stricter and less Rails-friendly PostgreSQL.
I'm running into this funny conundrum:
irb(main):015:0> puts w.to_sql
SELECT DISTINCT ON (vendors.id) vendors.*
FROM "vendors"
INNER JOIN "locations" ON "locations"."locatable_id" = "vendors"."id"
AND "locations"."locatable_type" = 'Vendor'
WHERE (locations.latitude IS NOT NULL AND locations.longitude IS NOT NULL)
But...
irb(main):017:0> puts w.order('vendors.id').to_sql
SELECT * FROM (
SELECT DISTINCT ON (vendors.id) vendors.*
FROM "vendors"
INNER JOIN "locations" ON "locations"."locatable_id" = "vendors"."id"
AND "locations"."locatable_type" = 'Vendor'
WHERE (locations.latitude IS NOT NULL AND locations.longitude IS NOT NULL)
) AS id_list ORDER BY id_list.alias_0
This, despite the fact that just adding ORDER BY vendors.id works just fine as a valid PostgreSQL query. Instead of just adding that, it does something super funny and produces an invalid query at the end of the day:
ActiveRecord::StatementInvalid: PGError: ERROR: column id_list.alias_0 does not exist
LINE 1: ...tions.longitude IS NOT NULL)) AS id_list ORDER BY id_list.al...
Any clue what I should look at?
I've run into the same problem. Looks like it was a bug in Arel causing DISTINCT ON not to play nice with ORDER when using PostgreSQL.
The original bug is no longer visible since Rails has moved from Lighthouse to github issues but you can take a look at the Google cache. There's a comment on a more recent pull request which indicates the issue was fixed after Rails 3.0.7 was released. Looks like it will be in 3.1 but I couldn't verify as some of the gems I'm using are not yet compatible.
The current work-around is to use find_by_sql.