Doctrine : leftJoin on table generates unnecessary queries - symfony1

Short description of my environment :
Symfony : 1.4.8
Doctrine : 1.2.3
Centos 5.5 (PHP 5.3.3)
MySQL 5.1.52
I've a new project built in Symfony and here is the project schema :
# Car
RjCar:
actAs: { Timestampable: ~ }
columns:
id: { type: integer(4), unsigned: true, primary: true, autoincrement: true }
year: { type: integer(2), unsigned: true, notnull: true }
engine_mod: { type: string(1000) }
exterior_mod: { type: string(1000) }
suspension_mod: { type: string(1000) }
audio_mod: { type: string(1000) }
vote_pos: { type: integer(4), notnull: true, unsigned: true, default: 0 }
vote_neg: { type: integer(4), notnull: true, unsigned: true, default: 0 }
views: { type: integer(4), notnull: true, unsigned: true, default: 0 }
# Foreign keys
category_id: { type: integer(1), unsigned: true, notnull: true }
category_check: { type: boolean, notnull: true, default: 0 }
user_id: { type: integer(5) }
relations:
RjCategory: { onDelete: CASCADE, local: category_id, foreign: id, foreignAlias: RjCars }
sfGuardUser: { onDelete: CASCADE, local: user_id, foreign: id, foreignAlias: RjCars }
# Category
RjCategory:
columns:
id: { type: integer(1), unsigned: true, primary: true, autoincrement: true }
name: { type: string(255), notnull: true}
# I do not include the sfGuardUser schema, but it's the default one from the plugin
When I want to retrieve the last 10 cars with category name and username, I use the following code in the RjCarTable.class.php :
$last_cars = $this->createQuery('car')
->leftJoin('car.sfGuardUser user')
->leftJoin('car.RjCategory categ')
->orderBy('car.created_at DESC')
->limit(10)
->execute();
return $last_cars;
On my page everything looks fine, I've all my results, but in the debug bar, I see 22 queries (instead of 2 as it should be).
Here is the query output for the first one that is normal :
SELECT
/* Everything about the car */
r.id AS r__id,
r.year AS r__year,
r.engine_mod AS r__engine_mod,
r.exterior_mod AS r__exterior_mod,
r.suspension_mod AS r__suspension_mod,
r.audio_mod AS r__audio_mod,
r.vote_pos AS r__vote_pos,
r.vote_neg AS r__vote_neg,
r.views AS r__views,
r.type_id AS r__type_id,
r.category_id AS r__category_id,
r.category_check AS r__category_check,
r.user_id AS r__user_id,
r.created_at AS r__created_at,
r.updated_at AS r__updated_at,
/* ... hidden because irrelevant... retrieve everything about the sfGuardUser and profile... */
/* Everything about the category */
r2.id AS r2__id,
r2.name AS r2__name,
FROM rj_car r
LEFT JOIN sf_guard_user s ON r.user_id = s.id
LEFT JOIN rj_category r2 ON r.category_id = r2.id
ORDER BY r.created_at DESC
LIMIT 10
So until then everything is normal, except this query is followed by 20 others to retrieve informations about each category (2 queries for each result apparently), while you can notice in the previous query these informations are available. I will not put all of them but here is some :
SELECT r.id AS r__id, r.name AS r__name FROM rj_category r WHERE (r.id = '8') LIMIT 1
SELECT r.id AS r__id, r.name AS r__name FROM rj_category r WHERE (r.id = '8') LIMIT 1
SELECT r.id AS r__id, r.name AS r__name FROM rj_category r WHERE (r.id = '9') LIMIT 1
SELECT r.id AS r__id, r.name AS r__name FROM rj_category r WHERE (r.id = '9') LIMIT 1
/* etc.. 20 times */
So my real questions are :
- Why is it performing all these unecessary queries while the first query should have these information?
- Why is this not happening for the sfGuardUser table? The relation between my RjCar object and sfGuardUser object is apparently the same as the one between RjCar and RjCategory.
If somebody already face the same problem I'll be really glad to ear about it. As I say everything is working fine, but I prefere this module not generate unecessary queries as it should perform on the homepage of my application.

This behavior seems odd and definitely not what is expected. Some things to try:
Are you caching Doctrine queries or results? Sometimes queries can stay cached after changes to the model with weird effects.
Those queries definitely look like Doctrine's lazy-fetching behavior. Have you tried narrowing down which calls specifically trigger those queries? Try tracking down which specific lines trigger the queries and see if there's anything special about them.

I slapped myself :
Here is what I found in the RjCar.class.php
/**
* Return the category of the car
*/
public function getRjCategory(){
return Doctrine_Core::getTable('RjCategory')->find($this->getCategoryId());
}
So it explains the unecessary queries... I don't even remember writing this piece of code, but considering I'm the only one working on this project I guess it's me.
As usual the problem was between the chair and the keyboard...
Thanks for your help.

When using ORM frameworks like Doctrine and Hibernate, it often generates redundant and ugly queries. The more complicated the query, the uglier it gets on these systems (try adding a 'limit' to your joined query and look at the code generated).
Frameworks in general provide great solution for 80% of your cases. For the remaining 20% you need to work slightly harder. Usually it's a fair trade-off.
If you are concerned about the quality of the output of this query, or think that you can't afford the latency it causes - Doctrine has a very simple method for writing and managing custom queries. Don't be afraid to go down that path...

Related

Swagger - How to write an example for a Free-Form Object?

We have a response type "Error" that may contain a field "extraInfo".
Error:
type: object
properties:
code:
type: string
message:
type: string
extraInfo:
description: any complementary information
type: object
// how to provide examples here?
One example for it can be :
"extraInfo": {
"doors": {
"frontLeftClosed": false,
"frontRightClosed": true,
"rearLeftClosed": true,
"rearRightClosed": true,
"trunkClosed": true
},
"windows": {
"frontLeftClosed": false,
"rearLeftClosed": true,
"trunkClosed": false
}
}
another could be :
"extraInfo": {
"transactionId": "77812783001"
}
Since it s a free form object, is there a way to provide examples for it in Swagger?
Couldn't find it in the spec : https://swagger.io/docs/specification/data-models/data-types/
Use the example keyword and specify the example value using the YAML or JSON object syntax:
extraInfo:
description: any complementary information
type: object
example: # <-------
doors:
frontLeftClosed: false
frontRightClosed: true
rearLeftClosed: true
rearRightClosed: true
trunkClosed: true
windows:
frontLeftClosed: false
rearLeftClosed: true
trunkClosed: false
OpenAPI 3.1 (which is compatible with JSON Schema 2020-12) also supports multiple examples for schemas and properties.
# openapi: 3.1.0
extraInfo:
description: any complementary information
type: object
# A list of examples
examples:
# Example 1
- transactionId: '77812783001'
# Example 2
- doors:
frontLeftClosed: false
frontRightClosed: true
rearLeftClosed: true
rearRightClosed: true
trunkClosed: true
windows:
frontLeftClosed: false
rearLeftClosed: true
trunkClosed: false

How can I find each ascendent of a record in a self-referential table without N+1

I am working with a self-referential model (Category) in a Ruby on Rails app, which has the following columns:
id, name, level, parent_id
belongs_to :parent, class_name: 'Category', optional: true
The concept is that a level 1 category can have a level 2 subcategory, which can have a level 3 subcategory, etc.
For example:
Category id: 1, name: 'Dessert', level: 1, parent_id: nil
Category id: 2, name: 'Cold', level: 2, parent_id: 1
Category id: 3, name: 'Cake', level: 3, parent_id: 2
Category id: 4, name: 'Ice Cream', level: 3, parent_id: 2
Category id: 5, name: 'Sponge', level: 4, parent_id: 3
I'd like to find each ascendent of a record, regardless of how many levels deep it is. I then want to concatenate all the names, in ascending order, into one string.
i.e., if I'm starting with Sponge, I'd like a method which returns "Dessert - Cold - Cake - Sponge"
What I have so far works but is an n+1 and doesn't feel very Railsy:
def self.concatenate_categories(order)
category = order.category
categories_array = []
order.category.level.times do
categories_array.push category.name
category = Category.find(category.parent_id) if category.parent_id.present?
end
categories_array.reverse.join(' - ')
end
If this order is for Sponge, I get "Dessert - Cold - Cake - Sponge".
If the order is for Cake, I get "Dessert - Cold - Cake".
You can try a recursive CTE to get each category parent based on its parent_id:
WITH bar AS (
WITH RECURSIVE foo AS (
SELECT
categories.id,
categories.name,
categories.parent_id
FROM categories
WHERE categories.id = 5
UNION
SELECT
p.id,
p.name,
p.parent_id
FROM categories p
INNER JOIN foo f
ON f.parent_id = p.id
) SELECT name FROM foo ORDER BY id
) SELECT STRING_AGG(name, ' - ') FROM bar
How about this? I haven't tested that code, but you get the idea, join as many times as there are levels and query once. Depending on how many levels there are, your solution can be faster than too many joins.
def self.concatenate_categories(order)
scope = order.category
categories_array = if category.level > 1
scope = scope.select('categories.name')
order.category.level.downto(1) do |l|
scope = scope.joins("JOIN categories as c#{l} ON categories.id = c#{l}.parent_id")
.select("c#{l}.name")
end
scope.to_a
else
Array.wrap(scope.name)
end
categories_array.reverse.join(' - ')
end

How to merge 2 activerecord records together and be left with 1? Rails

I have 2 apples:
{
id: 1,
rotten: true,
branch_on_tree: nil,
type: "red delicious"
},
{
id: 2,
rotten: nil,
branch_on_tree: 5,
type: "red delicious"
}
They are duplicate apples for red delicious. How do I merge the records together and then delete the one with missing data? Is there a convenient way to do this?
Note: There might be like 10 duplicates. I don't want any null values in my final record. Non-null values take precedence.
Not very convinient way but it will work
assuming apples is an array:
[
{
id: 1,
rotten: true,
branch_on_tree: nil,
type: "red delicious"
},
# ...
]
that can come from:
apples = Apple.where(type: "red delicious")
apples_attrs = apples.map(&:attributes)
Then,
apple_attrs = apples_attrs.reduce do |apple, next_apple|
apple.merge(next_apple) do |_, old_value, new_value|
old_value || new_value
end
end
apples.destroy_all
Apple.create(apple_attrs)
You might want to check this guide https://apidock.com/ruby/Hash/merge
Assuming type always has some value, you can use DISTINCT with where clause. The below should work
Apple.where('rotten IS NOT NULL AND branch_on_tree IS NOT NULL').select('DISTINCT ON (type) rotten,branch_on_tree,type').take

How can I express this SQL in Elasticsearch-rails and Elasticsearch-model?

I used gem elasticsearch-rails and elasticsearch-model and I have difficult to write this query in elasticsearch-rails.
SELECT "news".* FROM "news"
WHERE "news"."is_active" = 'true' AND
((priority is not null AND created_at > '2014-07-08 08:55:52.888587') OR
(created_at > '2014-07-08 08:55:52.888820' AND is_persisted = 't') )
ORDER BY "news"."priority" ASC, "news"."created_at" DESC
LIMIT 10 OFFSET 0
In my previous project I used "Tire Model", I used something like this:
filter :bool, must: {and: [{term: {field with options}}, {term: {field with options}}]}, It works in tire model
But if I use something like this in elasticsearch-rails, it throws missing filtered error
I write something like this for filtering active record:
def self.news_index(page = 1)
query =:
response=self.elasticsearch.search query: { match: { is_active: true }}
response=response.page(page)
end
In the above method, I want to add combined filter with bool option. Can anyone guide me?
Elasticsearch-ruby is far closer to the elasticsearch DSL when it comes to querying. Most of the time you'll be passing in hashes (or hash-like objects) to the query method.
Something like this should get you close:
self.search query: {
filtered: {
query: { match: { is_active: true }},
filter: {
bool: {
must: {
and: [{term: {field with options}}, {term: {field with options}}]
}
}
}
}
}
The difference is that instead of filter being a method call in the tire query which took arguments :bool and the filter. You now need to specify a :filter key with a hash value which then contains a :bool key with your existing filter as the value.

Doctrine Rollback Issue

I have a weird phenomenom. I added database tables to the schema which are already existing in the database the connect 'virtuemart'. But he can't connect to this tables.
I tried it with
$taxId = Doctrine::getTable('JosVmTaxRate')
->findOneBy('tr.tax_country', 'NZL');
and
$taxId = Doctrine::getTable('JosVmTaxRate')
->findBy('tr.tax_country', 'NZL');
and
$taxId = Doctrine::getTable('JosVmTaxRate')
->findOneBy('tr.tax_country', '"NZL"');
and
$taxId = Doctrine::getTable('JosVmTaxRate')
->createQuery('tr')
->addWhere('tr.tax_country =?', 'NZL')
->execute()
->get(0);
but always the error message
500 | Internal Server Error | Doctrine_Transaction_Exception
Rollback failed. There is no active transaction.
appears.
I checked the schema but I can't find any fault in there:
JosVmTaxRate:
columns:
tax_rate_id: { type: int, notnull: true, unique: true, primary: true, autoincrement: true }
vendor_id { type: int, default: null }
tax_state { type: string(64), default: null }
tax_country { type: string(64), default: null }
mdate { type: int, default: null }
tax_rate { type: decimal(10), scale(4), default: null }
Any idea?
I hope I provided enough information. If not don't hesitate and ask for it.
The code looks alright for me, but the schema not.
What about the colons after vendor_id, tax_state, tax_country, mdate and tax_rate? They are missing.
scale(4) is wrong. scale: 4 is correct.

Resources