Not null query extremely slow in ajax request partial - ruby-on-rails
I have this simple scope
scope :has_video_link,
-> { where.not(video_link: nil) }
I use it like so in my controller
#videos = Message .featured
.includes(:user, :company, :forum, :topic)
.published
.unremoved
.approved
.has_video_link
.order(created_at: :desc)
.paginate(
page: page,
per_page: limit)
Which I call like so
before_action only: :index
skip_before_action :verify_authenticity_token
def index
self.page_title = 'Home'
fetch_videos
end
That works just fine, and my page loads in under a second on prod no problem. My issue is I have an ajax request that calls this
def videos
fetch_videos
respond_to do |format|
format.js
end
end
On dev this works great but in prod this takes over 60 seconds....
First image is from the first times its called (4ms)
2nd image is the ajax call 69000ms...
Any ideas? How do i fix this?? Thank you!!
edit: here is the explain
EXPLAIN for: SELECT `messages`.* FROM `messages` WHERE `messages`.`is_featured` = 1 AND (messages.created_at<='2021-06-04 12:36:22.241601') AND `messages`.`is_removed` = 0 AND `messages`.`is_approved` = 1 AND (`messages`.`video_link` IS NOT NULL) ORDER BY `messages`.`created_at` DESC LIMIT 5 OFFSET 0
+----+-------------+----------+------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+---------+------+------+----------+------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+---------+------+------+----------+------------------------------------+
| 1 | SIMPLE | messages | NULL | range | index_messages_on_is_removed_and_user_id_and_created_at,index_messages_on_is_removed_and_message_type_and_created_at,index_messages_on_is_removed_company_id_type_created_at,index_messages_on_is_removed_company_id_created_at_id_user_id,index_messages_on_is_removed_and_created_at,index_messages_on_is_removed_company_id_created_at_rating_count,index_messages_on_is_removed_company_id_user_role_created_at,index_messages_on_is_removed_and_company_id_and_rating_total,index_messages_on_is_removed_and_forum_id_and_created_at,index_messages_on_is_approved,index_messages_on_is_removed_and_is_approved | index_messages_on_is_removed_and_created_at | 7 | NULL | 3470 | 9.0 | Using index condition; Using where |
+----+-------------+----------+------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+---------+------+------+----------+------------------------------------+
1 row in set (0.00 sec)
EXPLAIN for: SELECT `users`.* FROM `users` WHERE `users`.`id` = 594568
+----+-------------+-------+------------+-------+---------------+---------+---------+-------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+---------+---------+-------+------+----------+-------+
| 1 | SIMPLE | users | NULL | const | PRIMARY | PRIMARY | 4 | const | 1 | 100.0 | NULL |
+----+-------------+-------+------------+-------+---------------+---------+---------+-------+------+----------+-------+
1 row in set (0.00 sec)
EXPLAIN for: SELECT `companies`.* FROM `companies` WHERE `companies`.`id` IN (564013, 562440)
+----+-------------+-----------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | companies | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.0 | Using where |
+----+-------------+-----------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set (0.01 sec)
EXPLAIN for: SELECT `forums`.* FROM `forums` WHERE `forums`.`id` IN (12224, 7759)
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | forums | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.0 | Using where |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set (0.00 sec)
EXPLAIN for: SELECT `topics`.* FROM `topics` WHERE `topics`.`id` IN (684474, 684473, 684472, 684470, 684467)
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | topics | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 5 | 100.0 | Using where |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set (0.00 sec)
Solved this issue. All I had to do was simply index video_links and that fixed my issue right up!
Related
Cypher query aggregate sum of values
I have a Cypher query that shows the following output: +---------------- | usid | count | +---------------- | "000" | 1 | | "000" | 0 | | "000" | 0 | | "001" | 1 | | "001" | 1 | | "001" | 0 | | "002" | 2 | | "002" | 2 | | "002" | 0 | | "003" | 4 | | "003" | 2 | | "003" | 2 | | "004" | 4 | | "004" | 4 | | "004" | 4 | +---------------- How can I get the below result with the condition SUM(count) <= 9. +---------------- | usid | count | +---------------- | "000" | 1 | | "001" | 2 | | "002" | 4 | | "003" | 8 | +---------------- Note: I have used the below query to get the 1st table data. MATCH (us:USER) WITH us WHERE us.count <= 4 RETURN us.id as usid, us.count as count;
I don't know how you get your original data, so I will just use a WITH clause and assume the data is there: // original data WITH usid, count // aggregate and filter WITH usid, sum(count) as new_count WHERE new_count <= 9 RETURN usid, new_count Based on the updated question, the new query would look like: MATCH (us:USER) WHERE us.count <= 4 WITH us.id as usid, sum(us.count) as count WHERE new_count <= 9 RETURN usid, count ˙˙˙
DISTINCT, MAX and list value from join table
How to combine distinct and max within join table below? Table_details_usage UID | VE_NO | START_MILEAGE | END_MILEAGE ------------------------------------------------ 1 | ASD | 410000 | 410500 2 | JWQ | 212000 | 212350 3 | WYS | 521000 | 521150 4 | JWQ | 212360 | 212400 5 | ASD | 410520 | 410600 Table_service_schedule SID | VE_NO | SV_ONMILEAGE | SV_NEXTMILEAGE ------------------------------------------------ 1 | ASD | 400010 | 410010 2 | JWQ | 212120 | 222120 3 | WYS | 511950 | 521950 4 | JWQ | 212300 | 222300 5 | ASD | 410510 | 420510 How to get display as below (only max value)? Get Max value from Table_service_schedule (SV_NEXTMILEAGE) and Get Max value from Table_details_usage (END_MILEAGE) SID | VE_NO | SV_NEXTMILEAGE | END_MILEAGE -------------------------------------------- 5 | ASD | 420510 | 410600 4 | JWQ | 222300 | 212400 3 | WYS | 521950 | 521150
Something in the lines of: SELECT SID, VE_NO, SV_NEXTMILEAGE, (select max(END_MILEAGE) from Table_details_usage d where d.VE_NO = s.VE_NO) END_MILEAGE FROM Table_service_schedule s WHERE SID = (SELECT max(SID) FROM Table_service_schedule s2 WHERE s2.VE_NO = s.VE_NO) Probably could need to change the direct value ov SV_NEXTMILEAGE to max as well if the id:s aren't in order...
Querying in Rails
How can I get the first name and last name from the table User through table Friend which is referenced by the user_id with the condition of user_id 5 and status is pending and place it in a each loop but i also need the contents of friend controller in rails. Thank a lot for the help Controller #add = Friend.where(user_id: 5, status: 'Pending') User Database +-----------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-----------------+--------------+------+-----+---------+----------------+ | id | int | NO | PRI | NULL | auto_increment | | first_name | text | YES | | NULL | | | last_name | text | YES | | NULL | | | email | text | YES | | NULL | | | contact_number | varchar | YES | | NULL | | | birth_date | date | YES | | NULL | | | created_at | datetime | NO | | NULL | | | updated_at | datetime | NO | | NULL | | Friend +--------------+--------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +--------------+--------------+------+-----+---------+----------------+ | id | int | NO | PRI | NULL | auto_increment | | request_date | date | YES | | NULL | | | reason | text | YES | | NULL | | | status | varchar(255) | YES | | NULL | | | userid | int | YES | MUL | NULL | |
How about this: #add = Friend.joins(:user).select("users.first_name, users.last_name").where(user_id: 5, status: 'Pending')
Seems like youre already fetching the id of the user, stored in the user_id field. In order to get the associated user, i would do #user = User.find(#add.id) This retrieves the user based on the id you pass, and from there you can do stuff like #user.first_name to get the values
All you need to do in Friend model belongs_to user and then #add =Friend.where("user_id = ? AND status = ?", 5,"Pending") and <#add.each do |s|> <%s.user.first_name%> <%=<%s.user.last_name%> // and so on.. <%end%>
PSQL code missing
My problem on getting one row shown with two null values. FROM tuotemerkki; tmtunnus | tmnimi | maa ----------+----------+------------- 1 | McCee | Yhdysvallat 2 | KooTek | Italia 3 | Giardino | Italia (3 rows) FROM tuote; ttunnus | tnimi | kuvaus | suositushinta | tmtunnus ---------+-----------------------+--------------------+---------------+---------- 111 | Trimmeri TRCee | tehokas 4-tahtinen | 179.00 | 1 112 | Trimmerisiima Cee | laadukas siima | 6.99 | 1 113 | Moottorisaha MSCee RR | robusti ja raskas | 559.00 | 1 114 | Trimmerisiima Y | yleissiima | 3.99 | 2 115 | Lapio L | kevyt yleislapio | 23.95 | 2 (5 rows) I need to get this selected with the NULL VALUE on giardino from tmnimi. tmnimi | tnimi | kuvaus ----------+-----------------------+-------------------- McCee | Trimmeri TRCee | tehokas 4-tahtinen McCee | Trimmerisiima Cee | laadukas siima McCee | Moottorisaha MSCee RR | robusti ja raskas KooTek | Trimmerisiima Y | yleissiima KooTek | Lapio L | kevyt yleislapio Giardino | | (6 rows) I get only this selected SELECT tmnimi, tnimi, kuvaus FROM tuote CROSS JOIN tuotemerkki WHERE tuote.tmtunnus = tuotemerkki.tmtunnus; tmnimi | tnimi | kuvaus --------+-----------------------+-------------------- McCee | Trimmeri TRCee | tehokas 4-tahtinen McCee | Trimmerisiima Cee | laadukas siima McCee | Moottorisaha MSCee RR | robusti ja raskas KooTek | Trimmerisiima Y | yleissiima KooTek | Lapio L | kevyt yleislapio (5 rows)
Use a left join between the two tables: SELECT t1.tmnimi, t2.tnimi, t2.kuvaus FROM tuotemerkki t1 LEFT JOIN tuote t2 ON t1.tmtunnus = t2.tmtunnus When you left join from tuotemerkki to tuote, then every record in the former table is guaranteed to appear in the result set. Since the Giardino record does not match to anything in the tuote table, all the columns from that table would have a NULL value for the Giardino record.
Neo4j/Cypher effective pagination with order by over large sub-graph
I have following simple relationship between (:User) nodes. (:User)-[:FOLLOWS {timestamp}]->(:User) If I paginate followers ordered by FOLLOWS.timestamp I'm running into performance problems when someone has millions of followers. MATCH (u:User {Id:{id}})<-[f:FOLLOWS]-(follower) WHERE f.timestamp <= {timestamp} RETURN follower ORDER BY f.timestamp DESC LIMIT 100 What is suggested approach for paginating big sets of data when ordering is required? UPDATE follower timestamp --------------------------------------- id(1000000) 1455967905 id(999999) 1455967875 id(999998) 1455967234 id(999997) 1455967123 id(999996) 1455965321 id(999995) 1455964123 id(999994) 1455963645 id(999993) 1455963512 id(999992) 1455961343 .... id(2) 1455909382 id(1) 1455908432 I want to slice this list down using timestamp which set on :FOLLOWS relationship. If I want to return batches of 4 followers I take current timestamp first and return 4 most recent, then 1455967123 and 4 most recent and so on. In order to do this the whole list should be order by timestamp which results in performance issues on millions of records.
If you're looking for the most recent followers, i.e. where the timestamp is greater than a given time, it only has to traverse the most recent ones. You can do it with (2) in 20ms If you are really looking for the oldest (first) followers it makes sense to skip ahead and don't look at the timestamp of every of the million followers (which takes about 1s on my system, see (3)). If you skip ahead the time goes down to 230ms, see (1) In general we can see that on my laptop it does 2M db-operations per core and second. (1) Look at first / oldest followers PROFILE > MATCH (u)<-[f:FOLLOWS]-(follower) WHERE id(u) = 0 > // skip ahead > WITH f,follower SKIP 999000 > // do the actual check > WITH f,follower WHERE f.ts < 500 > RETURN f, follower > ORDER BY f.ts > LIMIT 10; +---------------------------------+ | f | follower | +---------------------------------+ | :FOLLOWS[0]{ts:1} | Node[1]{} | ... +---------------------------------+ 10 rows 243 ms Compiler CYPHER 2.3 Planner COST Runtime INTERPRETED +-----------------+----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | Operator | Estimated Rows | Rows | DB Hits | Identifiers | Other | +-----------------+----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +ProduceResults | 1 | 10 | 0 | f, follower | f, follower | | | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +Projection | 1 | 10 | 0 | anon[142], anon[155], anon[158], anon[178], f, follower, f, follower, u | anon[155]; anon[158] | | | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +Top | 1 | 10 | 0 | anon[142], anon[155], anon[158], anon[178], f, follower, u | Literal(10); | | | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +Projection | 1 | 499 | 499 | anon[142], anon[155], anon[158], anon[178], f, follower, u | anon[155]; anon[158]; anon[155].ts | | | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +Projection | 1 | 499 | 0 | anon[142], anon[155], anon[158], f, follower, u | f; follower | | | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +Filter | 1 | 499 | 0 | anon[142], f, follower, u | anon[142] | | | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +Projection | 1 | 1000 | 1000 | anon[142], f, follower, u | f; follower; f.ts < { AUTOINT2} | | | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +Skip | 1 | 1000 | 0 | f, follower, u | { AUTOINT1} | | | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +Expand(All) | 1 | 1000000 | 1000001 | f, follower, u | (u)<-[ f#12:FOLLOWS]-( follower#24) | | | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ | +NodeByIdSeek | 1 | 1 | 1 | u | | +-----------------+----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+ Total database accesses: 1001501 (2) Look at most recent followers PROFILE > MATCH (u)<-[f:FOLLOWS]-(follower) WHERE id(u) = 0 > AND f.ts > 999500 > RETURN f, follower > LIMIT 10; +----------------------------------------------+ | f | follower | +----------------------------------------------+ | :FOLLOWS[999839]{ts:999840} | Node[999840]{} | ... +----------------------------------------------+ 10 rows 23 ms Compiler CYPHER 2.3 Planner COST Runtime INTERPRETED +-----------------+----------------+-------+---------+----------------+---------------------------------------------------------------+ | Operator | Estimated Rows | Rows | DB Hits | Identifiers | Other | +-----------------+----------------+-------+---------+----------------+---------------------------------------------------------------+ | +ProduceResults | 1 | 10 | 0 | f, follower | f, follower | | | +----------------+-------+---------+----------------+---------------------------------------------------------------+ | +Limit | 1 | 10 | 0 | f, follower, u | Literal(10) | | | +----------------+-------+---------+----------------+---------------------------------------------------------------+ | +Filter | 1 | 10 | 16394 | f, follower, u | AndedPropertyComparablePredicates(f,f.ts,f.ts > { AUTOINT1}) | | | +----------------+-------+---------+----------------+---------------------------------------------------------------+ | +Expand(All) | 1 | 16394 | 16395 | f, follower, u | (u)<-[f:FOLLOWS]-(follower) | | | +----------------+-------+---------+----------------+---------------------------------------------------------------+ | +NodeByIdSeek | 1 | 1 | 1 | u | | +-----------------+----------------+-------+---------+----------------+---------------------------------------------------------------+ Total database accesses: 32790 (3) Find oldest followers without skipping ahead PROFILE > MATCH (u)<-[f:FOLLOWS]-(follower) WHERE id(u) = 0 > AND f.ts < 500 > RETURN f, follower > LIMIT 10; +-------------------------------------+ | f | follower | +-------------------------------------+ ... | :FOLLOWS[491]{ts:492} | Node[492]{} | +-------------------------------------+ 10 rows 1008 ms Compiler CYPHER 2.3 Planner COST Runtime INTERPRETED +-----------------+----------------+--------+---------+----------------+---------------------------------------------------------------+ | Operator | Estimated Rows | Rows | DB Hits | Identifiers | Other | +-----------------+----------------+--------+---------+----------------+---------------------------------------------------------------+ | +ProduceResults | 1 | 10 | 0 | f, follower | f, follower | | | +----------------+--------+---------+----------------+---------------------------------------------------------------+ | +Limit | 1 | 10 | 0 | f, follower, u | Literal(10) | | | +----------------+--------+---------+----------------+---------------------------------------------------------------+ | +Filter | 1 | 10 | 999498 | f, follower, u | AndedPropertyComparablePredicates(f,f.ts,f.ts < { AUTOINT1}) | | | +----------------+--------+---------+----------------+---------------------------------------------------------------+ | +Expand(All) | 1 | 999498 | 999499 | f, follower, u | (u)<-[f:FOLLOWS]-(follower) | | | +----------------+--------+---------+----------------+---------------------------------------------------------------+ | +NodeByIdSeek | 1 | 1 | 1 | u | | +-----------------+----------------+--------+---------+----------------+---------------------------------------------------------------+ Total database accesses: 1998998