Not null query extremely slow in ajax request partial - ruby-on-rails

I have this simple scope
scope :has_video_link,
-> { where.not(video_link: nil) }
I use it like so in my controller
#videos = Message .featured
.includes(:user, :company, :forum, :topic)
.published
.unremoved
.approved
.has_video_link
.order(created_at: :desc)
.paginate(
page: page,
per_page: limit)
Which I call like so
before_action only: :index
skip_before_action :verify_authenticity_token
def index
self.page_title = 'Home'
fetch_videos
end
That works just fine, and my page loads in under a second on prod no problem. My issue is I have an ajax request that calls this
def videos
fetch_videos
respond_to do |format|
format.js
end
end
On dev this works great but in prod this takes over 60 seconds....
First image is from the first times its called (4ms)
2nd image is the ajax call 69000ms...
Any ideas? How do i fix this?? Thank you!!
edit: here is the explain
EXPLAIN for: SELECT `messages`.* FROM `messages` WHERE `messages`.`is_featured` = 1 AND (messages.created_at<='2021-06-04 12:36:22.241601') AND `messages`.`is_removed` = 0 AND `messages`.`is_approved` = 1 AND (`messages`.`video_link` IS NOT NULL) ORDER BY `messages`.`created_at` DESC LIMIT 5 OFFSET 0
+----+-------------+----------+------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+---------+------+------+----------+------------------------------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+----------+------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+---------+------+------+----------+------------------------------------+
| 1 | SIMPLE | messages | NULL | range | index_messages_on_is_removed_and_user_id_and_created_at,index_messages_on_is_removed_and_message_type_and_created_at,index_messages_on_is_removed_company_id_type_created_at,index_messages_on_is_removed_company_id_created_at_id_user_id,index_messages_on_is_removed_and_created_at,index_messages_on_is_removed_company_id_created_at_rating_count,index_messages_on_is_removed_company_id_user_role_created_at,index_messages_on_is_removed_and_company_id_and_rating_total,index_messages_on_is_removed_and_forum_id_and_created_at,index_messages_on_is_approved,index_messages_on_is_removed_and_is_approved | index_messages_on_is_removed_and_created_at | 7 | NULL | 3470 | 9.0 | Using index condition; Using where |
+----+-------------+----------+------------+-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+---------+------+------+----------+------------------------------------+
1 row in set (0.00 sec)
EXPLAIN for: SELECT `users`.* FROM `users` WHERE `users`.`id` = 594568
+----+-------------+-------+------------+-------+---------------+---------+---------+-------+------+----------+-------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+-------+---------------+---------+---------+-------+------+----------+-------+
| 1 | SIMPLE | users | NULL | const | PRIMARY | PRIMARY | 4 | const | 1 | 100.0 | NULL |
+----+-------------+-------+------------+-------+---------------+---------+---------+-------+------+----------+-------+
1 row in set (0.00 sec)
EXPLAIN for: SELECT `companies`.* FROM `companies` WHERE `companies`.`id` IN (564013, 562440)
+----+-------------+-----------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-----------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | companies | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.0 | Using where |
+----+-------------+-----------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set (0.01 sec)
EXPLAIN for: SELECT `forums`.* FROM `forums` WHERE `forums`.`id` IN (12224, 7759)
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | forums | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 2 | 100.0 | Using where |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set (0.00 sec)
EXPLAIN for: SELECT `topics`.* FROM `topics` WHERE `topics`.`id` IN (684474, 684473, 684472, 684470, 684467)
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | topics | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 5 | 100.0 | Using where |
+----+-------------+--------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
1 row in set (0.00 sec)

Solved this issue. All I had to do was simply index video_links and that fixed my issue right up!

Related

Cypher query aggregate sum of values

I have a Cypher query that shows the following output:
+----------------
| usid | count |
+----------------
| "000" | 1 |
| "000" | 0 |
| "000" | 0 |
| "001" | 1 |
| "001" | 1 |
| "001" | 0 |
| "002" | 2 |
| "002" | 2 |
| "002" | 0 |
| "003" | 4 |
| "003" | 2 |
| "003" | 2 |
| "004" | 4 |
| "004" | 4 |
| "004" | 4 |
+----------------
How can I get the below result with the condition SUM(count) <= 9.
+----------------
| usid | count |
+----------------
| "000" | 1 |
| "001" | 2 |
| "002" | 4 |
| "003" | 8 |
+----------------
Note: I have used the below query to get the 1st table data.
MATCH (us:USER)
WITH us
WHERE us.count <= 4
RETURN us.id as usid, us.count as count;
I don't know how you get your original data, so I will just use a WITH clause and assume the data is there:
// original data
WITH usid, count
// aggregate and filter
WITH usid, sum(count) as new_count
WHERE new_count <= 9
RETURN usid, new_count
Based on the updated question, the new query would look like:
MATCH (us:USER)
WHERE us.count <= 4
WITH us.id as usid, sum(us.count) as count
WHERE new_count <= 9
RETURN usid, count
˙˙˙

DISTINCT, MAX and list value from join table

How to combine distinct and max within join table below?
Table_details_usage
UID | VE_NO | START_MILEAGE | END_MILEAGE
------------------------------------------------
1 | ASD | 410000 | 410500
2 | JWQ | 212000 | 212350
3 | WYS | 521000 | 521150
4 | JWQ | 212360 | 212400
5 | ASD | 410520 | 410600
Table_service_schedule
SID | VE_NO | SV_ONMILEAGE | SV_NEXTMILEAGE
------------------------------------------------
1 | ASD | 400010 | 410010
2 | JWQ | 212120 | 222120
3 | WYS | 511950 | 521950
4 | JWQ | 212300 | 222300
5 | ASD | 410510 | 420510
How to get display as below (only max value)?
Get Max value from Table_service_schedule (SV_NEXTMILEAGE) and Get Max value from Table_details_usage (END_MILEAGE)
SID | VE_NO | SV_NEXTMILEAGE | END_MILEAGE
--------------------------------------------
5 | ASD | 420510 | 410600
4 | JWQ | 222300 | 212400
3 | WYS | 521950 | 521150
Something in the lines of:
SELECT
SID,
VE_NO,
SV_NEXTMILEAGE,
(select max(END_MILEAGE) from Table_details_usage d where d.VE_NO = s.VE_NO) END_MILEAGE
FROM Table_service_schedule s
WHERE SID = (SELECT max(SID) FROM Table_service_schedule s2 WHERE s2.VE_NO = s.VE_NO)
Probably could need to change the direct value ov SV_NEXTMILEAGE to max as well if the id:s aren't in order...

Querying in Rails

How can I get the first name and last name from the table User through table Friend which is referenced by the user_id with the condition of user_id 5 and status is pending and place it in a each loop but i also need the contents of friend controller in rails. Thank a lot for the help
Controller
#add = Friend.where(user_id: 5, status: 'Pending')
User Database
+-----------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+-----------------+--------------+------+-----+---------+----------------+
| id | int | NO | PRI | NULL | auto_increment |
| first_name | text | YES | | NULL | |
| last_name | text | YES | | NULL | |
| email | text | YES | | NULL | |
| contact_number | varchar | YES | | NULL | |
| birth_date | date | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| updated_at | datetime | NO | | NULL | |
Friend
+--------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+----------------+
| id | int | NO | PRI | NULL | auto_increment |
| request_date | date | YES | | NULL | |
| reason | text | YES | | NULL | |
| status | varchar(255) | YES | | NULL | |
| userid | int | YES | MUL | NULL | |
How about this:
#add = Friend.joins(:user).select("users.first_name, users.last_name").where(user_id: 5, status: 'Pending')
Seems like youre already fetching the id of the user, stored in the user_id field. In order to get the associated user, i would do
#user = User.find(#add.id)
This retrieves the user based on the id you pass, and from there you can do stuff like
#user.first_name
to get the values
All you need to do in Friend model belongs_to user and then
#add =Friend.where("user_id = ? AND status = ?", 5,"Pending")
and
<#add.each do |s|>
<%s.user.first_name%>
<%=<%s.user.last_name%>
// and so on..
<%end%>

PSQL code missing

My problem on getting one row shown with two null values.
FROM tuotemerkki;
tmtunnus | tmnimi | maa
----------+----------+-------------
1 | McCee | Yhdysvallat
2 | KooTek | Italia
3 | Giardino | Italia
(3 rows)
FROM tuote;
ttunnus | tnimi | kuvaus | suositushinta | tmtunnus
---------+-----------------------+--------------------+---------------+----------
111 | Trimmeri TRCee | tehokas 4-tahtinen | 179.00 | 1
112 | Trimmerisiima Cee | laadukas siima | 6.99 | 1
113 | Moottorisaha MSCee RR | robusti ja raskas | 559.00 | 1
114 | Trimmerisiima Y | yleissiima | 3.99 | 2
115 | Lapio L | kevyt yleislapio | 23.95 | 2
(5 rows)
I need to get this selected with the NULL VALUE on giardino from tmnimi.
tmnimi | tnimi | kuvaus
----------+-----------------------+--------------------
McCee | Trimmeri TRCee | tehokas 4-tahtinen
McCee | Trimmerisiima Cee | laadukas siima
McCee | Moottorisaha MSCee RR | robusti ja raskas
KooTek | Trimmerisiima Y | yleissiima
KooTek | Lapio L | kevyt yleislapio
Giardino | |
(6 rows)
I get only this selected
SELECT tmnimi, tnimi, kuvaus
FROM tuote
CROSS JOIN tuotemerkki
WHERE tuote.tmtunnus = tuotemerkki.tmtunnus;
tmnimi | tnimi | kuvaus
--------+-----------------------+--------------------
McCee | Trimmeri TRCee | tehokas 4-tahtinen
McCee | Trimmerisiima Cee | laadukas siima
McCee | Moottorisaha MSCee RR | robusti ja raskas
KooTek | Trimmerisiima Y | yleissiima
KooTek | Lapio L | kevyt yleislapio
(5 rows)
Use a left join between the two tables:
SELECT
t1.tmnimi,
t2.tnimi,
t2.kuvaus
FROM tuotemerkki t1
LEFT JOIN tuote t2
ON t1.tmtunnus = t2.tmtunnus
When you left join from tuotemerkki to tuote, then every record in the former table is guaranteed to appear in the result set. Since the Giardino record does not match to anything in the tuote table, all the columns from that table would have a NULL value for the Giardino record.

Neo4j/Cypher effective pagination with order by over large sub-graph

I have following simple relationship between (:User) nodes.
(:User)-[:FOLLOWS {timestamp}]->(:User)
If I paginate followers ordered by FOLLOWS.timestamp I'm running into performance problems when someone has millions of followers.
MATCH (u:User {Id:{id}})<-[f:FOLLOWS]-(follower)
WHERE f.timestamp <= {timestamp}
RETURN follower
ORDER BY f.timestamp DESC
LIMIT 100
What is suggested approach for paginating big sets of data when ordering is required?
UPDATE
follower timestamp
---------------------------------------
id(1000000) 1455967905
id(999999) 1455967875
id(999998) 1455967234
id(999997) 1455967123
id(999996) 1455965321
id(999995) 1455964123
id(999994) 1455963645
id(999993) 1455963512
id(999992) 1455961343
....
id(2) 1455909382
id(1) 1455908432
I want to slice this list down using timestamp which set on :FOLLOWS relationship. If I want to return batches of 4 followers I take current timestamp first and return 4 most recent, then 1455967123 and 4 most recent and so on. In order to do this the whole list should be order by timestamp which results in performance issues on millions of records.
If you're looking for the most recent followers, i.e. where the timestamp is greater than a given time, it only has to traverse the most recent ones.
You can do it with (2) in 20ms
If you are really looking for the oldest (first) followers it makes sense to skip ahead and don't look at the timestamp of every of the million followers (which takes about 1s on my system, see (3)). If you skip ahead the time goes down to 230ms, see (1)
In general we can see that on my laptop it does 2M db-operations per core and second.
(1) Look at first / oldest followers
PROFILE
> MATCH (u)<-[f:FOLLOWS]-(follower) WHERE id(u) = 0
> // skip ahead
> WITH f,follower SKIP 999000
> // do the actual check
> WITH f,follower WHERE f.ts < 500
> RETURN f, follower
> ORDER BY f.ts
> LIMIT 10;
+---------------------------------+
| f | follower |
+---------------------------------+
| :FOLLOWS[0]{ts:1} | Node[1]{} |
...
+---------------------------------+
10 rows
243 ms
Compiler CYPHER 2.3 Planner COST Runtime INTERPRETED
+-----------------+----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| Operator | Estimated Rows | Rows | DB Hits | Identifiers | Other |
+-----------------+----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +ProduceResults | 1 | 10 | 0 | f, follower | f, follower |
| | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +Projection | 1 | 10 | 0 | anon[142], anon[155], anon[158], anon[178], f, follower, f, follower, u | anon[155]; anon[158] |
| | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +Top | 1 | 10 | 0 | anon[142], anon[155], anon[158], anon[178], f, follower, u | Literal(10); |
| | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +Projection | 1 | 499 | 499 | anon[142], anon[155], anon[158], anon[178], f, follower, u | anon[155]; anon[158]; anon[155].ts |
| | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +Projection | 1 | 499 | 0 | anon[142], anon[155], anon[158], f, follower, u | f; follower |
| | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +Filter | 1 | 499 | 0 | anon[142], f, follower, u | anon[142] |
| | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +Projection | 1 | 1000 | 1000 | anon[142], f, follower, u | f; follower; f.ts < { AUTOINT2} |
| | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +Skip | 1 | 1000 | 0 | f, follower, u | { AUTOINT1} |
| | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +Expand(All) | 1 | 1000000 | 1000001 | f, follower, u | (u)<-[ f#12:FOLLOWS]-( follower#24) |
| | +----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
| +NodeByIdSeek | 1 | 1 | 1 | u | |
+-----------------+----------------+---------+---------+-------------------------------------------------------------------------+---------------------------------------+
Total database accesses: 1001501
(2) Look at most recent followers
PROFILE
> MATCH (u)<-[f:FOLLOWS]-(follower) WHERE id(u) = 0
> AND f.ts > 999500
> RETURN f, follower
> LIMIT 10;
+----------------------------------------------+
| f | follower |
+----------------------------------------------+
| :FOLLOWS[999839]{ts:999840} | Node[999840]{} |
...
+----------------------------------------------+
10 rows
23 ms
Compiler CYPHER 2.3 Planner COST Runtime INTERPRETED
+-----------------+----------------+-------+---------+----------------+---------------------------------------------------------------+
| Operator | Estimated Rows | Rows | DB Hits | Identifiers | Other |
+-----------------+----------------+-------+---------+----------------+---------------------------------------------------------------+
| +ProduceResults | 1 | 10 | 0 | f, follower | f, follower |
| | +----------------+-------+---------+----------------+---------------------------------------------------------------+
| +Limit | 1 | 10 | 0 | f, follower, u | Literal(10) |
| | +----------------+-------+---------+----------------+---------------------------------------------------------------+
| +Filter | 1 | 10 | 16394 | f, follower, u | AndedPropertyComparablePredicates(f,f.ts,f.ts > { AUTOINT1}) |
| | +----------------+-------+---------+----------------+---------------------------------------------------------------+
| +Expand(All) | 1 | 16394 | 16395 | f, follower, u | (u)<-[f:FOLLOWS]-(follower) |
| | +----------------+-------+---------+----------------+---------------------------------------------------------------+
| +NodeByIdSeek | 1 | 1 | 1 | u | |
+-----------------+----------------+-------+---------+----------------+---------------------------------------------------------------+
Total database accesses: 32790
(3) Find oldest followers without skipping ahead
PROFILE
> MATCH (u)<-[f:FOLLOWS]-(follower) WHERE id(u) = 0
> AND f.ts < 500
> RETURN f, follower
> LIMIT 10;
+-------------------------------------+
| f | follower |
+-------------------------------------+
...
| :FOLLOWS[491]{ts:492} | Node[492]{} |
+-------------------------------------+
10 rows
1008 ms
Compiler CYPHER 2.3 Planner COST Runtime INTERPRETED
+-----------------+----------------+--------+---------+----------------+---------------------------------------------------------------+
| Operator | Estimated Rows | Rows | DB Hits | Identifiers | Other |
+-----------------+----------------+--------+---------+----------------+---------------------------------------------------------------+
| +ProduceResults | 1 | 10 | 0 | f, follower | f, follower |
| | +----------------+--------+---------+----------------+---------------------------------------------------------------+
| +Limit | 1 | 10 | 0 | f, follower, u | Literal(10) |
| | +----------------+--------+---------+----------------+---------------------------------------------------------------+
| +Filter | 1 | 10 | 999498 | f, follower, u | AndedPropertyComparablePredicates(f,f.ts,f.ts < { AUTOINT1}) |
| | +----------------+--------+---------+----------------+---------------------------------------------------------------+
| +Expand(All) | 1 | 999498 | 999499 | f, follower, u | (u)<-[f:FOLLOWS]-(follower) |
| | +----------------+--------+---------+----------------+---------------------------------------------------------------+
| +NodeByIdSeek | 1 | 1 | 1 | u | |
+-----------------+----------------+--------+---------+----------------+---------------------------------------------------------------+
Total database accesses: 1998998

Resources