I'm currently developing some kind of a configurator using neo4j as a backend. Now I ran into a problem, I don't know how to solve best.
I've got nodes created like this:
(A:Product {name:'ProductA', minWidth:20, maxWidth:200, minHeight:10, maxHeight:400})
(B:Product {name:'ProductB', minWidth:40, maxWidth:100, minHeight:20, maxHeight:300})
...
There is an interface where the user can input a desired width & height, f.e. Width=30, Height=250. Now I'd like to check which products match the input criteria. As the input might be any long value, the approach used in http://neo4j.com/blog/modeling-a-multilevel-index-in-neoj4/ with dates doesn't seem to be suitable for me. How can I run a cypher query giving me all the nodes matching the input criteria?
I don't know if I understand well what you are asking for, but if I do, here a simple query to get this:
Assuming the user wants width = 30 and height = 50
Match (p:Product)
WHERE
p.minWidth < 30 AND p.maxWidth > 30 AND
p.minHeight < 50 AND p.maxHeight > 50
RETURN
p
If this is not what you are looking for, feel free to say it as comment.
Related
Nodes with the Location node label have an index on Label.name
Profiling the following query gives me a smart plan, with a NodeHashJoin between the two sides of the graph on either side of Trip nodes. Very clever. Works great.
PROFILE MATCH (rosen:Location)<-[:OCCURS_AT]-(ev:Event)<-[:HAS]-(trip:Trip)-[:OPERATES_ON]->(date:Date)
WHERE rosen.name STARTS WITH "U Rosent" AND
ev.scheduled_departure_time > "07:45:00" AND
date.date = '2015-11-20'
RETURN rosen.name, ev.scheduled_departure_time, trip.headsign
ORDER BY ev.scheduled_departure_time
LIMIT 20;
However, just changing one line of the query from:
WHERE rosen.name STARTS WITH "U Rosent" AND
to
WHERE id(rosen) = 4752371 AND
seems to alter the entire behavior of the query plan, which now appears to become more "sequential", losing the parallel execution of (Trip)-[:OPERATES_ON]->(Date)
Much slower. 6x more DB hits in total.
Question
Why does changing the retrieval of one, seemingly-unrelated Location node via a different index/mechanism alter the behavior of the whole query?
(I'm not sure how best to convey more information about the graph model, but please advise, and I'd be happy to add details that are missing)
Edit:
It gets better. Changing that query line from:
WHERE rosen.name STARTS WITH "U Rosent" AND
to
WHERE rosen.name = "U Rosenthaler Platz." AND
results in the same loss of parallelism in the query plan!
Seems odd that a LIKE query is faster than an = ?
I am server engineer in company that provide dating service.
Currently I am building a PoC for our new recommendation engine.
I try to use neo4j. But performance of this database does not meet our needs.
I have strong feeling that I am doing something wrong and neo4j can do much better.
So can someone give me an advice how to improve performance of my Cypher’s query or how to tune neo4j in right way?
I am using neo4j-enterprise-2.3.1 which is running on c4.4xlarge instance with Amazon Linux.
In our dataset each user can have 4 types of relationships with others users - LIKE, DISLIKE, BLOCK and MATCH.
Also he has a properties like countryCode, birthday and gender.
I made import of all our users and relationships from RDBMS to neo4j using neo4j-import tool.
So each user is a node with properties and each reference is a relationship.
The report from neo4j-import tool said that :
2 558 667 nodes,
1 674 714 539 properties and
1 664 532 288 relationships
were imported.
So it’s huge DB :-) In our case some nodes can have up to 30 000 outgoing relationships..
I made 3 indexes in neo4j :
Indexes
ON :User(userId) ONLINE
ON :User(countryCode) ONLINE
ON :User(birthday) ONLINE
Then I try to build online recommendation engine using this query :
MATCH (me:User {userId: {source_user_id} })-[:LIKE | :MATCH]->()<-[:LIKE | :MATCH]-(similar:User)
USING INDEX me:User(userId)
USING INDEX similar:User(birthday)
WHERE similar.birthday >= {target_age_gte} AND
similar.birthday <= {target_age_lte} AND
similar.countryCode = {target_country_code} AND
similar.gender = {source_gender}
WITH similar, count(*) as weight ORDER BY weight DESC
SKIP {skip_similar_person} LIMIT {limit_similar_person}
MATCH (similar)-[:LIKE | :MATCH]-(recommendation:User)
WITH recommendation, count(*) as sheWeight
WHERE recommendation.birthday >= {recommendation_age_gte} AND
recommendation.birthday <= {recommendation_age_lte} AND
recommendation.gender= {target_gender}
WITH recommendation, sheWeight ORDER BY sheWeight DESC
SKIP {skip_person} LIMIT {limit_person}
MATCH (me:User {userId: {source_user_id} })
WHERE NOT ((me)--(recommendation))
RETURN recommendation
here is the execution plan for one of the user :
plan
When I executed this query for list of users I had the result :
count=2391, min=4565.128849, max=36257.170065, mean=13556.750555555178, stddev=2250.149335254768, median=13405.409811, p75=15361.353029999998, p95=17385.136478, p98=18040.900481, p99=18426.811424, p999=19506.149138, mean_rate=0.9957385490980866, m1=1.2148195797996817, m5=1.1418078036067119, m15=0.9928564378521962, rate_unit=events/second, duration_unit=milliseconds
So even the fastest is too slow for Real-time recommendations..
Can you tell me what I am doing wrong?
Thanks.
EDIT 1 : plan with the expanded boxes :
I built an unmanaged extension to see if I could do better than Cypher. You can grab it here => https://github.com/maxdemarzi/social_dna
This is a first shot, there are a couple of things we can do to speed things up. We can pre-calculate/save similar users, cache things here and there, and random other tricks. Give it a shot, let us know how it goes.
Regards,
Max
If I'm reading this right, it's finding all matches for users by userId and separately finding all matches for users by your various criteria. It's then finding all of the places that they come together.
Since you have a case where you're starting on the left with a single node, my guess is that we'd be better served by following the paths and then filtering what it gotten via relationship traversal.
Let's see how starting like this works for you:
MATCH
(me:User {userId: {source_user_id} })-[:LIKE | :MATCH]->()
<-[:LIKE | :MATCH]-(similar:User)
WITH similar
WHERE similar.birthday >= {target_age_gte} AND
similar.birthday <= {target_age_lte} AND
similar.countryCode = {target_country_code} AND
similar.gender = {source_gender}
WITH similar, count(*) as weight ORDER BY weight DESC
SKIP {skip_similar_person} LIMIT {limit_similar_person}
MATCH (similar)-[:LIKE | :MATCH]-(recommendation:User)
WITH recommendation, count(*) as sheWeight
WHERE recommendation.birthday >= {recommendation_age_gte} AND
recommendation.birthday <= {recommendation_age_lte} AND
recommendation.gender= {target_gender}
WITH recommendation, sheWeight ORDER BY sheWeight DESC
SKIP {skip_person} LIMIT {limit_person}
MATCH (me:User {userId: {source_user_id} })
WHERE NOT ((me)--(recommendation))
RETURN recommendation
[UPDATED]
One possible (and nonintuitive) cause of inefficiency in your query is that when you specify the similar:User(birthday) filter, Cypher uses an index seek with the :User(birthday) index (and additional tests for countryCode and gender) to find all possible DB matches for similar. Let's call that large set of similar nodes A.
Only after finding A does the query filter to see which of those nodes are actually connected to me, as specified by your MATCH pattern.
Now, if there are relatively few me to similar paths (as specified by the MATCH pattern, but without considering its WHERE clause) as compared to the size of A -- say, 2 or more orders of magnitude smaller -- then it might be faster to remove the :User label from similar (since I presume they are probably all going to be users anyway, in your data model), and also remove the USING INDEX similar:User(birthday) clause. In this case, not using the index for similar may actually be faster for you, since you will only be using the WHERE clause on a relatively small set of nodes.
The same considerations also apply to the recommendation node.
Of course, this all has to be verified by testing on your actual data.
I am using Neo4J to represent texts; in the simplest case a text is a sequence of words joined by the relationship LEMMA_TEXT.
I am trying to find the Nth word after a known word, with a query that looks something like this.
MATCH (anchor)-[:LEMMA_TEXT*32]->(word)
WHERE id(anchor) = 3275
RETURN word
In one particular case, if I increase the path length to 33, I get this error:
Neo.DatabaseError.Statement.ExecutionFailure: There is no spoon.
And yet the following query returns the correct result.
MATCH (anchor)-[:LEMMA_TEXT*32]->(word)-[:LEMMA_TEXT]->(next)
WHERE id(anchor) = 3275
RETURN next
which demonstrates that the node I want exists and is reachable.
Where is the section of the manual that tells me how to bend the spoon with my mind? More importantly, what does this actually mean?!
If anything breaks at number like 33, it means that there was a restriction upto 32, why 32? 2^5.
It's not trivial that most of the restrictions are in a factor of 2, MongoDB document size cannot be more than 16 MB, on a collection there could be maximum index, no more than 64. etc.
why it works as 32 and then next, because till 32 it can achieve in one operation and for last one it can see the next one as another operation. But it cannot go for 33 in one operation.
Most of these restrictions are basically sanity check though and not really technical boundary.
As for why it is almost always a factor of 2, I want someone else to answer or in other words I don't know.
Have you tried splitting the landing and the search statements in 2?
Plus you should add the label for the text word (forormance)
Example:
MATCH (anchor)
WHERE id(anchor) = 3275
WITH anchor
MATCH (anchor)-[:LEMMA_TEXT*32]->(word)
RETURN word
You get the same error?
My model is like this:
(wine_critic)-[REVIEW]->(wine_bottle)
A wine_critic can do many reviews of the same wine_bottle,
(they don't know it, it is blind tasting)
with each review having a timestamp and a score
So, I want to get the total number of reviews a wine_bottle has been reviewed by a certain wine_critic, across with the MAX/MIN score that same critic has given to that same bottle.
START critic=node:critics('ID:5')
MATCH (critic)-[rev:REVIEW]->(bottle)
WITH critic, bottle, COUNT(rev) as total_reviews
RETURN critic.name, bottle.name, total_reviews
but also, in the same query I want MAX(rev.score) and MIN(rev.score).
Is it possible?
You don't really need the WITH in this case:
START critic=node:critics('ID:5')
MATCH (critic)-[rev:REVIEW]->(bottle)
RETURN critic.name, bottle.name, COUNT(rev) as total_reviews, MAX(rev.score) as max, MIN(rev.score) as min
May as well also throw AVG(rev.score) and STDEV(rev.score) in too.
Update:
Ugly workaround in 1.8.x:
START critic=node:critics('ID:5')
MATCH (critic)-[rev:REVIEW]->(bottle)
WITH critic, bottle, MIN(rev.score) as min
MATCH (critic)-[rev:REVIEW]->(bottle)
RETURN critic.name, bottle.name, COUNT(rev) as total_reviews, MAX(rev.score) as max, min
Does it possible to have an order by "property" with a where clause and now the "index/position" of the result?
I mean, when using order for sorting we need to be able to know the position of the result in the sort.
Imagine a scoreboard with 1 million user node, i do an order by on user node.score with a where "name = user_name" and i wan't to know the current rank of the user. I do not find how to do this using order by ...
start game=node(1)
match game-[:has_child_user]->user
with user
order by user.score
with user
where user.name = "my_user"
return user , "the position in the sort";
the expected result would be :
node_user | rank
(i don't want to fetch one million entries at client side to know the current rank/position of a node in the ORDER BY!)
This functionality does not exist today in Cypher. Do you have an example of what this would look like in SQL? Would the below be something that fits the bill? (just a sketch, not working!)
(your code)
start game=node(1)
match game-[:has_child_user]->user
with user
order by user.score
(+ this code)
with user, index() as rank
return user.name, rank;
If you have more thoughts or want to start hacking on this please open an issue at https://github.com/neo4j/neo4j/issues
For the time being there is a work around that you can do:
start n=node(0),rank_node=node(1)
match n-[r:rank]->rn
where rn.score <= rank_node.score
return rank_node,count(*) as pos;
For live example see: http://console.neo4j.org/?id=bela20