I have a simple query
MATCH (n:TYPE {id:123})<-[:CONNECTION*]<-(m:TYPE) RETURN m
and when executing the query "manually" (i.e. using the browser interface to follow edges) I only get a single node as a result as there are no further connections. Checking this with the query
MATCH (n:TYPE {id:123})<-[:CONNECTION]<-(m:TYPE)<-[n:CONNECTION]-(o:TYPE) RETURN m,o
shows no results and
MATCH (n:TYPE {id:123})<-[:CONNECTION]<-(m:TYPE) RETURN m
shows a single node so I have made no mistake doing the query manually.
However, the issue is that the first question takes ages to finish and I do not understand why.
Consequently: What is the reason such trivial query takes so long even though the maximum result would be one?
Bonus: How to fix this issue?
As Tezra mentioned, the variable-length pattern match isn't in the same category as the other two queries you listed because there's no restrictions given on any of the nodes in between n and m, they can be of any type. Given that your query is taking a long time, you likely have a fairly dense graph of :CONNECTION relationships between nodes of different types.
If you want to make sure all nodes in your path are of the same label, you need to add that yourself:
MATCH path = (n:TYPE {id:123})<-[:CONNECTION*]-(m:TYPE)
WHERE all(node in nodes(path) WHERE node:TYPE)
RETURN m
Alternately you can use APOC Procedures, which has a fairly efficient means of finding connected nodes (and restricting nodes in the path by label):
MATCH (n:TYPE {id:123})
CALL apoc.path.subgraphNodes(n, {labelFilter:'TYPE', relationshipFilter:'<CONNECTION'}) YIELD node
RETURN node
SKIP 1 // to avoid returning `n`
MATCH (n:TYPE {id:123})<-[:CONNECTION]<-(m:TYPE)<-[n:CONNECTION]-(o:TYPE) RETURN m,o Is not a fair test of MATCH (n:TYPE {id:123})<-[:CONNECTION*]<-(m:TYPE) RETURN m because it excludes the possibility of MATCH (n:TYPE {id:123})<-[:CONNECTION]<-(m:ANYTHING_ELSE)<-[n:CONNECTION]-(o:TYPE) RETURN m,o.
For your main query, you should be returning DISTINCT results MATCH (n:TYPE {id:123})<-[:CONNECTION*]<-(m:TYPE) RETURN DISTINCT m.
This is for 2 main reasons.
Without distinct, each node needs to be returned the number of times for each possible path to it.
Because of the previous point, that is a lot of extra work for no additional meaningful information.
If you use RETURN DISTINCT, it gives the cypher planner the choice to do a pruning search instead of an exhaustive search.
You can also limit the depth of the exhaustive search using ..# so that it doesn't kill your query if you run against a much older version of Neo4j where the Cypher Planner hasn't learned pruning search yet. Example use MATCH (n:TYPE {id:123})<-[:CONNECTION*..10]<-(m:TYPE) RETURN m
Related
I am currently working with neo4j and I need to find path between 2 nodes in large graph. I am using this cypher query:
MATCH p=(acq:Acquisition {id:'1'})-[r*]->(ecs:ExternalCommunicationService {id:'1'})
RETURN p
Everything works as expected (query returns path of any length between nodes) but I am getting warning message displayed bellow:
Warning: This feature is deprecated and will be removed in future versions. Binding relationships to a list in a variable length pattern is deprecated.
In official documentation is used same pattern with *.
What is the correct way of finding paths of any lengths between nodes without getting any warning(without using deprecated syntax) ?
Binding relationships to a list in a variable length pattern is deprecated since 3.2.0-rc1.
According this pull request Cypher queries like:
MATCH (n)-[rs*]-() RETURN rs
will generate a warning and the canonical way to write the same query is:
MATCH p=(n)-[*]-() RETURN relationships(p) AS rs
Since you are not using the r variable in your query you can simply remove it from the query and the warning will disappear. This way:
MATCH p=(acq:Acquisition {id:'1'})-[*]->(ecs:ExternalCommunicationService {id:'1'})
RETURN p
All that means is that is a future release, your use of r in your pattern will no longer be permitted. Your query will need to look like this or it will break in a future release. Since you are not directly trying to use r in your results it should no problem for you to remove it.
MATCH p=(acq:Acquisition {id:'1'})-[*]->(ecs:ExternalCommunicationService {id:'1'})
RETURN p
You could always get the relationships using the relationships(p) to return a collection of relationships from the path if you needed to do something with them after the match.
Depending on the nature of your graph (size and complexity) though your query may become unwieldy because it is virtually unconstrained except for the ending node and the direction. There are a number of ways you could make it safer.
1 - Use shortestPath
You could use shortestPath or allShortestPaths
MATCH p=allShortestPaths((acq:Acquisition {id:'1'})-[*]->(ecs:ExternalCommunicationService {id:'1'}))
RETURN p
2 - Limit the Depth
You could add a limit on the depth of the match. It could be fixed
MATCH p=(acq:Acquisition {id:'1'})-[*10]->(ecs:ExternalCommunicationService {id:'1'})
RETURN p
or a range
MATCH p=(acq:Acquisition {id:'1'})-[*5..10]->(ecs:ExternalCommunicationService {id:'1'})
RETURN p
3 - Add Relationship TYPE
You could add one or more labels to reduce the potential number of paths you match. That can be used in conjunction with the depth parameters.
MATCH p=(acq:Acquisition {id:'1'})-[:TYPE_A|TYPE_B*5..10]->(ecs:ExternalCommunicationService {id:'1'})
RETURN p
4 - Use APOC
You could use APOC procedures.
I have some questions regarding Neo4j's Query profiling.
Consider below simple Cypher query:
PROFILE
MATCH (n:Consumer {mobileNumber: "yyyyyyyyy"}),
(m:Consumer {mobileNumber: "xxxxxxxxxxx"})
WITH n,m
MATCH (n)-[r:HAS_CONTACT]->(m)
RETURN n,m,r;
and output is:
So according to Neo4j's Documentation:
3.7.2.2. Expand Into
When both the start and end node have already been found, expand-into
is used to find all connecting relationships between the two nodes.
Query.
MATCH (p:Person { name: 'me' })-[:FRIENDS_WITH]->(fof)-->(p) RETURN
> fof
So here in the above query (in my case), first of all, it should find both the StartNode & the EndNode before finding any relationships. But unfortunately, it's just finding the StartNode, and then going to expand all connected :HAS_CONTACT relationships, which results in not using "Expand Into" operator. Why does this work this way? There is only one :HAS_CONTACT relationship between the two nodes. There is a Unique Index constraint on :Consumer{mobileNumber}. Why does the above query expand all 7 relationships?
Another question is about the Filter operator: why does it requires 12 db hits although all nodes/ relationships are already retrieved? Why does this operation require 12 db calls for just 6 rows?
Edited
This is the complete Graph I am querying:
Also I have tested different versions of same above query, but the same Query Profile result is returned:
1
PROFILE
MATCH (n:Consumer{mobileNumber: "yyyyyyyyy"})
MATCH (m:Consumer{mobileNumber: "xxxxxxxxxxx"})
WITH n,m
MATCH (n)-[r:HAS_CONTACT]->(m)
RETURN n,m,r;
2
PROFILE
MATCH (n:Consumer{mobileNumber: "yyyyyyyyy"}), (m:Consumer{mobileNumber: "xxxxxxxxxxx"})
WITH n,m
MATCH (n)-[r:HAS_CONTACT]->(m)
RETURN n,m,r;
3
PROFILE
MATCH (n:Consumer{mobileNumber: "yyyyyyyyy"})
WITH n
MATCH (n)-[r:HAS_CONTACT]->(m:Consumer{mobileNumber: "xxxxxxxxxxx"})
RETURN n,m,r;
The query you are executing and the example provided in the Neo4j documentation for Expand Into are not the same. The example query starts and ends at the same node.
If you want the planner to find both nodes first and see if there is a relationship then you could use shortestPath with a length of 1 to minimize the DB hits.
PROFILE
MATCH (n:Consumer {mobileNumber: "yyyyyyyyy"}),
(m:Consumer {mobileNumber: "xxxxxxxxxxx"})
WITH n,m
MATCH Path=shortestPath((n)-[r:HAS_CONTACT*1]->(m))
RETURN n,m,r;
Why does this do this?
It appears that this behaviour relates to how the query planner performs a database search in response to your cypher query. Cypher provides an interface to search and perform operations in the graph (alternatives include the Java API, etc.), queries are handled by the query planner and then turned into graph operations by neo4j's internals. It make sense that the query planner will find what is likely to be the most efficient way to search the graph (hence why we love neo), and so just because a cypher query is written one way, it won't necessarily search the graph in the way we imagine it will in our head.
The documentation on this seemed a little sparse (or, rather I couldn't find it properly), any links or further explanations would be much appreciated.
Examining your query, I think you're trying to say this:
"Find two nodes each with a :Consumer label, n and m, with contact numbers x and y respectively, using the mobileNumber index. If you find them, try and find a -[:HAS_CONTACT]-> relationship from n to m. If you find the relationship, return both nodes and the relationship, else return nothing."
Running this query in this way requires a cartesian product to be created (i.e., a little table of all combinations of n and m - in this case only one row - but for other queries potentially many more), and then relationships to be searched for between each of these rows.
Rather than doing that, since a MATCH clause must be met in order to continue with the query, neo knows that the two nodes n and m must be connected via the -[:HAS_CONTACT]-> relationship if the query is to return anything. Thus, the most efficient way to run the query (and avoid the cartesian product) is as below, which is what your query can be simplified to.
"Find a node n with the :Consumer label, and value x for the index mobileNumber, which is connected via a -[:HAS_CONTACT]-> relationshop to a node m with the :Consumer label, and value y for its proprerty mobileNumber. Return both nodes and the relationship, else return nothing."
So, rather than perform two index searches, a cartesian product and a set of expand into operations, neo performs only one index search, an expand all, and a filter.
You can see the result of this simplification by the query planner through the presence of AUTOSTRING parameters in your query profile.
How to Change Query to Implement Search as Desired
If you want to change the query so that it must use an expand into relationship, make the requirement for the relationship optional, or use explicitly iterative execution. Both these queries below will produce the initially expected query profiles.
Optional example:
PROFILE
MATCH (n:Consumer{mobileNumber: "xxx"})
MATCH (m:Consumer{mobileNumber: "yyy"})
WITH n,m
OPTIONAL MATCH (n)-[r:HAS_CONTACT]->(m)
RETURN n,m,r;
Iterative example:
PROFILE
MATCH (n1:Consumer{mobileNumber: "xxx"})
MATCH (m:Consumer{mobileNumber: "yyy"})
UNWIND COLLECT(n1) AS n
MATCH (n)-[r:HAS_CONTACT]->(m)
RETURN n,m,r;
This is a follow-up question to:
How to ask Neo4j to take cycles into account
In my previous question, #stdob-- kindly helped me find the following query:
MATCH (n1:S)
OPTIONAL MATCH (n1)-[:R]->(n2:R)<-[:R]-(n3:E)
OPTIONAL MATCH (n3t)-[:R]->(n4:R:L)
WHERE n3t = n3
RETURN labels(n1), labels(n3t), labels(n4);
The above query is a replacement for the following:
MATCH (n1:S)
OPTIONAL MATCH (n1)-[:R]->(n2:R)<-[:R]-(n3:E)-[:R]->(n4:R:L)
RETURN labels(n1), labels(n3t), labels(n4);
And I have to use the first one because in my data there's the possibility that n2 and n4 are the same nodes and since Neo4j refuses to take the same node twice, it will return null.
While the first query is valid and working, it has got a really bad performance. It forces the database to restart the search for the whole data and at the end, it will match the selected nodes using n3t = n3. Just to give you a hint on how bad its performance is, on a dataset of 200k magnitude, it takes 5 seconds to return the result while if I omit the second OPTIONAL MATCH and its WHERE the result is generated in less than 10 milliseconds for the same query. If anyone's interested, here's the execution plan for the query:
The right branch is the part I mentioned earlier (which I tried to fool Neo4j to take a node for the second time). As you can see 2M db hits were incurred in order to make Neo4j take a node for the second time. The actual query for this execution plan is:
PROFILE MATCH (n5_1:Revision:`Account`)<-[:RevisionOf]-(n5_2:Entity:`Account`)
WITH n5_2, n5_1
ORDER BY n5_1.customer_number ASC
LIMIT 100
OPTIONAL MATCH (n5_1)-[:`Main Contact`]->(n4_1:Wrapper)<-[:Wrapper]-(:Revision:`Contact`)<-[:RevisionOf]-(n4_2:Entity:`Contact`)
OPTIONAL MATCH (n4_4)-[:RevisionOf]->(n4_3:Revision:Latest:`Contact`:Active)
WHERE (n4_2) = (n4_4)
RETURN n5_1, n5_2, n4_1, n4_2, n4_3
So my question is, how can I write a Cypher query in which a node is taken for the second time while the performance is not gonna suffer?
For some example data and testbed, please go to the other question.
I posted on your linked question an answer that should give the result table you described. If that fits what you're looking for, this query uses the same approach, and may be the solution for this question:
PROFILE MATCH (n5_1:Revision:`Account`)<-[:RevisionOf]-(n5_2:Entity:`Account`)
WITH n5_2, n5_1
ORDER BY n5_1.customer_number ASC
LIMIT 100
OPTIONAL MATCH (n5_1)-[:`Main Contact`]->(n4_1:Wrapper)<-[:Wrapper]-(:Revision:`Contact`)<-[:RevisionOf]-(n4_2:Entity:`Contact`)
WHERE (n4_2)-[:RevisionOf]->(:Revision:Latest:`Contact`:Active)
OPTIONAL MATCH (n4_2)-[:RevisionOf]->(n4_3:Revision:Latest:`Contact`:Active)
RETURN n5_1, n5_2, n4_1, n4_2, n4_3
This keeps the n4_2 in your last OPTIONAL MATCH, which should solve the performance issue you observed.
As you noted in your previous question, you want to avoid circumstances where the first OPTIONAL MATCH succeeds, but the second fails, leaving the variables as non-null from the first OPTIONAL MATCH when you don't want them to be.
We solve that issue by adding a WHERE after the first OPTIONAL MATCH, forcing the match to succeed only if the pattern you're looking for in the second OPTIONAL MATCH exists off of the last node (this will work even if such a pattern reuses relationships and nodes from the OPTIONAL MATCH).
You can try collect the tail in additionally:
PROFILE
MATCH (n1:S)
OPTIONAL MATCH (n1)-[:R]->(n2:R)<-[:R]-(n3:E)
WITH n1, [null] + ( (n3)-[:R]->(:R:L) ) as tail
WITH n1, tail, size(tail) as tailSize
UNWIND tail as t
WITH n1, tailSize, t WHERE (tailSize = 2 AND NOT t is NULL) OR tailSize = 1
WITH n1, nodes(t) as nds
WITH n1, nds[0] as n3t, nds[1] as n4
RETURN labels(n1), labels(n3t), labels(n4)
I want to write a query in Cypher and run it on Neo4j.
The query is:
Given some start vertexes, walk edges and find all vertexes that is connected to any of start vertex.
(start)-[*]->(v)
for every edge E walked
if startVertex(E).someproperty != endVertex(E).someproperty, output E.
The graph may contain cycles.
For example, in the graph above, vertexes are grouped by "group" property. The query should return 7 rows representing the 7 orange colored edges in the graph.
If I write the algorithm by myself it would be a simple depth / breadth first search, and for every edge visited if the filter condition is true, output this edge. The complexity is O(V+E)
But I can't express this algorithm in Cypher since it's very different language.
Then i wrote this query:
find all reachable vertexes
(start)-[*]->(v), reachable = start + v.
find all edges starting from any of reachable. if an edge ends with any reachable vertex and passes the filter, output it.
match (reachable)-[]->(n) where n in reachable and reachable.someprop != n.someprop
so the Cypher code looks like this:
MATCH (n:Col {schema:"${DWMDATA}",table:"CHK_P_T80_ASSET_ACCT_AMT_DD"})
WITH n MATCH (n:Col)-[*]->(m:Col)
WITH collect(distinct n) + collect(distinct m) AS c1
UNWIND c1 AS rn
MATCH (rn:Col)-[]->(xn:Col) WHERE rn.schema<>xn.schema and xn in c1
RETURN rn,xn
The performance of this query is not good as I thought. There are index on :Col(schema)
I am running neo4j 2.3.0 docker image from dockerhub on my windows laptop. Actually it runs on a linux virtual machine on my laptop.
My sample data is a small dataset that contains 0.1M vertexes and 0.5M edges. For some starting nodes it takes 60 or more seconds to complete this query. Any advice for optimizing or rewriting the query? Thanks.
The following code block is the logic I want:
VertexQueue1 = (starting vertexes);
VisitedVertexSet = (empty);
EdgeSet1 = (empty);
While (VertexSet1 is not empty)
{
Vertex0 = VertexQueue1.pop();
VisitedVertexSet.add(Vertex0);
foreach (Edge0 starting from Vertex0)
{
Vertex1 = endingVertex(Edge0);
if (Vertex1.schema <> Vertex0.schema)
{
EdgeSet1.put(Edge0);
}
if (VisitedVertexSet.notContains(Vertex1)
and VertexQueue1.notContains(Vertex1))
{
VertexQueue1.push(Vertex1);
}
}
}
return EdgeSet1;
EDIT:
The profile result shows that expanding all paths has a high cost. Looking at the row number, it seems that Cypher exec engine returns all paths but I want distint edge list only.
LEFT one:
match (start:Col {table:"F_XXY_DSMK_ITRPNL_IDX_STAT_W"})
,(start)-[*0..]->(prev:Col)-->(node:Col)
where prev.schema<>node.schema
return distinct prev,node
RIGHT one:
MATCH (n:Col {schema:"${DWMDATA}",table:"CHK_P_T80_ASSET_ACCT_AMT_DD"})
WITH n MATCH (n:Col)-[*]->(m:Col)
WITH collect(distinct n) + collect(distinct m) AS c1
UNWIND c1 AS rn
MATCH (rn:Col)-[]->(xn:Col) WHERE rn.schema<>xn.schema and xn in c1
RETURN rn,xn
I think Cypher lets this be much easier than you're expecting it to be, if I'm understanding the query. Try this:
MATCH (start:Col {schema:"${DWMDATA}",table:"CHK_P_T80_ASSET_ACCT_AMT_DD"})-->(node:Col)
WHERE start.schema <> node.schema
RETURN start, node
Though I'm not sure why you're comparing the schema property on the nodes. Isn't the schema for the start node fixed by the value that you pass in?
I might not be understanding the query though. If you're looking for more than just the nodes connected to the start node, you could do:
MATCH
(start:Col {schema:"${DWMDATA}",table:"CHK_P_T80_ASSET_ACCT_AMT_DD"})
(start)-[*0..]->(prev:Col)-->(node:Col)
WHERE prev.schema <> node.schema
RETURN prev, node
That open-ended variable length relationship specification might be slow, though.
Also note that when Cypher is browsing a particular path it stops which it finds that it's looped back onto some node (EDIT relationship, not node) in the path matched so far, so cycles aren't really a problem.
Also, is the DWMDATA value that you're passing in interpolated? If so, you should think about using parameters for security / performance:
http://neo4j.com/docs/stable/cypher-parameters.html
EDIT:
Based on your comment I have a couple of thoughts. First limiting to DISTINCT path isn't going to help because every path that it finds is distinct. What you want is the distinct set of pairs, I think, which I think could be achieved by just adding DISTINCT to the query:
MATCH
(start:Col {schema:"${DWMDATA}",table:"CHK_P_T80_ASSET_ACCT_AMT_DD"})
(start)-[*0..]->(prev:Col)-->(node:Col)
WHERE prev.schema <> node.schema
RETURN DISTINT prev, node
Here is another way to go about it which may or may not be more efficient, but might at least give you an idea for how to go about things differently:
MATCH
path=(start:Col {schema:"${DWMDATA}",table:"CHK_P_T80_ASSET_ACCT_AMT_DD"})-->(node:Col)
WITH rels(path) AS rels
UNWIND rels AS rel
WITH DISTINCT rel
WITH startNode(rel) AS start_node, endNode(rel) AS end_node
WHERE start_node.schema <> end_node.schema
RETURN start_node, end_node
I can't say that this would be faster, but here's another way to try:
MATCH (start:Col)-[*]->(node:Col)
WHERE start.property IN {property_values}
WITH collect(ID(node)) AS node_ids
MATCH (:Col)-[r]->(node:Col)
WHERE ID(node) IN node_ids
WITH DISTINCT r
RETURN startNode(r) AS start_node, endNode(r) AS end_node
I suspect that the problem in all cases is with the open-ended variable length path. I've actually asked on the Slack group to try to get a better understanding of how it works. In the meantime, for all the queries that you try I would suggest prefixing them with the PROFILE keyword to get a report from Neo4j on what parts of the query are slow.
// this is very inefficient!
MATCH (start:Col)-[*]->(node:Col)
WHERE start.property IN {property_values}
WITH distinct node
MATCH (prev)-[r]->(node)
RETURN distinct prev, node;
you might be better off with this:
MATCH (start:Col)
WHERE start.property IN {property_values}
MATCH (node:Col)
WHERE shortestPath((start)-[*]->(node)) IS NOT NULL
MATCH (prev)-[r]->(node)
RETURN distinct prev, node;
I am using Neo4j to track relationships in OOP architecture. Let us assume that nodes represent classes and (u) -[:EXTENDS]-> (v) if class u extends class v (i.e. for each node there is at most one outgoing edge of type EXTENDS). I am trying to find out a chain of predecessor classes for a given class (n). I have used the following Cypher query:
start n=node(...)
match (n) -[:EXTENDS*]-> (m)
return m.className
I need to process nodes in such an order that the direct predecessor of class n comes first, its predecessor comes as second etc. It seems that the Neo4j engine returns the nodes in exactly this order (given the above query) - is this something I should rely on or could this behavior suddenly change in some of the future releases?
If I should not rely on this behavior, what Cypher query would allow me to get all predecessor nodes in given ordering? I was thinking about following query:
start n=node(...)
match p = (n) -[:EXTENDS*]-> (m {className: 'Object'})
return p
Which would work quite fine, but I would like to avoid specifying the root class (Object in this case).
It's unlikely to change anytime soon as this is really the nature of graph databases at work.
The query you've written will return ALL possible "paths" of nodes that match that pattern. But given that you've specified that there is at most one :EXTENDS edge from each such node, the order is implied with the direction you've included in the query.
In other words, what's returned won't start "skipping" nodes in a chain.
What it will do, though, is give you all "sub-paths" of a path. That is, assuming you specified you wanted the predecessors for node "a", for the following path...
(a)-[:EXTENDS]->(b)-[:EXTENDS]->(c)
...your query (omitting the property name) will return "a, b, c" and "a, b". If you only want ALL of its predecessors, and you can use Cypher 2.x, consider using the "path" way, something like:
MATCH p = (a)-[:EXTENDS*]->(b)
WITH p
ORDER BY length(p) DESC
LIMIT 1
RETURN extract(x in nodes(p) | p.className)
Also, as a best practice, given that you're looking at paths of indefinite length, you should likely limit the number of hops your query makes to something reasonable, e.g.
MATCH (n) -[:EXTENDS*0..10]-> (m)
Or some such.
HTH