Neo4j query for complete paths - neo4j

I have the following structure.
CREATE
(`0` :Sentence {`{text`:'This is a sentence'}}) ,
(`1` :Word {`{ text`:'This' }}) ,
(`2` :Word {`{text`:'is'}}) ,
(`3` :Sentence {`{'text'`:'Sam is a dog'}}) ,
(`0`)-[:`RELATED_TO`]->(`1`),
(`0`)-[:`RELATED_TO`]->(`2`),
(`3`)-[:`RELATED_TO`]->(`2`)
schema example
So my question is this. I have a bunch of sentences that I have decomposed into word objects. These word objects are all unique and therefore will point to different sentences. If I perform a search for one word it's very easy to figure out all of the sentences that word is related to. How can I structure a query to figure out the same information for two words instead of one.
I would like to submit two or more words and find a path that includes all words submitted picking up all sentences of interest.

I just remembered an alternate approach that may work better. Compare the PROFILE on this query with the profiles for the others, see if it works better for you.
WITH {myListOfWords} as wordList
WITH wordList, size(wordList) as wordCnt
MATCH (s)-[:RELATED_TO]->(w:Word)
WHERE w.text in wordList
WITH s, wordCnt, count(DISTINCT w) as cnt
WHERE wordCnt = cnt
RETURN s

Unfortunately it's not a very pretty approach, it basically comes down to collecting :Word nodes and using the ALL() predicate to ensure that the pattern you want holds true for all elements of the collection.
MATCH (w:Word)
WHERE w.text in {myListOfWords}
WITH collect(w) as words
MATCH (s:Sentence)
WHERE ALL(word in words WHERE (s)-[:RELATED_TO]->(word))
RETURN s
What makes this ugly is that the planner isn't intelligent enough right now to infer that when you say MATCH (s:Sentence) WHERE ALL(word in words ... that the initial matches for s ought to come from the match from the first w in your words collection, so it starts out from all :Sentence nodes first, which is a major performance hit.
So to get around this, we have to explicitly match from the first of the words collection, and then use WHERE ALL() for the remaining.
MATCH (w:Word)
WHERE w.text in {myListOfWords}
WITH w, size(()-[:RELATED_TO]->(w)) as rels
WITH w ORDER BY rels ASC
WITH collect(w) as words
WITH head(words) as head, tail(words) as words
MATCH (s)-[:RELATED_TO]->(head)
WHERE ALL(word in words WHERE (s)-[:RELATED_TO]->(word))
RETURN s
EDIT:
Added an optimization to order your w nodes by the degree of their incoming :RELATED_TO relationships (this is a degree lookup on very few nodes), as this will mean the initial match to your :Sentence nodes is the smallest possible starting set before you filter for relationships from the rest of the words.

As an alternative, you could consider using manual indexing (also called "legacy indexing") instead of using Word nodes and RELATED_TO relationships. Manual indexes support "fulltext" searches using lucene.
There are many apoc procedures that help you with this.
Here is an example that might work for you. In this example, I assume case-insensitive comparisons are OK, you retain the Sentence nodes (and their text properties), and you want to automatically add the text properties of all Sentence nodes to a manual index.
If you are using neo4j 3.2+, you have to add this setting to the neo4j.conf file to make some expensive apoc.index procedures (like apoc.index.addAllNodes) available:
dbms.security.procedures.unrestricted=apoc.*
Execute this Cypher code to initialize a manual index named "WordIndex" with the text text from all existing Sentence nodes, and to enable automatic indexing from that point onwards:
CALL apoc.index.addAllNodes('WordIndex', {Sentence: ['text']}, {autoUpdate: true})
YIELD label, property, nodeCount
RETURN *;
To find (case insensitively) the Sentence nodes containing all the words in a collection (passed via a $words parameter), you'd execute a Cypher statement like the one below. The WITH clause builds the lucene query string (e.g., "foo AND bar") for you. Caveat: since lucene's special boolean terms (like "AND" and "OR") are always in uppercase, you should make sure the words you pass in are lowercased (or modify the WITH clause below to use the TOLOWER()` function as needed).
WITH REDUCE(s = $words[0], x IN $words[1..] | s + ' AND ' + x) AS q
CALL apoc.index.search('WordIndex', q) YIELD node
RETURN node;

Related

Neo4j all the ways to run a `MATCH`

I'm trying to fill in my understanding of the fundamentals of neo4j (version 3.4).
I believe that all of the following produce the same results -- ie they are different syntax for doing exactly the same thing:
MATCH (ee {name: "Emil"}) RETURN ee;
MATCH (ee) WHERE ee.name = "Emil" RETURN ee;
MATCH (ee:Person {name: "Emil"}) RETURN ee;
MATCH (ee:Person) WHERE ee.name = "Emil" RETURN ee;
MATCH (ee:Person) WHERE (ee).name = "Emil" RETURN ee;
I actually have several questions:
Among this list is there a "best" way of doing a Node MATCH? Obviously using a :Label makes it more efficient, but the effect of WHERE vs prop maps is mysterious.
Are any of these out-right incorrect? That is to say that although they work, it's kind of unintended or a particularly bad pattern.
Are there additional ways of making a MATCH operation, further to this list? I'm curious for an exhaustive list (wrong or right).
Your clauses are NOT all the same.
The first group of (2) MATCH clauses DO NOT require the Person label for matched nodes. But the second group of (3) clauses DO require the Person label.
Putting a label (or many labels, as appropriate) on a node is generally a good idea, as matching on a label is a quick way to filter your nodes when performing a MATCH. And labels also aid in data model understandability. In addition, a label is required if you wanted to index a node property.
But whether or not you should specify a label (or several) on a specific a MATCH clause depends on the amount of filtering you are trying to do. Omitting all labels from a MATCH would be perfectly appropriate if at that point in your query you really wanted to match nodes with any label (or even no labels).
For an exhaustive list, you might include:
MATCH (ee) WHERE ee.name in ["Emil"] RETURN ee
This is not particularly helpful with a single item, but you can have multiple comma delimited items in a list or a collection. For example,
match (n:order{item:'widget'})
with collect(distinct n.customer_id) as customerCollection
match (c:Customers) where c.customer_id in [customerCollection]
return c.Name, c.City

Leaf nodes and paths in cypher

I have a graph in Neo4J that looks like this:
(a {flag:any})<- (many, 0 or more) <-(b {flag:true})<- (many, 0 or more) <-(c {flag: any})
-OR-
(a {flag:any})<- (many, 0 or more) <-(d)
-OR-
(a {flag:any})
Where a, b, c, and d all have the same type, and the relations are also the same. All the nodes have flag:false except where noted. Of course the real graph is a tree, not a vine.
In short, every path should begin with a and end with the first flag=true node, or should begin with a and get all children down to the leaf of the tree. Per the last example, a doesn't have to have any children - it can be a root and a leaf. Finally, in the first case, we'll never pull in c. b stops the traversal.
How can I write this query?
I have gotten it to work with a path and several unwind/collect statements that are basically horse****, lol. I want a better query, but I am so confused now it is not going to happen.
The following query should return all 3 kinds of paths. I assume that all relevant nodes are labeled Foo, and all relevant relationships have the BAR type.
The first term of the WHERE clause looks for paths (of length 0 or more, because of the variable-length relationship pattern used in the MATCH clasue) that end in a node with a true flag with no true flags earlier in the path (except for possibly the starting node). The second term looks for paths (of length 0 or more) ending with a leaf node, where no nodes (except for possibly the starting node) have a true flag.
MATCH p=(a:Foo)<-[:BAR*0..]-(b:Foo)
WHERE
(b.flag AND NONE(x IN NODES(p)[1..-1] WHERE x.flag)) OR
((NOT (b)<-[:BAR]-()) AND NONE(y IN NODES(p)[1..] WHERE y.flag))
RETURN p;
NOTE: Variable-length relationship patterns with no upper bound (like [:BAR*0..]) can be very expensive, and can take a very long time or cause an out of memory error. So, you may need to specify a reasonable upper bound (for example, [:BAR*0..5]).
I would approach this query as the UNION of the two cases:
MATCH shortestPath((a)<-[:REL_TYPE*1..]-(end:Label {flag: true}))
RETURN a, end
UNION
MATCH (a)<-[:REL_TYPE*0..]-(end:Label)
WHERE NOT (end)<-[:REL_TYPE]-()
RETURN a, end
Let's break it down:
To express that we only want to traverse until the first flag is true, we use shortestPath.
To express that we want to traverse down to the leaf, we use the following formalisation: a node is a leaf if it has no relationships that could be continued, captured by a WHERE NOT filter on patterns.
This should give an idea of the basic ideas to use for such queries -- please provide some feedback so that I can refine the answer.

Cypher - How to query multiple Neo4j Node property fragments with "STARTS WITH"

I'm looking for a way to combine the Cypher "IN" and "STARTS WITH" query. In other words I'm looking for a way to look up nodes that start with specific string sequences that are provided as Array using IN.
The goal is to have the query run in as less as possible calls against the DB.
I browsed the documentation and played around with Neo4j a bit but wasn't able to combine the following two queries into one:
MATCH (a:Node_type_A)-[]->(b:Node_type_B)
WHERE a.prop_A IN [...Array of Strings]
RETURN a.prop_A, COLLECT ({result_b: b.prop_B})
and
MATCH (a:Node_type_A)-[]->(b:Node_type_B)
WHERE a.prop_A STARTS WITH 'String'
RETURN a.prop_A, b.prop_B
Is there a way to combine these two approaches?
Any help is greatly appreciated.
Krid
You'll want to make sure there is an index or unique constraint (whichever is appropriate) on your :Node_type_A(prop_A) to speed up your lookups.
If I'm reading your requirements right, this query may work for you, adding your input strings as appropriate (parameterize them if you can).
WITH [...] as inputs
UNWIND inputs as input
// each string in inputs is now on its own row
MATCH (a:Node_type_A)
WHERE a.prop_A STARTS WITH input
// should be an fast index lookup for each input string
WITH a
MATCH (a)-[]->(b:Node_type_B)
RETURN a.prop_A, COLLECT ({result_b: b.prop_B})
Something like this should work:
MATCH (a:Node_type_A)-[]->(b:Node_type_B)
WITH a.prop_A AS pa, b.prop_B AS pb
WITH pa, pb,
REDUCE(s = [], x IN ['a','b','c'] |
CASE WHEN pa STARTS WITH x THEN s + pb ELSE s END) AS pbs
RETURN pa, pbs;

How to optimize the calculation of the shortest time in a certain path in Neo4j?

I have a database in Neo4j of modules that I imported through CSV. The data looks something like this. Each module has its name, it's module that is the successor, average time duration and another duration called medtime.
I have been able to import the data and to set the relationships through a Cypher Query script that looks like this:
LOAD CSV WITH HEADERS FROM "file:c:/users/Skelo/Desktop/Neo4J related/Statistic Dependencies/Simple.csv" AS row FIELDTERMINATOR ';'
CREATE (n:Module)
SET n = row, n.name = row.name, n.mafter = row.mafter, n.avgtime = row.avgtime, n.medtime = row.medtime
WITH n
RETURN n
Then I have set the relationships like this:
Match (p:Module),(q:Module)
Where p.mafter = q.name
Merge (p)-[:PRECEEDS]->(q)
Return p,q
Now to the point. I want to calculate the shortest path from a certain module to another, more specifically the time that it takes to get from a module to another and for this, I use the more or less copied part of the script from
http://www.neo4j.org/graphgist?8412907 and that is
MATCH p = (trop:Module {name:'BLSACXAMT0A_00'})-[prec:PRECEEDS*]->(hop:Module {name:'BL_LOAD_CLOSE'})
WITH p, REDUCE(x = 0, a IN NODES(p) | x + a.avgtime) AS cum_duration
ORDER BY cum_duration DESC
LIMIT 1
RETURN cum_duration AS `Total Average Time`
This, however, takes about 50 second to execute and that is outrageous. You can see it on the screenshot right below. The ammount of modules imported into the database is only about 2000 and what I want to achieve, is to successfully work with more than 50 000 nodes and perform such tasks much faster.
Other issue is, that the results are somehow suspicious. The format looks wrong, every number I have in the database has max 4 digits after the decimal point and I am only adding these values to zero, therefore if the result looks like this: 00103,68330,51670, I have serious doubts. Please, help me, if it is wrong, why is it so, and what can I do to correct it.
Neo4j claims that it is efficient and fast, therefore I presume that the fault is in my code (the performance of my computer is more than enough). Please, If you can, help me to shorten this time and explain the patterns needed to perform this.
A few observations that should help:
You have several errors in how you are importing. These errors will create many more nodes than you think, and create the "suspicious" issue you raised:
Your file has multiple rows with the same name, but your import is creating a new Module node every time. Therefore, you are ending up with multiple nodes for some of your modules. You should be using MERGE instead of CREATE.
Your mafter property needs to contain a collection of strings, not a single string.
You are importing the numeric values as strings, so code such as x + a.avgtime is just doing string concatenation, not numeric addition. Furthermore, even if you did attempt to convert your strings to numbers, that would fail because your numbers use a comma instead of a period to indicate the decimal place.
Try this for importing (into an empty DB):
LOAD CSV WITH HEADERS FROM "file:c:/users/Skelo/Desktop/Neo4J related/Statistic Dependencies/Simple.csv" AS row FIELDTERMINATOR ';'
MERGE (n:Module {name: row.name})
ON CREATE SET
n.mafter = [row.mafter],
n.avgtime = TOFLOAT(REPLACE(row.avgtime, ',', '.')),
n.medtime = TOFLOAT(REPLACE(row.medtime, ',', '.'))
ON MATCH SET
n.mafter = n.mafter + row.mafter;
You also need to change your current merge query so that you can handle an mafter that is a collection. Note that the following query is designed to NOT create any new nodes (even if a name in mafter does not yet have a module node).
MATCH (p:Module)
OPTIONAL MATCH (p)-[:PRECEEDS]->(z:Module)
WITH p, COLLECT(z.name) AS existing
WITH p, filter(x IN p.mafter
WHERE NOT x IN existing) AS todo
MATCH (q:Module)
WHERE q.name IN todo
MERGE (p)-[:PRECEEDS]->(q)
RETURN p, q;
You should create an index to speed up the matching of modules by name:
CREATE INDEX ON :Module(name)
Cypher does have a shortestPath function, see http://neo4j.com/docs/stable/query-match.html#_shortest_path. However this calculates the shortest path based on the number of hops and does not take a weight into account.
Neo4j has couple of graph algorithms on board, e.g. Dijekstra or AStar. Unfortunately these are not yet available via cypher. Instead you have two alternatives to use them:
1) write an unmanaged extension to Neo4j and use GraphAlgoFactory in the implmentation. This requires to write same java code and deploy it to the Neo4j server. Using a custom CostEvaluator you can use the avgTime property on your nodes as cost parameter.
2) use the REST API as documented on http://neo4j.com/docs/stable/rest-api-graph-algos.html#rest-api-execute-a-dijkstra-algorithm-and-get-a-single-path. This approach requires to have the weight as a property on the relationship and not on a node (like in your data model)

Any length relationship between two nodes that share the same label

I'm trying to find a query that will show me any length relationship that exists between two nodes that share the same index. Basically, if there is any overlap between for a specific label. My graph is Pretty simple and not particularly large:
(m:`Campaign`), (n:`Politician`), (o:`Assistant`), (p:`Staff`), (q:`Aid`), (s:`Contributor`)
(m)<-[:Campaigns_for]-(n)
(o)<-[:works_for]-(m)
(p)<-[:works_for]-(o)
(q)<-[:volunteers_for]-(p)
(m)<-[:contributes_to]-(s)
I want to find all the shared nodes and their relationships between Campaigns.
so far i have:
MATCH (n:`Campaign`)-[r*]-(m:`Campaign`)
RETURN n,count(r) as R,m
ORDER BY R DESC
but it's not returning everyhing I want, I want in addition to the counts, the labels of each relationship and the names of the nodes in between.
Assuming that "names of the nodes" means "return the name property of the node" (you could always substitute in "labels(n)" if you're after labels), then something like this might work, but you have some aggregation going on here so you may need to parse a bit:
MATCH p =(a:Campaign)-[r*]-(b:Campaign)
RETURN a, length(relationships(p)) AS count, b, extract(x IN relationships(p)| type(x)), extract(x IN nodes(p)| x.name)
ORDER BY count DESC
I'm also assuming that when you say "not returning everything you want", you mean that in addition to what's currently returned in your result set, you want just those other items you listed.
Keep in mind it might also be possible to have a cycle in your graph (not knowing too much about your particular graph), so, you may want to check your beginning and ending nodes.

Resources