How To Generate k-ary trees in Neo4j quickly? - neo4j

Question

I'm generating perfect k-ary trees in neo4j but my queries for doing so don't seem very efficient I was wondering if I could improve on them in anyway, my go code below shows all three queries Im running to generate the trees, k is number of children per node, h is tree height:
func createPerfectKaryTreeInNeo(k, h int, execNeo func(string) error) error {
lastNode := ((iPow(k, (h + 1)) - 1) / (k - 1)) - 1
err := execNeo(fmt.Sprintf("FOREACH(i IN RANGE(0, %d, 1) | CREATE (:NODE {id:i, value:i}))", lastNode))
if err != nil {
return err
}
err = execNeo(fmt.Sprintf("MATCH (a:NODE), (b:NODE) WHERE b.id = a.id * %d + 1 CREATE (a)-[:FIRST_CHILD]->(b)", k))
if err != nil {
return err
}
err = execNeo(fmt.Sprintf("MATCH (a:NODE), (b:NODE) WHERE b.id = a.id + 1 AND a.id %% %d <> 0 CREATE (a)-[:NEXT_SIBLING]->(b)", k))
if err != nil {
return err
}
return nil
}
I think this is slow for h > 9 because of the last 2 queries, the MATCH on the 2 unconnected nodes, when I run this in the neo4j web client it warns about:
This query builds a cartesian product between disconnected patterns.
If a part of a query contains multiple disconnected patterns, this
will build a cartesian product between all those parts. This may
produce a large amount of data and slow down query processing. While
occasionally intended, it may often be possible to reformulate the
query that avoids the use of this cross product, perhaps by adding a
relationship between the different parts or by using OPTIONAL MATCH
(identifier is: (b))
Is there a way I can reformulate these queries to be more efficient?
EDIT:
The code is here if you wish to run it: https://github.com/robsix/data_model_perf_test

Answer 1

Graphs are designed to quickly identify a single point and then traverse from there. Your query structure (write all the nodes, then sort them and add relationships) does pretty much the opposite, which is why you're getting all those warnings. Unfortunately, to provide variable children per node, you will need to be able to query the id property quickly, so make sure that you have an index on :Node(id) and then try a single big query like this:
WITH 3 AS k, 2 AS h
WITH k, REDUCE(s = toFloat(0), x IN RANGE(1, h-1)|s + k^x) AS max_parent_id
UNWIND RANGE(0, toInt(max_parent_id)) AS parent_id
WITH k, parent_id, k*parent_id+1 AS first_child_id
MERGE (parent:NODE {id:parent_id, value:parent_id})
MERGE (child:NODE {id: first_child_id, value:first_child_id})
MERGE (parent) - [:FIRST_CHILD] -> (child)
WITH k, first_child_id
UNWIND RANGE(first_child_id + 1, first_child_id + k - 1) AS next_child_id
MERGE (last_child:NODE {id:next_child_id -1, value:next_child_id -1})
MERGE (next_child:NODE {id:next_child_id, value:next_child_id})
MERGE (last_child) - [:NEXT_SIBLING] -> (next_child)
This will run through all the possible parent ids, and for each one, will MERGE (match or create) a node with the right ID. It will then MERGE the first child node, whose ID you can already calculate, along with the FIRST_CHILD relationship. This will avoid your cartesian problem. The query will then go through the ids of each possible sibling to the first one, MATCH the existing sibling, and MERGE the next sibling along with the relationship.
UPDATE: I am so sorry, I totally overlooked the node visualization when testing it. I've updated the query since, to solve an index error and account for some reordering that I didn't know Cypher did. You learn something every day! But yeah, what's up there now generates the right graph.

Answer 2

The best I can come up with is still to use three queries but the are used in an interesting way to create the K-ary tree without making neo4j do too many searches:
func createPerfectKaryTreeInNeo(k, h int, execNeo func(string) error) error {
lastNode := ((iPow(k, (h+1)) - 1) / (k - 1)) - 1
if lastNode % 2 != 0 {
err := execNeo(fmt.Sprintf("UNWIND RANGE(0, %d, 2) AS id CREATE (a:NODE {id:id, value: id})-[:NEXT_SIBLING]->(b:NODE {id: id+1, value: id+1}) WITH a, b MATCH (c:NODE {id: b.id+1}) CREATE (b)-[:NEXT_SIBLING]->(c)", lastNode - 1))
if err != nil {
return err
}
} else {
err := execNeo(fmt.Sprintf("UNWIND RANGE(1, %d, 2) AS id CREATE (a:NODE {id:id, value: id})-[:NEXT_SIBLING]->(b:NODE {id: id+1, value: id+1}) WITH a, b MATCH (c:NODE {id: b.id+1}) CREATE (b)-[:NEXT_SIBLING]->(c)", lastNode))
if err != nil {
return err
}
err = execNeo("MATCH (a:NODE {id:1}) CREATE (:NODE {id:0, value:0})-[:NEXT_SIBLING]->(a)")
}
lastParentNode := (lastNode - 1) / k
err := execNeo(fmt.Sprintf("UNWIND RANGE(0, %d, 1) AS id MATCH shortestPath((a:NODE {id:id})-[:NEXT_SIBLING *]->(b:NODE {id:id*%d+1})) CREATE (a)-[:FIRST_CHILD]->(b)", lastParentNode, k))
if err != nil {
return err
}
err = execNeo(fmt.Sprintf("MATCH (a:NODE)-[r:NEXT_SIBLING]->(b:NODE) WHERE a.id %% %d = 0 DELETE r", k))
if err != nil {
return err
}
return nil
}
I should note that this algorithm is specifically for perfect k-ary trees with node id's allocated in breadth first order, the way it works is:
1) generate all the nodes in pairs and assign them all in order as being NEXT_SIBLINGS of one another i.e. 0->1->2->3->4 so you end up with a straight graph.
2) loop through all the ids small enough to have children and match using the shortestPath function in the hope that neo4j is smart enough to work out that given the current shape of the graph as soon as it finds a match, that is the shortest possible path and so return early without continuing to search further.
3) the last query then grabs adjacent nodes that should not be considered NEXT_SIBLINGS and deletes the relationship leaving behind a perfect k-ary tree with depth h.
The changes have sped up the data creation by at least an order of magnitude.
UPDATE:
The accepted answer above is correct, this is just the go code that matches it:
func createPerfectKaryTreeInNeo(k, h int, execNeo func(string) error) error {
return execNeo(fmt.Sprintf(`
WITH %d AS k, %d AS h
WITH k AS k, REDUCE(s = toFloat(0), x IN RANGE(1, h-1)|s + k^x) AS max_parent_id
UNWIND RANGE(0, toInt(max_parent_id)) AS parent_id
WITH k AS k, parent_id, k*parent_id+1 AS first_child_id
MERGE (parent:NODE {id:parent_id, value:parent_id})
MERGE (child:NODE {id: first_child_id, value:first_child_id})
MERGE (parent) - [:FIRST_CHILD] -> (child)
WITH k AS k, first_child_id
UNWIND RANGE(first_child_id + 1, first_child_id + k - 1) AS next_child_id
MERGE (last_child:NODE {id:next_child_id -1, value:next_child_id -1})
MERGE (next_child:NODE {id:next_child_id, value:next_child_id})
MERGE (last_child) - [:NEXT_SIBLING] -> (next_child)
`, k, h))
}
It is several order of magnitude faster than my improvements I originally described in this answer