Slow performing cypher query that creates nodes to group existing nodes by property values - neo4j

Question

I have a performance issue with a modifying cypher query. Given is an origin node that has a huge amount of outgoing relationships to child nodes. These child nodes all have a key property. Now the goal is to create new nodes between the origin and the child nodes to group all child nodes which share the same key properties value. A plot of that idea can be found at the neo4j console: http://console.neo4j.org/?id=vinntj
I use the query together with spring-data-neo4j 2.2.2.RELEASE and neo4j 1.9.2 embedded. The parameter for that query must be a node id and the result of that query should be the modified root node.
The query currently looks like (a bit more complex than in the linked neo4j console):
START root=node({0})
MATCH (root)-[r:LEAF]->(child)
SET root.__type__='my.GroupedRoot'
DELETE r
WITH child.`custom-GROUP` AS groupingKey, root AS origin, child AS leaf
CREATE UNIQUE (origin)-[:GROUP]->(group{__type__:'my.Group',key:'GROUP',value:groupingKey,origin:ID(origin)})-[:LEAF]->(leaf)
RETURN DISTINCT origin
The property custom-GROUP is the key to group by. In SDN it is represented by a DynamicProperties object. I annotated it to be indexed as well as the groupingKey and origin property of the created group node.
With 5000 child nodes it takes ~50sec to group them. For 10000 nodes ~90sec. For 20000 nodes ~380s and for 30000 nodes > 50min! This looks like an o(log n) scale to me. But my goal is an o(n) scale and to get 500000+ child nodes processed below 30min. I assume that the CREATE UNIQUE part of that query causes that problem because for new group nodes it always need to check what kind of group nodes have already been created. And the amount to check grows with the amount of already grouped child nodes.
Does someone have an idea about how to get this query faster? Or to do the same thing faster with an other query?

Answer 1

If the CREATE UNIQUE is indeed the problem, then this will first create the groups, then map to them.
START root=node(*)
MATCH (root)-[r:LEAF]->(child)
WHERE HAS (root.key) AND root.key='root'
WITH DISTINCT child.key AS groupingKey, root as origin
CREATE UNIQUE (origin)-[:GROUP]->(intermediate { key:groupingKey,origin:ID(origin)})
WITH groupingKey, origin, intermediate
MATCH (origin)-[r:LEAF]->(leaf)
WHERE leaf.key = groupingKey
DELETE r
CREATE (intermediate)-[:LEAF]->(leaf)
RETURN DISTINCT origin
The console is not letting me view the execution plan for either of our queries for some reason so I don't know for sure if it will help.
You might also consider indexing the roots so that you aren't having to do a "WHERE" on all of the nodes. You could just check an index for key=root.
EDIT An alternative to the above query is as follows which will prevent having to match the leaf nodes a second time by using a collect.
START root=node(*)
MATCH (root)-[r:LEAF]->(child)
WHERE HAS (root.key) AND root.key='root'
DELETE r
WITH DISTINCT child.key AS groupingKey, root as origin, COLLECT(child) as children
CREATE UNIQUE (origin)-[:GROUP]->(intermediate { key:groupingKey,origin:ID(origin)})
WITH groupingKey, origin, intermediate, children
FOREACH(leaf IN children : CREATE (intermediate)-[:LEAF]->(leaf))
RETURN DISTINCT origin

Answer 2

Well, now I turned to not use this kind of cypher queries on such a big amount of data. I implemented the same functionality using the traversal API for extracting the groupable items and the Neo4jTemplate to create the new nodes and relationships. Now 50000 items can be grouped in 5474ms instead of ~1h with the previously used cypher query. This is a very big improvement.