Neo4j indexing for large number of nodes - neo4j

I am learning the basics of neo4j and I am looking at the following example with credit card fraud https://linkurio.us/stolen-credit-cards-and-fraud-detection-with-neo4j. Cypher query that finds stores where all compromised user shopped is
MATCH (victim:person)-[r:HAS_BOUGHT_AT]->(merchant)
WHERE r.status = “Disputed”
MATCH victim-[t:HAS_BOUGHT_AT]->(othermerchants)
WHERE t.status = “Undisputed” AND t.time < r.time
WITH victim, othermerchants, t ORDER BY t.time DESC
RETURN DISTINCT othermerchants.name as suspicious_store, count(DISTINCT t) as count, collect(DISTINCT victim.name) as victims
ORDER BY count DESC
However, when the number of users increase (let's say to millions of users), this query may become slow since the initial query will have to traverse through all nodes labeled person. Is it possible to speed up the query by asigning properties to nodes instead of transactions? I tried to remove "status" property from relationships and add it to nodes (users, not merchants). However, when I run query with constraint WHERE victim.status="Disputed" query doesn't return anything. So, in my case person has one additional property 'status'. I assume I did a lot of things wrong, but would appreciate comments. For example
MATCH (victim:person)-[r:HAS_BOUGHT_AT]->(merchant)
WHERE victim.status = “Disputed”
returns the correct number of disputed transactions. The same holds for separately quering number of undisputed transactions. However, when merged, they yield an empty set.
If I made a mistake in my approach, how can I speed up queries for large number of nodes (avoid traversing all nodes in the first step). I will be working with a data set with similar properties, but will have around 100 million users, so I would like to index users on additional properties.

[Edited]
Moving the status property from the relationship to the person node does not seem to be the right approach, since I presume the same person can be a customer of multiple merchants.
Instead, you can reify the relationship as a node (let's label it purchase), as in:
(:person)-[:HAS_PURCHASE]->(:purchase)-[:BOUGHT_AT]->(merchant)
The purchase nodes can have the status property. You just have to create the index:
CREATE INDEX ON :purchase(status)
Also, you can put the time property in the new purchase nodes.
With the above, your query would become:
MATCH (victim:person)-[:HAS_PURCHASE]->(pd:purchase)-[:BOUGHT_AT]->(merchant)
WHERE pd.status = “Disputed”
MATCH victim-[:HAS_PURCHASE]->(pu:purchase)-[:BOUGHT_AT]->(othermerchants)
WHERE pu.status = “Undisputed” AND pu.time < pd.time
WITH victim, othermerchants, pu ORDER BY pu.time DESC
RETURN DISTINCT othermerchants.name as suspicious_store, count(DISTINCT pu) as count, collect(DISTINCT victim.name) as victims
ORDER BY count DES

Related

"Query Optimization" : Neo4j Query builds a cartesian product between disconnected patterns -

I’m supposed to have graph of multiple nodes(more than 2) with their relationships at 1st degree, second degree, third degree.
For that right now I am using this query
WITH ["1258311979208519680","3294971891","1176078684270333952",”117607868427845”] as ids
MATCH (n1:Target),(n2:Target) WHERE n1.id in ids and n2.id in ids and n1.id<>n2.id and n1.uid=103 and n2.uid=103
MATCH p = ((n1)-[*..3]-(n2)) RETURN p limit 30
In which 4 nodes Id’s are mention in WITH[ ] and next [*..3] it is used to draw 3rd degree graph between the selected nodes.
WHAT the ABOVE QUERY DOING
After running the above query it will return the mutual nodes in case of second degree [*..2] if any of the 2 selected nodes have mutual relation it’ll return.
WHAT I WANT
*1) First of all I want to optimize the query, as it is taking so much time and this query causing the Cartesian product which slow down the query process.
2) As in this above query if any 2 nodes have mutual relationship it will return the data, I WANT, the query will return mutual nodes attached with all selected nodes. Means if we have some nodes in return, these nodes must have relation to all selected target nodes.
Any suggestions to modify the query, to optimize the query.
If you are looking for to avoid the cartesian product issue with the given query
WITH ["1258311979208519680","3294971891","1176078684270333952",”117607868427845”] as ids
MATCH (n1:Target),(n2:Target) WHERE n1.id in ids and n2.id in ids and n1.id<>n2.id and n1.uid=103 and n2.uid=103
MATCH p = ((n1)-[*..3]-(n2)) RETURN p limit 30
I suggest to use this one below
MATCH (node1:Target) WHERE node1.id IN ["1258311979208519680","3294971891","1176078684270333952"]
MATCH (node2:Target) WHERE node2.id IN ["1258311979208519680","3294971891","1176078684270333952"]
and node1.id <> node2.id
MATCH p=(node1)-[*..2]-(node2)
RETURN p
It will remove the cartesian product issue.
Try this..

Cypher pattern for getting self related nodes

Given that I'm very new to Neo4j. I have a schema which looks like the below image:
Here Has nodes are different for example Passport, Merchant, Driving License, etc. and also these nodes are describing the customer node (looking for future scope of filtering customers based on these nodes).
SIMILAR is a self-relation meaning there exists a customer with ID:1 is related to another customer with ID:2 with a score of 2800.
I have the following questions:
Is this a good schema given the condition of the future scope I mentioned above, or getting all the properties in a single customer node is viable? (Different nodes may have array of items as well, for example: ()-[:HAS]->(Phone) having {active: "+91-1231241", historic_phone_numbers: ["+91-121213", "+91-1231421"]})
I want to get the customer along with describing nodes in relation to other customers. For that, I tried the below query (w/o number of relation more than 1):
// With number_of_relation > 1
MATCH (searched:Customer)-[r:SIMILAR]->(matched:Customer)
WHERE r.score > 2700
WITH searched, COLLECT(matched.customer_id) AS MatchedList, count(r) as cnt
WHERE cnt > 1
UNWIND MatchedList AS matchedCustomer
MATCH (person:Customer {customer_id: matchedCustomer})-[:HAS|:LIVES_IN|:IS_EMPLOYED_BY]->(related)
RETURN searched, person, related
Result what I got is below, notice one customer node not having its describing nodes:
// without number_of_relation > 1
// second attempt - for a sample customer_id
MATCH (matched)<-[r:SIMILAR]-(c)-[:HAS|:LIVES_IN|:IS_EMPLOYED_BY]->(b)
WHERE size(keys(b)) > 0
AND c.customer_id = "1b093559-a39b-4f95-889b-a215cac698dc"
AND r.score > 2700
RETURN b AS props, c AS src_cust, r AS relation, matched
Result I got are below, notice related nodes are not having their describing nodes:
If I had two describing nodes with some property (some may have a list) upon which I wanted to query and build the expected graph specified in point 2 above, how can I do that?
I want the database to find a similar customer given the describing nodes. Example: A customer {name: "Dave"} has phone {active_number: "+91-12345"} is similar to customer {name: "Mike"} has phone {active_number: "+91-12345"}. How can get started with this?
If something is unclear, please ask. I can explain with examples.
[EDITED]
Yes, the schema seems fine, except that you should not use the same HAS relationship type between different node label pairs.
The main problem with your first query is that its top MATCH clause uses a directional relationship pattern, ()-->(), which does not allow all Customer nodes to have a chance to be the searched node (because some nodes may only be at the tail end of SIMILAR relationships). This tweaked query should work better:
MATCH (searched:Customer)-[r:SIMILAR]-(matched:Customer)
WHERE r.score > 2700
WITH searched, COLLECT(matched) AS matchedList
WHERE SIZE(matchedList) > 1
UNWIND matchedList AS person
MATCH (person)-[:HAS|LIVES_IN|IS_EMPLOYED_BY]->(pDesc)
WITH searched, person, COLLECT(pDesc) AS personDescribers
MATCH (searched)-[:HAS|LIVES_IN|IS_EMPLOYED_BY]->(sDesc)
RETURN searched, person, personDescribers, COLLECT(sDesc) AS searchedDescribers
It's not clear what you want are trying to do.
To get all Customers who have the same phone number:
MATCH (c:Customer)-[:HAS_PHONE]-(p:Phone)
WHERE p.activeNumber = '+91-12345'
WITH p.activeNumber AS phoneNumber, COLLECT(c) AS customers
WHERE SIZE(customers) > 1
RETURN phoneNumber, customers

Obtain pairs of nodes having exactly one relationship of a certain type which connects them to each other in Cypher

I have a graph database in Neo4j with drugs and drug-drug interactions, among other entities. In this regard, ()-[:IS_PARTICIPANT_IN]->() connects a drug to an interaction. I need to obtain those pairs of drugs a and b which are not involved in any other :IS_PARTICIPANT_IN relationship other than the one between them, i.e. (a)-[:IS_PARTICIPANT_IN]->(ddi:DrugDrugInteraction)<-[:IS_PARTICIPANT_IN]-(b), without any other IS_PARTICIPANT_IN relationships involving neither a nor b.
For that purpose, I have tried the following Cypher query. However, it ends up reaching heap size (raised to 8 GB), as collect operations consume too much memory.
MATCH (drug1:Drug)-[r1:IS_PARTICIPANT_IN]->(ddi:DrugDrugInteraction)
MATCH (drug2:Drug)-[r2:IS_PARTICIPANT_IN]->(ddi)
WHERE drug1 <> drug2
OPTIONAL MATCH (drug2)-[r3:IS_PARTICIPANT_IN]->(furtherDDI:DrugDrugInteraction)
WHERE furtherDDI <> ddi
WITH drug1, drug2, ddi, COLLECT(ddi) AS ddis, furtherDDI, COLLECT(furtherDDI) AS additionalDDIs
WITH drug1, drug2, ddi, COUNT(ddis) AS n1, COUNT(additionalDDIs) AS n2
WHERE n1 = 1 AND n2 = 0
RETURN drug1.name, drug2.name, ddi.name ORDER BY drug1;
How can I improve my code so as to get the desired results without exceeding the heap size limit?
This should work:
MATCH (d:Drug)
WHERE SIZE((d)-[:IS_PARTICIPANT_IN]->()) = 1
MATCH (d)-[:IS_PARTICIPANT_IN]->(ddi)
RETURN ddi.name AS ddiName, COLLECT(d.name) AS drugNames
ORDER BY drugNames[0]
The WHERE clause uses a very efficient degreeness check to filter for Drug nodes that have only a single outgoing IS_PARTICIPANT_IN relationship. This check is efficient because it does not have to actually get any DrugDrugInteraction nodes.
After the degreeness check, the query performs a second MATCH to actually get the associated DrugDrugInteraction node. (I assume that the IS_PARTICIPANT_IN relationship only points at DrugDrugInteraction nodes, and have therefore omitted the label from the search pattern, for efficiency).
The RETURN clause uses the aggregating function COLLECT to collect the Drug names for each ddi name. (I assume that ddi nodes have unique names.)
By the way, this query will also work if there are any number of Drugs (not just 2) that participate in the same DrugDrugInteraction, and no other ones. Also, if a matched DrugDrugInteraction happens to have a related Drug that participates in other interactions, this query will not include that Drug in the result (since this query only pays attention to d nodes that passed the initial degreeness check).

find all items that wasn't bought by a person and count the times it was bought

I have a graph that looks like this.
I want to find all the items bought by the people, who bought the same items as Gremlin using cypher.
Basically I want to imitate the query in the gremlin examples that looks like this
g.V().has("name","gremlin")
.out("bought").aggregate("stash")
.in("bought").out("bought")
.where(not(within("stash")))
.groupCount()
.order(local).by(values,desc)
I was trying to do it like this
MATCH (n)-[:BOUGHT]->(g_item)<-[:BOUGHT]-(r),
(r)-[:BOUGHT]->(n_item)
WHERE
n.name = 'Gremlin'
AND NOT (n)-[:BOUGHT]->(n_item)
RETURN n_item.id, count(*) as frequency
ORDER by frequency DESC
but it seems it doesn't count frequencies properly - they seem to be twice as big.
4 - 4
5 - 2
3 - 2
While 3 and 5 was bought only once and 4 was bought 2 times.
What's the problem?
Cypher is interested in paths, and your MATCH finds the following:
2 paths to item 3 both through Rexter (via items 2 and 1)
2 paths to item 5 through Pipes (via items 1 and 2)
4 paths to item 4 via Rexter and Pipes (via items 1 and 2 for each person)
Basically the items are being counted multiple times because there are multiple paths to that same item per individual person via different common items with Gremlin.
To get accurate counts, you either need to match to distinct r users, and only then match out to items the r users bought (as long as they aren't in the collection of items bought by Gremlin), OR you need to do the entire match, but before doing the counts, get distinct items with respect to each person so each item per person only occurs once...then get the count per item (counts across all persons).
Here's a query that uses the second approach
MATCH (n:Person)-[:BOUGHT]->(g_item)
WHERE n.name = 'Gremlin'
WITH n, collect(g_item) as excluded
UNWIND excluded as g_item // now you have excluded list to use later
MATCH (g_item)<-[:BOUGHT]-(r)-[:BOUGHT]->(n_item)
WHERE r <> n AND NOT n_item in excluded
WITH DISTINCT r, n_item
WITH n_item, count(*) as frequency
RETURN n_item.id, frequency
ORDER by frequency DESC
You should be using labels in your graph, and you should use them in your query in order to leverage indexes and quickly find a starting point in the graph. In your case, an index on :Person(name), and usage of the :Person label in the query, should make this quick even as more nodes and more :Persons are added to the graph.
EDIT
If you're just looking for conciseness of the query, and don't have a large enough graph where performance will be an issue, then you can use your original query but add one extra line to get distinct rows of r and n_item before you count the item. This ensures that you only count an item per person once when you get the count.
Note that forgoes optimizations for handling excluded items (it will do a pattern match per item rather than aggregating the collection of bought items and doing a collection membership check), and it aggregates on items while doing property access rather than doing property access only after aggregating by the node.
MATCH (n:Person)-[:BOUGHT*2]-(r)-[:BOUGHT]->(n_item)
WHERE n.name = 'Gremlin'
WITH DISTINCT n, r, n_item
WHERE NOT (n)-[:BOUGHT]->(n_item)
RETURN n_item.id, count(*) as frequency
ORDER by frequency DESC
I am adding a quick shortcut in your match, using :BOUGHT*2 to indicate two :BOUGHT hops to r, since we don't really care about the item in-between.

Neo4j relate nodes by same property

I have a Neo4J DB up and running with currently 2 Labels: Company and Person.
Each Company Node has a Property called old_id.
Each Person Node has a Property called company.
Now I want to establish a relation between each Company and each Person where old_id and company share the same value.
Already tried suggestions from: Find Nodes with the same properties in Neo4J and
Find Nodes with the same properties in Neo4J
following the first link I tried:
MATCH (p:Person)
MATCH (c:Company) WHERE p.company = c.old_id
CREATE (p)-[:BELONGS_TO]->(c)
resulting in no change at all and as suggested by the second link I tried:
START
p=node(*), c=node(*)
WHERE
HAS(p.company) AND HAS(c.old_id) AND p.company = c.old_id
CREATE (p)-[:BELONGS_TO]->(c)
RETURN p, c;
resulting in a runtime >36 hours. Now I had to abort the command without knowing if it would eventually have worked. Therefor I'd like to ask if its theoretically correct and I'm just impatient (the dataset is quite big tbh). Or if theres a more efficient way in doing it.
This simple console shows that your original query works as expected, assuming:
Your stated data model is correct
Your data actually has Person and Company nodes with matching company and old_id values, respectively.
Note that, in order to match, the values must be of the same type (e.g., both are strings, or both are integers, etc.).
So, check that #1 and #2 are true.
Depending on the size of your dataset you want to page it
create constraint on (c:Company) assert c.old_id is unique;
MATCH (p:Person)
WITH p SKIP 100000 LIMIT 100000
MATCH (c:Company) WHERE p.company = c.old_id
CREATE (p)-[:BELONGS_TO]->(c)
RETURN count(*);
Just increase the skip value from zero to your total number of people in 100k steps.

Resources