Find most frequently used distinct set of terms - neo4j

Question

Imagine a graph database composed of URLs and tags used to describe them. From this we want to find which sets of tags are most frequently used together and determine which URLs belong in each identified set.
I've tried to create a dataset which simplifies this problem as such in cypher:
CREATE (tech:Tag { name: "tech" }), (comp:Tag { name: "computers" }), (programming:Tag { name: "programming" }), (cat:Tag { name: "cats" }), (mice:Tag { name: "mice" }), (u1:Url { name: "http://u1.com" })-[:IS_ABOUT]->(tech), (u1)-[:IS_ABOUT]->(comp), (u1)-[:IS_ABOUT]->(mice), (u2:Url { name: "http://u2.com" })-[:IS_ABOUT]->(mice), (u2)-[:IS_ABOUT]->(cat), (u3:Url { name: "http://u3.com" })-[:IS_ABOUT]->(tech), (u3)-[:IS_ABOUT]->(programming), (u4:Url { name: "http://u4.com" })-[:IS_ABOUT]->(tech), (u4)-[:IS_ABOUT]->(mice), (u4)-[:IS_ABOUT]->(acc:Tag { name: "accessories" })
Using this as a reference (neo4j console example here), we can look at it and visually identify that the most commonly used tags are tech and mice (the query for this is trivial) both referencing 3 URLs. The most commonly used tag pair is [tech, mice] as it (in this example) is the only pair shared by 2 urls (u4, and u1). It's important to note that this tag pair is a subset of the matched URLs, it's not the entire set for either. There is no combination of 3 tags shared by any urls.
How can I write a cypher query to identify which tag combinations are the most frequently used together (either in pairs, or in N sized groups)? Perhaps there's a better way to structure this data which would make analysis easier? Or is this problem is not well suited for a Graph DB? Been struggling a bit trying to figure this one out, any help or thoughts would be appreciated!

Answer 1

It looks like the problem on combinatorics.
// The tags for each URL, sorted by ID
MATCH (U:Url)-[:IS_ABOUT]->(T:Tag)
WITH U, T ORDER BY id(T)
WITH U,
collect(distinct T) as TAGS
// Calc the number of combinations of tags for a node,
// independent of the order of tags
// Since the construction of the power in the cyper is not available,
// use the logarithm and exponent
//
WITH U, TAGS,
toInt(floor(exp(log(2) * size(TAGS)))) as numberOfCombinations
// Iterate through all combinations
UNWIND RANGE(0, numberOfCombinations) as combinationIndex
WITH U, TAGS, combinationIndex
// And check for each tag its presence in combination
// Bitwise operations are missing in the cypher,
// therefore, we use APOC
// https://neo4j-contrib.github.io/neo4j-apoc-procedures/#_bitwise_operations
//
UNWIND RANGE(0, size(TAGS)-1) as tagIndex
WITH U, TAGS, combinationIndex, tagIndex,
toInt(ceil(exp(log(2) * tagIndex))) as pw2
call apoc.bitwise.op(combinationIndex, "&", pw2) YIELD value
WITH U, TAGS, combinationIndex, tagIndex,
value WHERE value > 0
// Get all combinations of tags for URL
WITH U, TAGS, combinationIndex,
collect(TAGS[tagIndex]) as combination
// Return all the possible combinations of tags, sorted by frequency of use
RETURN combination, count(combination) as freq, collect(U) as urls
ORDER BY freq DESC
I think that it is best to calculate and store the tag combination with the use of this algorithm at the time of tagging. And the query will be something like this:
MATCH (Comb:TagsCombination)<-[:IS_ABOUT]-(U:Url)
WITH Comb, collect(U) as urls, count(U) as freq
MATCH (Comb)-[:CONTAIN]->(T:Tag)
RETURN Comb, collect(T) as Tags, urls, freq ORDER BY freq DESC

Answer 2

Start at the URL nodes, build a tuple of tag.name objects (order it first so they all group up the same). That'll give you all of the possible combinations of tags that exist. Then, use filters to find out how many url's match for each possible set of tags.
MATCH (u:url)
WITH u
MATCH (u) - [:IS_ABOUT] -> (t:tag)
WITH u, t
ORDER BY t.name
WITH u, [x IN COLLECT(t)|x.name] AS tags
WITH DISTINCT tags
MATCH (u)
WHERE ALL(tag IN tags WHERE (u) - [:IS_ABOUT] -> (tag))
RETURN tags, count(u)