I'm dealing with the following situation: many trips exist between many cities. Both have various properties. E.g the cities have a name and an amount of trips that passed them, whereas trips have a distance and time.
What is 'best practise' in Neo4j?
a) Add all cities and trips as nodes, and connect the trips to the start and end nodes by means of 'STARTED_AT' and 'ENDS_IN' relations.
or
b) Add only cities as a node, and represent each of the trips as a relation between 2 nodes. This means there are many of the same relations between nodes, where the only difference is that they have other properties.
Information that might be useful: we only need to do all kinds of queries. No insertion needed.
Thanks!
I would argue it's better to store trips as nodes because relationship properties cannot be indexed, and it will be slow to do more complex queries (like find shortest route by time) So if you are searching for trips by ID or something, you will need to store them as nodes.
On the other hand, an argument can be made for using relationships, because then you can take full advantage of APOC's weighted graph search functions.
A good way to decide if something should be a node or relation, is to ask yourself "are there any other relations that would make sense here?" If you are talking about if two cities are connected, a relationship makes more since because they either are or are not. If you are talking about road trips though, the trip can pass through multiple cities, can have participants in the trip (or groups there of) and can have an owner. In that case, for future flexibility, nodes will be much easier to maintain.
I would say it really depends on how you model these trips, lets assume we can generalize this as (city)-[trip]->(city). Notice that neo4j's relations always has a direction so we can go on adding an unlimited number of trips between cities without having to redefine each city for each trip -- this actually answers (a) by the way, we don't need to define where it starts and ends the relation does all that work for you.
'This means there are many of the same relations between nodes' <<- on this note, if you need to differ each trip based on the time the trip was taken you can add the date/timestamp in the relationship property or you can go with a time tree (see Mark Needham's Article on that here and Graphgrid's take)
Hope this helps.
Related
I am teaching myself graph modelling and use Neo4j 2.2.3 database with NodeJs and Express framework.
I have skimmed through the free neo4j graph database book and learned how to model a scenario, when to use relationship and when to create nodes, etc.
I have modelled a vehicle selling scenario, with following structure
NODES
(:VEHICLE{mileage:xxx, manufacture_year: xxxx, price: xxxx})
(:VFUEL_TYPE{type:xxxx}) x 2 (one for diesel and one for petrol)
(:VCOLOR{color:xxxx}) x 8 (red, green, blue, .... yellow)
(:VGEARBOX{type:xxx}) x 2 (AUTO, MANUAL)
RELATIONSHIPS
(vehicleNode)-[:VHAVE_COLOR]->(colorNode - either of the colors)
(vehicleNode)-[:VGEARBOX_IS]->(gearboxNode - either manual or auto)
(vehicleNode)-[:VCONSUMES_FUEL_TYPE]->(fuelNode - either diesel or petrol)
Assuming we have the above structure and so on for the rest of the features.
As shown in the above screenshot (136 & 137 are VEHICLE nodes), majority of the features of a vehicle is created as separate nodes and shared among vehicles with common feature with relationships.
Could you please advise whether roles (labels) like color, body type, driving side (left drive or right drive), gearbox and others should be seperate nodes or properties of vehicle node? Which option is more performance friendly, and easy to query?
I want to write a JS code that allows querying the graph with above structure with one or many search criteria. If majority of those features are properties of VEHICLE node then querying would not be difficult:
MATCH (v:VEHICLE) WHERE v.gearbox = "MANUAL" AND v.fuel_type = "PETROL" AND v.price > x AND v.price < y AND .... RETURN v;
However with existing graph model that I have it is tricky to search, specially when there are multiple criteria that are not necessarily a properties of VEHICLE node but separate nodes and linked via relationship.
Any ideas and advise in regards to existing structure of the graph to make it more query-able as well as performance friendly would be much appreciated. If we imagine a scenario with 1000 VEHICLE nodes that would generate 15000 relationship, sounds a bit scary and if it hits a million VEHICLE then at most 15 million relationships. Please comment if I am heading in the wrong direction.
Thank you for your time.
Modeling is full of tradeoffs, it looks like you have a decent start.
Don't be concerned at all with the number of relationships. That's what graph databases are good at, so I wouldn't be too concerned about over-using them.
Should something be a property, or a node? I can't answer for your scenario, but here are some things to consider:
If you look something up by a value all the time, and you have many objects, it's usually going to be faster to find one node and then everything connected to it, because graph DBs are good at exploiting relationships. It's less fast to scan all nodes of a label and find the items where a property=a value.
Relationships work well when you want to express a connection to something that isn't a simple primitive data type. For example, take "gearbox". There's manuals, and other types...if it's a property value, you won't later easily be able to decide to store 4 other sub-types/sub-aspects of "gearbox". If it were a node, that would later be easy because you could add more properties to the node, or relate other things.
If a piece of data really is a primitive (String, integer, etc) and you don't need extra detail about it, that usually makes a good property. Querying primitive values by connecting to other nodes will seem clunky later on. For example, I wouldn't model a person with a "date of birth" as a separate node, that would be irritating to query, and would give you flexibility you'd be very unlikely to need in the future.
Semantically, how is your data related? If two items are similar because they share an X, then that X probably should be a node. If two items happen to have the same Y value but that doesn't really mean much, then Y is probably better off as a node property.
Suppose I have a large knowledge base with many relationship types, e.g., hasChild, livesIn, locatedIn, capitalOf, largestCityOf...
The number of capicalOf relationships is relatively small (say, one hundred) compared to that of all nodes and other types of relationships.
I want to fetch any capital which is also the largest city in their country by the following query:
MATCH city-[:capitalOf]->country, city-[:largestCityOf]->country RETURN city
Apparently it would be wise to take the capitalOf type as clue, scan all 100 relationship with this type and refine by [:largestCityOf]. However the current execution plan engine of neo4j would do an AllNodesScan and Expand. Why not consider add an "RelationshipByTypeScan" operator into the current query optimization engine, like what NodeByLabelScan does?
I know that I can transform relationship types to relationship properties, index it using the legacy index and manually indicate
START r=relationship:rels(rtype = "capitalOf")
to tell neo4j how to make it efficient. But for a more complicated pattern query with many relationship types but no node id/label/property to start from, it is clearly a duty of the optimization engine to decide which relationship type to start with.
I saw many questions asking the same problem but getting answers like "negative... a query TYPICALLY starts from nodes... ". I just want to use the above typical scenario to ask why once more.
Thanks!
A relationship is local to its start and end node - there is no global relationship dictionary. An operation like "give me globally all relationships of type x" is therefore an expensive operation - you need to go through all nodes and collect matching relationships.
There are 2 ways to deal with this:
1) use a manual index on relationships as you've sketched
2) assign labels to your nodes. Assume all the country nodes have a Country label. Your can rewrite your query:
MATCH (city)-[:capitalOf]->(country:Country), (city)-[:largestCityOf]->(country) RETURN city
The AllNodesScan is now a NodeByLabelScan. The query grabs all countries and matches to the cities. Since every country does have one capital and one largest city this is efficient and scales independently of the rest of your graph.
If you put all relationships into one index and try to grab to ~100 capitalOf relationships that operation scales logarithmically with the total number of relationships in your graph.
I am trying to realize a datamodel in Neo4j. The model has points of interest in a city and streets. The streets connect the points.
Initially I thought that points and streets should both represented in the graph database as nodes.
Between these two different type of nodes there is a relationship ("point is connected with").
Now I am thinking the possibility that instead of representing the street as a node, perhaps is more correct to represent the street as relationship ("connects two points")
And this is my question actually. What is the more correct way to represent the network (line part) in a model: with nodes or with relationships?
The only major difference between relationships and nodes is that relationships must exist between two nodes. This means that you wouldn't be able to store a specific street if you didn't store two points of interest that it connects. So, if you see this being an issue, you may want to store streets as nodes. If you are certain that you will only want to store streets if there are points of interest in your database that exist on the street, then it'd make more sense to represent the streets as relationships.
In general, you should try to avoid storing properties in nodes that you only intend to use to find relationships between them. In this case, you mention possible storying a "point is connected with" property in each point of interest node. This would work, but is essentially just saying that a relationship exists between two points without actually using a relationship. Again, in the case where you want to be able to store streets that don't have points of interests existing on them, this may be necessary, and you could store streets that don't have points of interests on them by leaving the "point is connected with" property as NULL, but I would advise against this.
Another thing to think about is what you would store in the relationship. If you go with the model where streets are nodes, it becomes very difficult to represent quantities like distances between points of interest without adding relationships into your graph specifically for those properties, which may as well be properties of a street relationship.
UPDATE: Thought I'd add an example query to show how making the streets relationships can simplify your logic and make using your database much simpler and more intuitive.
Imagine you wish to find the path with the fewest points of interest between points A and B.
This is what the query would look like with the relationships model:
MATCH (a:Point {name: "foo"}), (b:Point {name: "bar"}),
p = shortestPath(a-[*:Street]-b)
RETURN p
By using relationships where appropriate, you enable the capabilities of Neo4j, allowing you to get a lot of work done with relatively simple queries. It's hard to think of a way to write this query in the model where you represent streets as nodes, but it would in all likelihood be much more complex and less efficient.
I can't seem to find any discussion on this. I had been imagining a database that was schemaless and node based and heirarchical, and one day I decided it was too common sense to not exist, so I started searching around and neo4j is about 95% of what I imagined.
What I didn't imagine was the concept of relationships. I don't understand why they are necessary. They seem to add a ton of complexity to all topics centered around graph databases, but I don't quite understand what the benefit is. Relationships seem to be almost exactly like nodes, except more limited.
To explain what I'm thinking, I was imagining starting a company, so I create myself as my first nodes:
create (u:User { u.name:"mindreader"});
create (c:Company { c.name:"mindreader Corp"});
One day I get a customer, so I put his company into my db.
create (c:Company { c.name:"Customer Company"});
create (u:User { u.name:"Customer Employee1" });
create (u:User { u.name:"Customer Employee2"});
I decide to link users to their customers
match (u:User) where u.name =~ "Customer.*"
match (c:Company) where c.name =~ "Customer.*
create (u)-[:Employee]->(c);
match (u:User where name = "mindreader"
match (c:Company) where name =~ "mindreader.*"
create (u)-[:Employee]->(c);
Then I hire some people:
match (c:Company) where c.name =~ "mindreader.*"
create (u:User { name:"Employee1"})-[:Employee]->(c)
create (u:User { name:"Employee2"})-[:Employee]->(c);
One day hr says they need to know when I hired employees. Okay:
match (c:Company)<-[r:Employee]-(u:User)
where name =~ "mindreader.*" and u.name =~ "Employee.*"
set r.hiredate = '2013-01-01';
Then hr comes back and says hey, we need to know which person in the company recruited a new employee so that they can get a cash reward for it.
Well now what I need is for a relationship to point to a user but that isn't allowed (:Hired_By relationship between :Employee relationship and a User). We could have an extra relationship :Hired_By, but if the :Employee relationship is ever deleted, the hired_by will remain unless someone remembers to delete it.
What I could have done in neo4j was just have a
(u:User)-[:hiring_info]->(hire_info:HiringInfo)-[:hired_by]->(u:User)
In which case the relationships only confer minimal information, the name.
What I originally envisioned was that there would be nodes, and then each property of a node could be a datatype or it could be a pointer to another node. In my case, a user record would end up looking like:
User {
name: "Employee1"
hiring_info: {
hire_date: "2013-01-01"
hired_by: u:User # -> would point to a user
}
}
Essentially it is still a graph. Nodes point to each other. The name of the relationship is just a field in the origin node. To query it you would just go
match (u:User) where ... return u.name, u.hiring_info.hiring_date, u.hiring_info.hired_by.name
If you needed a one to many relationship of the same type, you would just have a collection of pointers to nodes. If you referenced a collection in return, you'd get essentially a join. If you delete hiring_info, it would delete the pointer. References to other nodes would not have to be a disorganized list at the toplevel of a node. Furthermore when I query each user I will know all of the info about a user without both querying for the user itself and also all of its relationships. I would know his name and the fact that he hired someone in the same query. From the database backend, I'm not sure much would change.
I see quite a few questions from people asking whether they should use nodes or relationships to model this or that, and occasionally people asking for a relationship between relationships. It feels like the XML problem where you are wondering if a pieces of information should be its own tag or just a property its parent tag.
The query engine goes to great pains to handle relationships, so there must be some huge advantage to having them, but I can't quite see it.
Different databases are for different things. You seem to be looking for a noSQL database.
This is an extremely wide topic area that you've reached into, so I'll give you the short of it. There's a spectrum of database schemas, each of which have different use cases.
NoSQL aka Non-relational Databases:
Every object is a single document. You can have references to other documents, but any additional traversal means you're making another query. Times when you don't have relationships between your data very often, and are usually just going to want to query once and have a large amount of flexibly-stored data as the document that is returnedNote: These are not "nodes". Node have a very specific definition and implies that there are edges.)
SQL aka Relational Databases:
This is table land, this is where foreign keys and one-to-many relationships come into play. Here you have strict schemas and very fast queries. This is honestly what you should use for your user example. Small amounts of data where the relationships between things are shallow (You don't have to follow a relationship more than 1-2 times to get to the relevant entry) are where these excel.
Graph Database:
Use this when relationships are key to what you're trying to do. The most common example of a graph is something like a social graph where you're connecting different users together and need to follow relationships for many steps. (Figure out if two people are connected within a depth for 4 for instance)
Relationships exist in graph databases because that is the entire concept of a graph database. It doesn't really fit your application, but to be fair you could just keep more in the node part of your database. In general the whole idea of a database is something that lets you query a LOT of data very quickly. Depending on the intrinsic structure of your data there are different ways that that makes sense. Hence the different kinds of databases.
In strongly connected graphs, Neo4j is 1000x faster on 1000x the data than a SQL database. NoSQL would probably never be able to perform in a strongly connected graph scenario.
Take a look at what we're building right now: http://vimeo.com/81206025
Update: In reaction to mindreader's comment, we added the related properties to the picture:
RDBM systems are tabular and put more information in the tables than the relationships. Graph databases put more information in relationships. In the end, you can accomplish much the same goals.
However, putting more information in relationships can make queries smaller and faster.
Here's an example:
Graph databases are also good at storing human-readable knowledge representations, being edge (relationship) centric. RDF takes it one step further were all information is stored as edges rather than nodes. This is ideal for working with predicate logic, propositional calculus, and triples.
Maybe the right answer is an object database.
Objectivity/DB, which now supports a full suite of graph database capabilities, allows you to design complex schema with one-to-one, one-to-many, many-to-one, and many-to-many reference attributes. It has the semantics to view objects as graph nodes and edges. An edge can be just the reference attribute from one node to another or an edge can exist as an edge object that sits between two nodes.
An edge object can have any number of attribute and can have references off to other objects, as shown in the diagram below.
Being able to "hang" complex objects off of an edge allows Objectivity/DB to support weighted queries where the edge-weight can be calculated using a user-defined weight calculator operator. The weight calculator operator can build the weight from a static attribute on the edge or build the weight by digging down through the objects connected to the edge. In the picture, above, we could create a edge-weight calculator that computes the sum of the CallDetail lengths connected to the Call edge.
I'm trying to decide if i should implement Categories as nodes or labels.
Especially the query to get a count of nodes belonging to a category is not so easy.
Nodes have to be able to belong to more categories !
Categories as labels, variant 1
Keep a list of categories somewhere, then:
MATCH a:cat1, b:cat2, c:cat3, ...
With a lot of categories i will get a lot of columns .. so that's not really good. Also lot's of preprocessing on the query.
Not even sure if i could get a count easily from that.
Categories as labels, variant 2
MATCH n:category <-- the category label is used to limit the amount of nodes
RETURN DISTINCT labels(n), count(*) as count
Will return something like:
["category","the actual category label"], 2
Looks perfect, but this won't work when a node has multiple categories
["category","cat1","cat2"], 2 <-- two nodes found with category "cat1" and "cat2"
["category","cat1"], 4 <-- four nodes found with category "cat1"
Now i don't know how to get the count per category ...
Maybe something with extract(..labels()..) or filter(..labels()..) could be able to do it, but i don't know how.
Categories as nodes
Yes this works ... this is pretty straight forwarded. But aren't labels suppose to be THE thing for categorizing nodes? Plus all the extra relationships i would be creating ..
Maybe i should implement it as both labels and nodes?
Then with labels i can get every node with a category fast. And with a node i could get the category count.
I'm still searching for a good perspective on this problem, so i can not give a concrete implementation question yet.
My two cents.
For your kind of categories, I would go with a node per category and create a BELONGS_TO relationship from nodes belonging to that category. There are a number of reasons for this preference of mine.
One of the reasons labels were added is that many people were putting a "type" property on nodes. Another way to talk about labels is that they add a little bit of a "schema" to your graph - in the sense that you can categorise nodes.
With the introduction of labels, there's always the risk that they will be abused. It is just an extra tool in a database that is primarily designed for storing graphs. In an extreme case, you could use labels for almost everything, ending up with a store of "tagged" nodes.
Finally, traversing relationships is the fastest thing Neo4j does. We're talking units of microseconds. Don't be afraid adding thousands of relationships to a node. I'd leave labels for developer-defined "schema-like" information.
So in your case of user-added categories, I'd definitely create category nodes and BELONGS_TO relationships, in favour of labelling.
One last thing with a disclaimer that this is a bit of self-marketing. If you get to a point where you have tens of thousands or millions of relationships per node, and all you're after is counting the relationships, it might be a good idea to cache those counts on the nodes as properties. I've developed a module called "Relationship Count Module" for the GraphAware Framework, which does exactly that. I've demonstrated in my MSc. thesis, which is gonna be public in a couple of weeks, that the module speeds up count queries for high-degree vertices by several orders of magnitude, for as little as 10-25% write throughput penalty. Let me know if you need more detail about that.