Improve relationship performance using Spring Data Neo4J - neo4j

We are using Spring Boot 2.2.5 and Spring Data Neo4J. We have nodes with relationships which we have mapped with Spring NodeEntity and RelationshipEntity. What we are noticing is if there are nodes with lots of first level relationships (e.g. over 1500), its taking time (over 1 second) to get and update the entity/relationships.
Are there any best practices on how to improve performance for the relationships. Are there ways we can use pagination, limits etc.?
Code:
#NodeEntity
public class Node {
#Id
#GeneratedValue
private Long id;
private String name;
#Relationship(type = RelatedNode.TYPE, direction = Relationship.UNDIRECTED)
private Set<RelatedNode> relatedNodes = new HashSet<>();
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public Set<RelatedNode> getRelatedNodes() {
return relatedNodes;
}
public void setRelatedNodes(Set<RelatedNode> relatedNodes) {
this.relatedNodes = relatedNodes;
}
public RelatedNode addRelatedNode(Node relatedNode, long count) {
final RelatedNode node = this.relatedNodes.stream()
.filter(i -> (relatedNode.getId().equals(i.getEnd().getId())) || (relatedNode.getId().equals(i.getStart().getId())))
.findFirst()
.orElseGet(() -> {
RelatedNode newRelatedNode = new RelatedNode();
newRelatedNode.setStart(this);
newRelatedNode.setEnd(relatedNode);
newRelatedNode.setCount(count);
this.relatedNodes.add(newRelatedNode);
return newRelatedNode;
});
return node;
}
public RelatedNode updateRelatedNode(Node relatedNode, long count) {
final RelatedNode node = this.relatedNodes.stream()
.filter(i -> (relatedNode.getId().equals(i.getEnd().getId())) || (relatedNode.getId().equals(i.getStart().getId())))
.findFirst().get();
if (node != null) {
node.setCount(count);
}
return node;
}
public void deleteRelatedNode(Node relatedNode) {
final RelatedNode node = this.relatedNodes.stream()
.filter(i -> (relatedNode.getId().equals(i.getEnd().getId())) || (relatedNode.getId().equals(i.getStart().getId())))
.findFirst().get();
this.relatedNodes.remove(node);
}
}
#RelationshipEntity(type = RelatedNode.TYPE)
public class RelatedNode {
public static final String TYPE = "RELATED_TO";
#Id
#GeneratedValue
private Long id;
#StartNode
private Node start;
#EndNode
private Node end;
private long count;
public Long getId() {
return id;
}
public long getCount() {
return count;
}
public void setCount(long count) {
this.count = count;
}
public Node getEnd() {
return end;
}
public void setEnd(Node end) {
this.end = end;
}
public Node getStart() {
return start;
}
public void setStart(Node start) {
this.start = start;
}
public void setId(Long id) {
this.id = id;
}
}
Notes:
I am having to add/update related nodes by using addRelatedNode/updateRelatedNode methods. If I create/update relationships without these methods, the relationships are duplicated as the relationship entity has attributes.

Related

Creating relationships between nodes using Spring Data Rest and neo4j

I am starting to use Neo4J with Spring Data Rest. I have a node entity and a relationship entity for modelling nodes and edges. I'm able to create new nodes with the following using postman.
POST http://localhost:8080/nodes
{
"name" : "Test"
}
I am unsure of what the JSON format would be to create relationships between the nodes. For example:
Create a new node and relate to an existing node
Create a relationship between two existing nodes.
Any examples on what JSON I need to use would be very much appreciated.
My node entity and relationship entity are as follows:
#NodeEntity
public class Node {
#Id
#GeneratedValue
private Long id;
private String name;
private int count;
#Relationship(type = Edge.TYPE, direction = Relationship.UNDIRECTED)
private Set<Edge> edges = new HashSet<>();
public void addEdge(Node target, int count) {
this.edges.add(new Edge(this, target, count));
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getCount() {
return count;
}
public void setCount(int count) {
this.count = count;
}
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public Set<Edge> getEdges() {
return edges;
}
public void setEdges(Set<Edge> edges) {
this.edges = edges;
}
}
#RelationshipEntity(type = Edge.TYPE)
public class Edge {
public static final String TYPE = "LINKED_TO";
#Id
#GeneratedValue
private Long relationshipId;
#StartNode
private Node start;
#EndNode
private Node end;
private int count;
public Edge() {
super();
}
public Edge(Node start, Node end, int count) {
this.start = start;
this.end = end;
this.count = count;
}
public int getCount() {
return count;
}
public void setCount(int count) {
this.count = count;
}
public Node getStart() {
return start;
}
public void setStart(Node start) {
this.start = start;
}
public Node getEnd() {
return end;
}
public void setEnd(Node end) {
this.end = end;
}
public Long getRelationshipId() {
return relationshipId;
}
public void setRelationshipId(Long relationshipId) {
this.relationshipId = relationshipId;
}
}
OK I worked it out, you can do this:
PATCH http://localhost:8080/nodes/1
{
"name" : "Test",
"edges": [
{
"start": "http://localhost:8080/nodes/1",
"end": "http://localhost:8080/nodes/2"
}
]
}
This will add the relationships between the nodes.
Hope this helps someone.

Neo4j remove existing relationships itself during adding new relationships

I have a code responsible for creating new relationship between two nodes.
At first It get a two nodes by property (pk), check if such relationship between those nodes already exists and if It doesn't not, create one.
public void createNeo4jGraphLink(#Nonnull final String childPk, #Nonnull final String parentPk) {
final UUID measureId = measureService.startMeasure(MeasureService.MeasureEvent.NEO_PERSIST);
Node child = neoRepository.findOneByPk(childPk);
Node parent = neoRepository.findOneByPk(parentPk);
checkIfLinkingAllowed(child, parent);
if (child.getPartOf() == null || !child.getPartOf().contains(parent)) {
if (child.getPartOf() == null) {
child.setPartOf(new HashSet<>());
}
child.getPartOf().add(parent);
neoRepository.save(child);
}
measureService.fixMeasure(MeasureService.MeasureEvent.NEO_PERSIST, measureId);
}
I can see that number of relationships rises in db, but one moment it sharply decrease and so on.. What reason can lead to this behavior? I've tried to use #Transactional annotation, experimented with propagation and isolation of transaction and tried without transactions at all, but it all does not work. Btw I use Spring Data Neo4j repositories. Spring Boot 1.4.4.RELEASE.
Child and parent entity' type is "Node" which is defined as a follows
#NodeEntity(label = "node")
public class Node {
private String pk = UUID.randomUUID().toString();
#JsonIgnore
private Long id;
private String name;
private Set<Node> partOf;
private String type;
public Node() {
}
#Property(name = "pk")
public String getPk() {
return pk;
}
public void setPk(String pk) {
this.pk = pk;
}
#GraphId(name = "id")
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
#Property(name = "type")
public String getType() {
return type;
}
public void setType(String type) {
this.type = type;
}
#Property(name = "name")
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
#Relationship(type = "PART_OF", direction = Relationship.OUTGOING)
public Set<Node> getPartOf() {
return partOf;
}
public void setPartOf(Set<Node> partOf) {
this.partOf = partOf;
}
#Override
public boolean equals(Object o) {
if (this == o) return true;
if (o == null || getClass() != o.getClass()) return false;
Node node = (Node) o;
if (!pk.equals(node.pk)) return false;
return true;
}
#Override
public int hashCode() {
return pk.hashCode();
}
}
I perform adding the relationship 1800 times, but as result have 62 relationships. Code does not throw any errors. While processing those 1800 requests, I can see in database there is any number of relationships (less then 1800), but as result only 62.
Thank you for any information

Can relate to only one element

I am using Neo4j OGM 2.0.4 driver with Java. I have trouble with adding more than one relationship to element.
I do something like this:
Site site1 = new Site();
site1.setTitle("Site 1");
site1.setHtmlCode("Content of site 1");
Site site2 = new Site();
Site subsite1 = new Site();
subsite1.setTitle("Subsite 1");
subsite1.setHtmlCode("Content of subsite 1");
subsite1.setParent(site1);
Site subsite2 = new Site();
subsite2.setTitle("Subsite 2");
subsite2.setHtmlCode("Content of subsite 2");
subsite2.setParent(site1);
session.deleteAll(Site.class);
session.save(site1);
session.save(subsite1);
session.save(subsite2);
When I want to show all Site nodes (on localhost:7474) then "Subsite 1" has no relationship.
#NodeEntity
public class Site extends Entity
{
private String _title;
private String _htmlCode;
#Relationship(type = "SITE_CREATED_BY")
Author _author;
#Relationship(type = "IS_CHILD")
Set<Site> _parentSite;
#Relationship(type = "IS_CHILD", direction = Relationship.INCOMING)
Set<Site> _childSites;
public Site()
{
_parentSite = new HashSet();
_childSites = new HashSet();
}
public void setTitle(String title)
{
_title = title;
}
public String getTitle()
{
return _title;
}
public void setHtmlCode(String htmlCode)
{
_htmlCode = htmlCode;
}
public String getHtmlCode()
{
return _htmlCode;
}
public void setAuthor(Author author)
{
_author = author;
}
public void setParent(Site site)
{
_parentSite.add(site);
}
}
Entity:
public abstract class Entity
{
private Long id;
private final ZonedDateTime _dateOfCreation;
Entity()
{
_dateOfCreation = ZonedDateTime.now();
}
public Long getId()
{
return id;
}
public ZonedDateTime getDateOfCreation()
{
return _dateOfCreation;
}
#Override
public boolean equals(Object o)
{
if (this == o) return true;
if (o == null || id == null || getClass() != o.getClass()) return false;
Entity entity = (Entity) o;
return id.equals(entity.id);
}
#Override
public int hashCode()
{
return (id == null) ? -1 : id.hashCode();
}
}
What am I doing wrong?
In this case where you have two relationships in different directions between the same type of node, first, make sure that you annotate both the fields as well as setter/accessor methods with the #Relationship,specifying the direction.
Site in your object model has references to both the parent and children, but when you create sites, they do not seem consistent with the model. Subsite1 and Subsite2 both set their parents to site1 but site has no record of its children (should be both subsites). Should work if your object and graph models are consistent.

Simple relationships not mapped with queryForObject

I have one question. I am using neo4j-ogm snapshot 1.5 .
I have the following classes:
#NodeEntity
public abstract class Entity {
#GraphId
protected Long id;
#Expose
protected String code = null;
#Override
public boolean equals(Object o) {
if (this == o)
return true;
if (o == null || id == null || getClass() != o.getClass())
return false;
Entity entity = (Entity) o;
if (!id.equals(entity.id))
return false;
return true;
}
#Override
public int hashCode() {
return (id == null) ? -1 : id.hashCode();
}
public Long getId(){
return id;
}
public void setId(Long neo4jId){
this.id = neo4jId;
}
public String getCode(){
return code;
}
public void setCode(String code){
this.code = code;
}
}
public class PropertyGroup extends Entity{
#Expose
private String name;
public PropertyGroup(){
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
public class User extends Entity {
private Long registration_date;
private Long last_login_date;
private Boolean is_admin = false;
private String push_dev;
private String push_id;
private Boolean push_enabled = false;
#Expose
private String avatar;
#Expose
private String avatarUrl;
#Expose
private String name;
#Expose
private volatile String password;
#Expose
private int likes = 0;
#Expose
private int questionCount = 0;
#Expose
private int followersCount = 0;
#Expose
private boolean isFollowing = false;
// public Set<UserPropertyRelation> properties;
// #Relationship(type = ModelRelType.ANSWERED)
// public Set<UserAnsweredRelation> userAnswers;
//
// #Relationship(type = ModelRelType.LIKES)
// private Set<LikesQuestionRelation> questionsLiked;
#Expose
#Relationship(type = ModelRelType.HAS_PROPERTY)
private Set<PropertyGroup> properties;
// private Profile userProfile;
// private List<Fact> facts;
// #Expose
// #Relationship(type = ModelRelType.OWNS)
// private List<Question> questions;
public User(){
// this.properties = new LinkedHashSet<UserPropertyRelation>();
// this.userAnswers = new LinkedHashSet<UserAnsweredRelation>();
// this.userProperties = new HashSet<PropertyGroup>();
// this.setFacts(new ArrayList<Fact>());
this.properties = new HashSet<PropertyGroup>();
}
public User(long regDate, long lastLoginDate, boolean isAdmin,
String pushDev, String pushId, boolean pushEnabled){
this();
this.registration_date = regDate;
this.last_login_date = lastLoginDate;
this.is_admin = isAdmin;
this.push_dev = pushDev;
this.push_id = pushId;
this.push_enabled = pushEnabled;
}
// public void addUserAnsweredRelation(UserAnsweredRelation answer){
// answer.setStartNode(this);
// this.userAnswers.add(answer);
// }
//
// public Set<UserAnsweredRelation> getUserAnsweredRelations() {
// return this.userAnswers;
// }
// public void setUserAnsweredRelations(Set<UserAnsweredRelation> userAnswers){
// for(UserAnsweredRelation a : userAnswers){
// a.setStartNode(this);
// }
//
// this.userAnswers = userAnswers;
// }
//
// public void addUserPropertyRelation(UserPropertyRelation rel){
// rel.setUser(this);
// properties.add(rel);
// }
//
// public void setUserPropertyRelations(Set<UserPropertyRelation> properties){
// for(UserPropertyRelation r: properties){
// r.setUser(this);
// }
//
// this.properties = properties;
// }
// public Set<UserPropertyRelation> getUserPropertyRelations(){
// return this.properties;
// }
public long getRegistrationDate() {
return registration_date;
}
public void setRegistrationDate(long registrationDate) {
this.registration_date = registrationDate;
}
public long getLastLoginDate() {
return last_login_date;
}
public void setLastLoginDate(long lastLoginDate) {
this.last_login_date = lastLoginDate;
}
public boolean isAdmin() {
return is_admin;
}
public void setAdmin(boolean isAdmin) {
this.is_admin = isAdmin;
}
public String getPushDev() {
return push_dev;
}
public void setPushDev(String pushDev) {
this.push_dev = pushDev;
}
public String getPushId() {
return push_id;
}
public void setPushId(String pushId) {
this.push_id = pushId;
}
public boolean isPushEnabled() {
return push_enabled;
}
public void setPushEnabled(boolean pushEnabled) {
this.push_enabled = pushEnabled;
}
public String getAvatar() {
return avatar;
}
public void setAvatar(String avatar) {
this.avatar = avatar;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public Set<PropertyGroup> getProperties() {
return properties;
}
public void setProperties(Set<PropertyGroup> properties) {
this.properties = properties;
}
// public Profile getUserProfile() {
// return userProfile;
// }
//
// public void setUserProfile(Profile userProfile) {
// this.userProfile = userProfile;
// }
// public Set<LikesQuestionRelation> getQuestionsLiked() {
// return questionsLiked;
// }
//
// public void setQuestionsLiked(Set<LikesQuestionRelation> likes) {
// this.questionsLiked = likes;
// }
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
// public List<Fact> getFacts() {
// return facts;
// }
//
// public void setFacts(List<Fact> facts) {
// this.facts = facts;
// }
public String getAvatarUrl() {
return avatarUrl;
}
public void setAvatarUrl(String avatarUrl) {
this.avatarUrl = avatarUrl;
}
public int getLikes() {
return likes;
}
public void setLikes(int likes) {
this.likes = likes;
}
public int getQuestionCount() {
return questionCount;
}
public void setQuestionCount(int questionCount) {
this.questionCount = questionCount;
}
public int getFollowersCount() {
return followersCount;
}
public void setFollowersCount(int followersCount) {
this.followersCount = followersCount;
}
public boolean isFollowing() {
return isFollowing;
}
public void setFollowing(boolean isFollowing) {
this.isFollowing = isFollowing;
}
// public List<Question> getQuestions() {
// return questions;
// }
//
// public void setQuestions(List<Question> questions) {
// this.questions = questions;
// }
When I am trying to do the following:
SessionFactory sessionFactory = new SessionFactory(modelsPackageName);
Session session = sessionFactory.openSession(url);
String cypher = "MATCH (u:User {code: {CODE}})-[h:HAS_PROPERTY]->(pg:PropertyGroup) " +
"RETURN u, h, pg";
Map<String, Object> params = new HashMap<String, Object>();
params.put("CODE", "fc48b19ba6f8427a03d6e5990bcef99a28f55592b80fe38731cf805ed188cabf");
// System.out.println(Util.mergeParamsWithCypher(cypher, params));
User u = session.queryForObject(User.class, cypher, params);
The user Object (u) never contains any properties (PropertyGroup entity is not mapped).
What am I doing wrong?
Any help would be appreciated.
Regards,
Alex
If you're using queryForObject return just one column- the object, in your case u.
Neo4j OGM 1.x does not support mapping of custom query results to domain entities, so you will have to return the entity ID, and then do an additional load-by-id specifying a custom depth.
OGM 2.0 (currently 2.0.0-M01) does support mapping custom query results to entities. Your query will remain the same (i.e. return u,h,pg) but instead you'll use the query() method that returns a Result. From the result, you'll be able to get your User entity by column-name u and it'll be hydrated with the PropertyGroups it is related to.
Update:
The dependencies for OGM 2.0.0-M01 are
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-ogm-api</artifactId>
<version>2.0.0-M01</version>
</dependency>
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-ogm-core</artifactId>
<version>2.0.0-M01</version>
</dependency>
Be sure to read the about the configuration changes since you're upgrading from OGM 1.x http://neo4j.com/docs/ogm/java/2.0.0-M01/#reference_setup
A summary of new features: http://neo4j.com/blog/neo4j-ogm-2-0-milestone-1/

Neo4j slow saving

I am using neo4j version 1.8.1 and spring-data-neo4j version 2.2.0. RELEASE.
The problem is about neo4j saving speed. I could not realize why this happens. The node is persisting into db for about 30 second. Here is my model class.
#NodeEntity
public class GraphUser {
public static final String FACEBOOK_FRIEND = "FACEBOOK_FRIEND";
public static final String TWITTER_FOLLOW = "TWITTER_FOLLOW";
public static final String CONTACT = "CONTACT";
public static final String KNOWS = "KNOWS";
public static final String BLOCKED = "BLOCKED";
public static final String FAVORITE = "FAVORITE";
#GraphId
private Long id;
#Indexed(unique = true, indexType = IndexType.FULLTEXT, indexName = "userIdIndex")
private String userId;
#Indexed(indexType = IndexType.FULLTEXT, indexName = "facebookIdIndex")
private String facebookId;
#Indexed(indexType = IndexType.FULLTEXT, indexName = "twitterIdIndex")
private String twitterId;
#Indexed(indexType = IndexType.FULLTEXT, indexName = "emailIndex")
private String email;
#Indexed(indexType = IndexType.FULLTEXT, indexName = "phoneNumberIndex")
private String phoneNumber;
private String knowsLevel;
#RelatedTo(type = FACEBOOK_FRIEND, direction = Direction.BOTH)
#Fetch
private Set<GraphUser> facebookRelations = new HashSet<GraphUser>();
#RelatedTo(type = TWITTER_FOLLOW, direction = Direction.OUTGOING)
#Fetch
private Set<GraphUser> twitterRelations = new HashSet<GraphUser>();
#RelatedTo(type = CONTACT, direction = Direction.OUTGOING)
#Fetch
private Set<GraphUser> contacts = new HashSet<GraphUser>();
#RelatedTo(type = KNOWS, direction = Direction.OUTGOING)
#Fetch
private Set<GraphUser> knows = new HashSet<GraphUser>();
#RelatedTo(type = BLOCKED, direction = Direction.OUTGOING)
#Fetch
private Set<GraphUser> blocks = new HashSet<GraphUser>();
#RelatedTo(type = FAVORITE, direction = Direction.OUTGOING)
#Fetch
private Set<GraphUser> favorites = new HashSet<GraphUser>();
#Query(value = "start user=node({self}), user2=node(*), matchedUser=node(*) " +
"where has(user2.userId) and has(matchedUser.userId) and has(user.knowsLevel) and has(user2.knowsLevel) and has(matchedUser.knowsLevel) " +
"and " +
"user.userId<>matchedUser.userId " +
"and " +
"(" +
"(user.knowsLevel='ALL' and (matchedUser.knowsLevel='ALL' or (user)<-[:KNOWS]-(matchedUser) or ((user)<-[:KNOWS]-(user2)<-[:KNOWS]-(matchedUser) and matchedUser.knowsLevel='SECOND'))) " +
"or " +
"(user.knowsLevel='SECOND' and ((user)-[:KNOWS]->(matchedUser) or (user)-[:KNOWS]->(user2)-[:KNOWS]->(matchedUser)) and (matchedUser.knowsLevel='ALL' or (user)<-[:KNOWS]-(matchedUser) or ((user)<-[:KNOWS]-(user2)<-[:KNOWS]-(matchedUser) and matchedUser.knowsLevel='SECOND'))) " +
"or " +
"(user.knowsLevel='FIRST' and (user)-[:KNOWS]->(matchedUser) and (matchedUser.knowsLevel='ALL' or (user)<-[:KNOWS]-(matchedUser) or ((user)<-[:KNOWS]-(user2)<-[:KNOWS]-(matchedUser) and matchedUser.knowsLevel='SECOND')))" +
") " +
"return matchedUser")
#Fetch
private Iterable<GraphUser> matchedUsers;
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getUserId() {
return userId;
}
public void setUserId(String userId) {
this.userId = userId;
}
public String getFacebookId() {
return facebookId;
}
public void setFacebookId(String facebookId) {
this.facebookId = facebookId;
}
public String getTwitterId() {
return twitterId;
}
public void setTwitterId(String twitterId) {
this.twitterId = twitterId;
}
public String getEmail() {
return email;
}
public void setEmail(String email) {
this.email = email;
}
public String getPhoneNumber() {
return phoneNumber;
}
public String getKnowsLevel() {
return knowsLevel;
}
public void setKnowsLevel(String knowsLevel) {
this.knowsLevel = knowsLevel;
}
public void setPhoneNumber(String phoneNumber) {
this.phoneNumber = phoneNumber;
}
public Set<GraphUser> getFacebookRelations() {
return facebookRelations;
}
public void setFacebookRelations(Set<GraphUser> facebookRelations) {
this.facebookRelations = facebookRelations;
}
public Set<GraphUser> getTwitterRelations() {
return twitterRelations;
}
public void setTwitterRelations(Set<GraphUser> twitterRelations) {
this.twitterRelations = twitterRelations;
}
public Set<GraphUser> getContacts() {
return contacts;
}
public void setContacts(Set<GraphUser> contacts) {
this.contacts = contacts;
}
public Set<GraphUser> getKnows() {
return knows;
}
public void setKnows(Set<GraphUser> knows) {
this.knows = knows;
}
public Set<GraphUser> getBlocks() {
return blocks;
}
public void setBlocks(Set<GraphUser> blocks) {
this.blocks = blocks;
}
public Set<GraphUser> getFavorites() {
return favorites;
}
public void setFavorites(Set<GraphUser> favorites) {
this.favorites = favorites;
}
}
What may I be missing ?
Thank you.
Two suspects
Indexes: I see you have around 5 indexes. Write takes longer when you have more indexes as the write should also update the index as part of the transaction.
Eager Fetch: I see you have around 6 relationships that you annotated with #Fetch. I believe spring-data-neo4j tries to re-fetch everything after the commit. And if these relationships/nodes have #Fetch for their own properties annotated with #Fetch then they are recursively fetched.
I suggest you start testing this by first removing #Fetch and then #Index and see if that improves performance. Are you also doing batch inserts?

Resources