Neo4j cache configurations - neo4j

Question

I have been playing around with heap size for JVM and the file store cache size in Neo4j. It seems like setting the memory mapped buffer to be handled by the OS does not have any effect on the system.
I tried setting the JVM heap quite large with a tiny cache and it was exactly as fast as a if the cache was large.
So my question is: How can I configure the system to allow me to control the cache? Is this an issue with the batching as it says that this uses the JVM heap?
I am using the following python script to fill up the database
neo4j.GraphDatabaseService("http://localhost:7474/db/data/")
f = open('indexslowdown_exp.txt','w')
f.write("Properties\t,\tSpeed\n")
total_time = timedelta(0)
name = 0
for y in range(0,1000):
batch = neo4j.WriteBatch(graph_db)
for x in range(0,100):
batch.create({"name":name})
name += 1
for x in range(0,100):
rand_node_A = random.randint(0,name-1)
rand_node_B = random.randint(0,name-1)
batch.append_cypher("START n=node("+str(rand_node_A)+"), m=node("+str(rand_node_B)+") CREATE (n)-[r:CONNECTED]->(m)")
start_time = datetime.now()
batch.submit()
end_time = datetime.now()
total_time = (end_time-start_time)
f.write(str(name)+" , "+str((total_time)/200)+"\n")
print "Inserting nodes: " + str(total_time)
f.close()
Neo4j.properties file:
use_memory_mapped_buffers=true
/# Default values for the low-level graph engine
neostore.nodestore.db.mapped_memory=1k
neostore.relationshipstore.db.mapped_memory=1k
neostore.propertystore.db.mapped_memory=2k
neostore.propertystore.db.strings.mapped_memory=1k
neostore.propertystore.db.arrays.mapped_memory=1k
# Enable this to be able to upgrade a store from an older version
#allow_store_upgrade=true
# Enable this to specify a parser other than the default one.
#cypher_parser_version=2.0
# Keep logical logs, helps debugging but uses more disk space, enabled for
# legacy reasons To limit space needed to store historical logs use values such
# as: "7 days" or "100M size" instead of "true"
keep_logical_logs=true
# Autoindexing
# Enable auto-indexing for nodes, default is false
#node_auto_indexing=true
# The node property keys to be auto-indexed, if enabled
#node_keys_indexable=name,age
# Enable auto-indexing for relationships, default is false
#relationship_auto_indexing=true
# The relationship property keys to be auto-indexed, if enabled
#relationship_keys_indexable=name,age
neo4j-wrapper:
wrapper.java.additional=-Dorg.neo4j.server.properties=conf/neo4j-server.properties
wrapper.java.additional=-Djava.util.logging.config.file=conf/logging.properties
wrapper.java.additional=-Dlog4j.configuration=file:conf/log4j.properties
#********************************************************************
# JVM Parameters
#********************************************************************
wrapper.java.additional=-XX:+UseConcMarkSweepGC
wrapper.java.additional=-XX:+CMSClassUnloadingEnabled
# Uncomment the following lines to enable garbage collection logging
wrapper.java.additional=-Xloggc:data/log/neo4j-gc.log
wrapper.java.additional=-XX:+PrintGCDetails
wrapper.java.additional=-XX:+PrintGCDateStamps
wrapper.java.additional=-XX:+PrintGCApplicationStoppedTime
wrapper.java.additional=-XX:+PrintTenuringDistribution
# Initial Java Heap Size (in MB)
wrapper.java.initmemory=200
# Maximum Java Heap Size (in MB)
wrapper.java.maxmemory=400
#********************************************************************
# Wrapper settings
#********************************************************************
# Override default pidfile and lockfile
#wrapper.pidfile=../data/neo4j-server.pid
#wrapper.lockfile=../data/neo4j-server.lck
#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
# using this configuration file has been installed as a service.
# Please uninstall the service before modifying this section. The
# service can then be reinstalled.
# Name of the service
wrapper.name=neo4j
# User account to be used for linux installs. Will default to current
# user if not set.
wrapper.user=

Answer 1

What are you most concerned with? Batch insertion performance? If so, MMIO will be most effective if you can fit the entire graph into memory, so if you can estimate the number of nodes and relationships, you can come up with a rough calculation of the size you need for those two stores.
Also given that you look to only be inserting primitives, you can likely estimate the size for the property store that you need. If you're going to store strings and arrays (of a larger type), you can increase the MMIO settings for those two stores, as well, but if you don't need them, set them low.
Approx. size of node store: # of nodes * 14 bytes (if you're using the latest Neo4j; 9 bytes if Neo4j is < 2.0)
Approx. size of relationship store: # of rels * 33 bytes
Remember: There's a near 1:1 correspondence between the store sizes on disk and in memory for the filesystem cache.
As well, a larger JVM heap doesn't necessarily mean greater performance; in fact, the MMIO sizes (depending on your value for the setting use_memory_mapped_buffers) may lie outside of the JVM heap. Regardless, a large JVM heap can also introduce longer GC pauses and other GC-related issues, so don't make it too big.
HTH

Answer 2

For your queries:
Use Parameters!
Use Labels & Indexes in 2.0.x
Try not to use random in performance test loops
Make sure to use streaming