I see the cpu getting close to 100% for lots of cypher queries, which being run of a strong machine (16 cores, 110 GB RAM, SSD etc for 10GB neo4j database)
is there a way to profile a cypher query for other aspects than db hits? for example: cpu usage of each statement, locks etc
server config:
neo4j-wrapper:
wrapper.java.initmemory=31768
wrapper.java.maxmemory=31768
neo4j.properties:
dbms.pagecache.memory=70g
java -XX:+PrintFlagsFinal -version | grep -e '(Initial\|Max)HeapSize'
Picked up _JAVA_OPTIONS: -Xmx32g -Xms32g
uintx InitialHeapSize := 34359738368 {product}
uintx MaxHeapSize := 34359738368 {product}
jstack output:
https://www.dropbox.com/s/ktvflamjqoz9dnu/sudo_jstack%20_13453?dl=0
I don't know of a way to profile specific queries for CPU usage, but have you configured Neo4j to allow it to use more memory? How much memory is it using?
Also, is this in production with a number of queries hitting the database, or are you making queries yourself with the web console? If the latter, sometimes a query will continue running even if you've cancelled it. You can help alleviate that by setting a query timeout (say for 60 seconds).
Related
I've set up a spark-jobserver to enable complex queries on a reduced dataset.
The jobserver executes two operations:
Sync with the main remote database, it makes a dump of some of the server's tables, reduce and aggregates the data, save the result as a parquet file and cache it as a sql table in memory. This operation will be done every day;
Queries, when the sync operation is finished, users can perform SQL complex queries on the aggregated dataset, (eventually) exporting the result as csv file. Every user can do only one query at time, and wait for its completion.
The biggest table (before and after the reduction, which include also some joins) has almost 30M of rows, with at least 30 fields.
Actually I'm working on a dev machine with 32GB of ram dedicated to the job server, and everything runs smoothly. Problem is that in the production one we have the same amount of ram shared with a PredictionIO server.
I'm asking how determine the memory configuration to avoid memory leaks or crashes for spark.
I'm new to this, so every reference or suggestion is accepted.
Thank you
Take an example,
if you have a server with 32g ram.
set the following parameters :
spark.executor.memory = 32g
Take a note:
The likely first impulse would be to use --num-executors 6
--executor-cores 15 --executor-memory 63G. However, this is the wrong approach because:
63GB + the executor memory overhead won’t fit within the 63GB capacity
of the NodeManagers. The application master will take up a core on one
of the nodes, meaning that there won’t be room for a 15-core executor
on that node. 15 cores per executor can lead to bad HDFS I/O
throughput.
A better option would be to use --num-executors 17 --executor-cores 5
--executor-memory 19G. Why?
This config results in three executors on all nodes except for the one
with the AM, which will have two executors. --executor-memory was
derived as (63/3 executors per node) = 21. 21 * 0.07 = 1.47. 21 – 1.47
~ 19.
This is explained here if you want to know more :
http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/
I am running simple queries on neo4j 2.1.7
I am trying to execute that query:
MATCH (a:Caller)-[:MADE_CALL]-(c:Call)-[:RECEIVED_CALL]-(b:Receiver) CREATE(a)-[:CALLED]->(b) RETURN a,b
While the query is executing, I am getting the following error
Disconnected from Neo4j. Please check if the cord is unplugged.
Then another error:
GC overhead limit exceeded
I'm working on windows server 2012 with 16G of RAM and here is my nodes.properties file:
**
`neostore.nodestore.db.mapped_memory=1800M
neostore.relationshipstore.db.mapped_memory=1G
#neostore.relationshipgroupstore.db.mapped_memory=10M
neostore.propertystore.db.mapped_memory=500M
neostore.propertystore.db.strings.mapped_memory=250M
neostore.propertystore.db.arrays.mapped_memory=10M
cache_type=weak
keep_logical_logs=100M size**`
and my neo4j-community.vmoption file:
**
-Xmx8192
-Xms4098
-Xmn1G
-include-options ${APPDATA}\Neo4j Community\neo4j-community.vmoptions**
I have 6 128 644 Nodes, 6 506 355 Relationships and 10 488 435 properties
Any solution?
TL;DR: Neo4j disconnected because your query is too inefficient. The solution is to improve the query.
Your Neo4j instance appears to have timed out and undergone a GC dump due to the computational intensiveness of your query. When you initialize the Neo4j database using the bash shell, you have the option of configuring certain JVM variables, of which include the amount of memory and heap size available to Neo4j. Should a query exceed these computational limitations, Neo4j automatically terminates the query, undergoes a GC dump, and disconnects.
Looking at the information you gave on the database, there are 6M nodes with 6M relationships. Considering that your query essentially looks for all pathways from Callers to Receivers across 6M nodes, then tries to perform bulk write operations, it's not surprising that Neo4j crashes/disconnects. I would suggest finding a way to limit the query (even with a simple LIMIT keyword) and running multiple smaller queries to get the job done.
I'm trying to query ~8 million nodes in a neo4j database. I can do queries which hit the index for exact matches easily enough, but is there a performant way to do aggregations?
MATCH (r:Resident) RETURN r.forename, count(r.forename) ORDER BY count(r.forename)
This query just sits there until I eventually restart my server. I've read the performance guides and I'm watching vm_stat and it seems to be quickly running out of pages free. I've tried tuning the memory / JVM heap settings to various things, but I'm not sure I completely know what I'm doing ;) I've got an 8 GB MacBook Air with an SSD drive in case that's helpful for suggesting settings. Also, here's my stats on my DB from webadmin:
10,236,226 nodes
56,280,161 properties
10,190,430 relationships
2 relationship types
14,535 MB database disk usage
I inserted 8M nodes with just 1 prop, and got this query down to ~20s without changing the default settings (after warming up the cache--first time took 90s), which is comparable to other databases like postgres (which I also tested).
Some things you could try to do:
raise the sizes on the appropriate files mmio settings (as per the file sizes in data/graph.db/) in conf/neo4j.properties (at the top)
increase the node cache size in neo4j.properties
increase the heap init/max in neo4j-wrapper.conf
make sure you have enough RAM left over
We are evaluating Neo4J for our application, testing it against a small test database with a total of around 20K nodes, 150K properties, and 100K relationships. The branching factor is ~100 relationships/node. Server and version information is below [1]. The Cypher query is:
MATCH p = ()-[r1:RATES]-(m1:Movie)-[r2:RATES]-(u1:User)-[r3:RATES]-(m2:Movie)-[r4:RATES]-()
RETURN r1.id as i_id, m1.id, r2.id, u1.id, r3.id, m2.id, r4.id as t_id;
(The first and last empty nodes aren't important to us, but I didn't see how to start with relationships.)
I killed it after a couple of hours. Maybe I'm expecting too much by hoping Neo4J would avoid combinatorial explosion. I tried tweaking some server parameters but got no further.
My main question is whether what I'm trying to do (a nine-step path query) is reasonable for Neo4J, or, for that matter, any graph database. I realize nine steps is a very deep search, and one that touches every node in the database multiple times, but unfortunately that's what our research needs to do.
Looking forward to your thoughts.
[1] System info:
The Linux server has 32 processors and 64GB of memory.
Neo4j - Graph Database Kernel (neo4j-kernel), version: 2.1.2.
java version "1.7.0_60", Java(TM) SE Runtime Environment (build 1.7.0_60-b19), Java HotSpot(TM) 64-Bit Server VM (build 24.60-b09, mixed mode)
To answer your main question, Neo4j has no problem doing a variable length query that does not result in a combinatorial explosion in the search space (an exponential time complexity as a result of your branching factor).
There is however an optimization that can be done to your Cypher query.
MATCH ()-[r1:RATES]->(m1:Movie),
(m1)<-[r2:RATES]-(u1:User),
(u1)-[r3:RATES]->(m2:Movie),
(m2)<-[r4:RATES]-()
RETURN r1.id as i_id, m1.id, r2.id, u1.id, r3.id, m2.id, r4.id as t_id;
That being said, Cypher has some current limitations with these kinds of queries. We call these queries "graph global operations". When you are running a query that touches the graph globally without a specific starting point, computation as well as writes and reads to disc can cause performance bottlenecks. When returning large payloads over HTTP REST, you'll encounter data transfer limitations within your network.
To test the difference between query response times due to network data transfer constraints, compare the previous query to the following:
MATCH ()-[r1:RATES]->(m1:Movie),
(m1)<-[r2:RATES]-(u1:User),
(u1)-[r3:RATES]->(m2:Movie),
(m2)<-[r4:RATES]-()
RETURN count(*)
The difference between the queries in response time should be significant.
So what are your options?
Option 1:
Write a Neo4j unmanaged extension in Java that runs on-heap embedded in the JVM using Neo4j's Java API. Your Cypher query can be translated imperatively into a traversal description that operates on your graph in-memory. Seeing that you have 64GB of memory, your Java heap should be configured so that Neo4j has access to 70-85% of your available memory.
You can learn more about the Neo4j Java API here: http://docs.neo4j.org/chunked/stable/server-unmanaged-extensions.html
Option 2:
Tune the performance configurations of Neo4j to run your graph in-memory and optimize your Cypher queries to limit the amount of data transferred over the network. Performance will still be sub-optimal for graph global operations.
When I start my Rails application and open a page which needs to query my MongoDB database, then there is the following problem:
on my local machine it takes about 1600ms to perform the queries and render all
on my linode it takes about 4min to perform the first query and render all
After that everything is faster, caching, pages are loaded instantly, etc.
But really, 4min? Why is that? Is that the loading from disk to memory for MongoDB? Why does it take so much longer than on my local machine?
Is this due to the hard drive being shared on Linode? I noticed a lot of activity when running iostat
$ iostat -d 2
Linux 3.12.6-x86_64-linode36 (linode) 01/31/2014 _x86_64_ (8 CPU)
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
xvda 1129.69 43026.47 17.62 1940251345 794504
xvdb 248.43 2572.50 698.08 116005452 31479356
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
xvda 4491.50 179012.00 0.00 358024 0
xvdb 0.00 0.00 0.00 0 0
It's my understanding that Mongo loads all the data from disk into memory, so I guess it's likely that you're experiencing slow performance during that phase. Perhaps it makes sense to hit the db with several queries to warm it up before you enable your application.