When I start my Rails application and open a page which needs to query my MongoDB database, then there is the following problem:
on my local machine it takes about 1600ms to perform the queries and render all
on my linode it takes about 4min to perform the first query and render all
After that everything is faster, caching, pages are loaded instantly, etc.
But really, 4min? Why is that? Is that the loading from disk to memory for MongoDB? Why does it take so much longer than on my local machine?
Is this due to the hard drive being shared on Linode? I noticed a lot of activity when running iostat
$ iostat -d 2
Linux 3.12.6-x86_64-linode36 (linode) 01/31/2014 _x86_64_ (8 CPU)
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
xvda 1129.69 43026.47 17.62 1940251345 794504
xvdb 248.43 2572.50 698.08 116005452 31479356
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
xvda 4491.50 179012.00 0.00 358024 0
xvdb 0.00 0.00 0.00 0 0
It's my understanding that Mongo loads all the data from disk into memory, so I guess it's likely that you're experiencing slow performance during that phase. Perhaps it makes sense to hit the db with several queries to warm it up before you enable your application.
Related
I am a newbie to Glusterfs. I have currently setup Glusterfs in two servers with following options:
performance.cache-size 2GB
performance.io-thread-count 16
performance.client-io-threads on
performance.io-cache on
performance.readdir-ahead on
When I run my binary in following manner:
./binary > shared_location/log
It takes roughly 1.5 mins
log size is roughly 100M
Whereas running in this manner:
./binary > local_location/log
Takes roughly 10 secs.
This is a huge difference in time. No of cores in glusterfs server: 2, current machine: 2
Is there any way I can reduce time?
Also, is there any standard configuration to start off with, so I can avoid basic issues like above?
Issue: After expected clear and rebuild of specific redis keys, worker dynos doesn't allocate memory (until restart of dyno).
I am experience an issue where my Heroku worker-dynos are hitting 95%-100% max memory usage during a delete and rebuild on about 4000 keys. I have a scheduled rebuild that starts every day at 4:00am. Based on logs I assume the DEL of the keys + the rebuild of keys take about ~1490 seconds.
Jun 29 04:01:41 app app/worker.2: 4 TID-...io8w RedisWorker JID-...cd2a7 INFO: start
Jun 29 04:06:28 app app/worker.1: 4 TID-...mtks RedisWorker JID-...bf170 INFO: start
Jun 29 04:26:32 app app/worker.1: 4 TID-...mtks RedisWorker JID-...bf170 INFO: done: 1203.71 sec
Jun 29 04:26:33 app app/worker.2: 4 TID-...io8w RedisWorker JID-...cd2a7 INFO: done: 1490.938 sec
The memory will hover max usage until the dyno restarts (which is scheduled) or we deploy. Example image: Heroku Memory Usage
This is a high level what gets triggered at 4am:
def full_clear
RedisWorker.delete_keys("key1*")
RedisWorker.delete_keys("key2*")
RedisWorker.delete_keys("key3*")
self.build
return true
end
def build
... rebuilds keys based on models ...
return true
end
def self.delete_keys(regex)
$redis.scan_each(match: regex) do |key|
$redis.del(key)
end
end
What I have researched so far or my thoughts:
After redis DEL is being invoked the memory doesn't allocate?
Could there be a better implementation of finding all keys that match and doing a batch delete?
I am using defaults for puma; would configuring puma+sidekiq to better match our resources help be the best starting action? Deploying Rails Applications with the Puma Web Server. After a restart the memory is only about 30%-40% until the next full-rebuild (even during high usage of hmsets).
I noticed that my ObjectSpace counts is comparably a lot lower after the dyno gets restart/rest of day until next scheduled full_rebuild.
Any thoughts how I can go about trying to figure out whats causing the dynos to hang memory? Seems isloated to side / the worker dynos being used do rebuild Redis.
Solution:
I installed New Relic to see if there was a potential memory bloat. One of our most used function calls was doing a N+1 query. Fixed the N+1 query and watched our 60k call in New Relic drop to ~5K.
GC also wasn't collecting because it wasn't hitting our threshold. Later on there might be potential GC optimization -- but for now we our immediate issue has been resolved.
I also reached out to Heroku for their thoughts and this is what was discussed:
Memory usage on the Dyno will be managed by the Ruby VM, and is likely that you're keep ing too much information in memory during the key rebuild. You should look into freeing memory used to generate the key-values after the data has been added redis.
Spending time fixing your N+1 queries will definitely help!
I've set up a spark-jobserver to enable complex queries on a reduced dataset.
The jobserver executes two operations:
Sync with the main remote database, it makes a dump of some of the server's tables, reduce and aggregates the data, save the result as a parquet file and cache it as a sql table in memory. This operation will be done every day;
Queries, when the sync operation is finished, users can perform SQL complex queries on the aggregated dataset, (eventually) exporting the result as csv file. Every user can do only one query at time, and wait for its completion.
The biggest table (before and after the reduction, which include also some joins) has almost 30M of rows, with at least 30 fields.
Actually I'm working on a dev machine with 32GB of ram dedicated to the job server, and everything runs smoothly. Problem is that in the production one we have the same amount of ram shared with a PredictionIO server.
I'm asking how determine the memory configuration to avoid memory leaks or crashes for spark.
I'm new to this, so every reference or suggestion is accepted.
Thank you
Take an example,
if you have a server with 32g ram.
set the following parameters :
spark.executor.memory = 32g
Take a note:
The likely first impulse would be to use --num-executors 6
--executor-cores 15 --executor-memory 63G. However, this is the wrong approach because:
63GB + the executor memory overhead won’t fit within the 63GB capacity
of the NodeManagers. The application master will take up a core on one
of the nodes, meaning that there won’t be room for a 15-core executor
on that node. 15 cores per executor can lead to bad HDFS I/O
throughput.
A better option would be to use --num-executors 17 --executor-cores 5
--executor-memory 19G. Why?
This config results in three executors on all nodes except for the one
with the AM, which will have two executors. --executor-memory was
derived as (63/3 executors per node) = 21. 21 * 0.07 = 1.47. 21 – 1.47
~ 19.
This is explained here if you want to know more :
http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/
I see the cpu getting close to 100% for lots of cypher queries, which being run of a strong machine (16 cores, 110 GB RAM, SSD etc for 10GB neo4j database)
is there a way to profile a cypher query for other aspects than db hits? for example: cpu usage of each statement, locks etc
server config:
neo4j-wrapper:
wrapper.java.initmemory=31768
wrapper.java.maxmemory=31768
neo4j.properties:
dbms.pagecache.memory=70g
java -XX:+PrintFlagsFinal -version | grep -e '(Initial\|Max)HeapSize'
Picked up _JAVA_OPTIONS: -Xmx32g -Xms32g
uintx InitialHeapSize := 34359738368 {product}
uintx MaxHeapSize := 34359738368 {product}
jstack output:
https://www.dropbox.com/s/ktvflamjqoz9dnu/sudo_jstack%20_13453?dl=0
I don't know of a way to profile specific queries for CPU usage, but have you configured Neo4j to allow it to use more memory? How much memory is it using?
Also, is this in production with a number of queries hitting the database, or are you making queries yourself with the web console? If the latter, sometimes a query will continue running even if you've cancelled it. You can help alleviate that by setting a query timeout (say for 60 seconds).
We are using sidekiq pro 1.7.3 and sidekiq 3.1.4, Ruby 2.0, Rails 4.0.5 on heroku with the redis green addon with 1.75G of memory.
We run a lot of sidekiq batch jobs, probably around 2 million jobs a day. What we've noticed is that the redis memory steadily increases over the course of a week. I would have expected that when the queues are empty and no workers are busy that redis would have low memory usage, but it appears to stay high. I'm forced to do a flushdb pretty much every week or so because we approach our redis memory limit.
I've had a series of correspondence with Redisgreen and they suggested I reach out to the sidekiq community. Here are some stats from redisgreen:
Here's a quick summary of RAM use across your database:
The vast majority of keys in your database are simple values taking up 2 bytes each.
200MB is being consumed by "queue:low", the contents of your low-priority sidekiq queue.
The next largest key is "dead", which occupies about 14MB.
And:
We just ran an analysis of your database - here is a summary of what we found in 23129 keys:
18448 strings with 1048468 bytes (79.76% of keys, avg size 56.83)
6 lists with 41642 items (00.03% of keys, avg size 6940.33)
4660 sets with 3325721 members (20.15% of keys, avg size 713.67)
8 hashs with 58 fields (00.03% of keys, avg size 7.25)
7 zsets with 1459 members (00.03% of keys, avg size 208.43)
It appears that you have quite a lot of memory occupied by sets. For example - each of these sets have more than 10,000 members and occupies nearly 300KB:
b-3819647d4385b54b-jids
b-3b68a011a2bc55bf-jids
b-5eaa0cd3a4e13d99-jids
b-78604305f73e44ba-jids
b-e823c15161b02bde-jids
These look like Sidekiq Pro "batches". It seems like some of your batches are getting filled up with very large numbers of jobs, which is causing the additional memory usage that we've been seeing.
Let me know if that sounds like it might be the issue.
Don't be afraid to open a Sidekiq issue or email prosupport # sidekiq.org directly.
Sidekiq Pro Batches have a default expiration of 3 days. If you set the Batch's expires_in setting longer, the data will sit in Redis longer. Unlike jobs, batches do not disappear from Redis once they are complete. They need to expire over time. This means you need enough memory in Redis to hold N days of Batches, usually not a problem for most people, but if you have a busy Sidekiq installation and are creating lots of batches, you might notice elevated memory usage.