I'm implementing a ruby on rails server(Ruby 2.2.1 and rails 4.1.10) and I'm facing some memory issues where the server process (puma) which can take 500MB and more. Mainly when i'm uploading large file to the server, i'm getting this kind of value. I'm using carrierwave.
My question is related to the ruby management system and garbage collection. Seen that my server is dedicated to embedded system , i really need to cap or control the memory, my process is taking from the system.
Is there a way to see which objects (with its size) are still alive and should not?
Is it right that the memory system in ruby does not retrieve back the free memory to the system if the memory is fragmented?
Please help me to figure out whats goin on when my memory is larger than 150MB idle.
Stéph
After reading a lot of posts talking about that problem, it seems that the real cause come from the Ruby GC, which is consuming a lot of server memory and causing a lot of server swapping since version 2.1.x .
To fix those performance issues, I just set RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR to a value around 1.25. You can play with that setting to find the best value for your environment.
FYI my app is running with Ruby 2.1.5 on Heroku Cedar 14 and it worked like a charm.
Hope it helps :)
I guess my problem is not related to the way GC is starting because if i ask for a manual garbage collection gc.start, i dont get back my memory. Maybe i'm having a memory leak somewhere but i would like to find a way to track it.
It fact, in your case carrierewave gem itself seems to be the source of your issue.
This gem seems to use the entire file to play with it and not chunks... So when your uploading large files you can easily hit the memory limit of your server.
This post seems to confirm what I'm saying:
https://github.com/carrierwaveuploader/carrierwave/issues/413
What I can suggest in your case is to consider using direct upload on S3 to save your server from processing the upload by doing it in background directly on Amazon servers.
I'm not very familiar with carrierwave but this gem should do the trick:
https://github.com/dwilkie/carrierwave_direct
I don't know if it could be a valid solution for you, but regarding your issue it seems to be your best alternative.
Hope it helps :) !
Related
I'm using have a server which running on 4gb ram...
Each uploaded picture will be watermark, so i decided to put in background process. However, when there are a lot of requests of uploading pictures..the server will facing high memory issue, and the memory don't deallocate themselves.
My Questions:
- why sidekiq worker terminate?
- is rmagick memory leak?
- how to handle this situation?
You've provided nowhere near the amount of details needed for us to give you informed advice, but I'll try something general: How many Sidekiq workers are you running? Consider reducing the #, then queue up tons of request to simulate a heavy load; keep doing that until you have few enough workers that Sidekiq can comfortably handle the worst load. (Or until you confirm that the issue appears the same even when there's only 1 Sidekiq worker!)
Once you've done that, then you'll have a better feel for the contours of the problem: something that looks like an Rmagick memory leak issue when your server is overloaded, may look different (or give you more ideas on how to address it) when you reduce the workload.
Also take a look at this similar SO question about Rmagick and memory leaks; it might be worth forcing garbage collection to limit how much damage any given leak can cause.
I have a Rails 3.2.8 app running on Heroku Cedar with Ruby 1.9.3. The app runs fine when it launches but after a day or so of continuous use, I start to see R14 errors on my logs. Once the memory errors start, they never go away, even if the app is idle for several hours.
Shouldnt the garbage collector clean up unused objects after a while and reduce the memory load? It seems this is not happening on Heroku. Generally, memory usage starts to creep up after running some reports with several thousand rows of data, although results are paginated.
How can I find the memory leak? Plugins like bleak_house are way out of date or dont run nicely in the Heroku environment. Can I adjust the GC settings to make it more aggressive?
The GC should do the clean up, and probably does.
You can force the GC with GC.start; if many objects were not collected this will, but I suspect that is not the issue.
Is it possible you somehow create a bunch of objects and never release them, by keeping cached copies or something?
I'm unfamiliar with the existing tools to check this, but you may want to check which objects exist using ObjectSpace. For example:
ObjectSpace.each_object.with_object(Hash.new(0)){|obj, h| h[obj.class] +=1 }
# => a Hash with the number of objects by class
If you get an unexpected number for one of your classes, for instance, you would have a better idea of where to look for.
Install the New Relic add-on. It has a bunch of useful metrics that you can use to find out the source of the leak. I think its generally a better idea to try to see which part of the code takes the longest to execute and perhaps try to optimize that, rather than tweak the GC outright.
Some of the nice features New Relic includes is being able to pinpoint the source of the longest running SQL query, for example. I encourage you to give it a try.
Some odd behavior I encountered while optimizing a rails view:
After tweaking the amount of garbage collect calls in a request I didnt see real improvements on the performance. Investigating the problem I found out garbage collect didn't really remove that much dead objects!
I got a very heavy view with LOADS of objects.
Using scrap I found out after a fresh server start and page load the amount of objects was about 670.000, after reloading the page 3 times the amount has risen to 19.000.000!
RAILS_GC_MALLOC_LIMIT is set to 16.000.000 and I can read GC has been called 1400 times.
Why does the memory keep increasing on a refresh of the page? And is there a way to make sure the old objects are removed by GC?
PS: Running on REE 1.8.7 2011.03 with rails 3.2.1
I highly recommended to use newrelic for optimalization, got way more performance boost there then with a little gc tweaking..
You dont need to gc objects you never create :)
I have an Jruby On Rails app which uses several WS's to gather data. The app processes the data displays it to the user, the user make his changes and then is sent back to the WS.
The problem here is that i store everything in the cache (session based) which is using memory store. But from time to time with no clear reason (for me at least) this error pops up:
ActionView::Template::Error (GC overhead limit exceeded)
I read what I could find about it and apparently this means that the Garbage Collector spends to much time in trying to free memory and no real progress is being made in that direction. My guess is that since everything is stored cache like into memory the GC tries to free it and can't do it and throws this error.
So here are the questions.
How can I get around this ?
If I switch from memory store to Redis, if my assumptions are correct, will this problem still appear.
Will the GC try to free Redis's memory area ? (Might be a stupid question but ... please help nonetheless :) )
Thank you.
Redis is a separate process, so your app garbage collector won't touch it. However, it is possible that redis is using up all the memory that would otherwise be available for the app, or that you are actually using a process memory cache and not redis, which is a different type of memory store.
I am using memcached and it seems to be crashing a lot lately. Sometimes a deploy can cause it to crash on ActionController::Base.cache_store.clear and sometimes it happens out of nowhere.
How can I get to the root cause of this? Does it have it's own log somewhere?
How can I make it more robust? Our site relies heavily on it and it going down brings the site down too. (We obviously need to figure out how to make our app still operate without it)
Any recommendations?
Check the config files where the stdio/stderror goes
Check the debug messages verbosity. Put it on max available.
What is the memory limit/memory segment sizes you use (again, in the config).
make sure they are not too small.
If memcached is crashing a lot, it's likely because you're using bad tools. Is this CentOS perhaps? libevent 1.1 or similarly ancient version?