I'm using ejabberd + mochiweb on our server. The longer I keep ejabberd and mochiweb running, the more memory is consumed (last night it was consuming 35% of memory. right now it's a bit above 50%). I thought this was just a mnesia garbage collection issue - so I installed Erlang R13B3 and restarted ejabberd. This didn't fix it though.
So I'm noticing now that at a bit above 50% of full memory consumption, it looks like ejabberd's starting to "let go" of memory and stay at around 50%. Is this normal? Is ~50% a threshold for ejabberd, so that when it reaches it it says, "hey time to actually let some memory go..." and maybe it keeps the rest around for quick access (like caching mnesia?)
I appreciate any input. Thanks!
Run erlang:memory(). in your shell every now and then. You can also give erlang:system_info(Type). with allocated_areas and allocator a try.
These should give you a hint on what kind of memory is leaking.
You can also setup memsup to warn you about processes allocating too much memory.
Turns out, there is no memory leak (yay!) Ejabberd is taking up only < 40MB. Finally I saw the light when I saw the Usage Graphs on EngineYard - only 288MB is actually being used, 550MB is being buffered, and 175MB is being cached. My ejabberd server an update every 2.5 seconds from each client so that may explain why so much is being buffered/cached.
Thanks for all of your help.
Newly created atoms in erlang processes get never garbage collected. This might be an issue when processes are registered by an algorith that creates atom names from random eg. randomly created strings.
Related
I have a ruby on rails app, where we validate records from huge excel files(200k records) in background via sidekiq. We also use docker and hence a separate container for sidekiq. When the sidekiq is up, memory used is approx 120Mb, but as the validation worker begins, the memory reaches upto 500Mb (that's after a lot of optimisation).
Issue is even after the job is processed, the memory usage stays at 500Mb and is never freed, not allowing any new jobs to be added.
I manually start garbage collection using GC.start after every 10k records and also after the job is complete, but still no help.
This is most likely not related to Sidekiq, but to how Ruby allocates from and releases memory back to the OS.
Most likely the memory can not be released because of fragmentation. Besides optimizing your program (process data chunkwise instead of reading it all into memory) you could try and tweak the allocator or change the allocator.
There has been a lot written about this specific issue with Ruby/Memory, I really like this post by Nate Berkopec: https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html which goes into all the details.
The simple "solution" is:
Use jemalloc or, if not possible, set MALLOC_ARENA_MAX=2.
The more complex solution would be to try and optimize your program further, so that it does not load that much data in the first place.
I was able to cut memory usage in a project from 12GB to < 3GB by switching to jemalloc. That project dealt with a lot of imports/exports and was written quite poorly and it was an easy win.
I'm using Puma server and DelayedJob.
It seems that the memory taken by each job isn't released and I slowly get a bloat causing me to restart my dyno (Heroku).
Any reason why the dyno won't return to the same memory usage figure before the job was performed?
Any way to force releasing it? I tried calling GC but it doesn't seem to help.
You can have one of the following problems. Or actually all of them:
Number 1. This is not an actual problem, but a misconception about how Ruby releases memory to operating system. Short answer: it doesn't. Long answer: Ruby manages an internal list of free objects. Whenever your program needs to allocate new objects, it will get those objects from this free list. If there are no more objects there, Ruby will allocate new memory from operating system. When objects are garbage collected they go back to the free list. So Ruby still have the allocated memory. To illustrate it better, imagine that your program is normally using 100 MB. When at some point program will allocate 1 GB, it will hold this memory until you restart it.
There are some good resource to learn more about it here and here.
What you should do is to increase your dyno size and monitor your memory usage over time. It should stabilize at some level. This will show you your normal memory usage.
Number 2. You can have an actual memory leak. It can be in your code or in some gem. Check out this repository, it contains information about well known memory leaks and other memory issues in popular gems. delayed_job is actually listed there.
Number 3. You may have unoptimized code that is using more memory than needed and you should try to investigate memory usage and try to decrease it. If you are processing large files, maybe you should do it in smaller batches etc.
I'm using have a server which running on 4gb ram...
Each uploaded picture will be watermark, so i decided to put in background process. However, when there are a lot of requests of uploading pictures..the server will facing high memory issue, and the memory don't deallocate themselves.
My Questions:
- why sidekiq worker terminate?
- is rmagick memory leak?
- how to handle this situation?
You've provided nowhere near the amount of details needed for us to give you informed advice, but I'll try something general: How many Sidekiq workers are you running? Consider reducing the #, then queue up tons of request to simulate a heavy load; keep doing that until you have few enough workers that Sidekiq can comfortably handle the worst load. (Or until you confirm that the issue appears the same even when there's only 1 Sidekiq worker!)
Once you've done that, then you'll have a better feel for the contours of the problem: something that looks like an Rmagick memory leak issue when your server is overloaded, may look different (or give you more ideas on how to address it) when you reduce the workload.
Also take a look at this similar SO question about Rmagick and memory leaks; it might be worth forcing garbage collection to limit how much damage any given leak can cause.
I was reading an article on memory fragmentation when I recalled that there are several examples of software that claim to defragment memory. I got curious, how does it work? Does it work at all?
EDIT:
xappymah gave a good argument against memory defragmentation in that a process might be very surprised to learn that its memory layout suddenly changed. But as I see it there's still the possibility of the OS providing some sort of API for global memory control. It does seem a bit unlikely however since it would give rise to the possibility of using it in malicious intent, if badly designed. Does anyone know if there is an OS out there that supports something of the sort?
The real memory defragmentation on a process level is possible only in managed environments such as, for example, Java VMs when you have some kind of an access to objects allocated in memory and can manage them.
But if we are talking about the unmanaged applications then there is no possibility to control their memory with third-party tools because every process (both the tool and the application) runs in its own address space and doesn't have access to another's one, at least without help from OS.
However even if you get access to another process's memory (by hacking your OS or else) and start modifying it I think the target application would be very "surprised".
Just imagine, you allocated a chunk of memory, got it's starting address and on the next second this chunk of memory is moved somewhere else because of "VeryCoolMemoryDefragmenter" :)
In my opinion memory it's a kind of Flash Drive, and this chip don't get fragmented because there aren't turning disks pins recording and playing information, in a random way, like a lie detector. This is the way that Hard Disk Fragmentation it's done. That's why SSD drives are so fast, effective, reliable and maintenance free. SSD it's a BIG piece of memory and it kind of look alike.
This is either ridiculously simple, or too complex . . . .
In our application there is a form that loads some data from the database and displays it in the grid (putting it simply). When the data is refreshed the total memory usage climbs by about 50K (depending on how much data is displayed no doubt). Sounds like a memory leak, but when we shut down the application, FastMM is set with ReportMemoryLeakOnShutDown := True, and it doesn't report any abnormal memory leaks.
So it appears we have a memory bubble or bag. Something that is accumulating more memory each time it is run. Like a TList that keeps getting new items added to it, but the old ones never get removed. Then in the shutdown process all the items get destroyed. The rows displayed in the grid do not increase, but there are a lot of object lists behind the scenes that make this work, so it could be anywhere.
So my question is if anyone knows of a good trick for finding out what parts of an application are using how much memory . . . . I can think of lots of tedious ways of doing it (which I am in the process of doing - checking each list I can find), so I am hoping someone has a trick or technique I have not thought of.
Thanks in advance!
Update: Every refresh results in an additional 10-50K of memory being used. The users are reporting that eventually the application stops responding. It certainly acts like a memory leak, but FastMM (the memory manager) does not see anything leaking. I'll try some other memory tools . . .
Just F8 through the critical part and look at the process usage graph (Process Explorer from Mark Russinovich works great for that). When you find the culprit method, repeat the process but descend into that method.
Tools like AQTime can report difference in memory/object usage between snapshots. This might help you find out what keeps growing.
It looks like there is some memory allocated via custom AllocMem() calls, bypassing FastMM.
This can be midas. Andreas has a solution for this
Or some other InitXXX WinAPI call that allocates something, without freeing. Or some other third-party or windows dll used by project.
Does this happen every time you refresh the data or only the first time? If it's only the first time it could be that the system just reserves the memory for your application, despite the fact that it's not used at this time. (Maybe at some point the old and new data existed simultaneously in memory?)
There are many tools which provide you with informations about memory leaks, have you tried a different one?
Im not a FastMM expert, but I suppose that after a memory manager get memory, after you free the objects/components, it holds for future use with some zeroes or flag, I dont know, avoiding the need to ask the OS for more memory any time, like a cache.
How about you create the same form/open same data, N times in a row?
Will increase 50K each time?
Once I had the same problem. The application was certainly leaking, but I got no report on shutdown. The reason for this was that I had included sharemem in the uses-section of the project.
Have you tried the full FastMM-version? I have found that tweaking its settings gives me a more verbose information of memory usage.
As Lars Truijens mentioned, AQTime provides a live memory consumption graph, so in runtime, you can see what objects are using more memory whenever you refresh data.