I have a ruby on rails app, where we validate records from huge excel files(200k records) in background via sidekiq. We also use docker and hence a separate container for sidekiq. When the sidekiq is up, memory used is approx 120Mb, but as the validation worker begins, the memory reaches upto 500Mb (that's after a lot of optimisation).
Issue is even after the job is processed, the memory usage stays at 500Mb and is never freed, not allowing any new jobs to be added.
I manually start garbage collection using GC.start after every 10k records and also after the job is complete, but still no help.
This is most likely not related to Sidekiq, but to how Ruby allocates from and releases memory back to the OS.
Most likely the memory can not be released because of fragmentation. Besides optimizing your program (process data chunkwise instead of reading it all into memory) you could try and tweak the allocator or change the allocator.
There has been a lot written about this specific issue with Ruby/Memory, I really like this post by Nate Berkopec: https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html which goes into all the details.
The simple "solution" is:
Use jemalloc or, if not possible, set MALLOC_ARENA_MAX=2.
The more complex solution would be to try and optimize your program further, so that it does not load that much data in the first place.
I was able to cut memory usage in a project from 12GB to < 3GB by switching to jemalloc. That project dealt with a lot of imports/exports and was written quite poorly and it was an easy win.
Related
I'm pretty new to Ruby, Rails, and everything else in that ecosystem. I've joined a team that has a Ruby 3.1.2 / Rails 6.1.7 app backed by a Postgres database.
We have a scenario where, sometimes, memory usage on one of our running instances jumps up significantly and is never relinquished (we've waited days). Until today, we didn't know what was causing it or how to reproduce it.
It turns out that it's caused by an internal tool which was running an unbounded ActiveRecord query -- no limit and no paging. When pointing this tool at a more active customer, it takes many seconds, returns thousands of records, and memory usage increases by tens of MB. No amount of waiting will lead to the memory usage going back down again.
Having discovered that, we recently added paging to that particular tool, and in the ~week since, we have not seen usage increasing in giant chunks anymore. However, there are other scenarios which have similar behavior but with smaller payloads; these cause memory usage to increase gradually over time. We deploy this application often enough that it hasn't been a big deal, but I am looking to gain a better understanding of what's happening and to determine if there's a problem here, because that's not what we should see from a stable application that's free of memory leaks.
My first suspicion was a memoized instance variable on the controller, but a quick check indicates that Rails controllers are discarded as soon as the request finishes processing, so I don't think that's it.
My next suspicion was that ActiveRecord was caching my resultset, but I've done a bunch of research on how this works and my understanding is that any cached queries/relations should be released when the request completes. Even if I have that wrong, a subsequent identical request takes just as long and causes another jump in memory usage, so either that's not it, or caching is broken on our system.
My Google searches turn up lots of results about various caching capabilities in Rails 2, 3, and 5 -- but not much about 6.x, so maybe something significant has changed and I just haven't found it.
I did find ruby-prof, memory-profiler, and get_process_mem -- these all seem like they are only suitable for high-level analysis and wouldn't help me here.
Can I explore the contents of the object graph currently in memory on an existing, live instance of my app? I'm imagining that this would happen in the Rails console, but that's not a constraint on the question. If not, is there some other way that I could find out what is currently in memory, and whether it's just a bunch of fragmented pages or if there's actually something that isn't getting garbage collected?
EDIT
#engineersmnky pointed out in the comments that maybe everything is fine and that perhaps Ruby is just still holding on to the OS page due to some other still-valid object therein. However, if this is the case, it strikes me as unlikely that memory usage would not go back down to the previous baseline after several days of production usage.
Loading tens of MB worth of resultset into memory should result in the allocation of >1000 16kb memory pages in just a handful of seconds. It seems reasonable to assume that the vast majority of those would contain exclusively this resultset, and could therefore be released as soon as the resultset is garbage collected.
Furthermore, I can reproduce the increased memory usage by running the same unbounded ActiveRecord query in the Rails console, and when I close that console, the memory goes down almost immediately -- exactly what I was expecting to see when the web request completes. I don't fully understand how the Rails console works when connecting to a running application, though, so this may not be relevant.
I am facing some strange issue with Sidekiq, I have a few heavy jobs running in the background with Sidekiq. but even after Sidekiq finished the job it's still holding the memory. what could be the reason?
Versions
ruby : 2.2.4
rails : 4.2.7
sidekiq : 3.5.4
I have attached the memory log also.
Even I haved check the link but it couldn't help. I have even manually started the GC also.
My understanding is that Ruby MRI will not release memory back to the OS. If your Sidekiq job consumes a lot of memory, even if those objects are garbage collected, the memory will just be released back to Ruby, not back to the OS. You should try to find a way to make your Sidekiq job(s) consume less memory, and assume that your workers will eventually allocate the maximum amount of memory that your most memory-consuming job requires.
Hi we were also facing the same problem. I researched about it a lot. Jim said it right. It's Ruby MRI which is handling the resources. Actually your sidekiq workers are heavy or they're performing heavy operations and I guess it's making more object allocations as well. They more the resources it needs ruby takes the resources from the OS and then use it for the operations being performed in sidekiq. It doesn't release memory to the OS. It uses the same memory space and reuses it to provide resources to other objects.
The things you have to keep in mind to stay out of this problem is to optimise your code.
If you're performing multiple heavy operations and they can be divided then use separate workers for that.
You can run your fetch query in batches to decrease the load on the server.
Use array operations whenever they're necessary.
you can run GC.start to force run Garbage collector after the end of your job to release the memory as well.
when you kill a sidekiq process it releases the memory to OS because process is being run by ruby and you killed it so in the end ruby will release memory to the OS.
I hope this helps.
You'll have to optimise your code and this is the same suggestion which is given by mperham.
People above have already delineated the problem - MRI will not release the memory that Sidekiq allocated after a huge job is completed, and also a possible solution - run GC.start manually after you finish your job.
There are scenarios where you might have a big job that's split into smaller ones and that stills generates a lot of memory. I will kindly suggest an approach here:
The SmallerJobs have a way to understand whether their counterparts have finished - maybe a flag on your Cache Engine** of choice:
class BigJob
def perform(record_ids)
Rails.cache.write("big_job/in_progress", record_ids)
record_ids.map { SmallerJob.perform_later(_1) }
end
end
class SmallerJob
def perform(record_id)
jobs_in_progress = Rails.cache.read("big_job/in_progress/")
# perform you action against the record
jobs_in_progress -= [ record_id ]
Rails.cache.write("big_job/in_progress/", jobs_in_progress)
ForceGCJob.perform_later if jobs_in_progress.empty?
end
end
After you're done, you queue a ForceGCJob, which as the name says, simply does this:
def perform
logger.debug(GC.stats) # if you're curious
GC.start
end
** concurrent writes to the cache might not be Serializable and you might lose removing some of them due to race conditions, this is just a simple example. In one scenario, I have a Database column for each record, which guarantees these operations are ACID
I'm using Puma server and DelayedJob.
It seems that the memory taken by each job isn't released and I slowly get a bloat causing me to restart my dyno (Heroku).
Any reason why the dyno won't return to the same memory usage figure before the job was performed?
Any way to force releasing it? I tried calling GC but it doesn't seem to help.
You can have one of the following problems. Or actually all of them:
Number 1. This is not an actual problem, but a misconception about how Ruby releases memory to operating system. Short answer: it doesn't. Long answer: Ruby manages an internal list of free objects. Whenever your program needs to allocate new objects, it will get those objects from this free list. If there are no more objects there, Ruby will allocate new memory from operating system. When objects are garbage collected they go back to the free list. So Ruby still have the allocated memory. To illustrate it better, imagine that your program is normally using 100 MB. When at some point program will allocate 1 GB, it will hold this memory until you restart it.
There are some good resource to learn more about it here and here.
What you should do is to increase your dyno size and monitor your memory usage over time. It should stabilize at some level. This will show you your normal memory usage.
Number 2. You can have an actual memory leak. It can be in your code or in some gem. Check out this repository, it contains information about well known memory leaks and other memory issues in popular gems. delayed_job is actually listed there.
Number 3. You may have unoptimized code that is using more memory than needed and you should try to investigate memory usage and try to decrease it. If you are processing large files, maybe you should do it in smaller batches etc.
I have a website hosted on a Heroku Dyno that allows max 512MB of memory.
My site allows users to upload raw time series data in CSV format, and I wanted to load test the performance of uploading a CSV with ~100k rows (3.2 MB in size). The UI lets the user upload the file, which in turns kicks of a Sidekiq job to import each row in the file into my database. It stores the uploaded file under /tmp storage on the dyno, which I believe gets cleared on each periodic restart of the dyno.
Everything actually finished without error, and all 100k rows were inserted. But several hours later I noticed my site was almost unresponsive and I checked Heroku metrics.
At the exact time I had started the upload, the memory usage began to grow and quickly exceeded the maximum 512MB.
The logs confirmed this fact -
# At the start of the job
Aug 22 14:45:51 gb-staging heroku/web.1: source=web.1 dyno=heroku.31750439.f813c7e7-0328-48f8-89d5-db79783b3024 sample#memory_total=412.68MB sample#memory_rss=398.33MB sample#memory_cache=14.36MB sample#memory_swap=0.00MB sample#memory_pgpgin=317194pages sample#memory_pgpgout=211547pages sample#memory_quota=512.00MB
# ~1 hour later
Aug 22 15:53:24 gb-staging heroku/web.1: source=web.1 dyno=heroku.31750439.f813c7e7-0328-48f8-89d5-db79783b3024 sample#memory_total=624.80MB sample#memory_rss=493.34MB sample#memory_cache=0.00MB sample#memory_swap=131.45MB sample#memory_pgpgin=441565pages sample#memory_pgpgout=315269pages sample#memory_quota=512.00MB
Aug 22 15:53:24 gb-staging heroku/web.1: Process running mem=624M(122.0%)
I can restart the Dyno to clear this issue, but I don't have much experience in looking at metrics so I wanted to understand what was happening.
If my job finished in ~30 mins, what are some common reasons why the memory usage might keep growing? Prior to the job it was pretty steady
Is there a way to tell what data is being stored in memory? It would be great to do a memory dump, although I don't know if it will be anything more than hex address data
What are some other tools I can use to get a better picture of the situation? I can reproduce the situation by uploading another large file to gather more data
Just a bit lost on where to start investigating.
Thanks!
Edit: - We have the Heroku New Relic addon which also collects data. Annoyingly enough, New Relic reports a different/normal memory usage value for that same time period. Is this common? What's it measuring?
There are most probable reasons for that:
Scenario 1. You process the whole file, first by loading every record from CSV to memory, doing some processing and then iterating over it and storing into database.
If that's the case then you need to change your implementation to process this file in batches. Load 100 records, process them, store in the database, repeat. You can also look at activerecord-import gem to speed up your inserts.
Scenario 2. You have memory leak in your script. Maybe you process in batches, but you hold references to unused object and they are not garbage collected.
You can find out by using ObjectSpace module. It has some pretty useful methods.
count_objects will return hash with counts for different object currently created on the heap:
ObjectSpace.count_objects
=> {:TOTAL=>30162, :FREE=>11991, :T_OBJECT=>223, :T_CLASS=>884, :T_MODULE=>30, :T_FLOAT=>4, :T_STRING=>12747, :T_REGEXP=>165, :T_ARRAY=>1675, :T_HASH=>221, :T_STRUCT=>2, :T_BIGNUM=>2, :T_FILE=>5, :T_DATA=>1232, :T_MATCH=>105, :T_COMPLEX=>1, :T_NODE=>838, :T_ICLASS=>37}
It's just a hash so you can look for specific type of object:
ObjectSpace.count_objects[:T_STRING]
=> 13089
You can plug this snippet in different points in your script to see how many objects are on the heap at specific time. To have consistent results you should manually trigger garbage collector before checking the counts. It will ensure that you will see only live objects.
GC.start
ObjectSpace.count_objects[:T_STRING]
Another useful method is each_object which iterates over all objects actually on the heap:
ObjectSpace.each_object { |o| puts o.inspect }
Or you can iterate over objects of one class:
ObjectSpace.each_object(String) { |o| puts o.inspect }
Scenario 3. You have memory leak in a gem or system library.
This like previous scenario, but the problem lies not in your code. You can find this also by using ObjectSpace. If you see there are some objects retained after calling library method, there is a chance that this library may have a memory leak. The solution would be to update such library.
Take a look at this repo. It maintains the list of gems with known memory leak problems. If you have something from this list I suggest to update it quickly.
Now addressing your other questions. If you have perfectly healthy app on Heroku or any other provider, you will always see memory increase over time, but it should stabilise at some point. Heroku is restarting dynos once a day or so. On your metrics you will see sudden drops and the slow increase over span of 2 days or so.
And New Relic by default shows average data from all instances. You should probably switch to showing data only from your worker dyno to see correct memory usage.
At the end I recommend to read this article about how Ruby uses memory. There are many useful tools mentioned there, derailed_benchmarks in particular. It was created by guy from Heroku (at that time) and it is a collection of many benchmarks related to most common problems people have on Heroku.
I'm using ejabberd + mochiweb on our server. The longer I keep ejabberd and mochiweb running, the more memory is consumed (last night it was consuming 35% of memory. right now it's a bit above 50%). I thought this was just a mnesia garbage collection issue - so I installed Erlang R13B3 and restarted ejabberd. This didn't fix it though.
So I'm noticing now that at a bit above 50% of full memory consumption, it looks like ejabberd's starting to "let go" of memory and stay at around 50%. Is this normal? Is ~50% a threshold for ejabberd, so that when it reaches it it says, "hey time to actually let some memory go..." and maybe it keeps the rest around for quick access (like caching mnesia?)
I appreciate any input. Thanks!
Run erlang:memory(). in your shell every now and then. You can also give erlang:system_info(Type). with allocated_areas and allocator a try.
These should give you a hint on what kind of memory is leaking.
You can also setup memsup to warn you about processes allocating too much memory.
Turns out, there is no memory leak (yay!) Ejabberd is taking up only < 40MB. Finally I saw the light when I saw the Usage Graphs on EngineYard - only 288MB is actually being used, 550MB is being buffered, and 175MB is being cached. My ejabberd server an update every 2.5 seconds from each client so that may explain why so much is being buffered/cached.
Thanks for all of your help.
Newly created atoms in erlang processes get never garbage collected. This might be an issue when processes are registered by an algorith that creates atom names from random eg. randomly created strings.