I have a ruby service running with unicorn which spawns 20 child worker. When I start my service, unicorn master starts with approx 520 MB of memory and then master spawn 20 child worker. My service runs perfectly for 7-8 day, but gradually unicorn master memory is kept increasing and goes up to 1.3 GB, which apparently is causing OOM error when unicorn master trying to fork a new child (which now need 1.3 GB of memory to fork new child)and due to increased memory footprint, memory was not available.
So my concern here is,
As per my understanding, Unicorn master doesn't serve any request, its
sole purpose is to spawn missing child then why master process memory is increasing?
I have unicorn-worker-killer configured with my app which is working absolutely
fine. If there is any memory leak in my Ruby app then unicorn-worker-killer should already be handling it. Is this assumption wrong? Can there be any memory leak in master process due to my Ruby app?
After killing the child worker, the unicorn is creating new child worker process with memory approx same as master process. Is it true that all child worker which Unicorn master will create over the time will start with the same memory as Unicorn master process?
Here below is my service and machine configuration.
Ruby: 2.1.2
Unicorn: 4.6.3
Unicorn worker count: 20
OS: Debian GNU/Linux 8, Jessie
RAM: 24 GB
Related
We are using Puma with cluster mode where we are using 3 workers with 5 threads. Once the container application is up it starts consuming memory by Puma workers at a high rate and after some time memory utilization by workers slows down but increases step by step.
After the memory is utilized 100% it stops the container and starts again.
This is really a big problem we are facing.
Attaching the Memory utilization Graph for Rails Puma Application.
I ran htop in my production server to see what was eating my RAM. A lot of sidekiq process is running, is this normal?
Press Shift-H. htop shows individual threads as processes by default. There is only one actual sidekiq process.
You seem to have configured it to have 25 workers.
By default, one sidekiq process creates 25 threads.
If that's crushing your machine with I/O, you can adjust it down:
sidekiq -c 10
https://github.com/mperham/sidekiq/wiki/Advanced-Options
If you are not using JRuby then it's likely these all are seperate processes that consume memory.
I have Rails app which uses Sidekiq for background process. To deploy this application I use capistrano, ubuntu server and apache passenger. To start and restart Sidekiq I use capistrano-sidekiq gem.
My problem is - when Sidekiq is running, amount of memory (RAM) used by Sidekiq is growing up. And when Sidekiq finished all processes (workers) it keeps holding a large amount of RAM and not reseting it.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ubuntu 2035 67.6 45.4 3630724 1838232 ? Sl 10:03 133:59 sidekiq 3.5.0 my_app [0 of 25 busy]
How to make Sidekiq to reset used memory after workers finished their work?
Sidekiq uses thread to execute jobs.
And threads share the same memory as the parent process.
So if one job uses a lot of memory the Sidekiq process memory usage will grow up and won't be released by Ruby.
Resque uses another technique it executes every jobs in another process therefore when the job is done, the job's process exits and the memory is released.
One way to prevent your Sidekiq process from using too much memory is to use Resque's forking method.
You could have your job main method executed in another process and wait until that new process exits
ex:
class Job
include Process
def perform
pid = fork do
# your code
end
waitpid(pid)
end
end
Using the sidekiq gem - I have sidekiq worker that runs a process (git-tf clone of big repository) using IO.popen and tracks the stdout to check the progress of the clone.
When I am running the worker, I see that sidekiq memory is getting larger over the time until I get kernel OOM and the process get killed. the subprocess (java process) is taking only 5% of the total memory.
How I can debug/check the memory leak I have in my code? and does the sidekiq memory is the total of my workers memory with the popen process?
And does anyone have any idea how to fix it?
EDIT
This is the code of my worker -
https://gist.github.com/yosy/5227250
EDIT 2
I ran the code without sidekiq, and I have no memory leaks.. this is something strange with sidekiq and big repositories in tfs
I didn't find the cause for the memory leak in sidekiq, but I found a away to get a way from sidekiq.
I have modified git-tf to have server command that accepts command from redis queue, it removes lot of complexity from my code.
The modified version of git-tf is here:
https://github.com/yosy/gittf
I will add documentation later about the sever command when I will fix some bugs.
Suppose I make a little change to my rails app such as changing the html layout. How would I do a rolling restart with Unicorn? Effectively one would like to bring up unicorn processes(or workers instead?) for the newest version of the rails app and then switch traffic from the old unicorn processes/workers to the new ones atomically. From Google searches I couldn't quite get a concrete definitive explanation of how to do this and all the gotchas surrounding it.
There are multiple methods, but one of them is as follows:
Send SIGUSR2 to the master process. Unicorn start a new master with worker processes, that live in parallel with your old master and old worker processes.
Wait until the new master and worker processes have started.
Kill the old master.
Source: http://unicorn.bogomips.org/SIGNALS.html
This is not very memory friendly though. You temporarily need twice the memory usage.
Phusion Passenger Enterprise supports rolling restarts (along with other cool features) but it restarts processes one-by-one and so does not need as much memory. It is possible to script one-by-one rolling restarts in Unicorn using the TTIN and TTOUT signals but Phusion Passenger does everything automatically for you without scripting.