We're running 3 Apache Passenger servers sharing the same file system, each running 11 Rails applications.
We've set
PassengerPoolIdleTime = 0 to ensure none of the applications ever die out completely, and
PassengerMaxPoolSize = 20 to ensure we have enough processes to run all the applications.
The problem is that when I run passenger-memory-stats on one of the servers I see 210 VM's!
And when I run passenger-status I see 20 application instances (as expected)!
Anyone know what's going on? How do I figure out which of those 210 instances are still being used, and how do I kill those on a regular basis? Would PassengerMaxInstancesPerApp do anything to reduce those seemingly orphaned instances?
Turns out we actually have that many Apache worker processes running, and only 24 of them are Passenger processes (asked someone a little more experienced than myself). We are actually hosting many more websites and shared hosting accounts than I thought.
Thanks for all the replies though!
You can get a definitive answer as to how many Rails processes you have by running this command:
ps auxw | grep Rails | wc -l
I doubt you really do have over 100 processes running though, as at about 50 mb each they'd collectively be consuming over 5 gb of RAM and/or swap and your system would likely have slowed to a crawl.
Not so much an answer, but some useful diagnostic advice.
Try adding the Rack::ProcTitle middleware to your stack. We have been using it in production for years. Running ps aux should give info on what the worker is up to (idle, handing a specific request etc…).
I would not assume that these processes are being spawned directly by passenger. It could be something forking deeper into your app.
Maybe the processes are stuck during shutdown. Try obtaining their backtraces to see what they're doing.
Related
I ran htop in my production server to see what was eating my RAM. A lot of sidekiq process is running, is this normal?
Press Shift-H. htop shows individual threads as processes by default. There is only one actual sidekiq process.
You seem to have configured it to have 25 workers.
By default, one sidekiq process creates 25 threads.
If that's crushing your machine with I/O, you can adjust it down:
sidekiq -c 10
https://github.com/mperham/sidekiq/wiki/Advanced-Options
If you are not using JRuby then it's likely these all are seperate processes that consume memory.
I have a Rails application which is using Puma. I'm using nginx for load balancing. I would like to dockerize and deploy to a DigitalOcean (Docker) droplet.
After reading lots of blogs and examples (most of which are a year old and that's a long time in the Docker world), I'm still confused about 2 things. Let's say that I select a DigitalOcean box with 4 CPUs. How am I supposed to set up the Rails containers? Should I set up 4 different containers, where Puma is configured with 1 worker process? Or should I set up 1 container where Puma is configured with 4 worker processes?
And the second thing I'm confused about: should I run nginx inside the Rails container, or should I run them in separate containers?
These 2 questions allow 4 permutations that I diagramed below.
option 1
option 2
option 3
option 4
Docker likes to push the single process per container style of design. When running multiple processes in a single container there is the extra layer of a service manager in between Docker and the underlying processes which causes Docker to lose visibility of the real service status. This more often than not makes services harder to manage with Docker and it's associated tools. Puma managing workers will not be as bad as a generic service manager running multiple processes.
You may also need to consider the next step in the application, hosting across multiple droplets/hosts and how to easy it will be to move to that next step.
Option 1 and 3 follow Dockers preferred design. If you are using MRI, Puma can run in clustered mode so it just depends on whether you want to manage the Ruby processes yourself (1) or have Puma do the worker management (3). There will be differences between how nginx and Puma distribute requests between workers. Puma can also schedule zero down time updates which would require a bit of effort to get working via Docker. If you are using Rubinius or JRuby you would probably lean towards option 3 and let threads do the work.
Option 1 may allow you to more easily scale across different sized hosts with Docker tools.
Option 2 looks like it adds an unnecessary application hop and Docker no longer maintains the service state in your app tier as you need something else in the container to launch both nginx and Puma.
Option 4 might perform a bit better than others due to the local sockets, but again Docker is no longer aware of the service state.
In any case try a couple of solutions and benchmark both with something like JMeter. You will quickly get an idea of what works and what doesn't, both in performance and management.
I'm running a rails app (dev server) with passenger in Amazon AWS with t2.micro instance. But i'm getting -bash: fork: Cannot allocate memory error constantly.
I'm running redis server on it with 50 sidekiq concurrency. Normally sites runs fine but when i start 2-3 sidekiq process simultaneously do do some batch process. The site take take time to redirect and eventylly crash with
502 Bad Gateway
nginx/1.10.0
Then i have to nginx restart every to to get the site running again. This is my dev server so i don't want to put more financial resources for upgrading to t2.small (as of now, this is our last option) as this is dev servre and will be using twice in 15 days. Is there any way i can solve this otherwise? Previously i had same 120 concurrency as production but then i changed to 50. That help a bit but still memory problems.
here are few stats with htop
This stats are while the server is idle. But when i run few task with sidekiq it crashing with 502.
I check few post suggesting swap memory but not sure this is preferable with t2.micro. Is this advisable for this server setup. Here in pic you can see i don't swap memory. Is it okay yo add swap memory to tackle this issue or there is any other better option.
Your server has a lack of memory, to fix it:
or: buy more operative memory
or: mount a swap
Then try again
In my case, redis used 2.5G memory, the server has 4.5G in total, and 3G is used.
1.5G free.
and Redis kept throwing this error.
solution:
add vm.overcommit_memory=1 to file: /etc/sysctl.conf
sudo sysctl -p /etc/sysctl.conf
refer to: redis bgsave failed because fork Cannot allocate memory
Using heroku logs --tail which works great for a few minutes. Then it stops displaying the logs. It seems that the ssh connection is timing out and dying. There is no error or message. Working in Ubuntu 11.04 on wired conneciton.
I added the following to ~/.ssh/config:
ServerAliveInterval 5
But it didn't work. Do I need anything else in the config file? How do I know if it is doing anything? How can I monitor the traffic and see the keepalive request? I am looking at System Monitor but don't see anything every 5 seconds.
Thanks.
Have you done all of this:
$ heroku config:add LOG_LEVEL=DEBUG
$ heroku addons:upgrade logging:expanded
$ heroku logs --tail
It turns out that I was looking for an answer to the wrong question. Why use the tail to save logs? It is problematic, labor intensive, and error prone.
The solution I found was Papertrail. Small sites are free. papertrailapp.com.
Here's the complete story from my blog: http://www.onlineinvestingai.com/blog/2011/08/07/better-logging-with-papertrail-on-heroku/
I have seen the same issue. I'm not certain that it is ssh that times out but something does. For now, we've placed our monitor in a loop so it resumes automatically in case of time-out.
We also use PaperTrail, however it has limits in the amount that you can transfer. We use PaperTrail for general purpose use, and tail the logs for the excrutiating-detail logs that quickly use up all avaiable PaperTrail capacity.
My Rails application has a number of tasks which are offloaded into background processes, such as image resizing and uploading to S3. I'm using delayed_job to manage these processes.
These processes, particularly thumbnailing PDFs (using Ghostscript) and resizing images (using ImageMagick), are CPU intensive and often consume 100% CPU time. Since these jobs are running on the same (RedHat Linux) server as the web application itself, as well as the DB, they can lead to our web application being unresponsive.
One solution is to get another server on which to run only the background jobs. I guess this would be the optimal solution? However, since this isn't something I can do immediately I wonder whether it would be possible to somehow make the background jobs run at a lower operating system priority, and hence consume less CPU cycles in doing their work?
Thoughts appreciated.
If I'm not mistaken, delayed_job uses worker processes that will handle all the background jobs. It should be easily possible to alter the OS scheduling priority of the process when you start it.
So instead of, for example:
ruby script/delayed_job -e production -n 2 start
try:
nice -n 15 ruby script/delayed_job -e production -n 2 start