I'm running an application that kicks off a Rufus Scheduler process in an initializer. The application is running with Passenger in production and I've noticed a couple weird behaviors:
First, in order to restart the server and make sure the initializer gets run, you have to both touch tmp/restart.txt and load the app in a browser. At that point, the initializer fires. The horrible thing is that if you only do the touch, the processes scheduled by Rufus get reset and aren't rescheduled until you load the app in a browser.
This alone I can deal with. But this leads to the second problem: I'll notice that the scheduled process hasn't run, so I load a page and suddenly the log file is telling me that it's running the initializers as if I'd rebooted. So, at some point, Passenger is randomly rebooting as if I'd touched tmp/restart.txt and wiping out my scheduled processes.
I have an incredibly poor understanding of Passenger and Rails's integration, so I don't know whether this occasional rebooting is aberrant or all part of the architecture. Can anyone offer any wisdom on this situation?
What you describe is the way Passenger works. It spawns new instances of the application when traffic warrants them, and shuts them down after periods of inactivity to free resources.
You should read the Passenger documentation, particularly the Resource Control and Optimization section. There are settings which can prevent the application from being shut down by Passenger, if that is what you want.
Using the PassengerPoolIdleTime setting, you could keep at least one process running, but you'll almost certainly want Passenger to start up other instances of the app as necessary. This thread on the Rufus Scheduler Google Group mentions using lock files to prevent more than one process from starting the scheduler, that may be useful to you.
Related
How can I do real-time profiling of Rails when it's in a development environment?
I've got a large task being run (it isn't done in the background on the development machine, but that wouldn't help anyway because I need to have the task run before I can interact with the web application further), and the task is taking so long I haven't yet seen it finish. So I assume something like ruby-prof probably won't work, because that relies upon there being a start, and a finish, and calculating how much time was taken up in the meantime by what methods.
Control-C profiling won't work either, because the Rails application doesn't respond to control-C, and doing kill -9 kills it, but doesn't produce a stacktrace.
I'd like something that can display how much time was spent in what methods (or in garbage collection) in the past few seconds or the past few minutes, preferably without killing the Rails process in the process.
I could try reducing the size of the task being run, but that feels like a hack.
I came across https://stackoverflow.com/questions/28370195/rails-real-time-profiler , but that's for applications in production, and doesn't have any answers.
I'm not able to run the application in production mode on my local machine.
I am running Apache, Phusion Passenger, Ruby on Rails website. I have a critical section that I can only allow 1 execution at the same time. I noticed that phusion passenger uses fork() to spawn the processes for each request according to this documentation. Since they are individual process, I noticed that all my Mutex.synchronize do not work. I added some loggings inside the critical section, and I still noticed logs being printed interleaving.
I tried all these:
Mutex.synchronize
Thread.exclusive
Monitor.synchronize
None of them works.
How do you do synchronization in this case? Thanks a lot.
Mutex, Monitor and Thread.exclusive only work within a single process. For inter-process synchronization, use lock files. See File#flock.
I serve my software using passenger. It spawns many ruby processes.
Sometimes one of these rubies becomes bloated and I want it to die.
I was hoping to use god to that intent. My idea was to monitor all these rubies and if it is consuming more than 500MB of memory for 3 cycles, god should try to gracefuly kill it. If it remains alive for more than 5 minutes then god should kill it not gracefully.
It seems to me that god always tries to run the service again, so it forces us to provide a start command. Is it possible to use god only to kill bad behaviored processes and let the passenger spawner to bring them back to live when necessary?
Answer to your question lies in question itself. you can kill ruby processes using god gem which is ruby process process monitor framework by github guys.
basically, here is how it works:
configure god to monitor process it can be anything from apache,passenger,mongrel or just simple file doing a long-running task.
Set conditionals in god's configuration file based upon which god will execute some predefined code.
here is a simple example(taken from docs). consider this as file long running process that runs undefiantly which we want to monitor for memory usage, lets call it simple.rb
loop do
puts 'Hello'
sleep 1
end
now, we install the god gem & configure it to as run as superuser so it can kill/spawn processes and next create a configuration file. example(also taken from docs):
God.watch do |w|
w.name = "simple"
w.start = "ruby /full/path/to/simple.rb"
w.keepalive(:memory_max => 500.megabytes)
end
Here, as you may have got the idea if the process memory usage goes above 500 megabytes, god will restart it. here are few resources that might help, if you are getting started with process management using god gem:
Example gist - Passenger worker monitor to kill workers which use too much RAM(Don't use god, but spawns a new passenger worker instead)
Project Homepage
Github Page
An indepth tutorial using god with rails & passenger
Now, please remember ALL configuration for god is actually legal ruby code so you can get creative & do all sorts of things.
lastly, if you are frequently finding yourself running long running process, I advice you to try JRuby which is works much better with long running processes due to JVM & LOT faster than MRI
I use the same setup on many of my projects and had the same memory leaking issues. After messing around with monitoring, we decided to use the passenger features to tackle it. Specifically it allows the setting (e.g.) PassengerMaxRequests 300 which shuts down any instance when it has served that number of requests.
If you use it, make sure that PassengerMinInstances is set to 0 because it preceedes the setting for max requests.
Is it possible to change the Passenger application pool size at runtime? Ie, without restarting apache, and without disrupting active visitors?
The same time every day we have a background job run. It is very memory intensive. Since during that time, traffic on the site tends to be relatively low, I would like to automatically scale down the number of application servers running just before the jobs runs, and then scale up again when it is finished.
Passenger should automatically shut down instances when they are not in use. Since your traffic is low then you should not have any unnecessary passenger instances running.
One variable you can tweak is PassengerPoolIdleTime. This tells passenger how long to wait before shutting down idle instances.
There is no PoolSize variable. There is however a MaxPoolSize but this limit should not be hit unless you are receiving heavy traffic.
You can find all the variables along with what they do here: http://www.modrails.com/documentation/Users%20guide.html
If you really want to change the settings dynamically, you can try calling sudo /etc/init.d/apache2 reload. This will tell apache to reload its settings, including the passenger configuration.
Good luck!
To jump on the band-wagon of Phusion Passenger we've setup a staging server for a small rails app to test things out.
So far it has been very nice to use, it makes installing/configuring and deploying apps a breeze. The problem is the site we're using doesn't get hit very often and it seems to shut down the servers in the background. Meaning when someone goes to the site they have a really long wait until it starts up a new server to handle the request. We've read through the documentation, tried quite a few different set-ups (smart/smart-lv2 modes, passengeridletime etc) and still haven't found a real solution.
After ploughing through Google results we can't really find useful information. Currently we have a cron job that makes a request every-so-often in an attempt to keep the servers running.
Is anyone else experiencing this problem and do you have any advice for a fix?
What's happening is that your Application and/or ApplicationSpawners are shutting down due to time-out. To process your new request, Passenger has to startup a new copy of your application, which can take several seconds, even on a fast machine. To fix the issue, there are a few Apache configuration options you can use to keep your Application alive.
Here's specifically what I've done on my servers. The PassengerSpawnMethod and PassengerMaxPreloaderIdleTime are the configuration options most important in your situation.
# Speeds up spawn time tremendously -- if your app is compatible.
# RMagick seems to be incompatible with smart spawning
# Older versions of Passenger called this RailsSpawnMethod
PassengerSpawnMethod smart
# Keep the application instances alive longer. Default is 300 (seconds)
PassengerPoolIdleTime 1000
# Keep the spawners alive, which speeds up spawning a new Application
# listener after a period of inactivity at the expense of memory.
# Older versions of Passenger called this RailsAppSpawnerIdleTime
PassengerMaxPreloaderIdleTime 0
# Just in case you're leaking memory, restart a listener
# after processing 5000 requests
PassengerMaxRequests 5000
By using "smart" spawning mode and turning off PassengerMaxPreloaderIdleTime, Passenger will keep 1 copy of your application in memory at all times (after the first request after starting Apache). Individual Application listeners will be forked from this copy, which is a super-cheap operation. It happens so quickly you can't tell whether or not your application has had to spawn a listener.
If your app is incompatible with smart spawning, I'd recommend keeping a large PassengerPoolIdleTime and hitting your site periodically using curl and a cronjob or monit or something to ensure the listener stays alive.
The Passenger User Guide is an awesome reference for these and more configuration options.
edit:
If your app is incompatible with smart spawning, there are some new options that are very nice
# Automatically hit your site when apache starts, so that you don't have to wait
# for the first request for passenger to "spin up" your application. This even
# helps when you have smart spawning enabled.
PassengerPreStart http://myexample.com/
PassengerPreStart http://myexample2.com:3500/
# the minimum number of application instances that must be kept around whenever
# the application is first accessed or after passenger cleans up idle instances
# With this option, 3 application instances will ALWAYS be available after the
# first request, even after passenger cleans up idle ones
PassengerMinInstances 3
So, if you combine PassengerPreStart and PassengerMinInstances, Passenger will spin up 3 instances immediately after apache loads, and will always keep at least 3 instances up, so your users will rarely (if ever) see a delay.
Or, if you're using smart spawning (recommended) with PassengerMaxPreloaderIdleTime 0 already, you can add PassengerPreStart to get the additional benefit of immediate startup.
Many thanks to the heroes at phusion.nl!
Just incase there are any nginx server users stumbling upon this question, both the 'PassengerMaxRequests' and 'PassengerStatThrottleRate' directives don't translate to nginx. However the others do:
rails_spawn_method smart;
rails_app_spawner_idle_time 0;
rails_framework_spawner_idle_time 0;
passenger_pool_idle_time 1000;
HTH!
EDIT rails_spawn_method is deprecated in passenger 3
instead use
passenger_spawn_method smart;
everything else is just good till date.
You can also use PassengerMinInstances:
http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerMinInstances
This can be combined with PassengerPreStart
RE:
# Additionally keep a copy of the Rails framework in memory. If you're
# using multiple apps on the same version of Rails, this will speed up
# the creation of new RailsAppSpawners. This isn't necessary if you're
# only running one or 2 applications, or if your applications use
# different versions of Rails.
RailsFrameworkSpawnerIdleTime 0
Just something to add and might be useful.
The default spawn method in the current release
is "smart-lv2", which skips the framework spawner, so setting
the framework spawner timeout wouldn't have effect anyway unless you
explicitly set the spawn method to "smart".
Source:
http://groups.google.com/group/phusion-passenger/browse_thread/thread/c21b8d17cdb073fd?pli=1
If your host is a shared server, like mine, you can't change the settings and are stuck with a cron job.
I had also this problem but I was not able to change the passenger settings because I had no write permission to this file. I found a tool ( http://www.wekkars.com ) that keeps my app responding fast. Maybe this can also be a solution for you.
check the version of passenger. it was RailsSpawnMethod <string> for old versions.
If so (if I remember correctly), replace Passenger with Rails in all the configuration directives or look for old passenger docs for more details