To jump on the band-wagon of Phusion Passenger we've setup a staging server for a small rails app to test things out.
So far it has been very nice to use, it makes installing/configuring and deploying apps a breeze. The problem is the site we're using doesn't get hit very often and it seems to shut down the servers in the background. Meaning when someone goes to the site they have a really long wait until it starts up a new server to handle the request. We've read through the documentation, tried quite a few different set-ups (smart/smart-lv2 modes, passengeridletime etc) and still haven't found a real solution.
After ploughing through Google results we can't really find useful information. Currently we have a cron job that makes a request every-so-often in an attempt to keep the servers running.
Is anyone else experiencing this problem and do you have any advice for a fix?
What's happening is that your Application and/or ApplicationSpawners are shutting down due to time-out. To process your new request, Passenger has to startup a new copy of your application, which can take several seconds, even on a fast machine. To fix the issue, there are a few Apache configuration options you can use to keep your Application alive.
Here's specifically what I've done on my servers. The PassengerSpawnMethod and PassengerMaxPreloaderIdleTime are the configuration options most important in your situation.
# Speeds up spawn time tremendously -- if your app is compatible.
# RMagick seems to be incompatible with smart spawning
# Older versions of Passenger called this RailsSpawnMethod
PassengerSpawnMethod smart
# Keep the application instances alive longer. Default is 300 (seconds)
PassengerPoolIdleTime 1000
# Keep the spawners alive, which speeds up spawning a new Application
# listener after a period of inactivity at the expense of memory.
# Older versions of Passenger called this RailsAppSpawnerIdleTime
PassengerMaxPreloaderIdleTime 0
# Just in case you're leaking memory, restart a listener
# after processing 5000 requests
PassengerMaxRequests 5000
By using "smart" spawning mode and turning off PassengerMaxPreloaderIdleTime, Passenger will keep 1 copy of your application in memory at all times (after the first request after starting Apache). Individual Application listeners will be forked from this copy, which is a super-cheap operation. It happens so quickly you can't tell whether or not your application has had to spawn a listener.
If your app is incompatible with smart spawning, I'd recommend keeping a large PassengerPoolIdleTime and hitting your site periodically using curl and a cronjob or monit or something to ensure the listener stays alive.
The Passenger User Guide is an awesome reference for these and more configuration options.
edit:
If your app is incompatible with smart spawning, there are some new options that are very nice
# Automatically hit your site when apache starts, so that you don't have to wait
# for the first request for passenger to "spin up" your application. This even
# helps when you have smart spawning enabled.
PassengerPreStart http://myexample.com/
PassengerPreStart http://myexample2.com:3500/
# the minimum number of application instances that must be kept around whenever
# the application is first accessed or after passenger cleans up idle instances
# With this option, 3 application instances will ALWAYS be available after the
# first request, even after passenger cleans up idle ones
PassengerMinInstances 3
So, if you combine PassengerPreStart and PassengerMinInstances, Passenger will spin up 3 instances immediately after apache loads, and will always keep at least 3 instances up, so your users will rarely (if ever) see a delay.
Or, if you're using smart spawning (recommended) with PassengerMaxPreloaderIdleTime 0 already, you can add PassengerPreStart to get the additional benefit of immediate startup.
Many thanks to the heroes at phusion.nl!
Just incase there are any nginx server users stumbling upon this question, both the 'PassengerMaxRequests' and 'PassengerStatThrottleRate' directives don't translate to nginx. However the others do:
rails_spawn_method smart;
rails_app_spawner_idle_time 0;
rails_framework_spawner_idle_time 0;
passenger_pool_idle_time 1000;
HTH!
EDIT rails_spawn_method is deprecated in passenger 3
instead use
passenger_spawn_method smart;
everything else is just good till date.
You can also use PassengerMinInstances:
http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerMinInstances
This can be combined with PassengerPreStart
RE:
# Additionally keep a copy of the Rails framework in memory. If you're
# using multiple apps on the same version of Rails, this will speed up
# the creation of new RailsAppSpawners. This isn't necessary if you're
# only running one or 2 applications, or if your applications use
# different versions of Rails.
RailsFrameworkSpawnerIdleTime 0
Just something to add and might be useful.
The default spawn method in the current release
is "smart-lv2", which skips the framework spawner, so setting
the framework spawner timeout wouldn't have effect anyway unless you
explicitly set the spawn method to "smart".
Source:
http://groups.google.com/group/phusion-passenger/browse_thread/thread/c21b8d17cdb073fd?pli=1
If your host is a shared server, like mine, you can't change the settings and are stuck with a cron job.
I had also this problem but I was not able to change the passenger settings because I had no write permission to this file. I found a tool ( http://www.wekkars.com ) that keeps my app responding fast. Maybe this can also be a solution for you.
check the version of passenger. it was RailsSpawnMethod <string> for old versions.
If so (if I remember correctly), replace Passenger with Rails in all the configuration directives or look for old passenger docs for more details
Related
I've setup my website with Rails 3 and Passenger (via nginx) and, although its only being used by one person, the web server has to essentially wake up the rails instance to render the page. This happens only when the website is not accessed for a while (hence its sleeping), but I'm a little paranoid that could it still lag like so when the website is operating on a production level (don't get mixed with the development/production mode, the sleeping website is running in production mode when I'm checking it out).
Any ideas? Or this just a sleep and wakeup thing when nobody's using the website.
There is an easy fix just edit your nginx.conf and set passenger_min_instances to a value larger then zero. This way passenger always keeps one instance alive; this will prevent the "lag" as you describe it. Read more about it in the Passenger Nginx documentation.
Take a look at passenger_pool_idle_time. It states the maximum number of seconds that an application instance may be idle. That is, if an application instance hasn’t received any traffic after the given number of seconds, then it will be shutdown in order to conserve memory.
I'm running an application that kicks off a Rufus Scheduler process in an initializer. The application is running with Passenger in production and I've noticed a couple weird behaviors:
First, in order to restart the server and make sure the initializer gets run, you have to both touch tmp/restart.txt and load the app in a browser. At that point, the initializer fires. The horrible thing is that if you only do the touch, the processes scheduled by Rufus get reset and aren't rescheduled until you load the app in a browser.
This alone I can deal with. But this leads to the second problem: I'll notice that the scheduled process hasn't run, so I load a page and suddenly the log file is telling me that it's running the initializers as if I'd rebooted. So, at some point, Passenger is randomly rebooting as if I'd touched tmp/restart.txt and wiping out my scheduled processes.
I have an incredibly poor understanding of Passenger and Rails's integration, so I don't know whether this occasional rebooting is aberrant or all part of the architecture. Can anyone offer any wisdom on this situation?
What you describe is the way Passenger works. It spawns new instances of the application when traffic warrants them, and shuts them down after periods of inactivity to free resources.
You should read the Passenger documentation, particularly the Resource Control and Optimization section. There are settings which can prevent the application from being shut down by Passenger, if that is what you want.
Using the PassengerPoolIdleTime setting, you could keep at least one process running, but you'll almost certainly want Passenger to start up other instances of the app as necessary. This thread on the Rufus Scheduler Google Group mentions using lock files to prevent more than one process from starting the scheduler, that may be useful to you.
I am setting up an Apache2 webserver running multiple Ruby on Rails web applications with Phusion Passenger. I know that Passenger spawns Ruby processes for handling requests. I have the following questions:
If more than one request has to be handled at the same time, will Passenger spawn multiple processes or multiple (Ruby) threads? How do I configure it so it always spawns single-threaded processes?
If I have two Rails applications, imagine that a request for app A goes to process 1, then later request for app B arrives. Is it possible that process 1 will handle this request as well? When and how is this possible? In other words, is one process allowed to handle requests for multiple Rails applications?
I have the same Rails application exported in multiple URLs and multiple virtual hosts (such as http:// and https://). Will the same process be able to serve different virtual hosts? (The answer to this seems to be yes, I've set a global variable in answering a request to virtual host A, and I was able to retrieve the value in virtual host B.)
Generally speaking, Passenger spawns new processes by forking an ApplicationSpawner, which has the framework and application code pre-loaded into memory, or a FrameworkSpawner, which just has the framework code.
Passenger, as far as I know, doesn't deal in threads. Instead, as the load increases on an application, it will fork that Application's ApplicationSpawner and initialize another instance. When load decreases, one or more application instances are killed off.
If Passenger is configured in a certain way (I believe by choosing the "smart" spawn method), it will create a FrameworkSpawner, which loads the rails code, but no application code, which can then be forked to load and application using that version of Rails.
So to answer your questions:
It will serve them sequentially, then spawn additional processes if it decides the load is high enough.
No. One process can only belong to a single Rails Application.
I'm kind of sketchy on this one, but your experiment makes sense. Passenger should be smart enough to figure out that even though it's running from different places in the server config, you're talking about the same application. It's probably based on the application's filesystem path.
EDIT: I went and read up on this a bit. Turns out I was mostly right, but the technical details were a bit off. See the Passenger documentation
Yup, Burke is right. In case of the third question, Phusion Passenger recognizes applications by their application root path. So even if you have two virtual hosts, if they both point to the same DocumentRoot then Phusion Passenger will think that they're the same app.
I use Apache + Passenger to host some Rails applications. Something seems to go in a sleep mode when there is no request for a longer time. It then takes 10-20 seconds for the site to load. Feels like there is something that has to wake up when there have been no requests for a longer time.
How can I fix that? I have enough RAM so it should be no problem if whatever goes to sleep just stays awake. ;)
Take a look at the PassengerPoolIdleTime parameter for Passenger.
It states the maximum number of seconds an application instance can be idle before it shuts down to conserve memory.
The default is 300, but you could try to set a higher number and see if that helps.
Also, if you're on a shared host and can't change that setting, you could always write a cron script to hit your site once every x seconds (where x is slightly less than PassengerPoolIdleTime), and update your analytics package to ignore requests from the IP address of the box that's doing the polling.
The passenger documentation recommends setting the PassengerPoolIdleTime to 0 on non-shared hosts that are running only a few Rails apps. That should prevent it from getting unloaded unless it's absolutely necessary.
#x0ne, you can set PoolIdleTime (pool_idle_time in nginx) in the global server configuration. In my installation of Nginx that's /opt/nginx/conf/nginx.conf.
Here's the part of passenger's documentation that covers PoolIdleTime: http://www.modrails.com/documentation/Users%20guide.html#PassengerPoolIdleTime
I'm running into a problem in a Rails application.
After some hours, the application seems to start hanging, and I wasn't able to find where the problem was. There was nothing relevant in the log files, but when I tried to get the url from a browser nothing happened (like mongrel accept the request but wasn't able to respond).
What do you think I can test to understand where the problem is?
I might get voted down for dodging the question, but I recently moved from nginx + mongrel to mod_rails and have been really impressed. Moving to a much simpler setup will undoubtedly save me headaches in the future.
It was a really easy transition, I'd highly recommend it.
Are you sure the problem is caused by Mongrel? Have you tried running your application under WEBrick?
There are a few things you can check, but since you say there's nothing in the logs to indicate error, it sounds like you might be running into a bug when using the log rotation feature of the Logger class. It causes mongrel to lock up. Instead of relying on Logger to rotate your logs, consider using logrotate or some other external log rotation service.
Does this happen at a set number of hours/days every time? How much RAM do you have?
I had this same problem. The couple options I had narrowed it down to were MySQL adapter related. I was running on Red Hat Enterprise Linux 4 (or 5) and the app would hang after a given amount of idle time.
One suggested solution was to compile native MySQL bindings, I had been using the pure Ruby one.
The other was to set the timeout on the MySQL adapter higher than what the connection would idle out on. (I don't have the specific configuration recorded, but as I recall it was in environment.rb and it was some class variable in the mysql adapter.)
I don't recall if either of those solutions fixed it, we moved to Ubuntu shortly after that and hadn't had a problem since.
Check the Mongrel FAQ:
http://mongrel.rubyforge.org/wiki/FAQ
From my experience, mongrel hangs when:
the log file got too big (hundreds of megabytes in size). you have to setup log rotation.
the MySQL driver times out
you have to change the timeout settings of your MySQL driver by adding this to your environment.rb:
ActiveRecord::Base.verification_timeout = 14400
(this is further explained in the deployment section of the FAQ)
Unfortunately, Rails (and thereby Mongrel) using up too much memory over time and crashing is a known problem (50K+ Google entries for "Ruby, rails, crashing, memory"). The current ruby interpreter has the property that it sometimes simply fails entirely to give memory back to the system - it may reuse the memory it has but it won't give it up.
There are numerous schemes for monitoring, killing and restarting Mongrel instances in a production environment - for example: (choosing at random) rails monitor . Until the problem is fixed more decisively, one of these may be your best bet.
We have experienced this same issue. First off, install the mongrel_proctitle gem
http://github.com/rtomayko/mongrel_proctitle/tree/master
This gem/plugin will allow you to view the mongrel processes via "ps" and you can see if a Mongrel is hung. An issue we have seen with Mongrel is that it will happily accept connections and enqueue them, then wedge itself. This plugin will help you see when a Mongrel has been wedged but then you must use another monitoring app to actually restart a a wedged Mongrel, something like Monit or God
You might also want to consider putting a more balanced reverse proxy in front of your Mongrels, something HAproxy, instead of nginx, Apache or Lighttpd. With a setting of "maxconn 1" in HAproxy you can assure that the queue is being maintained by HAproxy versus Mongrel. The other reverse proxies (nginx, Apache, Lighttpd) only do round-robin which means that they can load up your Mongrel queue, inadvertently.
My personal choice is God as it is much more flexible.
tl;dr Install this gem plugin and keep an eye on your Mongrels. Try Apache+Phusion Passenger.