Is it possible to change the Passenger application pool size at runtime? Ie, without restarting apache, and without disrupting active visitors?
The same time every day we have a background job run. It is very memory intensive. Since during that time, traffic on the site tends to be relatively low, I would like to automatically scale down the number of application servers running just before the jobs runs, and then scale up again when it is finished.
Passenger should automatically shut down instances when they are not in use. Since your traffic is low then you should not have any unnecessary passenger instances running.
One variable you can tweak is PassengerPoolIdleTime. This tells passenger how long to wait before shutting down idle instances.
There is no PoolSize variable. There is however a MaxPoolSize but this limit should not be hit unless you are receiving heavy traffic.
You can find all the variables along with what they do here: http://www.modrails.com/documentation/Users%20guide.html
If you really want to change the settings dynamically, you can try calling sudo /etc/init.d/apache2 reload. This will tell apache to reload its settings, including the passenger configuration.
Good luck!
Related
I am using the Amazon Auto Scaling group for Rails application deployed on an EC2 instance using NGINX and Puma. I am facing some challenges with the configuring of the Auto Scaling policy.
I am using r5.xlarge for the main instance that is hosting my corn jobs and r5.large for the autoscaling instance. My current scaling trigger is defined on the 50% CPU but apparently, that does not work due to the following reasons
Since the main instance has 4 CPUs the overall consumption did not hit 50% unless there is some corn job running that is consuming all resources.
Even if the CPU will hit 50% the startup time of rails application is 30-40 seconds and in the meantime, all requests received by the server returns 503.
If the CPU consumption is less than 50% but the system receives a lot of concurrent requests it does not start a new instance and either start returning 503 or the response time increases significantly.
I have tried changing the auto-scaling group from CPU consumption to the number of requests but the start time issue of instance still prevails and sometimes it starts a new instance when it is not even needed.
Have you ever faced any such issue with Rails deployment, anything that you thinks worked for your out of the box?
We are running Ruby application with PUMA in ECS Tasks, but should be quite the same problematic that with EC2.
Since Ruby is single threaded, your Ruby Process running your PUMA server is only going to use one CPU at a time. If you have 4 CPU, I imagine one PUMA process will never manage to saturate more than 25% of the overall machine.
Note: Also have a look at your configuration regarding the number of PUMA Threads. This is also critical to configure, since you are doing auto-scaling, your application NEED to be able to saturate the CPU it's using, to be able to kick in. With too few Puma Thread it will not be the case, with too much your application will become unstable, this is something to fine tune.
Recommendation:
Run one PUMA process per CPU you have available with the EC2 class you have chosen, each PUMA server listening on a different port, have your load-balancer manage that. This should allow your machine to reach potentially 100% CPU during saturation (in theory), allowing auto-scaling base on CPU to work
Preferred solution: Pick smaller machines, with 1 CPU, so you only need to run one PUMA server per machine.
From my experience with ECS, Ruby and other single threaded languages should not use more than 1 (v)CPU machines, and you should instead really on heavy horizontal scaling if necessary (some of our service are running 50x ECS instances).
Hope this helps.
Our rails 3.2 app (deployed on passenger/nginx) uses gem ruote. The ruote worker (responsible for writing to database) needs to be kept running all the time for ruote to work. What we did is to keep min one instance running all the time by setting in nginx.conf:
passenger_min_instances 1;
However after about 5min idle time, the ruote worker stops responding. After restarting nginx, then ruote worker starts to work again. We don't know what passenger kills to cause the problem. enter code hereWhat else we need to set up in passenger to keep ruote worker running all the time?
This specifies the minimum number of application processes that should exist for a given application. You should set this option to a non-zero value if you want to avoid potentially long startup times after a website has been idle for an extended period.
Please note that this option does not pre-start application processes during Nginx startup. It just makes sure that when the application is first accessed:
at least the given number of processes will be spawned.
the given number of processes will be kept around even when processes are being idle cleaned (see passenger_pool_idle_time).
If you want to pre-start application processes during Nginx startup, then you should use the passenger_pre_start directive, possibly in combination with passenger_min_instances. This behavior might seem counter-intuitive at first sight, but passenger_pre_start explains the rationale behind it.
I've setup my website with Rails 3 and Passenger (via nginx) and, although its only being used by one person, the web server has to essentially wake up the rails instance to render the page. This happens only when the website is not accessed for a while (hence its sleeping), but I'm a little paranoid that could it still lag like so when the website is operating on a production level (don't get mixed with the development/production mode, the sleeping website is running in production mode when I'm checking it out).
Any ideas? Or this just a sleep and wakeup thing when nobody's using the website.
There is an easy fix just edit your nginx.conf and set passenger_min_instances to a value larger then zero. This way passenger always keeps one instance alive; this will prevent the "lag" as you describe it. Read more about it in the Passenger Nginx documentation.
Take a look at passenger_pool_idle_time. It states the maximum number of seconds that an application instance may be idle. That is, if an application instance hasn’t received any traffic after the given number of seconds, then it will be shutdown in order to conserve memory.
I'm running an application that kicks off a Rufus Scheduler process in an initializer. The application is running with Passenger in production and I've noticed a couple weird behaviors:
First, in order to restart the server and make sure the initializer gets run, you have to both touch tmp/restart.txt and load the app in a browser. At that point, the initializer fires. The horrible thing is that if you only do the touch, the processes scheduled by Rufus get reset and aren't rescheduled until you load the app in a browser.
This alone I can deal with. But this leads to the second problem: I'll notice that the scheduled process hasn't run, so I load a page and suddenly the log file is telling me that it's running the initializers as if I'd rebooted. So, at some point, Passenger is randomly rebooting as if I'd touched tmp/restart.txt and wiping out my scheduled processes.
I have an incredibly poor understanding of Passenger and Rails's integration, so I don't know whether this occasional rebooting is aberrant or all part of the architecture. Can anyone offer any wisdom on this situation?
What you describe is the way Passenger works. It spawns new instances of the application when traffic warrants them, and shuts them down after periods of inactivity to free resources.
You should read the Passenger documentation, particularly the Resource Control and Optimization section. There are settings which can prevent the application from being shut down by Passenger, if that is what you want.
Using the PassengerPoolIdleTime setting, you could keep at least one process running, but you'll almost certainly want Passenger to start up other instances of the app as necessary. This thread on the Rufus Scheduler Google Group mentions using lock files to prevent more than one process from starting the scheduler, that may be useful to you.
I use Apache + Passenger to host some Rails applications. Something seems to go in a sleep mode when there is no request for a longer time. It then takes 10-20 seconds for the site to load. Feels like there is something that has to wake up when there have been no requests for a longer time.
How can I fix that? I have enough RAM so it should be no problem if whatever goes to sleep just stays awake. ;)
Take a look at the PassengerPoolIdleTime parameter for Passenger.
It states the maximum number of seconds an application instance can be idle before it shuts down to conserve memory.
The default is 300, but you could try to set a higher number and see if that helps.
Also, if you're on a shared host and can't change that setting, you could always write a cron script to hit your site once every x seconds (where x is slightly less than PassengerPoolIdleTime), and update your analytics package to ignore requests from the IP address of the box that's doing the polling.
The passenger documentation recommends setting the PassengerPoolIdleTime to 0 on non-shared hosts that are running only a few Rails apps. That should prevent it from getting unloaded unless it's absolutely necessary.
#x0ne, you can set PoolIdleTime (pool_idle_time in nginx) in the global server configuration. In my installation of Nginx that's /opt/nginx/conf/nginx.conf.
Here's the part of passenger's documentation that covers PoolIdleTime: http://www.modrails.com/documentation/Users%20guide.html#PassengerPoolIdleTime