I am monitoring my server on New Relic and the memory consumption of my app is rather high about 1 GB. Currently I am the only visitor to the site. When I drill down, I see that most of the consumption is because of Ruby. It says 17 instances running. What does this mean and how can I lower it?
Unicorn is configured to run X number of instances by default. You can explicitly configure this number in config/unicorn.rb using worker_processes 4 (to run 4 instances). Each unicorn instance will load up the entire stack for your application and keep it memory. A mid-sized rails applications tends to be around ~100 MB and up, it should stay at that level given there aren't any memory leaks. The memory consumption is generally affected by the number of dependencies and the complexity of the application.
Related
I am using the Amazon Auto Scaling group for Rails application deployed on an EC2 instance using NGINX and Puma. I am facing some challenges with the configuring of the Auto Scaling policy.
I am using r5.xlarge for the main instance that is hosting my corn jobs and r5.large for the autoscaling instance. My current scaling trigger is defined on the 50% CPU but apparently, that does not work due to the following reasons
Since the main instance has 4 CPUs the overall consumption did not hit 50% unless there is some corn job running that is consuming all resources.
Even if the CPU will hit 50% the startup time of rails application is 30-40 seconds and in the meantime, all requests received by the server returns 503.
If the CPU consumption is less than 50% but the system receives a lot of concurrent requests it does not start a new instance and either start returning 503 or the response time increases significantly.
I have tried changing the auto-scaling group from CPU consumption to the number of requests but the start time issue of instance still prevails and sometimes it starts a new instance when it is not even needed.
Have you ever faced any such issue with Rails deployment, anything that you thinks worked for your out of the box?
We are running Ruby application with PUMA in ECS Tasks, but should be quite the same problematic that with EC2.
Since Ruby is single threaded, your Ruby Process running your PUMA server is only going to use one CPU at a time. If you have 4 CPU, I imagine one PUMA process will never manage to saturate more than 25% of the overall machine.
Note: Also have a look at your configuration regarding the number of PUMA Threads. This is also critical to configure, since you are doing auto-scaling, your application NEED to be able to saturate the CPU it's using, to be able to kick in. With too few Puma Thread it will not be the case, with too much your application will become unstable, this is something to fine tune.
Recommendation:
Run one PUMA process per CPU you have available with the EC2 class you have chosen, each PUMA server listening on a different port, have your load-balancer manage that. This should allow your machine to reach potentially 100% CPU during saturation (in theory), allowing auto-scaling base on CPU to work
Preferred solution: Pick smaller machines, with 1 CPU, so you only need to run one PUMA server per machine.
From my experience with ECS, Ruby and other single threaded languages should not use more than 1 (v)CPU machines, and you should instead really on heavy horizontal scaling if necessary (some of our service are running 50x ECS instances).
Hope this helps.
I have a Rails application which has 100% CPU utilization most of the time.
I am not able to figure out why there is so much load on the server. I am using the Puma web server with a default configuration, and am running multiple background jobs using the sucker-punch gem. There are 7 files which are using sucker punch jobs with 5 workers:
include SuckerPunch::Job
workers 5
I ran the top -i query and found the following processes running on the server. I can see multiple Ruby commands on the server. Can someone tell me whether this is normal behavior on a server, or if something is wrong?
Some Ways to Reduce Resource Contention
Your user space load is high (~48%), so you'll probably want to reduce the number of workers in your web application, increase the number of CPUs available on your instance, move to a version of Ruby that has better concurrency and real multi-core support (e.g. Rubinius or JRuby), or some combination of these options. Depending on what your code is actually doing, you may also need to re-architect your application to offload expensive I/O from the application server.
In addition, your steal time is quite high (~41%), so your EC2 instance is probably overloaded. Simply moving your application to a less-loaded instance may free up sufficient resources to reduce application wait times.
I've been reading information about how different rails application servers work for a while and some things got me confused probably because of my lack of knowledge in this field. Anyway, the following things got me confused:
Puma server has the following line about its clustered mode workers number in its readme:
On a ruby implementation that offers native threads, you should tune this number to match the number of cores available
So if I have, lets say, 2 cores and use rubinius as a ruby implementation, should I still use more than 1 process considering that rubinius use native threads and doesn't have the lock thus it uses all the CPU cores anyway, even with 1 process?
I understand it that I'd need to only increase the threads pool of the only process if I upgrade to a machine with more cores and memory, if it's not correct, please explain it to me.
I've read some articles on using Server-Sent Events with puma which, as far as I understand, blocks the puma thread since the browser keeps the connection open, so if I have 16 threads and 16 people are using my site, then the 17th would have to wait before one of those 16 leaves so it could connect? That's not very efficient, is it? Or what do I miss?
If I have a 1 core machine with 3Gb of RAM, just for the sake of the question, and using unicorn as my application server and 1 worker takes 300 MBs of memory and its CPU usage is insignificant, how many workers should I have? Some say that the number of workers should be equal to the number of cores, but if I set the workers number to, lets say, 7 (since I have enough RAM for it), it will be able to handle 7 concurrent requests, won't it? So it's just a question of memory and cpu usage and amount of RAM? Or what do I miss?
I'm developing an API on Appfog and want to know what to focus on (more memory with one instances or more instances with lower memory).
Appfog gives you free 2GB of RAM and up to 16 instances if each instances get 128 MB RAM.
My application uses PHP, MySql and Memcachier.
I want to launch it soon and want to know which configuration is best for my server.
What is the benefit with more RAM or instances?
Thanks for helping :)
Best Regards,
Johnny
You want as many instances as your app will run without running out of memory :). More instances means better performance and uptime. However, if an instance runs out of memory it will be shut down leaving your app running with fewer instances until they all collapse. You can diagnose this problem with the af apps and af logs <appname> --all commands. If the app is running at < 100% regularly then the instance memory budget may be too low. When there are down instances, the logs command may reveal memory limit reached errors.
Memory Recommendations
Here are some memory recommendations to start out with: Wordpress with several installed plugins will need > 512mb to be stable. For lean custom PHP apps 128mb is usually sufficient but should be watched. If an app is using a framework try 256mb. These memory limits may seem high but it's really the peak memory usage not the average usage.
Load Test
Load testing using Seige can help find a memory / instance balance. It does this by determining if your app is peaking out over the memory limit. Scale the app down to 1 instance and siege with 5, 10, and 15 concurrent connections progressively increasing by 5 until the app falls over. If the app does stop, bump the memory up and try again.
I am running ubuntu EC2 small instance with ruby on rails and mysql in the instance, my rails app consists of delayed jobs.
I have observed it suddenly my instance CPU utilization spikes out more then 70 % and the persists for more then a minute.
Ruby on rails app is not a CPU intense app, but I am not able to figure why and where exactly its happening,I have activated the Cloud watch monitoring, and also created alarm .
Kindly suggest me to figure out some tools to figure out which processes is a CPU intense.
How to I handle it.
Run top and see what's at the top of the list when the CPU usage is high.