I am running ubuntu EC2 small instance with ruby on rails and mysql in the instance, my rails app consists of delayed jobs.
I have observed it suddenly my instance CPU utilization spikes out more then 70 % and the persists for more then a minute.
Ruby on rails app is not a CPU intense app, but I am not able to figure why and where exactly its happening,I have activated the Cloud watch monitoring, and also created alarm .
Kindly suggest me to figure out some tools to figure out which processes is a CPU intense.
How to I handle it.
Run top and see what's at the top of the list when the CPU usage is high.
Related
I am using the Amazon Auto Scaling group for Rails application deployed on an EC2 instance using NGINX and Puma. I am facing some challenges with the configuring of the Auto Scaling policy.
I am using r5.xlarge for the main instance that is hosting my corn jobs and r5.large for the autoscaling instance. My current scaling trigger is defined on the 50% CPU but apparently, that does not work due to the following reasons
Since the main instance has 4 CPUs the overall consumption did not hit 50% unless there is some corn job running that is consuming all resources.
Even if the CPU will hit 50% the startup time of rails application is 30-40 seconds and in the meantime, all requests received by the server returns 503.
If the CPU consumption is less than 50% but the system receives a lot of concurrent requests it does not start a new instance and either start returning 503 or the response time increases significantly.
I have tried changing the auto-scaling group from CPU consumption to the number of requests but the start time issue of instance still prevails and sometimes it starts a new instance when it is not even needed.
Have you ever faced any such issue with Rails deployment, anything that you thinks worked for your out of the box?
We are running Ruby application with PUMA in ECS Tasks, but should be quite the same problematic that with EC2.
Since Ruby is single threaded, your Ruby Process running your PUMA server is only going to use one CPU at a time. If you have 4 CPU, I imagine one PUMA process will never manage to saturate more than 25% of the overall machine.
Note: Also have a look at your configuration regarding the number of PUMA Threads. This is also critical to configure, since you are doing auto-scaling, your application NEED to be able to saturate the CPU it's using, to be able to kick in. With too few Puma Thread it will not be the case, with too much your application will become unstable, this is something to fine tune.
Recommendation:
Run one PUMA process per CPU you have available with the EC2 class you have chosen, each PUMA server listening on a different port, have your load-balancer manage that. This should allow your machine to reach potentially 100% CPU during saturation (in theory), allowing auto-scaling base on CPU to work
Preferred solution: Pick smaller machines, with 1 CPU, so you only need to run one PUMA server per machine.
From my experience with ECS, Ruby and other single threaded languages should not use more than 1 (v)CPU machines, and you should instead really on heavy horizontal scaling if necessary (some of our service are running 50x ECS instances).
Hope this helps.
I have a Rails application which has 100% CPU utilization most of the time.
I am not able to figure out why there is so much load on the server. I am using the Puma web server with a default configuration, and am running multiple background jobs using the sucker-punch gem. There are 7 files which are using sucker punch jobs with 5 workers:
include SuckerPunch::Job
workers 5
I ran the top -i query and found the following processes running on the server. I can see multiple Ruby commands on the server. Can someone tell me whether this is normal behavior on a server, or if something is wrong?
Some Ways to Reduce Resource Contention
Your user space load is high (~48%), so you'll probably want to reduce the number of workers in your web application, increase the number of CPUs available on your instance, move to a version of Ruby that has better concurrency and real multi-core support (e.g. Rubinius or JRuby), or some combination of these options. Depending on what your code is actually doing, you may also need to re-architect your application to offload expensive I/O from the application server.
In addition, your steal time is quite high (~41%), so your EC2 instance is probably overloaded. Simply moving your application to a less-loaded instance may free up sufficient resources to reduce application wait times.
I have a Rails app that among other things, have several background jobs which are computationally expensive (image manipulation :O).
I am using Sidekiq to manage those jobs. I currently have set a concurrency of 5 threads per Sidekiq process and here is what I do in order to see the memory usage:
ps faux | grep sidekiq
Results are this:
hommerzi 3874 3.5 5.7 287484 231756 pts/5 Sl+ 17:17 0:10 | \_ sidekiq 2.17.0 imageparzer [3 of 3 busy]
However, I have a feeling that there must be a way to monitor this correctly from within the Rails app, or am I wrong?
My question would be: How can I monitor my memory usage in this scenario?
My advice would be to use Monit (or God) to manage your processes. This goes for database, server, application; not just background jobs.
Here's an example: Monit Ruby on Rails Sidekiq
Monitor your application for a while and set realistic memory limits. Then, if one of your processes dies or goes above that limit for a given amount of cycles (usually 2 minute checks), it will (re)start the process.
You can also setup an alert email address and a web frontend (with basic HTTP auth). This will prove essential for running stable applications in production. For example, recently I had a sidekiq process get carried away with itself and chew up 250mb memory. Monit then restarted the process (which is now hovering around 150mb) and sent me an alert. Now I can check the logs/system to see why that might have happened. This all happened while I was asleep. Much better than the alternative: waking up and finding your server on its knees with a runaway process or downed service.
https://www.digitalocean.com/community/articles/how-to-install-and-configure-monit
I am monitoring my server on New Relic and the memory consumption of my app is rather high about 1 GB. Currently I am the only visitor to the site. When I drill down, I see that most of the consumption is because of Ruby. It says 17 instances running. What does this mean and how can I lower it?
Unicorn is configured to run X number of instances by default. You can explicitly configure this number in config/unicorn.rb using worker_processes 4 (to run 4 instances). Each unicorn instance will load up the entire stack for your application and keep it memory. A mid-sized rails applications tends to be around ~100 MB and up, it should stay at that level given there aren't any memory leaks. The memory consumption is generally affected by the number of dependencies and the complexity of the application.
I'm developing an API on Appfog and want to know what to focus on (more memory with one instances or more instances with lower memory).
Appfog gives you free 2GB of RAM and up to 16 instances if each instances get 128 MB RAM.
My application uses PHP, MySql and Memcachier.
I want to launch it soon and want to know which configuration is best for my server.
What is the benefit with more RAM or instances?
Thanks for helping :)
Best Regards,
Johnny
You want as many instances as your app will run without running out of memory :). More instances means better performance and uptime. However, if an instance runs out of memory it will be shut down leaving your app running with fewer instances until they all collapse. You can diagnose this problem with the af apps and af logs <appname> --all commands. If the app is running at < 100% regularly then the instance memory budget may be too low. When there are down instances, the logs command may reveal memory limit reached errors.
Memory Recommendations
Here are some memory recommendations to start out with: Wordpress with several installed plugins will need > 512mb to be stable. For lean custom PHP apps 128mb is usually sufficient but should be watched. If an app is using a framework try 256mb. These memory limits may seem high but it's really the peak memory usage not the average usage.
Load Test
Load testing using Seige can help find a memory / instance balance. It does this by determining if your app is peaking out over the memory limit. Scale the app down to 1 instance and siege with 5, 10, and 15 concurrent connections progressively increasing by 5 until the app falls over. If the app does stop, bump the memory up and try again.