I am using Rails 4.2 and Ruby 2.2.0. Recently, I have noticed that my app will hang for long periods of time after a certain controller has been invoked. This is an app that has been under development for several years and I have never seen this sort of hanging behavior before (nor am I unfortunately completely aware of what changes I've made since it was last not hanging).
What I mean by "hang" is: No pages will load (no controllers can be reached, and even the default homepage which is basically a static page won't load). None of my logs will update. Standard output is not being written to. The application also has many background worker processes (managed by Resque), which also appear to be hung. This behavior will persist for minutes at a time.
In order to narrow down the culprit, I removed everything from my app except for one controller. That controller looks like:
puts "Peforming"
respond_to do |format|
format.xml {
render :xml => "<Status>OK</Status>"
}
end
It will accept any POST from a client, and will respond with "OK," after printing a message. The app is hosted on a server, and I am hitting it with an XML POST as fast as I can from my local machine. After less than a minute of hitting it, the app will hang entirely, as described above.
I have also looked at netstat but see nothing out of the ordinary. There are lots of connections in the TIME_WAIT state, but I understand that that is fine. I also looked at top, and the machine is 90% idle, even when hanging.
Finally, this is running on Amazon EC2 in case that is relevant.
Related
I am running nginx with rails on a small production server to which I submit several jobs per second, each of which return a json result that I have obtained using render json: my_result.
Every time I do a capistrano deployment to production I get very long (5-10 seconds, sometimes more) delay in the rendering step. Once several minutes have passed all this slowness is over.
I tried looking online whether Rails needs to "purge" any previous data that remains in memory due to capistrano having restarted it, but could not find anything, neither how to avoid this issue.
I have the same thing, but I use passenger, every restart it takes a few seconds for it to get itself set. I just kinda excepted the fact that it take a few seconds. but I do see your point it is a nuisance when you are trying to make simple changes quickly to get things the way you need it.
I am using Rails 4.2 and Ruby 2.2.0. Recently, I have noticed that my app will hang for long periods of time when a certain worker is working. This is an app that has been under development for several years and I have never seen this sort of hanging behavior before (nor am I unfortunately completely aware of what changes I've made since it was last not hanging).
What I mean by "hang" is: No pages will load (no controllers can be reached, and even the default homepage which is basically a static page won't load). None of my logs will update. Standard output is not being written to. The application also has many background worker processes (managed by Resque), which also appear to be hung.
I have looked at top and netstat and various other Unix utilities and I don't see anything alarming there, either.
Finally, I'm not sure if it's worth mentioning but this application is running via Foreman on an EC2 instance.
EDIT: I don't believe that it is query related, because nothing abnormal shows up on top. Also, I'm a little inexperienced with rails, but each job has its own entry in the process table, if that helps.
Does anyone have any experience with Rail development page display erratically slowing down and speeding up (page appears 15-20 seconds after the console says the entire page has been rendered).
My development environment is rails 3.0.17 on Mac (Lion), WEBrick 1.3.1, ruby 1.9.2, with Postgres v11 (app supplied by Heroku) for the development database.
Recently, I've noticed some very long delays loading pages... 15 to 20 seconds at times, and the delay is completely unrelated to the complexity of the page, and a given page might load fast several times, then load slow. It might be pretty bad for several minutes, then go away for an hour.
And whether the page loads slow or fast, the rails log always shows the page has rendered fairly quickly... I'll see something like "Completed 200 OK in 486ms" but then the browser might say "waiting for localhost" for another 15 seconds before displaying the page.
Does not seem to depend on the browser (FF or Safari act the same).
Dos not sem to relate to updating the code base. I cna be testing some UI elements and everything is snappy, then suddenly several pages hang for 15-120 seconds.
Still happens even after I added the gem 'rails-dev-boost' to my Gemfile in development.
I also added to my development machine gem 'http_logger' to log calls to external http requests like S3, but don't see that as a factor (the delays often happen on pages that don't access external APIs).
My layouts have a couple of external .js dependencies such as http://static.twilio.com/libs/twiliojs/1.0/twilio.min.js, but I have added that to the bottom of my layout so I would (resumably) still see the page contents quickly if that were being delayed. (Besides, it is probably cached by my browser).
Of course I have tried rebooting the machine, verified activity monitor seems normal for CPU and memory usage. It does not seem to correlate when timemachine is doing its thing.
MORE INFO: per suggestion in comment from AKG below, I used Firebug's web console and most of the time the stylesheets load in a couple of milliseocnds, but when the slowdown occurs I'm seeing delays of 8-20 seconds for most of the stylesheets... which suggests webrick is failing to serve the pages timely?
It could be that buggy/slow middleware is executing after the render is called, but before the response is sent. I recently ran into this issue with pauses of more than 1 minute in some cases, and the culprit turned out to be the Bullet gem, which is designed to help us detect N+1 queries. I removed the middleware and the issue went away!
I got the same problem a few times. I switched to Mongrel. That automatically resolved the issue. However, I got this with Rails 2.3.8.
If using in the controller:
def index
#time_start_in_controller
...
end
and at the end of view, use
<%= "took #(Time.now - #time_start_in_controller} seconds" %>
but isn't the time at the view not the true ending of rendering, because it needs to mix with the layout and so forth. What is a more accurate way (just as accurate as possible) to print out the page generation time right on the webpage?
(update: also, the console showing the log as taking 61ms, but the page definitely took 2 to 3 seconds to load, and the network I am using is super fast, at home or at work, at 18mbps or higher with a ping of maybe 30ms)
update: it is a bit strange that if I use the http performance test ab
ab -n 10 http://www.my-web-site.com:8080
it takes 3 seconds total for 10 requests. But if I use Firefox or Chrome to load the page, each page load is about 3 seconds. This is tunneling to my work's computer back to my notebook running Rails 3, but shouldn't make a difference because I run Bash locally for the above statement and use Firefox locally too.
in a typical production environment, static content (images, css, js) are handled by the web server (eg. apache, nginx etc) not you rails server. so you should check their logs as well. if you are serving static content from your rails server that could be your problem right there.
If your browser time is slow but the time taken in rails (According to the logs) is fast, that can mean many things including but not quite limited to:
network speed is slow
your dns server is slow and browser can't resolve your dns quickly (this happens with for instance if you use godaddy for your dns server they throttle dns lookups)
the requests concurrency exceeds how many threads you have in rails
one way to debug these types of performance issues is to put something in front of the rails server (for example Haproxy) and turn the logging to full. As they will show how many waiting requests there are and how long the actual request/response transferring took along with how long it took your rails thread to process.
I'm running a deployed rails webserver, and i'm experiencing memory issues. I get an Errno::ENOMEM error each time i try to invoke sendmail. When i run 'top' on my server, it shows me that 2 ruby1.8 processes are running, each digesting around 40% of memory. Is this caused by a leak in my code, or have I launched two simultaneous ruby processes?
Both processes are run by my deployer account, and following that they are invoked when I push to my production git repository (using pushand). I'm trying to find out if this is natural (ie. server needs to be upgraded) or if I have some kind of fault in my setup. I upgraded my RAM from 256 mb to 512 mb recently, and didn't have this problem before.
The problem is preventing visitors from creating a user, because the application throws an error when they should get an activation email.
Note that I'm running rails 2.1.0 on an ubuntu setup, with apache and passenger.
Ok guys, I guess this was one of the cases where I had to help myself, and I'm actually pretty glad I did. I Didn't use any of the tools I linked to.
I logged in my server via. ssh and ran two commands in two screen windows: top and tail -f /var/www/mysite.com/log/production.log.
I looked in the live log file and noticed a page request that took a long time to load – a page where a user can get his/her images shown, where each user is identified by their name, via. acts_as_taggable_on. It took 6 seconds for this single request.
And it kept popping up. I think I saw it 6 or 7 times, and I decided to investigate it further. It was always the same user id, so I decided to look up the user. It turns out, the user had a blank name, "", and the tagged images is found by
Asset.tagged_with(#user.name, :on => "users")
Now in my app, there are 10s of thousands of assets, and Asset.tagged_with("", :on => "users") would return ALL of them, thus eating up my memory.
Now I've handled the case where a user has a blank name, and has it required for all new users. Even though no one helped, I still feel some gratitude towards stackoverflow, because sometimes its as simple as asking the questions to solve the problem.