I build a backend service (nginx + passenger + rails) on AWS.
My service is running as expected most of the time but sometimes goes wrong when passenger creates a weird new process to handle request:
[INFO] Started POST "/apixxxxxx"
[FATAL] LoadError (Expected /xxxxxxxx/app/controllers/application_controller.rb to define ApplicationController):
[INFO] Started POST "/apixxxxxx"
[FATAL] ActionController::RoutingError (uninitialized constant ApplicationController):
[INFO] Started POST "/apioooooo"
[FATAL] ActionController::RoutingError (uninitialized constant ApplicationController):
...
It has some signature on this weird process:
Other ruby process still works fine when this specific weird ruby process keeps failing on every new request.
If error happens, it always starts by the "LoadError" in the beginning of the process's first request.
This kind of "uninitialized constant" error keeps forever as long as this process is alive.
*new added information:
According to rails log, the same api can reply without error in other 'normal' passenger process. I exclude the problem about typo or rails naming convention.
Here is some information of my machine:
ec2 type : m4.large
storage : general purpose SSD (use EBS)
system : ubuntu 14.04.2 (not AWS version)
application : version 1.8.1, passenger 5.0.26, Rails 3.2.22.2, ruby 2.2.4p230, sidekiq 4.1.1
I also googled some keyword like "AWS EBS volume's IOPS limit" or "sidekiq thread" with this LoadError issue but no luck. Is there any new direction to investigate this problem?
Related
I'm getting the below error when trying to run migrations locally. But it only happens sometimes, so I'm trying to understand what might cause it.
Full error:
rake aborted!
PG::ConnectionBad: could not fork new process for connection: Resource temporarily unavailable
could not fork new process for connection: Resource temporarily unavailable
E, [2017-04-22T10:09:26.381554 #17026] ERROR -- : fsevent: running worker failed: Resource temporarily unavailable
Otherwise, the rails console, and accessing the db directly through a client works fine.
I was pointed to another thread which recommended checking,
ulimit -Hn
unlimited
So my descriptor limit doesn't appear to be the issue.
Source: fork: retry: Resource temporarily unavailable
Update:
It consistently fails when the rails server (Puma) is running, but works when its not.
I'm attempting to host my rails app on a vps. The issue that I'm running into is when I try to navigate to my rails app, I get the following error message:
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, [no address given] and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
When I check out the virtual server's error log it tells me the following:
Premature end of script headers:
And sometimes that's followed with a feed or a contact or a referer: myurl.com/ or nothing at all.
I've seen plenty of posts regarding this error messages elsewhere but none so far have resolved the matter. The general consensus seems to be that it likely has to do with the rails environment variable or file.
As this is my first attempt at putting a rails app on a production server, I figure this is plausible. Below is my environment.rb
cat config/environment.rb
# Load the Rails application.
require File.expand_path('../application', __FILE__)
# Initialize the Rails application.
Rails.application.initialize!
As for the server, I'm using Ubuntu 12.04 with Webmin/Virtualmin and apache2 with passenger. Ruby 2.1.5p273 was installed using RVM and all gems are up to date.
Any thoughts would be helpful!
Today I was confronted with exactly the same problem. In my case it was the line 'secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>' in config/secrets.yml (but the env variable was not defined on the server). Please, look in §3.3 of http://edgeguides.rubyonrails.org/upgrading_ruby_on_rails.html.
I am running a small Rails application under Ubuntu 14.04.1, using RVM to manage rubies. I have started using the Phusion Passenger (4.0.48)/nginx package so I can eventually serve multiple Rails applications from the same server (an older Dell laptop) in a low volume production environment. The application uses RSolr to make calls to a SOLR server to get text from old lab notebooks.
When I run the application using WEBrick or the Passenger standalone application server, it works fine. But when I run it under nginx, I get an error when certain text is returned from SOLR:
ArgumentError (invalid byte sequence in US-ASCII):
<internal:prelude>:10:in 'synchronize'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:74:in 'process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:141:in 'accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:109:in 'main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:448:in 'block (3 levels) in start_threads'
I assume this is due to non-ASCII text being returned by SOLR to be presented on the display page. Since I don’t get this error when running under WEBrick or the passenger stand-alone server, I assume this is due to some configuration setting(s) related to how Passenger and nginix interact. I've Googled around on this issue, most posts relate to using utf8 in your ruby code, or getting the correct ENV setting, e.g. here, but nothing that seems to address this situation.
Any suggestions on where I should be peeking or troubleshooting, or where I should be changing a configuration setting? Thank you.
Having experienced a few periods of downtime, we've recently upgraded to a production environment in Heroku (Crane database plus 2 x web dynos) however we've seen no improvement. In fact reliability seems to have decreased since upgrading.
The root cause seems to be the following exception:
PG::Error (SSL SYSCALL error: EOF detected
which causes the dyno to fail and - eventually - restart, but not before causing some downtime.
I've no idea what's causing it. Common culprits appear to be Resque and Unicorn, neither of which I'm using. We're on rails 3.2.11, on Heroku Cedar, using pg gem 1.14.1
Logs report the following at crash time:
2013-05-23T19:01:33+00:00 app[heroku-postgres]: source=HEROKU_POSTGRESQL_PINK measure.current_transaction=34490 measure.db_size=38311032bytes measure.tables=19 measure.active-connections=7 measure.waiting-connections=0 measure.index-cache-hit-rate=0.99438 measure.table-cache-hit-rate=0.8824
2013-05-23T19:01:35.123633+00:00 app[web.2]:
2013-05-23T19:01:35.123633+00:00 app[web.2]: PG::Error (SSL SYSCALL error: EOF detected
2013-05-23T19:01:35.123633+00:00 app[web.2]: ):
I have read the following: https://groups.google.com/forum/?fromgroups#!topic/heroku/a6iviwAFgdY but can't find anything that might help.
https://gist.github.com/ktopping/5657474
The above fixes the exception, which is useful (as it should declutter my logs, and even help speed up reconnecting to the database) but doesn't actually stop my main issue which is Heroku web dynos crashing more often than I would like.
Am investigating some other routes (Unicorn, rack-timeout).
Recently, we are facing a strange problem where some XHR requests, sent from Passenger to our application server, fail to execute properly.
Please have look at following chain of events:
LB Passenger sends a request to the application server.
Rails returns a NoMethodError exception when trying to figure out which controller action this request is supposed to be processed by.
Rails error log:
Started POST "/user/" for 194.29.65.21 at 2013-03-25 13:14:38 +0100
NoMethodError (undefined method `[]' for nil:NilClass):
Here is a log when Rails is able to figure out which controller/action a request should be routed to:
Started POST "/User/" for 194.29.65.21 at 2013-03-25 13:14:38 +0100
Processing by Web::UserController#index as
Parameters: {"c"=>"1", "fref"=>""}
This error gets propagated to the application server passenger which throws the following error:
[ pid=20071 thr=3074915216 file=ext/nginx/HelperAgent.cpp:577
time=2013-03-24 22:25:58.257 ]: Uncaught exception in PassengerServer
client thread: exception: Cannot read response from backend
process: Connection reset by peer (104) backtrace:
in 'void Client::forwardResponse(Passenger::SessionPtr&, Passenger::FileDescriptor&)' (HelperAgent.cpp:361)
in 'void Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:503)
in 'void Client::threadMain()' (HelperAgent.cpp:596)
And finally LB Nginx gives this error: up stream status = 500 error.
This error comes randomly for a given route. Even the frequency of occurrence of this error changes randomly.
System details:
Ruby: 1.9.2p180
Rails : 3.0.5
Phusion Passenger version 3.0.5
nginx version: n
We think this happens because Passenger is failing to send the whole request data to Rails which results in Rails failing to decide the valid controller action.
Review passenger config for nginx, I have experienced a lot issues with passenger for nginx.
http://www.modrails.com/documentation/Users%20guide%20Nginx.html#PassengerSpawnMethod specially verify this configuration and and add this in nginx.conf in parent node.
We just debugged and found that issue is not with XHR but with POST requests.
Also right now we are using passenger 3.0.5, does any know if upgrading to passenger 3.0.19 help?