Recently, we are facing a strange problem where some XHR requests, sent from Passenger to our application server, fail to execute properly.
Please have look at following chain of events:
LB Passenger sends a request to the application server.
Rails returns a NoMethodError exception when trying to figure out which controller action this request is supposed to be processed by.
Rails error log:
Started POST "/user/" for 194.29.65.21 at 2013-03-25 13:14:38 +0100
NoMethodError (undefined method `[]' for nil:NilClass):
Here is a log when Rails is able to figure out which controller/action a request should be routed to:
Started POST "/User/" for 194.29.65.21 at 2013-03-25 13:14:38 +0100
Processing by Web::UserController#index as
Parameters: {"c"=>"1", "fref"=>""}
This error gets propagated to the application server passenger which throws the following error:
[ pid=20071 thr=3074915216 file=ext/nginx/HelperAgent.cpp:577
time=2013-03-24 22:25:58.257 ]: Uncaught exception in PassengerServer
client thread: exception: Cannot read response from backend
process: Connection reset by peer (104) backtrace:
in 'void Client::forwardResponse(Passenger::SessionPtr&, Passenger::FileDescriptor&)' (HelperAgent.cpp:361)
in 'void Client::handleRequest(Passenger::FileDescriptor&)' (HelperAgent.cpp:503)
in 'void Client::threadMain()' (HelperAgent.cpp:596)
And finally LB Nginx gives this error: up stream status = 500 error.
This error comes randomly for a given route. Even the frequency of occurrence of this error changes randomly.
System details:
Ruby: 1.9.2p180
Rails : 3.0.5
Phusion Passenger version 3.0.5
nginx version: n
We think this happens because Passenger is failing to send the whole request data to Rails which results in Rails failing to decide the valid controller action.
Review passenger config for nginx, I have experienced a lot issues with passenger for nginx.
http://www.modrails.com/documentation/Users%20guide%20Nginx.html#PassengerSpawnMethod specially verify this configuration and and add this in nginx.conf in parent node.
We just debugged and found that issue is not with XHR but with POST requests.
Also right now we are using passenger 3.0.5, does any know if upgrading to passenger 3.0.19 help?
Related
I build a backend service (nginx + passenger + rails) on AWS.
My service is running as expected most of the time but sometimes goes wrong when passenger creates a weird new process to handle request:
[INFO] Started POST "/apixxxxxx"
[FATAL] LoadError (Expected /xxxxxxxx/app/controllers/application_controller.rb to define ApplicationController):
[INFO] Started POST "/apixxxxxx"
[FATAL] ActionController::RoutingError (uninitialized constant ApplicationController):
[INFO] Started POST "/apioooooo"
[FATAL] ActionController::RoutingError (uninitialized constant ApplicationController):
...
It has some signature on this weird process:
Other ruby process still works fine when this specific weird ruby process keeps failing on every new request.
If error happens, it always starts by the "LoadError" in the beginning of the process's first request.
This kind of "uninitialized constant" error keeps forever as long as this process is alive.
*new added information:
According to rails log, the same api can reply without error in other 'normal' passenger process. I exclude the problem about typo or rails naming convention.
Here is some information of my machine:
ec2 type : m4.large
storage : general purpose SSD (use EBS)
system : ubuntu 14.04.2 (not AWS version)
application : version 1.8.1, passenger 5.0.26, Rails 3.2.22.2, ruby 2.2.4p230, sidekiq 4.1.1
I also googled some keyword like "AWS EBS volume's IOPS limit" or "sidekiq thread" with this LoadError issue but no luck. Is there any new direction to investigate this problem?
I have a situation where I'm dynamically generating a pdf document using Ruby on Rails [send_data call] (http://apidock.com/rails/v2.3.8/ActionController/Streaming/send_data).(Rails 2.3.18 ... its a legacy app),
The pdf is always generated fine the first time, but on reload, or making subsequent calls to the same endpoint (doesn't matter what the parameters are in the url) the request just stalls forever. Its the same from the browser, using curl from the commandline ... doesn't matter where or what browser is used.
If I kill the request (by x'ing out of it in the browser, for example, and retrying it) the call succeeds
In tracing the output using logs, I found out that the hung request never gets to the Rails server (Phusion passenger instance), because there are no Rails logs produced for the hung call. They also don't show up in Apache Access Logs since apparently the request never succeeds.
Sometimes in the log, I'll see
[ 2015-12-30 16:16:14.0423 14334/7fbb1a32d700 apache2/Hooks.cpp:718 ]: Unexpected error in mod_passenger: Cannot connect to the helper agent at /tmp/passenger.1.0.11825/generation-5/request
Backtrace:
in 'Passenger::FileDescriptor Hooks::connectToHelperAgent()' (Hooks.cpp:283)
in 'int Hooks::handleRequest(request_rec*)' (Hooks.cpp:532)
Any ideas?
Since this afternoon, access to the backend of my spree shop is suddenly unavailable. When I try to visit any page in the backend (/admin/users, /admin/orders, etc) the page just loads for a long time, until it times out and I get the generic error page.
When I look into the logs I always see either :
Processing by Spree::Admin::OrdersController#index as HTML
Completed 500 Internal Server Error in 127259ms
** [Airbrake] Success: Net::HTTPOK
Errno::ETIMEDOUT (Connection timed out - connect(2)):
app/middleware/flash_session_cookie_middleware.rb:18:in `call'
or
Processing by Spree::Admin::OrdersController#index as HTML
Completed 500 Internal Server Error in 127520ms
** [Airbrake] Success: Net::HTTPOK
SocketError (getaddrinfo: Name or service not known):
app/middleware/flash_session_cookie_middleware.rb:17:in `call'
This started happening after the last deployment to production, which only changed images and stylesheets. I can't reproduce the error locally, despite having the same code and an exact copy of the production database.
I'm using Spree version 2.0.3
Run Spree::Config[:check_for_spree_alerts] = false in your console to fix this. You may also want to add this line to your initializers/spree.rb to ensure check_for_spree_alerts is not re-enabled in the future.
This is happening because the Spree Alerts website has been discontinued. See: https://github.com/spree/spree/pull/6516
To be specific, this is happening because when logging into the backend, Spree 2.0.x checks for any alerts from the Spree website here https://github.com/spree/spree/blob/2-0-stable/backend/app/controllers/spree/admin/base_controller.rb#L39 which then calls alert.rb:14 :
HTTParty.get('http://alerts.spreecommerce.com/alerts.json', query: params).parsed_response
At the moment, alerts.spreecommerce.com has been discontinued and is timing out, which explains the errors you're receiving.
The Spree Alerts code has been removed as of Spree 2.3, so you could also upgrade to that version to resolve this issue.
I'm using Rails 3.2.3 and on Heroku Bamboo stack using Mailgun to send emails. I randomly get this error and have no idea what it means or where it comes from:
Started POST "/invites" for 67.250.49.100 at 2012-05-14 20:17:41 -0700
Sent mail to example#example.com (10147ms)
SocketError (getaddrinfo: Name or service not known):
app/controllers/invites_controller.rb:43:in `invite_friend'
app/controllers/invites_controller.rb:9:in `create'
Weirdly, sometimes it works and rarely it'll throw this error. Any ideas? I can't seem to regularly reproduce...
Need some help here.
I have deployed spree (0.70.3) on slicehost (ubuntu, ruby1.8.7, Rails
3.1.3), installed the spree gem, loaded in the sample data
successfully following the steps on spree startup manual and I am
using passenger (it works on localhost:3000). When I upload to my
server, however, instead of seeing the spree interface, I am getting
an error message. From the production.log:
Processing by HomeController#index as HTML Completed 500 Internal
Server Error in 22ms ActiveRecord::StatementInvalid (Could not find
table 'product_groups'):
Any suggestions how I can get this to work?
Website: http://www.nutxboutique.com
production.log is the rails log. It appears your problem is occurring before rails completes loading. If you're using apache, check/post the apache logs, it might have more clues as to what's going on.