I am running Unicorn with a standard Ruby on Rails application on Ubuntu. The problem I am having is that intermittently my unicorn processes will timeout for no reason.
The same request will work 90% of the time and timeout the other 10% of the time.
The Rails log looks like this
2015-07-30 14:33:52,519 DEBG 'web' stdout output:
Jul 30 14:33:52 3f48d861780b rails[148]: Started GET "/request_url" for 00.00.00.00 at 2015-07-30 14:33:52 +0000
2015-07-30 14:34:23,257 DEBG 'web' stderr output:
E, [2015-07-30T14:34:23.252900 #18] ERROR -- : worker=4 PID:148 timeout (31s > 30s), killing
E, [2015-07-30T14:34:23.256257 #18] ERROR -- : reaped #<Process::Status: pid 148 SIGKILL (signal 9)> worker=4
2015-07-30 14:34:23,261 DEBG 'web' stderr output:
I, [2015-07-30T14:34:23.261294 #215] INFO -- : worker=4 ready
The nginx error.log shows this error:
2015/07/30 14:20:43 [error] 2368#0: *365 upstream prematurely closed connection while reading response header from upstream, client: 00.00.00.00, server: ~^(www\.)?(.+)$, request: "GET /request_url HTTP/1.1", upstream: "http://00.00.00.00:5000/request_url", host: "myapp.com"
Does anyone know what could be causing this?
Related
I'm new to Ruby on Rails and helping on an existing project. I ran rails server, and received the following output
=> Booting Unicorn
=> Rails 4.2.5.2 application starting in development on http://localhost:3000
=> Run `rails server -h` for more startup options
=> Ctrl-C to shutdown server
I, [2016-12-05T13:44:05.166061 #15228] INFO -- : listening on addr=0.0.0.0:8080 fd=11
I, [2016-12-05T13:44:05.185700 #15228] INFO -- : master process ready
I, [2016-12-05T13:44:05.187404 #15242] INFO -- : worker=0 ready
I, [2016-12-05T13:44:05.187827 #15243] INFO -- : worker=1 ready
I, [2016-12-05T13:44:05.188412 #15244] INFO -- : worker=2 ready
^CI, [2016-12-05T13:45:11.233928 #15228] INFO -- : reaped #<Process::Status: pid 15244 exit 0> worker=2
I, [2016-12-05T13:45:11.234060 #15228] INFO -- : reaped #<Process::Status: pid 15243 exit 0> worker=1
I, [2016-12-05T13:45:11.234227 #15228] INFO -- : reaped #<Process::Status: pid 15242 exit 0> worker=0
Which seems to indicate that the server is up and running. However, I am unable to connect to http://localhost:3000, it gives me an error connection refused. Does anybody know how I can go about debugging this?
With rails 4.2.0, the server binds to localhost by default, instead of 0.0.0.0. When working with a rails in a virtual box, accessing the server from the host computer, the binding address needs to be 0.0.0.0
Start rails server with -b0.0.0.0 to make the rails server accessible from the host computer/browser.
http://guides.rubyonrails.org/4_2_release_notes.html#default-host-for-rails-server https://github.com/samuelkadolph/unicorn-rails/issues/12#issuecomment-60875268
or try passing different port
rails s -p 3001
Try checking your hosts file.
Linux /etc/hosts
Windows c:\Windows\System32\drivers\etc\hosts
and editing the localhosts to serve 127.0.0.1. or move it so it is last. Rails will take the last mention of localhost and use it.
Or you can go even deper and try using telnet localhost 3000 to tell if the port is unavailable at the TCP level.
My hosts file looks something like
127.0.0.1 localhost
127.0.1.1 myPC
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
I'm developing an application where a user can upload 1-3 images at some point. However, this does not work. When submitting the form, nothing will come through to Unicorn. The only piece of log information I have is this from nginx:
2014/09/24 15:28:05 [error] 7301#0: *23953 readv() failed (104: Connection reset by peer) while reading upstream, client: <my IP>, server: localhost, request: "POST /products HTTP/1.1", upstream: "http://unix:/tmp/<appname>/unicorn.sock:/products", host: "<domain>", referrer: "<domain>/products/new"
Also, my browser will try to load the page for some time, and then show a blank page, with the response header giving a 500.
EDIT: okay, as it seems, rack causes a crash when processing file uploads:
E, [2014-09-24T21:22:33.505511 #18252] ERROR -- : app error: invalid byte sequence in UTF-8 (ArgumentError)
E, [2014-09-24T21:22:33.506288 #18252] ERROR -- : kernel/common/string.rb:103:in `=~'
E, [2014-09-24T21:22:33.506392 #18252] ERROR -- : /srv/<appname>/.rvm/gems/rbx-2.2.10/gems/rack-1.5.2/lib/rack/multipart/parser.rb:92:in 'get_current_head_and_filename_and_content_type_and_name_and_body'
Check your unicorn configuration in unicorn.rb and try to modify the timeout
After Serval days testing for how to deploy my ROR project on AWS EC2.
Rubber,eb,at last I decided to use OPSWORKS because of a video in youtube.
But the deployment still hard.
OK,My question is:How to debug Ruby On Rails on AWS OpsWorks?
I've know
In the web console,we can get deploy log.
Via ssh,we can get [apptest].access.log,error.log,access.log in the path
/var/log/nginx/(I test with nginx and Unicorn)
But all of below is log about deployment,and few logs/infos about access website.
For example:(test.access.log)
133.255.255.124 - - [25/Jun/2014:15:10:03 +0000] "GET / HTTP/1.1" 500 5 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.76.4 (KHTML, like Gecko) Version/7.0.4 Safari/537.76.4"
133.255.255.124 - - [25/Jun/2014:15:10:19 +0000] "GET / HTTP/1.1" 500 5 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.76.4 (KHTML, like Gecko) Version/7.0.4 Safari/537.76.4"
while I got 500 .where can I found detail about this information?
Here is error.log in nginx
1. 2014/06/25 13:35:59 [emerg] 5502#0: still could not bind()
2. 2014/06/25 15:23:06 [crit] 11593#0: *2 connect() to
unix:/srv/www/test/shared/sockets/unicorn.sock failed (2: No such
file or directory) while connecting to upstream, client:
133.255.255.124, server: test, request: "GET / HTTP/1.1", upstream: "http://unix:/srv/www/test/shared/sockets/unicorn.sock:/", host:
"55.218.118.37"
you can find during 500 return ,error.log info is useless,and access.log is nothing to show.
Where Can I get detail Debug info while visit test website,like tomcat server
Thanks a lot.
Here is something in unicorn.stderr.log,and production.log unicorn.stdout.log
is nothing to show.even I set config.log_level = :debug in production.rb and restart the web server.
I, [2014-06-25T23:45:18.172877 #6436] INFO -- : executing ["/home/deploy/.bundler/test/ruby/2.0.0/bin/unicorn_rails", "--env", "production", "--daemonize", "-c", "/srv/www/test/shared/config/unicorn.conf", {10=>#<Kgio::UNIXServer:fd 10>}] (in /srv/www/test/releases/20140625234506)
I, [2014-06-25T23:45:19.196006 #6436] INFO -- : inherited addr=/srv/www/test/shared/sockets/unicorn.sock fd=10
I, [2014-06-25T23:45:19.196505 #6436] INFO -- : Refreshing Gem list
I, [2014-06-25T23:45:23.769647 #6449] INFO -- : worker=0 ready
I, [2014-06-25T23:45:23.811102 #6436] INFO -- : master process ready
I, [2014-06-25T23:45:23.848092 #6452] INFO -- : worker=1 ready
I, [2014-06-25T23:45:24.129596 #5830] INFO -- : reaped #<Process::Status: pid 5852 exit 0> worker=0
I, [2014-06-25T23:45:24.129878 #5830] INFO -- : reaped #<Process::Status: pid 5855 exit 0> worker=1
I, [2014-06-25T23:45:24.129967 #5830] INFO -- : master complete
You would need to do something like the following after logging in :
sudo su deploy
cd /srv/www/#{application_name}/shared/log/
The files are stored under :
/srv/www/#{application_name}/shared/log
➜ log ls -lsh *.log
32K -rw-r--r-- 1 deploy www-data 32K Jun 25 20:45 cron-error.log
4.0K -rw-r--r-- 1 deploy www-data 1.6K Jun 25 20:30 cron.log
192K -rw-r--r-- 1 deploy www-data 188K Jun 25 20:45 newrelic_agent.log
48M -rw-r--r-- 1 deploy www-data 47M Jun 25 20:55 staging.log
0 -rw-r--r-- 1 deploy www-data 0 Jun 24 06:46 unicorn.stderr.log
4.0K -rw-r--r-- 1 deploy www-data 3.1K Jun 25 08:49 unicorn.stdout.log
NOTE : #{application_name} is whatever your shortcode is for the application.
I'm running a Rails 4 app on a Digital Ocean VPS and I'm getting an Nginx 504 gateway time out error on a particular HTTP request. All requests succeed, but this particular one does not. I am running in production mode and there are no errors or exceptions in my Rails log.
The Nginx log looks like this:
2014/04/23 05:08:22 [error] 6946#0: *109805 upstream timed out
(110: Connection timed out) while reading response header from upstream, client:
174.126.176.107, server: new.pilotpro.com, request:
"GET /dropbox/sync?right_now=1398115006.949 HTTP/1.1", upstream:
"http://127.0.0.1:8080/dropbox/sync?right_now=1398115006.949",
host: "new.pilotpro.com"
And the Unicorn log looks like this:
E, [2014-04-23T04:52:06.285058 #5424] ERROR -- : worker=1 PID:5429 timeout (61s > 60s), killing
E, [2014-04-23T04:52:06.297947 #5424] ERROR -- : reaped #<Process::Status: pid 5429 SIGKILL (signal 9)> worker=1
I, [2014-04-23T04:52:06.298207 #5424] INFO -- : worker=1 spawning...
I, [2014-04-23T04:52:06.299481 #5457] INFO -- : worker=1 spawned pid=5457
As far as I can tell nothing is wrong other than the fact that the request is timing out. Is this simply a matter of increasing Nginx's timeout limits?
I'd appreciate any advice. Thanks!
Using carrier wave to upload a pdf document via CarrierWave. Then using RMagick to save the first page of the pdf as a thumbnail.jpg. I have this working correctly in my development environment (Mac OS X 10.9), however its failing in production (Nginx + Unicorn on Ubuntu 12.04 LTS).
It appears that the upload completes successfully, based on watching the upload percentage in the browser status bar. However, something errors in the processing, and I get Rails error page. The problem is, there is nothing in my production.log.
I get the following in unicorn.log:
I, [2013-12-12T01:22:55.232431 #2935] INFO -- : worker=1 ready
I, [2013-12-12T01:22:55.246992 #2932] INFO -- : worker=0 ready
E, [2013-12-12T01:26:00.666715 #2929] ERROR -- : worker=1 PID:2935 timeout (31s > 30s), killing
E, [2013-12-12T01:26:00.676250 #2929] ERROR -- : reaped #<Process::Status: pid 2935 SIGKILL (signal 9)> worker=1
I, [2013-12-12T01:26:00.676382 #2929] INFO -- : worker=1 spawning...
I, [2013-12-12T01:26:00.681087 #3502] INFO -- : worker=1 spawned pid=3502
I, [2013-12-12T01:26:00.681399 #3502] INFO -- : Refreshing Gem list
SECURITY WARNING: No secret option provided to Rack::Session::Cookie.
This poses a security threat. It is strongly recommended that you
provide a secret to prevent exploits that may be possible from crafted
cookies. This will not be supported in future versions of Rack, and
future versions will even invalidate your existing user cookies.
Called from: /home/deployer/apps/______/shared/bundle/ruby/1.9.1/gems/actionpack-3.2.9/lib/action_dispatch/middleware/session/abstract_store.rb:28:in `initialize'.
And this in my nginx error.log
2013/12/12 01:26:00 [error] 2597#0: *9 upstream prematurely closed connection while reading response header from upstream, client: ###.###.###.###, server: ___.org, request: "POST /admin/explores HTTP/1.1", upstream: "http://unix:/tmp/unicorn.sock:/admin/explores", host: "___.org", referrer: "https://___.org/admin/explores/new"
Not sure how to get a better error description of what's going on. Would appreciate any help.
Thanks!
Ruby 1.9.3 + Rails 3.2.9
The line E, [2013-12-12T01:26:00.666715 #2929] ERROR -- : worker=1 PID:2935 timeout (31s > 30s), killing from your unicorn.log tells you the problem: you have a timeout. Increase unicorn's timeout to 60 seconds by setting timeout 60 in your unicorn.rb. More information: http://unicorn.bogomips.org/Unicorn/Configurator.html#method-i-timeout