I'm developing an AngularJS (front-end) running in an Nginx web server that sends request to a Rails API (backend) running in a Unicorn application server.
I recognize I'm just a developer and I have no idea about servers administration, so I just put the servers and start them.
The application is working, however, Unicorn has a strange behaviour. When I start it I always get this error:
roberto#ubuntu:~/dev/scripts$ ./start_unicorn.sh
I, [2014-06-14T11:46:06.085834 #4258] INFO -- : Refreshing Gem list
I, [2014-06-14T11:46:11.591592 #4258] INFO -- : listening on addr=0.0.0.0:8080 fd=10
I, [2014-06-14T11:46:12.087321 #4258] INFO -- : master process ready
I, [2014-06-14T11:46:12.151320 #4263] INFO -- : worker=0 ready
I, [2014-06-14T11:46:12.150526 #4266] INFO -- : worker=1 ready
E, [2014-06-14T11:46:39.112668 #4258] ERROR -- : worker=0 PID:4263 timeout (16s > 15s), killing
E, [2014-06-14T11:46:39.112898 #4258] ERROR -- : worker=1 PID:4266 timeout (16s > 15s), killing
E, [2014-06-14T11:46:39.118081 #4258] ERROR -- : reaped #<Process::Status: pid 4263 SIGKILL (signal 9)> worker=0
E, [2014-06-14T11:46:39.118634 #4258] ERROR -- : worker=1 PID:4266 timeout (16s > 15s), killing
E, [2014-06-14T11:46:39.121820 #4258] ERROR -- : reaped #<Process::Status: pid 4266 SIGKILL (signal 9)> worker=1
I, [2014-06-14T11:46:39.172067 #4284] INFO -- : worker=1 ready
I, [2014-06-14T11:46:39.172620 #4281] INFO -- : worker=0 ready
It takes some seconds until it responds. And this happens continuosly.
I guess I'm missing some configuration but no idea...
If you need any more details, such as config files, just let me know
Do you have your assets precompiled ? are you in production ?
If not when you start your server and get your first request, rails will try to compile your assets which can take more than 15 seconds and reach the unicorn timeout.
In your start.sh you should have somewhere
export RAILS_ENVIRONEMENT=production
And during you deployment you should have :
rake assets:precompile
Related
I'm using SideKiq and Redis to deliver emails asynchronously on my Rails (4.2) site but some emails take too long to send. Most emails take between 5 and 15 minutes to to be delivered (I'm not sure if this is normal and may be part of the problem). For most emails I send this wait is fine, but for confirmation emails, I need them to be sent more or less immediately.
Normally when I send an email, I send it like this:
UserMailer.delay.my_email(args)
To send an email immediately, I have tried all of the following:
UserMailer.my_email(args).deliver
UserMailer.my_email(args).deliver!
UserMailer.my_email(args).deliver_now
UserMailer.my_email(args).deliver_now!
However all of these result in a 504 error. When I try to send an email, the page waits for several seconds and then I get in my unicorn.log is:
Started POST "/my-path/" for 0.0.0.0 at 0000-00-00 00:00:00 +0000
Processing by MyController#my_path as HTML
Parameters: {"utf8"=>"✓", "authenticity_token"=>"...", ...}
Rendered user_mailer/my_email.html.haml within layouts/user_mailer (0.0ms)
E, [0000-00-00T00:00:00.000000 #00000] ERROR -- : worker=1 PID:00000 timeout (61s > 60s), killing
E, [0000-00-00T00:00:00.000000 #00000] ERROR -- : reaped #<Process::Status: pid 00000 SIGKILL (signal 9)> worker=1
I, [0000-00-00T00:00:00.000000 #00000] INFO -- : worker=1 spawning...
I, [0000-00-00T00:00:00.000000 #00001] INFO -- : worker=1 spawned pid=00001
I, [0000-00-00T00:00:00.000000 #00001] INFO -- : Refreshing Gem list
I, [0000-00-00T00:00:00.000000 #00001] INFO -- : worker=1 ready
This is my sidekiq.yml (the both production and preview are taking a very long time):
---
:concurrency: 5
:logfile: log/sidekiq.log
preview:
:pidfile: /home/unicorn/sidekiq_preview.pid
production:
:pidfile: /home/unicorn/sidekiq_production.pid
:concurrency: 15
I'm not getting anything in my sidekiq.log or mail.log.
Rails 4.2
I run "vagrant up".
I have Virtual Box installed.
I am working on a Mac OSX 10.11
I ssh to the vagrant environment.
I start my app and go to the front page of my app. This is slow, but it works.
Then try to see another page in the browser. It takes almost 2 minutes for the page to render, during which time I see:
./bin/unicorn_rails
I, [2016-02-12T23:59:43.291050 #20214] INFO -- : listening on addr=0.0.0.0:8080 fd=11
I, [2016-02-12T23:59:43.291348 #20214] INFO -- : worker=0 spawning...
I, [2016-02-12T23:59:43.293053 #20214] INFO -- : master process ready
I, [2016-02-12T23:59:43.297287 #20243] INFO -- : worker=0 spawned pid=20243
I, [2016-02-12T23:59:43.297441 #20243] INFO -- : Refreshing Gem list
I, [2016-02-13T00:00:03.677384 #20243] INFO -- : worker=0 ready
E, [2016-02-13T00:06:42.781557 #20214] ERROR -- : worker=0 PID:20243 timeout (61s > 60s), killing
E, [2016-02-13T00:06:43.051335 #20214] ERROR -- : reaped #<Process::Status: pid 20243 SIGKILL (signal 9)> worker=0
I, [2016-02-13T00:06:43.074907 #20214] INFO -- : worker=0 spawning...
I, [2016-02-13T00:06:43.259004 #20263] INFO -- : worker=0 spawned pid=20263
I, [2016-02-13T00:06:43.270408 #20263] INFO -- : Refreshing Gem list
I, [2016-02-13T00:07:07.999131 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.020710 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.020774 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.020798 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.020820 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.398987 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.399073 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.516064 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.564657 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.608567 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.656556 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.716912 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.770429 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.820702 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.872056 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.941071 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:09.010447 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:09.977588 #20263] INFO -- : worker=0 ready
I have two questions:
1.) why is nothing logged to the terminal when I go to the front page? I'm used to seeing a lot of data logged.
2.) Has anyone seen this error before? "SIGWINCH ignored because we're not daemonized"
It happened because of terminal is resized after running
./bin/unicorn_rails
Here is link of issue (issue link)
Unicorn error log:
INFO -- : Refreshing Gem list
I, [2014-10-26T07:46:40.418950 #25903] INFO -- : unlinking existing socket=/home/ubuntu/rails/shared/tmp/.sock
I, [2014-10-26T07:46:40.452871 #25903] INFO -- : listening on addr=/home/ubuntu/rails/shared/tmp/.sock fd=10
I, [2014-10-26T07:46:40.453466 #25903] INFO -- : listening on addr=0.0.0.0:8080 fd=11
I, [2014-10-26T07:46:43.816336 #25903] INFO -- : master process ready
I, [2014-10-26T07:46:49.218342 #25907] INFO -- : worker=0 ready
I, [2014-10-26T07:46:50.778582 #25910] INFO -- : worker=1 ready
I, [2014-10-26T07:46:51.022575 #25913] INFO -- : worker=2 ready
I, [2014-10-26T07:46:51.137159 #25916] INFO -- : worker=3 ready
E, [2014-11-14T06:53:26.084384 #25903] ERROR -- : reaped #<Process::Status: pid 25907 SIGKILL (signal 9)> worker=0
E, [2014-11-14T06:53:26.347536 #25903] ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM)
It is clearly a memory issue.
You can easily detect it by reading the error message:
Cannot allocate memory - fork(2) (Errno::ENOMEM)
Now you have two options:
Get an instance with more memory on EC2 -try t2.small I assume it would be enough.
Reduce the number of Unicorn workers as each of them consumes memory by its own.
I've got a large 2.3 Rails app running on Unicorn. I'm using Unicorn, so that I can have zero downtime deployments. However, I've noticed that the first request after a restart is very slow.
First request:
Completed 304 Not Modified in 2771.8ms (ActiveRecord: 98.6ms)
Second request:
Completed 304 Not Modified in 94.4ms (ActiveRecord: 26.9ms)
I do have preload_app true and I am re-establishing the db-connection in the after-fork.
I have no idea how to explain the 2600ms divergence between these two values.
Does anyone have any thoughts? Really, what I am looking for are ways to debug this issue.
UPDATE
Here is my unicorn.log after a restart:
I, [2014-05-16T13:46:26.529305 #11637] INFO -- : executing ["/data/app/current/ey_bundler_binstubs/unicorn", "-E", "staging", "-c", "/data/app/shared/config/custom_unicorn.rb", "-D", "/data/app/current/config.ru", {12=>#<Kgio::UNIXServer:fd 12>}] (in /data/app/releases/20140516184210)
I, [2014-05-16T13:46:27.566115 #11637] INFO -- : inherited addr=/var/run/engineyard/unicorn_afar.sock fd=12
I, [2014-05-16T13:46:27.566551 #11637] INFO -- : Refreshing Gem list
I, [2014-05-16T13:47:13.036963 #8247] INFO -- : reaped #<Process::Status: pid 8681 exit 0> worker=3
I, [2014-05-16T13:47:14.093196 #8247] INFO -- : reaped #<Process::Status: pid 8670 exit 0> worker=2
I, [2014-05-16T13:47:14.100269 #12047] INFO -- : worker=0 ready
I, [2014-05-16T13:47:15.105249 #12063] INFO -- : worker=1 ready
I, [2014-05-16T13:47:15.114038 #8247] INFO -- : reaped #<Process::Status: pid 8655 exit 0> worker=1
I, [2014-05-16T13:47:15.957970 #8247] INFO -- : reaped #<Process::Status: pid 8638 exit 0> worker=0
I, [2014-05-16T13:47:15.958159 #8247] INFO -- : master complete
I, [2014-05-16T13:47:16.087761 #12082] INFO -- : worker=2 ready
I, [2014-05-16T13:47:16.876129 #11637] INFO -- : master process ready
I, [2014-05-16T13:47:17.102994 #12095] INFO -- : worker=3 ready
And here is the first request on my rails logs:
Started GET "/" for 70.XX.XXX.XXX at 2014-05-16 13:47:51 -0700
Processing by HomeController#index as HTML
(1.1ms) SELECT ..... <regular controller/ActiveRecord queries>
Completed 304 Not Modified in 2724.8ms (ActiveRecord: 98.9ms)
First request to Rails app is very slow may be relevant.
Maybe there is a dependency that is loading / running on the first page load?
Some ideas:
Check the rails log to see if there's anything funky going on
Is this just happening for Unicorn or with other servers too?
Add log statements with time stamps to get a sense for what part of the app is taking a long time
Try using ruby prof
I'm following along with a screencasthttp://railscasts.com/episodes/335-deploying-to-a-vps?view=asciicastthat that instructs how to deploy to a VPS with nginx and unicorn (I'm installing a Ruby on Rails app on ubuntu). After installing the various services (nginx, postgres, unicorn, ruby) and running cap deploy:cold, my app (as expected) displayed the default nginx page, which the screencast instructs to remove in the following way so that nginx serves the production app
deployer#li349-144:~$ sudo rm /etc/nginx/sites-enabled/default
[sudo] password for deployer:
deployer#li349-144:~$ sudo service nginx restart
Restarting nginx: nginx.
I've previously deployed applications where, after removing this default page, the intended application appears. However, after I run these commands on my server this time and navigate to the ip address, the browser tells me it can't connect to the server. Not sure what was wrong, I tried to restart nginx, unicorn and postgres (all of which restarted successfully) but I got the same error message in the browser.
There's a log directory on my cloud server with a unicorn.log logfile and a production.log but neither of them indicate any problems. For example, the production.log only indicates that database migrations have been run, and this is the unicorn.log
I, [2013-07-11T23:26:57.717019 #6664] INFO -- : listening on addr=/tmp/unicorn.qbruby3.sock fd=10
I, [2013-07-11T23:26:57.717480 #6664] INFO -- : worker=0 spawning...
I, [2013-07-11T23:26:57.718428 #6664] INFO -- : worker=1 spawning...
I, [2013-07-11T23:26:57.719538 #6667] INFO -- : worker=0 spawned pid=6667
I, [2013-07-11T23:26:57.719713 #6667] INFO -- : Refreshing Gem list
I, [2013-07-11T23:26:57.722070 #6664] INFO -- : master process ready
I, [2013-07-11T23:26:57.726747 #6670] INFO -- : worker=1 spawned pid=6670
I, [2013-07-11T23:26:57.727030 #6670] INFO -- : Refreshing Gem list
I, [2013-07-11T23:27:09.930162 #6670] INFO -- : worker=1 ready
I, [2013-07-11T23:27:10.084362 #6667] INFO -- : worker=0 ready
I, [2013-07-12T01:05:52.638290 #6664] INFO -- : reloading config_file=/home/michael/apps/qbruby3/shared/config/unicorn.rb
I, [2013-07-12T01:05:52.668897 #6664] INFO -- : done reloading config_file=/home/michael/apps/qbruby3/shared/config/unicorn.rb
I, [2013-07-12T01:05:52.858858 #6664] INFO -- : reaped #<Process::Status: pid 6667 exit 0> worker=0
I, [2013-07-12T01:05:52.859032 #6664] INFO -- : worker=0 spawning...
I, [2013-07-12T01:05:52.860609 #7212] INFO -- : worker=0 spawned pid=7212
I, [2013-07-12T01:05:52.860839 #7212] INFO -- : Refreshing Gem list
I, [2013-07-12T01:05:52.875751 #6664] INFO -- : reaped #<Process::Status: pid 6670 exit 0> worker=1
I, [2013-07-12T01:05:52.875944 #6664] INFO -- : worker=1 spawning...
I, [2013-07-12T01:05:52.877405 #7215] INFO -- : worker=1 spawned pid=7215
I, [2013-07-12T01:05:52.877651 #7215] INFO -- : Refreshing Gem list
I, [2013-07-12T01:06:02.191290 #7212] INFO -- : worker=0 ready
I, [2013-07-12T01:06:02.269397 #7215] INFO -- : worker=1 ready
In this situation, what else could I check to identify the reason the app is not appearing?