Unicorn error log:
INFO -- : Refreshing Gem list
I, [2014-10-26T07:46:40.418950 #25903] INFO -- : unlinking existing socket=/home/ubuntu/rails/shared/tmp/.sock
I, [2014-10-26T07:46:40.452871 #25903] INFO -- : listening on addr=/home/ubuntu/rails/shared/tmp/.sock fd=10
I, [2014-10-26T07:46:40.453466 #25903] INFO -- : listening on addr=0.0.0.0:8080 fd=11
I, [2014-10-26T07:46:43.816336 #25903] INFO -- : master process ready
I, [2014-10-26T07:46:49.218342 #25907] INFO -- : worker=0 ready
I, [2014-10-26T07:46:50.778582 #25910] INFO -- : worker=1 ready
I, [2014-10-26T07:46:51.022575 #25913] INFO -- : worker=2 ready
I, [2014-10-26T07:46:51.137159 #25916] INFO -- : worker=3 ready
E, [2014-11-14T06:53:26.084384 #25903] ERROR -- : reaped #<Process::Status: pid 25907 SIGKILL (signal 9)> worker=0
E, [2014-11-14T06:53:26.347536 #25903] ERROR -- : Cannot allocate memory - fork(2) (Errno::ENOMEM)
It is clearly a memory issue.
You can easily detect it by reading the error message:
Cannot allocate memory - fork(2) (Errno::ENOMEM)
Now you have two options:
Get an instance with more memory on EC2 -try t2.small I assume it would be enough.
Reduce the number of Unicorn workers as each of them consumes memory by its own.
Related
I am getting following error while running diaspora pod (ruby application) deployed on ubuntu14.04 system
bundler: failed to load command: unicorn (/home/ubuntu/.rvm/gems/ruby-2.1.8#diaspora/bin/unicorn)
ubuntu#ip:~/diaspora$ ./script/server
WARNING: Setting port via configuration is deprecated, set listen instead. See the updated config/diaspora.yml.example.
Starting Diaspora in development mode on port 3000 with job processing inside the request cycle.
I, [2016-06-30T18:31:18.139991 #1580] INFO -- : [diaspora:default] send_command: monitor
I, [2016-06-30T18:31:18.142457 #1580] INFO -- : [diaspora:default] schedule :monitor (reason: monitor by user)
I, [2016-06-30T18:31:18.142780 #1580] INFO -- : [diaspora:default] => monitor (reason: monitor by user)
I, [2016-06-30T18:31:18.142940 #1580] INFO -- : [diaspora:default] starting async with 0.2s chain monitor []
I, [2016-06-30T18:31:18.143217 #1580] INFO -- : [Eye] <= loading: ["/home/ubuntu/diaspora/config/eye.rb"]
I, [2016-06-30T18:31:18.157191 #1580] INFO -- : [Eye] <= command: load /home/ubuntu/diaspora/config/eye.rb (0.181545199s)
I, [2016-06-30T18:31:18.158367 #1580] INFO -- : [diaspora:web] schedule :monitor (reason: monitor by user)
I, [2016-06-30T18:31:18.158632 #1580] INFO -- : [diaspora:web] => monitor (reason: monitor by user)
I, [2016-06-30T18:31:18.158705 #1580] INFO -- : [diaspora:default] <= monitor
I, [2016-06-30T18:31:18.159647 #1580] INFO -- : [diaspora:web] pid_file not found, starting...
I, [2016-06-30T18:31:18.160052 #1580] INFO -- : [diaspora:web] switch :starting [:unmonitored => :starting] (reason: monitor by user)
I, [2016-06-30T18:31:18.161873 #1580] INFO -- : [diaspora:web] daemonizing: bin/bundle exec unicorn -c config/unicorn.rb with start_grace: 2.5s, env: 'DB=mysql RAILS_ENV=development PORT=3000', <1720> (in /home/ubuntu/diaspora)
I, [2016-06-30T18:31:18.162008 #1580] INFO -- : [diaspora:web] sleeping for :start_grace 2.5
I, [2016-06-30T18:31:20.662786 #1580] INFO -- : [diaspora:web] switch :started [:starting => :up] (reason: monitor by user)
I, [2016-06-30T18:31:20.665149 #1580] INFO -- : [diaspora:web] <= monitor
bundler: failed to load command: unicorn (/home/ubuntu/.rvm/gems/ruby-2.1.8#diaspora/bin/unicorn)
W, [2016-06-30T18:31:25.663536 #1580] WARN -- : [diaspora:web] check_alive: process <1720> not found
I, [2016-06-30T18:31:25.663809 #1580] INFO -- : [diaspora:web] delete pid_file: /home/ubuntu/diaspora/tmp/pids/web.pid
I, [2016-06-30T18:31:25.664177 #1580] INFO -- : [diaspora:web] switch :crashed [:up => :down] (reason: crashed)
I, [2016-06-30T18:31:25.664615 #1580] INFO -- : [diaspora:web] schedule :check_crash (reason: crashed)
I, [2016-06-30T18:31:25.665000 #1580] INFO -- : [diaspora:web] => check_crash (reason: crashed)
W, [2016-06-30T18:31:25.665108 #1580] WARN -- : [diaspora:web] check crashed: process is down
I, [2016-06-30T18:31:25.665185 #1580] INFO -- : [diaspora:web] schedule :restore (reason: crashed)
I, [2016-06-30T18:31:25.665359 #1580] INFO -- : [diaspora:web] <= check_crash
I, [2016-06-30T18:31:25.665547 #1580] INFO -- : [diaspora:web] => restore (reason: crashed)
I, [2016-06-30T18:31:25.665635 #1580] INFO -- : [diaspora:web] pid_file not found, starting...
I, [2016-06-30T18:31:25.665866 #1580] INFO -- : [diaspora:web] switch :starting [:down => :starting] (reason: crashed)
I, [2016-06-30T18:31:25.667339 #1580] INFO -- : [diaspora:web] daemonizing: bin/bundle exec unicorn -c config/unicorn.rb with start_grace: 2.5s, env: 'DB=mysql RAILS_ENV=development PORT=3000', <1725> (in /home/ubuntu/diaspora)
I, [2016-06-30T18:31:25.667438 #1580] INFO -- : [diaspora:web] sleeping for :start_grace 2.5
I, [2016-06-30T18:31:28.168226 #1580] INFO -- : [diaspora:web] switch :started [:starting => :up] (reason: crashed)
I, [2016-06-30T18:31:28.170467 #1580] INFO -- : [diaspora:web] <= restore
bundler: failed to load command: unicorn (/home/ubuntu/.rvm/gems/ruby-2.1.8#diaspora/bin/unicorn)
W, [2016-06-30T18:31:33.168931 #1580] WARN -- : [diaspora:web] check_alive: process <1725> not found
I, [2016-06-30T18:31:33.169188 #1580] INFO -- : [diaspora:web] delete pid_file: /home/ubuntu/diaspora/tmp/pids/web.pid
I, [2016-06-30T18:31:33.169547 #1580] INFO -- : [diaspora:web] switch :crashed [:up => :down] (reason: crashed)
I, [2016-06-30T18:31:33.170001 #1580] INFO -- : [diaspora:web] schedule :check_crash (reason: crashed)
I, [2016-06-30T18:31:33.170419 #1580] INFO -- : [diaspora:web] => check_crash (reason: crashed)
W, [2016-06-30T18:31:33.170527 #1580] WARN -- : [diaspora:web] check crashed: process is down
I, [2016-06-30T18:31:33.170602 #1580] INFO -- : [diaspora:web] schedule :restore (reason: crashed)
I, [2016-06-30T18:31:33.170781 #1580] INFO -- : [diaspora:web] <= check_crash
I, [2016-06-30T18:31:33.170977 #1580] INFO -- : [diaspora:web] => restore (reason: crashed)
I, [2016-06-30T18:31:33.171062 #1580] INFO -- : [diaspora:web] pid_file not found, starting...
I, [2016-06-30T18:31:33.171282 #1580] INFO -- : [diaspora:web] switch :starting [:down => :starting] (reason: crashed)
I, [2016-06-30T18:31:33.172718 #1580] INFO -- : [diaspora:web] daemonizing: bin/bundle exec unicorn -c config/unicorn.rb with start_grace: 2.5s, env: 'DB=mysql RAILS_ENV=development PORT=3000', <1730> (in /home/ubuntu/diaspora)
I, [2016-06-30T18:31:33.172822 #1580] INFO -- : [diaspora:web] sleeping for :start_grace 2.5
Change line 172 or around there in diaspora.yml
#rails_environment: 'development'
to 'production' and uncomment it.
Setting port via configuration is deprecated, set listen instead
There's your problem right there. You need to manually compare your config/diaspora.yml file to the updated config/diaspora.yml.examle file, using a tool like meld.
they changed the line that specifies the port setting, it should now say something like:
listen: 'http://127.0.0.1:3000/'
this is line 178 in the latest template...
Rails 4.2
I run "vagrant up".
I have Virtual Box installed.
I am working on a Mac OSX 10.11
I ssh to the vagrant environment.
I start my app and go to the front page of my app. This is slow, but it works.
Then try to see another page in the browser. It takes almost 2 minutes for the page to render, during which time I see:
./bin/unicorn_rails
I, [2016-02-12T23:59:43.291050 #20214] INFO -- : listening on addr=0.0.0.0:8080 fd=11
I, [2016-02-12T23:59:43.291348 #20214] INFO -- : worker=0 spawning...
I, [2016-02-12T23:59:43.293053 #20214] INFO -- : master process ready
I, [2016-02-12T23:59:43.297287 #20243] INFO -- : worker=0 spawned pid=20243
I, [2016-02-12T23:59:43.297441 #20243] INFO -- : Refreshing Gem list
I, [2016-02-13T00:00:03.677384 #20243] INFO -- : worker=0 ready
E, [2016-02-13T00:06:42.781557 #20214] ERROR -- : worker=0 PID:20243 timeout (61s > 60s), killing
E, [2016-02-13T00:06:43.051335 #20214] ERROR -- : reaped #<Process::Status: pid 20243 SIGKILL (signal 9)> worker=0
I, [2016-02-13T00:06:43.074907 #20214] INFO -- : worker=0 spawning...
I, [2016-02-13T00:06:43.259004 #20263] INFO -- : worker=0 spawned pid=20263
I, [2016-02-13T00:06:43.270408 #20263] INFO -- : Refreshing Gem list
I, [2016-02-13T00:07:07.999131 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.020710 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.020774 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.020798 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.020820 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.398987 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.399073 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.516064 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.564657 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.608567 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.656556 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.716912 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.770429 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.820702 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.872056 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:08.941071 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:09.010447 #20214] INFO -- : SIGWINCH ignored because we're not daemonized
I, [2016-02-13T00:07:09.977588 #20263] INFO -- : worker=0 ready
I have two questions:
1.) why is nothing logged to the terminal when I go to the front page? I'm used to seeing a lot of data logged.
2.) Has anyone seen this error before? "SIGWINCH ignored because we're not daemonized"
It happened because of terminal is resized after running
./bin/unicorn_rails
Here is link of issue (issue link)
I'm developing an AngularJS (front-end) running in an Nginx web server that sends request to a Rails API (backend) running in a Unicorn application server.
I recognize I'm just a developer and I have no idea about servers administration, so I just put the servers and start them.
The application is working, however, Unicorn has a strange behaviour. When I start it I always get this error:
roberto#ubuntu:~/dev/scripts$ ./start_unicorn.sh
I, [2014-06-14T11:46:06.085834 #4258] INFO -- : Refreshing Gem list
I, [2014-06-14T11:46:11.591592 #4258] INFO -- : listening on addr=0.0.0.0:8080 fd=10
I, [2014-06-14T11:46:12.087321 #4258] INFO -- : master process ready
I, [2014-06-14T11:46:12.151320 #4263] INFO -- : worker=0 ready
I, [2014-06-14T11:46:12.150526 #4266] INFO -- : worker=1 ready
E, [2014-06-14T11:46:39.112668 #4258] ERROR -- : worker=0 PID:4263 timeout (16s > 15s), killing
E, [2014-06-14T11:46:39.112898 #4258] ERROR -- : worker=1 PID:4266 timeout (16s > 15s), killing
E, [2014-06-14T11:46:39.118081 #4258] ERROR -- : reaped #<Process::Status: pid 4263 SIGKILL (signal 9)> worker=0
E, [2014-06-14T11:46:39.118634 #4258] ERROR -- : worker=1 PID:4266 timeout (16s > 15s), killing
E, [2014-06-14T11:46:39.121820 #4258] ERROR -- : reaped #<Process::Status: pid 4266 SIGKILL (signal 9)> worker=1
I, [2014-06-14T11:46:39.172067 #4284] INFO -- : worker=1 ready
I, [2014-06-14T11:46:39.172620 #4281] INFO -- : worker=0 ready
It takes some seconds until it responds. And this happens continuosly.
I guess I'm missing some configuration but no idea...
If you need any more details, such as config files, just let me know
Do you have your assets precompiled ? are you in production ?
If not when you start your server and get your first request, rails will try to compile your assets which can take more than 15 seconds and reach the unicorn timeout.
In your start.sh you should have somewhere
export RAILS_ENVIRONEMENT=production
And during you deployment you should have :
rake assets:precompile
I've got a large 2.3 Rails app running on Unicorn. I'm using Unicorn, so that I can have zero downtime deployments. However, I've noticed that the first request after a restart is very slow.
First request:
Completed 304 Not Modified in 2771.8ms (ActiveRecord: 98.6ms)
Second request:
Completed 304 Not Modified in 94.4ms (ActiveRecord: 26.9ms)
I do have preload_app true and I am re-establishing the db-connection in the after-fork.
I have no idea how to explain the 2600ms divergence between these two values.
Does anyone have any thoughts? Really, what I am looking for are ways to debug this issue.
UPDATE
Here is my unicorn.log after a restart:
I, [2014-05-16T13:46:26.529305 #11637] INFO -- : executing ["/data/app/current/ey_bundler_binstubs/unicorn", "-E", "staging", "-c", "/data/app/shared/config/custom_unicorn.rb", "-D", "/data/app/current/config.ru", {12=>#<Kgio::UNIXServer:fd 12>}] (in /data/app/releases/20140516184210)
I, [2014-05-16T13:46:27.566115 #11637] INFO -- : inherited addr=/var/run/engineyard/unicorn_afar.sock fd=12
I, [2014-05-16T13:46:27.566551 #11637] INFO -- : Refreshing Gem list
I, [2014-05-16T13:47:13.036963 #8247] INFO -- : reaped #<Process::Status: pid 8681 exit 0> worker=3
I, [2014-05-16T13:47:14.093196 #8247] INFO -- : reaped #<Process::Status: pid 8670 exit 0> worker=2
I, [2014-05-16T13:47:14.100269 #12047] INFO -- : worker=0 ready
I, [2014-05-16T13:47:15.105249 #12063] INFO -- : worker=1 ready
I, [2014-05-16T13:47:15.114038 #8247] INFO -- : reaped #<Process::Status: pid 8655 exit 0> worker=1
I, [2014-05-16T13:47:15.957970 #8247] INFO -- : reaped #<Process::Status: pid 8638 exit 0> worker=0
I, [2014-05-16T13:47:15.958159 #8247] INFO -- : master complete
I, [2014-05-16T13:47:16.087761 #12082] INFO -- : worker=2 ready
I, [2014-05-16T13:47:16.876129 #11637] INFO -- : master process ready
I, [2014-05-16T13:47:17.102994 #12095] INFO -- : worker=3 ready
And here is the first request on my rails logs:
Started GET "/" for 70.XX.XXX.XXX at 2014-05-16 13:47:51 -0700
Processing by HomeController#index as HTML
(1.1ms) SELECT ..... <regular controller/ActiveRecord queries>
Completed 304 Not Modified in 2724.8ms (ActiveRecord: 98.9ms)
First request to Rails app is very slow may be relevant.
Maybe there is a dependency that is loading / running on the first page load?
Some ideas:
Check the rails log to see if there's anything funky going on
Is this just happening for Unicorn or with other servers too?
Add log statements with time stamps to get a sense for what part of the app is taking a long time
Try using ruby prof
I'm following along with a screencasthttp://railscasts.com/episodes/335-deploying-to-a-vps?view=asciicastthat that instructs how to deploy to a VPS with nginx and unicorn (I'm installing a Ruby on Rails app on ubuntu). After installing the various services (nginx, postgres, unicorn, ruby) and running cap deploy:cold, my app (as expected) displayed the default nginx page, which the screencast instructs to remove in the following way so that nginx serves the production app
deployer#li349-144:~$ sudo rm /etc/nginx/sites-enabled/default
[sudo] password for deployer:
deployer#li349-144:~$ sudo service nginx restart
Restarting nginx: nginx.
I've previously deployed applications where, after removing this default page, the intended application appears. However, after I run these commands on my server this time and navigate to the ip address, the browser tells me it can't connect to the server. Not sure what was wrong, I tried to restart nginx, unicorn and postgres (all of which restarted successfully) but I got the same error message in the browser.
There's a log directory on my cloud server with a unicorn.log logfile and a production.log but neither of them indicate any problems. For example, the production.log only indicates that database migrations have been run, and this is the unicorn.log
I, [2013-07-11T23:26:57.717019 #6664] INFO -- : listening on addr=/tmp/unicorn.qbruby3.sock fd=10
I, [2013-07-11T23:26:57.717480 #6664] INFO -- : worker=0 spawning...
I, [2013-07-11T23:26:57.718428 #6664] INFO -- : worker=1 spawning...
I, [2013-07-11T23:26:57.719538 #6667] INFO -- : worker=0 spawned pid=6667
I, [2013-07-11T23:26:57.719713 #6667] INFO -- : Refreshing Gem list
I, [2013-07-11T23:26:57.722070 #6664] INFO -- : master process ready
I, [2013-07-11T23:26:57.726747 #6670] INFO -- : worker=1 spawned pid=6670
I, [2013-07-11T23:26:57.727030 #6670] INFO -- : Refreshing Gem list
I, [2013-07-11T23:27:09.930162 #6670] INFO -- : worker=1 ready
I, [2013-07-11T23:27:10.084362 #6667] INFO -- : worker=0 ready
I, [2013-07-12T01:05:52.638290 #6664] INFO -- : reloading config_file=/home/michael/apps/qbruby3/shared/config/unicorn.rb
I, [2013-07-12T01:05:52.668897 #6664] INFO -- : done reloading config_file=/home/michael/apps/qbruby3/shared/config/unicorn.rb
I, [2013-07-12T01:05:52.858858 #6664] INFO -- : reaped #<Process::Status: pid 6667 exit 0> worker=0
I, [2013-07-12T01:05:52.859032 #6664] INFO -- : worker=0 spawning...
I, [2013-07-12T01:05:52.860609 #7212] INFO -- : worker=0 spawned pid=7212
I, [2013-07-12T01:05:52.860839 #7212] INFO -- : Refreshing Gem list
I, [2013-07-12T01:05:52.875751 #6664] INFO -- : reaped #<Process::Status: pid 6670 exit 0> worker=1
I, [2013-07-12T01:05:52.875944 #6664] INFO -- : worker=1 spawning...
I, [2013-07-12T01:05:52.877405 #7215] INFO -- : worker=1 spawned pid=7215
I, [2013-07-12T01:05:52.877651 #7215] INFO -- : Refreshing Gem list
I, [2013-07-12T01:06:02.191290 #7212] INFO -- : worker=0 ready
I, [2013-07-12T01:06:02.269397 #7215] INFO -- : worker=1 ready
In this situation, what else could I check to identify the reason the app is not appearing?