I can execute rails-commands in production (FreeBSD 10) as long as Spring is disabled:
# DISABLE_SPRING=1 bin/rails runner -e production 'p "test"'
"test"
# script/rails runner -e production 'p "test"'
"test"
With Spring, the rails-command fails with the following error:
# bin/rails runner -e production 'p "test"'
/usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/client/run.rb:20:in `initialize': Connection refused - "/tmp/spring/6e31dd16c1146fc209ac385f36aac9a1" (Errno::ECONNREFUSED)
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/client/run.rb:20:in `open'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/client/run.rb:20:in `server'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/client/run.rb:54:in `verify_server_version'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/client/run.rb:25:in `call'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/client/command.rb:7:in `call'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/client/rails.rb:23:in `call'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/client/command.rb:7:in `call'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/client.rb:26:in `run'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/bin/spring:48:in `<top (required)>'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/binstub.rb:11:in `load'
from /usr/local/rvm/gems/ruby-2.0.0-p481/gems/spring-1.1.3/lib/spring/binstub.rb:11:in `<top (required)>'
from /usr/local/rvm/rubies/ruby-2.0.0-p481/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/local/rvm/rubies/ruby-2.0.0-p481/lib/ruby/site_ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /s/usr-local/www/the_project/releases/20140724133941/bin/spring:16:in `<top (required)>'
from bin/rails:3:in `load'
from bin/rails:3:in `<main>'
So far, I verified that the according files in /tmp/spring exist as expected.
What could be the cause of this error and how can I achieve a correctly working Spring?
UPDATE 1
Here is an outline of several checks I did so far.
# ls -al /tmp/spring
total 32
drwxr-xr-x 2 proj wheel 8 Jul 27 19:35 .
drwxrwxrwt 36 root wheel 43 Jul 28 08:22 ..
srwxr-xr-x 1 proj wheel 0 Jul 27 19:35 3004493e4fa38d91e78f82a2e74b4bc8
-rw-r--r-- 1 proj wheel 6 Jul 27 19:35 3004493e4fa38d91e78f82a2e74b4bc8.pid
srwxr-xr-x 1 proj wheel 0 Jun 30 13:00 383a7b299457953fc0eda45a4eca65f8
-rw-r--r-- 1 proj wheel 5 Jun 30 13:00 383a7b299457953fc0eda45a4eca65f8.pid
srwxr-xr-x 1 proj wheel 0 Jul 24 14:01 6e31dd16c1146fc209ac385f36aac9a1
-rw-r--r-- 1 proj wheel 6 Jul 24 14:01 6e31dd16c1146fc209ac385f36aac9a1.pid
# groups proj
proj wheel rvm
# bin/spring status
Spring is running:
25240 ruby: spring server | 20140727193408 | started 14 hours ago (ruby)
25245 ruby: spring app | 20140727193408 | started 0 secs ago | production mode (ruby)
# fuser /tmp/spring/*
/tmp/spring/3004493e4fa38d91e78f82a2e74b4bc8:
/tmp/spring/3004493e4fa38d91e78f82a2e74b4bc8.pid: 25240wa
/tmp/spring/383a7b299457953fc0eda45a4eca65f8:
/tmp/spring/383a7b299457953fc0eda45a4eca65f8.pid:
/tmp/spring/6e31dd16c1146fc209ac385f36aac9a1:
/tmp/spring/6e31dd16c1146fc209ac385f36aac9a1.pid: 17260wa
# ps aux | grep -E '25240|17260'
proj 17260 0.0 0.1 79040 3544 - IJ Thu02PM 0:00.47 ruby: spring server | 20140724133941 | started 92 hours ago (ruby)
proj 25240 0.0 0.1 79040 3372 - IJ 7:35PM 0:00.28 ruby: spring server | 20140727193408 | started 14 hours ago (ruby)
# head -n1 /tmp/spring.log
[2014-07-28 10:16:40 +0000] [66008] [server] started on /tmp/spring/3004493e4fa38d91e78f82a2e74b4bc8
# tail /tmp/spring.log
[2014-07-28 10:18:00 +0000] [66008] [server] accepted client
[2014-07-28 10:18:00 +0000] [66008] [server] running command rails_runner
[2014-07-28 10:18:00 +0000] [66128] [application:production] got client
[2014-07-28 10:18:00 +0000] [66128] [application:production] forked 66370
[2014-07-28 10:18:00 +0000] [66008] [application_manager:production] got worker pid 66370
[2014-07-28 10:18:03 +0000] [66128] [application:production] 66370 exited with 0
I was able to resolve the problem by changing the users login shell to bash (before, it was set to zsh) and adding this PATH to ~/.profile: PATH=$GEM_HOME/bin:$GEM_HOME/wrappers:$PATH
To ensure only RVM's ruby is being used, I also forcefully removed the system ruby (1.9.3): pkg remove -f ruby
(Note: Without the PATH setting mentioned above, this now causes the expected error env: ruby: No such file or directory)
It appears a killall ruby was necessary in order for the changes to take effect.
Maybe a bin/spring stop would've been sufficient too.
Some thoughts on this, it seems the spring server can not be started or you are not allowed to connect to the socket:
Check the permissions and ownership of the socket file and the containing folder /tmp/spring.
Try to delete the socket file, change the folder permissions on /tmp/spring to 777 (just to rule this out as root cause) and restart the app / server. "Connecting to the socket object requires read/write permission."
Did you check that spring/server is actually running? The client tries to boot the server if it is not running), but if that does not work, you get the ECONNREFUSED. Try to debug into the spring sources.
Does lsof | grep '/tmp/spring/6e31dd16c1146fc209ac385f36aac9a1' give you anything if you think the server is running? What does ps -ax | grep [process id] say about the server process?
What is the output of bin/spring status?
Try to find out what spring is doing from the log file (look out for started on #{env.socket_name} message).
I stumbled on this question with the message Error connecting to Spring server.
The solution for me was as simple as:
spring stop
With that, my db:migrate task completed successfully.
Related
I cannot figure this out. Hope I can get some help
I have a hobby tier in Heroku running django. To this I would like to attach a Redis service. However, I would like to use the recent rejson (redislabs/rejson:latest) docker image (instead of redistogo or heroku-redis)because it has json support. This works great on my local env. I was able to push the docker image into the container registry and actually start the redis server
2021-07-23T00:14:48.576294+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.576 # Current maximum open files is 10000. maxclients has been reduced to 9968 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
2021-07-23T00:14:48.577054+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 * Running mode=standalone, port=6379.
2021-07-23T00:14:48.577124+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # Server initialized
2021-07-23T00:14:48.577184+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2021-07-23T00:14:48.577671+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 # <ReJSON> JSON data type for Redis v1.0.7 [encver 0]
2021-07-23T00:14:48.577789+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.577 * Module 'ReJSON' loaded from /usr/lib/redis/modules/rejson.so
2021-07-23T00:14:48.578232+00:00 app[worker.1]: 3:M 23 Jul 2021 00:14:48.578 * Ready to accept connections
Unfortunately, django is unable to connect to it
ConnectionError: Error -2 connecting to redis://localhost:6379. Name or service not known
There are no ENV variable exposed, so am unable to set any (I mean I can set them, but doubt if they will be relevant)
I did experiment with installing redistogo add on and am able to connect to it (this required setting the connection based on the REDIS_URL env variable that gets exposed when redistogo is added)
At my wits end...any help appreciated...I guess the question really boils do to:
I have everything installed as per this tutorial:
https://www.digitalocean.com/community/tutorials/how-to-setup-a-rails-4-app-with-apache-and-passenger-on-centos-6
But when I go to the website, I get an error, and in the error log it shows this:
[ 2015-02-26 16:17:47.3041 28207/7f7326058780 agents/Watchdog/Main.cpp:728 ]: All Phusion Passenger agents started!
[Thu Feb 26 16:17:47.305819 2015] [mpm_prefork:notice] [pid 28184] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 Phusion_Passenger/4.0.59 mod_perl/2.0.9-dev Perl/v5.16.3 configured -- resuming normal operations
[Thu Feb 26 16:17:47.305844 2015] [core:notice] [pid 28184] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[ 2015-02-26 16:22:04.7157 28225/7f2c65840840 apache2/Hooks.cpp:151 ]: A filesystem exception occured.
Message: Cannot stat '/var/www/vhosts/blah.blah/httpdocs/config.ru
Backtrace:
in 'void Passenger::DirectoryMapper::autoDetect()' (DirectoryMapper.h:144)
in 'bool Hooks::prepareRequest(request_rec*, Passenger::DirConfig*, const char*, bool)' (Hooks.cpp:374)
Can anyone help?
The config.ru file was auto generated by rails, and it has these lines:
# This file is used by Rack-based servers to start the application.
require ::File.expand_path('../config/environment', __FILE__)
run BlahBlah::Application
Your problem can be related to the following things:
config.ru file is not exists by the specified path
Try to execute command for check this:
ls -la /var/www/vhosts/blah.blah/httpdocs/config.ru
Passenger doesn't have permissions to read this file
Try to execute command for check this:
namei -lm /var/www/vhosts/blah.blah/httpdocs/config.ru
P.S.
I noticed you are using shared hosting. Don't use them if you want to avoid a lot of different problems. Use VPS servers instead. Like Digital Ocean or Linode for Rails applications.
I've been trying to set up webistrano on one of our servers. Everything went well, until I tried deploying my set up. Webistrano is currenlty set up to live on the test server, so I want it to deploy to 127.0.0.1
The configuration contains valid user, webistrano, that should have access to
/home/webistrano/.subversion/servers
the password for that user is correct. Could someone shed some light as to what might be going wrong with it?
*** SSH options: setting ssh_port to: 22
*** LOCAL SCM ERROR: svn: warning: Can't open file '/home/webistrano/.subversion/servers': Permission denied
* executing `deploy'
* executing `deploy:update'
** transaction: start
* executing `deploy:update_code'
* executing "svn checkout -q --username webistrano --password XXXXXXXX --no-auth-cache -r27687 http://server.com/repos/server/clients/site/trunk /var/www/vhosts/server.com/releases/20121001183038 && (echo 27687 > /var/www/vhosts/server.com/releases/20121001183038/REVISION)"
servers: ["127.0.0.1"]
*** [deploy:update_code] rolling back
* executing "rm -rf /var/www/vhosts/server.com/releases/20121001183038; true"
servers: ["127.0.0.1"]
** [deploy:update_code] exception while rolling back: Capistrano::ConnectionError, connection failed for: 127.0.0.1 (Errno::EACCES: Permission denied - connect(2))
*** connection failed for: 127.0.0.1 (Errno::EACCES: Permission denied - connect(2))
debugging with
puts system('id') in deployer.rb returns
uid=500(webistrano) gid=500(webistrano) groups=500(webistrano) context=unconfined_u:system_r:httpd_t:s0
and ls -all /home/webistrano returns:
drwxrwxr-x. 3 webistrano webistrano 4096 Sep 27 16:59 .subversion
ls -all /home/webistrano/.subversion returns:
drwxrwxr-x. 3 webistrano webistrano 4096 Oct 1 20:02 .
drwx------. 4 webistrano webistrano 4096 Oct 1 22:14 ..
drwx------. 6 webistrano webistrano 4096 Sep 27 16:59 auth
-rw-rw-r--. 1 webistrano webistrano 6866 Sep 27 16:59 config
-rw-rw-r--. 1 webistrano webistrano 4276 Sep 27 16:59 README.txt
-rw-rw-r--. 1 webistrano webistrano 7674 Oct 1 20:02 servers
Which shows that servers should be fully readable by everyone?
What's going on?
It turns out this was an issue with Passenger.
Phusion Passenger is supposed to run as the user who owns conf/environment.conf (which in my case was webistrano). Displaying the user id also showed webistrano, but if I changed it to run as nobody, the svn error stopped appearing. I was unable to svn export the code though, probably because nobody does not have a home directory.
I ended up running webistrano through mongrel, and starting it by doing:
$ su - webistrano
to make sure that the correct home environment is loaded and all the permissions are correct.
It works perfect now . Hope this helps somebody
Something is wrong with the logs. They work fine on development, but in production I can't write my own messages.
I haven't changed anything in the configuration files, other than installing Devise and Mongoid. But just in case, I tried uncommenting the line #config.log_level = :debug in production.rb
Controller:
class PagesController < ApplicationController
def home
logger.fatal 'bla'
end
def about
end
end
Terminal (log permissions):
root#ubuntu:/srv/www/myapp# ls log -lah
total 496K
drwxr-xr-x 2 myapp root 4.0K 2012-02-21 17:18 .
drwxr-xr-x 14 root root 4.0K 2012-02-20 14:54 ..
-rw-r--r-- 1 myapp myapp 35K 2012-02-21 16:23 development.log
-rw-r--r-- 1 myapp root 0 2012-02-17 18:27 .gitkeep
-rw-r--r-- 1 root root 447K 2012-02-21 17:47 passenger.80.log
-rw-r--r-- 1 myapp myapp 0 2012-02-21 17:18 production.log
Terminal (passenger):
root#ubuntu:/srv/www/myapp# passenger start -e production -p 80 --user=myapp
=============== Phusion Passenger Standalone web server started ===============
PID file: /srv/www/myapp/tmp/pids/passenger.80.pid
Log file: /srv/www/myapp/log/passenger.80.log
Environment: production
Accessible via: http://0.0.0.0/
You can stop Phusion Passenger Standalone by pressing Ctrl-C.
===============================================================================
cache: [HEAD /] miss
cache: [GET /] miss
Assuming you're on Rails 3.2.0 or Rails 3.2.1, this is a known issue (see https://github.com/rails/rails/issues/4277). It's already been fixed, but hasn't been released yet.
A workaround is to add an initializer with:
Rails.logger.instance_variable_get(:#logger).instance_variable_get(:#log_dest).sync = true if Rails.logger
Update: Rails 3.2.2 fixes this.
I am in the process of setting up a server to run a Ruby on Rails application on Fedora 12, using Passenger.
I am at the stage where I've installed Passenger, set it up as prescribed, but get the following errors when I restart Apache:
[Wed Jan 13 15:41:38 2010] [notice] caught SIGTERM, shutting down
[Wed Jan 13 15:41:40 2010] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Wed Jan 13 15:41:40 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jan 13 15:41:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.25235/.guard: Permission denied (13)
[Wed Jan 13 15:41:40 2010] [notice] Digest: generating secret for digest authentication ...
[Wed Jan 13 15:41:40 2010] [notice] Digest: done
[Wed Jan 13 15:41:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.25235/.guard: Permission denied (13)
[Wed Jan 13 15:41:40 2010] [error] python_init: Python version mismatch, expected '2.6', found '2.6.2'.
[Wed Jan 13 15:41:40 2010] [error] python_init: Python executable found '/usr/bin/python'.
[Wed Jan 13 15:41:40 2010] [error] python_init: Python path being used '/usr/lib/python26.zip:/usr/lib/python2.6/:/usr/lib/python2.6/plat-linux2:/usr/lib/python2.6/lib-tk:/usr/lib/python2.6/lib-old:/usr/lib/python2.6/lib-dynload'.
[Wed Jan 13 15:41:40 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Wed Jan 13 15:41:40 2010] [notice] mod_python: using mutex_directory /tmp
[Wed Jan 13 15:41:40 2010] [notice] Apache/2.2.14 (Unix) DAV/2 Phusion_Passenger/2.2.9 PHP/5.3.0 mod_python/3.3.1 Python/2.6.2 mod_ssl/2.2.14 OpenSSL/1.0.0-fips-beta3 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operations
As you can see, there is a permissions problem when Passenger is trying to initialize:
[Wed Jan 13 15:41:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.25235/.guard: Permission denied (13)
When Apache is starts, it does create a file in /tmp:
d-ws--x--x. 2 root root 4096 2010-01-13 16:04 passenger.26117
If instead I run the app by firing up mongrel directly with mongrel_rails start -e production, I see the following:
ActiveRecord::StatementInvalid (Mysql::Error: Can't create/write to file '/tmp/#sql_5d3_0.MYI' (Errcode: 13): SHOW FIELDS FROM `users`):
Again the error points to permission issues with the /tmp directory.
I am at a loss as to what the solution is. I'm not sure if it is related to simply directory permissions or Fedora's SELinux security.
Any help would be appreciated. Thanks.
I did the same as Fred, except that instead of doing it one error at a time:
Go into permissive mode by running setenforce 0
Restart apache, and hit your site and use it for a while as normal
Run grep httpd /var/log/audit/audit.log | audit2allow -M passenger
semodule -i passenger.pp
Go back to enforcing mode by running setenforce 1
Restart apache and test your site - hopefully it should all be working as before!
Note that this is basically a specific example of the procedure on the Centos SELinux help - check it out.
I'm having the same issue in CentOS 5.4, SELinux getting in the way of Passenger.
Setting PassengerTempDir to /var/run/passenger simply gives you the same permission errors in the new directory instead of /tmp :
[Mon Feb 22 11:42:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create directory '/var/run/passenger/passenger.3686'
I can then change the security context of /var/run/passenger to get past this error:
chcon -R -h -t httpd_sys_content_t /var/run/passenger/
...and that lets Passenger create the temp directory, but not files within that directory:
[Mon Feb 22 12:07:06 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /var/run/passenger/passenger.3686/.guard: Permission denied (13)
Oddly, re-running the recursive chcon again doesn't get past this error, it keeps dying at this point, and this is where my SELinux knowledge gets murky.
The Phusion Passenger guide sections 6.3.5 and 6.3.7 have some useful thoughts, but they don't seem to completely resolve the problem.
You need more than just the httpd_sys_content_t permission. I use the following technique to get things started:
start a tail on the audit log: tail -f /var/log/audit/audit.log
reload apache: apachectl restart
Go to the /tmp/directory: cd /tmp
If just 1 line is added use the command: tail -1 /var/log/audit/audit.log | audit2allow -M httpdfifo
Note that the name 'httpdfifo' is just a name chosen to reflect the kind of error that has been observed.
This will create a file named 'httpdfifo.pp'. To allow apache to create a FIFO from here on after you have to issue the command: semodule -i httpdfifo.pp
Continue to do this until all audit errors have been resolved (It took 4 different kind of permissions on my system running Centos 5.4)
Running setenforce 0 before starting will let you test if it's SELinux. Don't forget to run setenforce 1 afterwards.
I tried what Dan Sketcher and Fred Appleman suggested, i.e. repeat the following:
yum install setroubleshoot
echo > /var/log/audit/audit.log # clear irrelevant errors
cd ~
service httpd restart # try booting passenger -- audit.log now shows the relevant permission errors
tail -f /var/log/httpd/error_log # check that passenger is still failing due to permission errors
sealert -a /var/log/audit/audit.log > selinux-diag.txt # translate the permission errors
# read and check that you are happy with selinux-diag.txt
# and either follow its specific advice, or if it just wants you to grep into audit2allow, then:
cat /var/log/audit/audit.log | audit2allow -M mypol # grant everything just denied
semodule -i mypol.p # commit new permissions
But after doing this 5 or 6 times, I kept coming up against new errors, and some of the same errors came up even after I had tried to permit them with "audit2allow".
In the end I just turned off SELinux, with:
echo 0 >/selinux/enforce