At work, we have about 1500 test cases, and we manually clean the database using DB.recreate! method before each test. When running all tests using bundle exec rake spec, all tests rarely pass. There are number of tests that fail towards the end of suite with the "Errno::ECONNREFUSED Connection Refused - connect(2) error" errors.
Any help would be much appreciated!
I am using CouchDB 1.3.1, Ubuntu 12.04 LTS, Ruby 1.9.3, and Rails 3.2.12.
Thanks,
EDIT
I looked at the log file more carefully and matched the time tests started failing and error messages that were generated in couchdb log.
[Fri, 16 Aug 2013 19:39:46 GMT] [error] [<0.23790.0>] ** Generic server <0.23790.0> terminating
** Last message in was {'EXIT',<0.23789.0>,killed}
** When Server state == {file,{file_descriptor,prim_file,{#Port<0.14445>,20}},
79}
** Reason for termination ==
** killed
[Fri, 16 Aug 2013 19:39:46 GMT] [error] [<0.23790.0>] {error_report,<0.31.0>,
{<0.23790.0>,crash_report,
[[{initial_call,{couch_file,init,['Argument__1']}},
{pid,<0.23790.0>},
{registered_name,[]},
{error_info,
{exit,killed,
[{gen_server,terminate,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,[<0.23789.0>]},
{messages,[]},
{links,[]},
{dictionary,[]},
{trap_exit,true},
{status,running},
{heap_size,377},
{stack_size,24},
{reductions,916}],
[]]}}
[Fri, 16 Aug 2013 19:39:46 GMT] [error] [<0.23808.0>] {error_report,<0.31.0>,
{<0.23808.0>,crash_report,
[[{initial_call,
{couch_ref_counter,init,['Argument__1']}},
{pid,<0.23808.0>},
{registered_name,[]},
{error_info,
{exit,
{noproc,
[{erlang,link,[<0.23790.0>]},
{couch_ref_counter,'-init/1-lc$^0/1-0-',1},
{couch_ref_counter,init,1},
{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]},
[{gen_server,init_it,6},
{proc_lib,init_p_do_apply,3}]}},
{ancestors,[<0.23793.0>,<0.23792.0>,<0.23789.0>]},
{messages,[]},
{links,[]},
{dictionary,[]},
{trap_exit,false},
{status,running},
{heap_size,377},
{stack_size,24},
{reductions,114}],
[]]}}
[Fri, 16 Aug 2013 19:39:46 GMT] [error] [<0.103.0>] ** Generic server <0.103.0> terminating
** Last message in was {'EXIT',<0.88.0>,killed}
** When Server state == {db,<0.103.0>,<0.104.0>,nil,<<"1376681645837889">>,
<0.106.0>,<0.102.0>,<0.107.0>,
{db_header,6,1,0,
{1856,{1,0,1777},95},
{1951,1,83},
nil,0,nil,nil,1000},
1,
{btree,<0.102.0>,
{1856,{1,0,1777},95},
#Fun<couch_db_updater.10.55895019>,
#Fun<couch_db_updater.11.100913286>,
#Fun<couch_btree.5.25288484>,
#Fun<couch_db_updater.12.39068440>,snappy},
{btree,<0.102.0>,
{1951,1,83},
#Fun<couch_db_updater.13.114276184>,
#Fun<couch_db_updater.14.2340873>,
#Fun<couch_btree.5.25288484>,
#Fun<couch_db_updater.15.23651859>,snappy},
{btree,<0.102.0>,nil,
#Fun<couch_btree.3.20686015>,
#Fun<couch_btree.4.73514747>,
#Fun<couch_btree.5.25288484>,nil,snappy},
1,<<"_users">>,"/var/lib/couchdb/_users.couch",
[#Fun<couch_doc.8.106888048>],
[],nil,
{user_ctx,null,[],undefined},
nil,1000,
[before_header,after_header,on_file_open],
[create,
{before_doc_update,
#Fun<couch_users_db.before_doc_update.2>},
{after_doc_read,
#Fun<couch_users_db.after_doc_read.2>},
sys_db,
{user_ctx,
{user_ctx,null,[<<"_admin">>],undefined}},
nologifmissing,sys_db],
snappy,#Fun<couch_users_db.before_doc_update.2>,
#Fun<couch_users_db.after_doc_read.2>}
** Reason for termination ==
** killed
Ah.... The power of the community. I got the following answer from someone in the CouchDB mailing list.
In short, the solution is to change delayed_commit value to false. It's set to true by default, and rapidly recreating multiple databases at the beginning of each test case were creating a race condition (deleting non-existent db, etc.).
This definitely solved my problem.
One caveat is that it has doubled our test duration. That's another problem to tackle, but for now, I am happy with all passing tests.
I am in the process of setting up a server to run a Ruby on Rails application on Fedora 12, using Passenger.
I am at the stage where I've installed Passenger, set it up as prescribed, but get the following errors when I restart Apache:
[Wed Jan 13 15:41:38 2010] [notice] caught SIGTERM, shutting down
[Wed Jan 13 15:41:40 2010] [notice] SELinux policy enabled; httpd running as context unconfined_u:system_r:httpd_t:s0
[Wed Jan 13 15:41:40 2010] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jan 13 15:41:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.25235/.guard: Permission denied (13)
[Wed Jan 13 15:41:40 2010] [notice] Digest: generating secret for digest authentication ...
[Wed Jan 13 15:41:40 2010] [notice] Digest: done
[Wed Jan 13 15:41:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.25235/.guard: Permission denied (13)
[Wed Jan 13 15:41:40 2010] [error] python_init: Python version mismatch, expected '2.6', found '2.6.2'.
[Wed Jan 13 15:41:40 2010] [error] python_init: Python executable found '/usr/bin/python'.
[Wed Jan 13 15:41:40 2010] [error] python_init: Python path being used '/usr/lib/python26.zip:/usr/lib/python2.6/:/usr/lib/python2.6/plat-linux2:/usr/lib/python2.6/lib-tk:/usr/lib/python2.6/lib-old:/usr/lib/python2.6/lib-dynload'.
[Wed Jan 13 15:41:40 2010] [notice] mod_python: Creating 4 session mutexes based on 256 max processes and 0 max threads.
[Wed Jan 13 15:41:40 2010] [notice] mod_python: using mutex_directory /tmp
[Wed Jan 13 15:41:40 2010] [notice] Apache/2.2.14 (Unix) DAV/2 Phusion_Passenger/2.2.9 PHP/5.3.0 mod_python/3.3.1 Python/2.6.2 mod_ssl/2.2.14 OpenSSL/1.0.0-fips-beta3 mod_perl/2.0.4 Perl/v5.10.0 configured -- resuming normal operations
As you can see, there is a permissions problem when Passenger is trying to initialize:
[Wed Jan 13 15:41:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /tmp/passenger.25235/.guard: Permission denied (13)
When Apache is starts, it does create a file in /tmp:
d-ws--x--x. 2 root root 4096 2010-01-13 16:04 passenger.26117
If instead I run the app by firing up mongrel directly with mongrel_rails start -e production, I see the following:
ActiveRecord::StatementInvalid (Mysql::Error: Can't create/write to file '/tmp/#sql_5d3_0.MYI' (Errcode: 13): SHOW FIELDS FROM `users`):
Again the error points to permission issues with the /tmp directory.
I am at a loss as to what the solution is. I'm not sure if it is related to simply directory permissions or Fedora's SELinux security.
Any help would be appreciated. Thanks.
I did the same as Fred, except that instead of doing it one error at a time:
Go into permissive mode by running setenforce 0
Restart apache, and hit your site and use it for a while as normal
Run grep httpd /var/log/audit/audit.log | audit2allow -M passenger
semodule -i passenger.pp
Go back to enforcing mode by running setenforce 1
Restart apache and test your site - hopefully it should all be working as before!
Note that this is basically a specific example of the procedure on the Centos SELinux help - check it out.
I'm having the same issue in CentOS 5.4, SELinux getting in the way of Passenger.
Setting PassengerTempDir to /var/run/passenger simply gives you the same permission errors in the new directory instead of /tmp :
[Mon Feb 22 11:42:40 2010] [error] *** Passenger could not be initialized because of this error: Cannot create directory '/var/run/passenger/passenger.3686'
I can then change the security context of /var/run/passenger to get past this error:
chcon -R -h -t httpd_sys_content_t /var/run/passenger/
...and that lets Passenger create the temp directory, but not files within that directory:
[Mon Feb 22 12:07:06 2010] [error] *** Passenger could not be initialized because of this error: Cannot create FIFO file /var/run/passenger/passenger.3686/.guard: Permission denied (13)
Oddly, re-running the recursive chcon again doesn't get past this error, it keeps dying at this point, and this is where my SELinux knowledge gets murky.
The Phusion Passenger guide sections 6.3.5 and 6.3.7 have some useful thoughts, but they don't seem to completely resolve the problem.
You need more than just the httpd_sys_content_t permission. I use the following technique to get things started:
start a tail on the audit log: tail -f /var/log/audit/audit.log
reload apache: apachectl restart
Go to the /tmp/directory: cd /tmp
If just 1 line is added use the command: tail -1 /var/log/audit/audit.log | audit2allow -M httpdfifo
Note that the name 'httpdfifo' is just a name chosen to reflect the kind of error that has been observed.
This will create a file named 'httpdfifo.pp'. To allow apache to create a FIFO from here on after you have to issue the command: semodule -i httpdfifo.pp
Continue to do this until all audit errors have been resolved (It took 4 different kind of permissions on my system running Centos 5.4)
Running setenforce 0 before starting will let you test if it's SELinux. Don't forget to run setenforce 1 afterwards.
I tried what Dan Sketcher and Fred Appleman suggested, i.e. repeat the following:
yum install setroubleshoot
echo > /var/log/audit/audit.log # clear irrelevant errors
cd ~
service httpd restart # try booting passenger -- audit.log now shows the relevant permission errors
tail -f /var/log/httpd/error_log # check that passenger is still failing due to permission errors
sealert -a /var/log/audit/audit.log > selinux-diag.txt # translate the permission errors
# read and check that you are happy with selinux-diag.txt
# and either follow its specific advice, or if it just wants you to grep into audit2allow, then:
cat /var/log/audit/audit.log | audit2allow -M mypol # grant everything just denied
semodule -i mypol.p # commit new permissions
But after doing this 5 or 6 times, I kept coming up against new errors, and some of the same errors came up even after I had tried to permit them with "audit2allow".
In the end I just turned off SELinux, with:
echo 0 >/selinux/enforce
I have a mongrel server running behind Apache. It works fine; however, every now and then the Apache server shuts downs seemingly by itself. I'm not sure if there is configuration issue or if it's an attack. Here is Apache error log:
[Thu Apr 30 02:15:07 2009] [notice] SIGHUP received. Attempting to restart
[Thu Apr 30 02:15:07 2009] [warn] NameVirtualHost *:0 has no VirtualHosts
[Thu Apr 30 02:15:07 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Thu Apr 30 02:17:13 2009] [error] [client 61.139.105.163] File does not exist: /var/www/fastenv
[Thu Apr 30 02:24:06 2009] [error] [client 61.139.105.163] File does not exist: /var/www/fastenv
[Thu Apr 30 10:49:18 2009] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Thu Apr 30 10:49:18 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Thu Apr 30 12:53:08 2009] [notice] SIGHUP received. Attempting to restart
[Thu Apr 30 12:53:08 2009] [warn] NameVirtualHost *:0 has no VirtualHosts
[Thu Apr 30 12:53:08 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Thu Apr 30 12:59:15 2009] [notice] SIGHUP received. Attempting to restart
[Thu Apr 30 12:59:15 2009] [warn] NameVirtualHost *:0 has no VirtualHosts
[Thu Apr 30 12:59:15 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Thu Apr 30 13:58:49 2009] [notice] SIGHUP received. Attempting to restart
[Thu Apr 30 13:58:49 2009] [warn] NameVirtualHost *:0 has no VirtualHosts
[Thu Apr 30 13:58:49 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Fri May 01 10:59:07 2009] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri May 01 10:59:07 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
[Fri May 01 17:51:15 2009] [warn] pid file /var/run/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri May 01 17:51:15 2009] [notice] Apache/2.2.3 (Debian) PHP/5.2.0-8+etch13 configured -- resuming normal operations
Not quite sure what is /var/www/fastenv but I don't think there is anything in my application that calls that. Also, website is still in Beta mode with few users and I don't think any have 61.139.105.163 IP address but it's possible that they might have it.
Any ideas? It would be good if you can give me hints where to look or how to go about anaysing this problem
I have the exact same log from the same IP. Looking it up shows it to belong to the Chinese government. It appears to be a scan using server side includes to find out as much as they can about your server. I banned the IP.
Not sure this is entirely programming-related, but anyway... none of those look like serious errors to me. The accesses to /var/www/fastenv just mean that the computer at IP address 61.139.105.163 sent a request for http://www.example.com/fastenv or something like that (it depends on exactly how you've configured your virtual hosts); I'd look at the access log for more information, to see what other requests have been coming from that IP address. It's probably not anything to worry about.
The line about NameVirtualHost *:0 means that somewhere in your configuration file you have an incorrect NameVirtualHost directive, maybe with no arguments. You should probably look for that and remove it, but if the server is running fine anyway, it's not a big deal.
The reason your server is terminating (restarting, actually) appears to be a SIGHUP - that is, something on the system is sending Apache a signal telling it to restart. It's basically the same thing that happens if you run apache2 restart, I think. Without knowing what's sending that signal, there's not more I can say.
61.139.105.163 is known for doing all kinds of hacking type things, just google the IP address. You should definitly ban this IP address.
Click on Apache Config --> Apache(httpd.conf)
Search for #Listen 12.34.56.78:80 and replace it with #Listen 12.34.56.78:8081.
Search for Listen 80 and replace it with Listen 8081.
Now you can start Apache now, and can run it with this URL: localhost:8081/xampp/