I am getting a really strange error from ActiveResource. I have a method that calls another system to get a list of products. The list is quite large and takes about 3 minutes to generate and transfer. Since this is really only a once a day kinda thing I built a rake task to run it. In production whenever I run the rake task it fails with a 500 error. Here is some sample output
$ time RAILS_ENV=production rake api:sync
rake aborted!
Failed with 500 Internal Server Error
(See full trace by running task with --trace)
real 2m1.753s
user 0m1.188s
sys 0m0.188s
Then I tried to use script runner:
$ RAILS_ENV=production script/runner 'Product.synchronize!(ProductManager::Product.find_valid_products)'
/var/www/apps/api/releases/20091202203413/vendor/rails/railties/lib/commands/runner.rb:48: Failed with 500 Internal Server Error (ActiveResource::ServerError)
from /usr/local/lib/ruby/1.8/net/protocol.rb:133:in `rbuf_fill'
from /usr/local/lib/ruby/1.8/timeout.rb:62:in `timeout'
from /usr/local/lib/ruby/1.8/timeout.rb:93:in `timeout'
from /usr/local/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
from /usr/local/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from /usr/local/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from /usr/local/lib/ruby/1.8/net/http.rb:2020:in `read_status_line'
from /usr/local/lib/ruby/1.8/net/http.rb:2009:in `read_new'
... 17 levels...
from /var/www/apps/api/releases/20091202203413/vendor/rails/railties/lib/commands/runner.rb:48
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
from /usr/local/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
from script/runner:3
However if I jump into the console to run it, everything is fine:
$ script/console production
Loading production environment (Rails 2.3.5)
>> Product.synchronize!(ProductManager::Product.find_valid_products); nil # prevent dump to console
=> nil
>>
I suspected it was a timeout, so I set the timeout value to 5 minutes on the ProductManager::Product class, but that had no effect. The interesting thing is that when I run via rake or script/runner the error happens occurs at 2 minutes and 1 second without fail. In development and staging there is no issue. Does production of ActiveResource have some kind of hidden 2 minute override I can't find?
In case you didn't get it from the traces, we are using Rails 2.3.5 frozen into the application. Thanks for you help
Peer
Turns out it was an Apache setting. The RequestTimeout in the httpd.conf file was 120 seconds. Once we upped that everything was fine.
Related
I enabled enhanced language metrics for the ruby language metrics in Heroku for a Ruby on Rails application, following the documentation (barnes gem etc.). This worked successfully.
Later on I saw errors when inspecting something in the system through the rails console heroku console --app my-rails-app-name. Any call of a model constant example lead to the following output:
Running console on ⬢ my-rails-app-name... up, run.8724 (Standard-1X)
irb(main):001:0> MyModel.all.count
Traceback (most recent call last):
4: from /app/vendor/bundle/bin/console:29:in `<main>'
3: from /app/vendor/bundle/bin/console:29:in `load'
2: from /app/vendor/bundle/ruby/2.6.0/gems/countries-3.0.0/bin/console:14:in `<top (required)>'
1: from (irb):1
NameError (uninitialized constant MyModel)
The work around is to run the rails console explicitly
heroku run rails console --app my-rails-app-name
Because the enhanced language metrics needs loading another build pack we can suspect that this is the reason for this. But don't give up! The order in which the build packs are executed can be configured. So drag the heroku/metrics buildpack above the heroku/ruby build pack, deploy one more time and afterwards you should be able to stick to your workflow :)
After installing a new gem, Slackistrano to already succesfully deployed app with Capistrano, our deployments started failing with the following error during rake assets:precompile or rake db:migrate tasks
cap aborted!
SSHKit::Command::Failed: rake stdout: Nothing written
rake stderr: Nothing written
/Users/pete/.rvm/gems/ruby-2.2.2/gems/sshkit-1.3.0/lib/sshkit/command.rb:94:in `exit_status='
This error is obviously not very helpful.
If I ran the capistrano task cap staging assets:precompile, it would succeed.
So what's going on here?
There are a few other solutions to SSHKit failures during cap deployments, but most of them were for first time deployments and didn't work for me.
Here's how I figured out the solution(after much hair pulling).
I knew that it was probably the new gem that was causing the problem. So I went into the source code for sshkit/command.rb and outputted the command it was trying to execute on the remote server.
I then logged into the remote server and ran it manually and got the following output:
$ cd /home/web/sites/site.com/releases/20160106224143 && ( RAILS_ENV=production ~/.rvm/bin/rvm default do bundle exec rake db:migrate )
rake aborted!
NoMethodError: undefined method `each_pair' for "slack:deploy:updating":String
/home/web/sites/site.com/shared/bundle/ruby/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1602:in `compile!'
/home/web/sites/site.com/shared/bundle/ruby/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1344:in `add_filter'
/home/web/sites/site.com/shared/bundle/ruby/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1331:in `before'
/home/web/sites/site.com/shared/bundle/ruby/2.2.0/gems/sinatra-1.4.6/lib/sinatra/base.rb:1989:in `block (2 levels) in delegate'
/home/web/sites/site.com/shared/bundle/ruby/2.2.0/gems/slackistrano-1.0.0/lib/slackistrano/tasks/slack.rake:121:in `<top (required)>'
So, the problem started with Slackistrano, but then, what? why is Sinatra there? I don't use Sinatra in my app.
Well, I discovered that there was a gem in my Gemfile that was for development only(mailcatcher) but was in the production section of the Gemfile. I removed this and redeployed successfully.
However, there is obviously a deeper issue because there seems to be some sort of name collision on the "before" method of sinatra and capistrano that caused this in the first place.
I'm posting here to help others debug possible SSHKit errors during Cap deployments and also perhaps to guide others on weird name collisions between Sinatra and Capistrano.
For me this issue can arise if you're accidentally on an incompatible version of Ruby whilst running Capistrano. I had to add a .ruby-version to clamp it for me to 2.7.5 whilst using gem 'capistrano', '3.11.0'.
So I'm not sure if stating Ruby version 2 or higher is strictly true.
I'm trying to run Rails on JRuby 1.7.3 in userspace on Solaris and I get this error when running rake db:create.
-bash-3.2$ rake db:create
rake aborted!
undefined method `error' for #<ActiveRecord::JDBCError:0x109d7b8>
org/jruby/RubyArray.java:1613:in `each'
org/jruby/RubyProc.java:249:in `call'
org/jruby/RubyArray.java:1613:in `each'
org/jruby/RubyArray.java:1613:in `each'
org/jruby/RubyKernel.java:1046:in `load'
Tasks: TOP => db:create
(See full trace by running task with --trace)
I have looked at the JRuby source for this line, but I'm pretty far out of my depth on this and I'm not sure what's going on. Has anyone seen this before?
I haven't run on that kind of error before, but just to double-check .. have you used jruby specific gem for making a connection to databases:
https://github.com/jruby/activerecord-jdbc-adapter
If not, try using that instead the ones you have been using so far for "regular" ruby
I just run into this error. It seems like a bug in Rails. Your idea of inspecting the source for that line was a good one. I went there and changed $stdout.print error.error for just $stdout.print error because the error method didn't seem defined.
That revealed the actual error:
#<ActiveRecord::JDBCError: Access denied for user 'root'#'localhost' (using password: YES)>
In my case, I was mispelling the username key in the config/database.yml file, so it would assume the root user, but the password would be wrong.
I've run pg:reset on Heroku and on trying to run db:migrate, all migrations run but the migration fails with the following error and trace:
rake aborted!
Error dumping database
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0.rc2/lib/active_record/tasks/postgresql_database_tasks.rb:55:in `structure_dump'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0.rc2/lib/active_record/tasks/database_tasks.rb:142:in `structure_dump'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0.rc2/lib/active_record/railties/databases.rake:288:in `block (3 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0.rc2/lib/active_record/railties/databases.rake:51:in `block (2 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.0.0/gems/activerecord-4.0.0.rc2/lib/active_record/railties/databases.rake:45:in `block (2 levels) in <top (required)>'
As can be seen here the problematic line and the one above it are:
command = "pg_dump -i -s -x -O -f #{Shellwords.escape(filename)} #{search_path} #{Shellwords.escape(configuration['database'])}"
raise 'Error dumping database' unless Kernel.system(command)
This works locally without any problems, both in development and production environments.
Has anyone experienced anything like this?
This is an interesting bug that, as it turns out, can be ignored. Based on the fact it's attempting to do a db:structure:dump, you are using 'sql' as your active_record.schema_format. The rake task db:structure:dump will fail on heroku because pg_dump is (unsurprisingly) not in the binary path. Interestingly enough, Heroku states that db:schema:dump is not supported but if you set the schema format to ruby, it works fine.
In Rails 3, the dump task would only raise an error is the exit code of the command was 1. On unix based systems if the command is not found, the exit code is 127. So even if the pg_dump command fails on rails 3 (which it does), it won't raise an error and it won't halt the rake task. So anyone using a sql schema format with Rails 3 wouldn't have this issue because it would fail silently. The refactor in Rails 4 to properly raise an error if the dump failed causes db:migrate to raise an error on Heroku. However, even though it errors with rake aborted the ddl is actually performed and committed.
Possible solutions:
Ignore the error as the migrations are actually running.
Since you don't care about the structure dump on production, set the schema_format to ruby. In config/environments/production.rb:
config.active_record.schema_format = :ruby
If for some reason you don't want to change the config file: add a rake task with the following to suppress the error:
if Rails.env == 'production'
Rake::Task["db:structure:dump"].clear
end
The accepted solution is somewhat correct. Heroku does have pg_dump on its runtimes
$ heroku run bash
$ which pg_dump
/usr/bin/pg_dump
The problem comes from this issue: https://github.com/rails/rails/issues/21226 the command cannot be run correctly.
If you need to do a db:structure load instead you can use the $ heroku pg:psql
heroku pg:psql -a your-app-name <db/structure.sql
From heroku rake db:structure:load failure.
If you need to dump you can use this article https://devcenter.heroku.com/articles/heroku-postgres-import-export there's also dedicated commands:
$ heroku pg:pull <REMOTE_SOURCE_DATABASE> <TARGET_DATABASE> # pull from REMOTE_SOURCE_DATABASE to TARGET_DATABASE
$ heroku pg:push <SOURCE_DATABASE> <REMOTE_TARGET_DATABASE> # push from SOURCE_DATABASE to REMOTE_TARGET_DATABASE
If you need to reset your database before running migrations you can use $ heroku pg:reset
I originally posted a question relating to this problem on serverfault.com: https://serverfault.com/questions/152587/apache-mod-proxy-to-another-server
I have since realized that this is not an issue with my server setup, but my rails application. I have set this application up server-wise the exact same as another functioning rails app I have running on the same server. When I start the server my mongrel.log looks like this:
** Daemonized, any open files are closed. Look at /var/www/osuwebdev/tmp/pids/mongrel.pid and log/mongrel.log for info.
** Starting Mongrel listening at 0.0.0.0:8080
** Starting Rails with production environment...
** Rails loaded.
** Loading any Rails specific GemPlugins
** Signals ready. TERM => stop. USR2 => restart. INT => stop (no restart).
** Rails signals registered. HUP => reload (without restart). It might not work well.
** Mongrel 1.1.5 available at 0.0.0.0:8080
** Writing PID file to /var/www/osuwebdev/tmp/pids/mongrel.pid
When I load a page this pops up in the log:
Wed Jun 30 19:46:10 +0000 2010: Error calling Dispatcher.dispatch #<NoMethodError: undefined method `[]' for nil:NilClass>
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/cgi.rb:108:in `send_cookies'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/cgi.rb:136:in `out'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/http_response.rb:65:in `start'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/cgi.rb:135:in `out'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/rails.rb:81:in `process'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:159:in `process_client'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:158:in `each'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:158:in `process_client'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `run'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `initialize'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `new'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:285:in `run'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:in `initialize'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:in `new'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel.rb:268:in `run'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:282:in `run'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:in `each'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/configurator.rb:281:in `run'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:128:in `run'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/../lib/mongrel/command.rb:212:in `run'
/var/lib/gems/1.8/gems/mongrel-1.1.5/bin/mongrel_rails:281
/usr/local/bin/mongrel_rails:19:in `load'
/usr/local/bin/mongrel_rails:19
I cannot figure out what is different about this app that may cause this issue.
I believe this is a weird combination of bugs in Rack, Mongrel, and Rails.
Save this ruby code in your app to config/mongrel.rb: http://gist.github.com/471663
In your config/environment.rb, at the very end of the file add:
require File.join(File.dirname(File.expand_path(FILE)), 'mongrel')
In that same file add this line in the Rails::initializer.run block:
config.gem "mongrel"
Longer explanation of the problem here: https://rails.lighthouseapp.com/projects/8994/tickets/4690-mongrel-doesnt-work-with-rails-238
This has definitely worked for a Rails 2.3.8 app, though I patched it slightly differently than described above.
Do you still see the problem on subsequent GET requests? I had a similar problem where my Rails app would run fine under WEBrick but not under Mongrel but found somewhere on the internets (here's a link that shows a similar version of the problem) that older versions of Mongrel mishandle the first request to the page but handle subsequent requests ok.
The workaround we put in was that whenever we started the app, we'd have curl perform a GET on the app to 'prime' Mongrel.
This is usually a fail reading a yaml file for config. It has set up a config object to get values from and the entry in the yaml file for your environment isn't there so the object is nil.
Are you running any gems that have a yaml file that maybe isn't set up for development?
I don't know why going back to 2.3.5 would help - it might be that it loads things in a different order and the config information is available.