I have a Ruby Rails application using Resque.
The Resuqe daemon crashes after running for a while with the error. My first guess was that the daemon is not accessing the MySQL DB often enough (for several hours) so I trigger every 5 minutes a queue item that accesses the DB. But still I get:
rake aborted!
Connection timed out
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:204:in `rescue in io'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:202:in `io'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:210:in `read'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:84:in `block in call'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:183:in `block (2 levels) in process'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:287:in `ensure_connected'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:173:in `block in process'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:248:in `logging'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:172:in `process'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis/client.rb:84:in `call'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis.rb:715:in `block in get'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis.rb:36:in `block in synchronize'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis.rb:36:in `synchronize'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-3.0.2/lib/redis.rb:714:in `get'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/redis-namespace-1.2.1/lib/redis/namespace.rb:257:in `method_missing'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/resque-1.23.0/lib/resque/worker.rb:517:in `job'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/resque-1.23.0/lib/resque/worker.rb:450:in `unregister_worker'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/resque-1.23.0/lib/resque/worker.rb:169:in `rescue in work'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/resque-1.23.0/lib/resque/worker.rb:129:in `work'
/usr/local/rvm/gems/ruby-1.9.3-p194/gems/resque-1.23.0/lib/resque/tasks.rb:36:in `block (2 levels) in <top (required)>'
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/ruby_noexec_wrapper:14:in `eval'
/usr/local/rvm/gems/ruby-1.9.3-p194/bin/ruby_noexec_wrapper:14:in `<main>'
Tasks: TOP => resque:work
(See full trace by running task with --trace)
I put some log file output in my method that gets called when any asynch task is triggered but looks as if the connection timed out happens while not executing any of my application code.
Where to go from here?
According to the log here, the connection to redis is lost, not mysql.
Related
While trying to deploy in the "Ruby 3.0 running on 64bit Amazon Linux 2/3.6.2" platform, both using the eb deploy command or manually uploading a zipped version of my project, it seems the Postgresql service is not running in my instance since I get the next in eb-engile.log:
2023/01/17 17:31:04.238133 [ERROR] An error occurred during execution of command [app-deploy] - [rake tasks]. Stop running the command. Error: running rake task db:migrate failed with error command bundle exec rake db:migrate failed with error Command /bin/su webapp -c bundle exec rake db:migrate failed with error exit status 1. Stderr:rake aborted!
ActiveRecord::ConnectionNotEstablished: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:83:in `rescue in new_client'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:77:in `new_client'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:882:in `public_send'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:882:in `new_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:926:in `checkout_new_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:905:in `try_to_checkout_new_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:866:in `acquire_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:588:in `checkout'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:428:in `connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:1128:in `retrieve_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_handling.rb:327:in `retrieve_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_handling.rb:283:in `connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:90:in `each'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/elasticbeanstalk/.rbenv/versions/3.0.5/bin/bundle:25:in `load'
/opt/elasticbeanstalk/.rbenv/versions/3.0.5/bin/bundle:25:in `<main>'
Caused by:
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/pg-1.2.3/lib/pg.rb:58:in `initialize'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/pg-1.2.3/lib/pg.rb:58:in `new'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/pg-1.2.3/lib/pg.rb:58:in `connect'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:78:in `new_client'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:882:in `public_send'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:882:in `new_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:926:in `checkout_new_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:905:in `try_to_checkout_new_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:866:in `acquire_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:588:in `checkout'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:428:in `connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:1128:in `retrieve_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_handling.rb:327:in `retrieve_connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/connection_handling.rb:283:in `connection'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/tasks/database_tasks.rb:237:in `migrate'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:92:in `block (3 levels) in <main>'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:90:in `each'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/activerecord-6.1.7/lib/active_record/railties/databases.rake:90:in `block (2 levels) in <main>'
/var/app/staging/vendor/bundle/ruby/3.0.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/elasticbeanstalk/.rbenv/versions/3.0.5/bin/bundle:25:in `load'
/opt/elasticbeanstalk/.rbenv/versions/3.0.5/bin/bundle:25:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Although, when trying to run the /bin/su webapp -c bundle exec rake db:migrate command as root (please notice that, I am invoking bundler then as the webapp user), I get different kind of error (trying to install a missing gem?):
Bundler::PermissionError: There was an error while trying to write to
`/var/app/staging/vendor/bundle/ruby/3.0.0/cache/public_suffix-5.0.1.gem`. It is likely that you
need to grant write permissions for that path.
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/shared_helpers.rb:105:in `rescue in
filesystem_access'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/shared_helpers.rb:102:in
`filesystem_access'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/rubygems_integration.rb:479:in `block in
download_gem'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/retry.rb:40:in `run'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/retry.rb:30:in `attempt'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/rubygems_integration.rb:470:in
`download_gem'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/source/rubygems.rb:485:in `download_gem'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/source/rubygems.rb:447:in `fetch_gem'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/source/rubygems.rb:431:in
`fetch_gem_if_possible'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/source/rubygems.rb:158:in `install'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer/gem_installer.rb:54:in
`install'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer/gem_installer.rb:16:in
`install_from_spec'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer/parallel_installer.rb:155:in
`do_install'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer/parallel_installer.rb:140:in
`install_serially'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer/parallel_installer.rb:91:in
`call'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer/parallel_installer.rb:67:in
`call'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer.rb:244:in
`install_in_parallel'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer.rb:201:in `install'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer.rb:89:in `block in run'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/process_lock.rb:19:in `rescue in lock'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/process_lock.rb:15:in `lock'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer.rb:71:in `run'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/installer.rb:23:in `install'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/cli/install.rb:62:in `run'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:260:in `block in install'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/settings.rb:131:in `temporary'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:259:in `install'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/command.rb:27:in
`run'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in
`invoke_command'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor.rb:392:in
`dispatch'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:34:in `dispatch'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/vendor/thor/lib/thor/base.rb:485:in
`start'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/cli.rb:28:in `start'
/opt/rubies/ruby-3.0.5/lib/ruby/gems/3.0.0/gems/bundler-2.4.1/exe/bundle:45:in `block in <top
(required)>'
/opt/rubies/ruby-3.0.5/lib/ruby/site_ruby/3.0.0/bundler/friendly_errors.rb:117:in
`with_friendly_errors'
/opt/rubies/ruby-3.0.5/lib/ruby/gems/3.0.0/gems/bundler-2.4.1/exe/bundle:33:in `<top
(required)>'
/opt/elasticbeanstalk/.rbenv/versions/3.0.5/bin/bundle:25:in `load'
/opt/elasticbeanstalk/.rbenv/versions/3.0.5/bin/bundle:25:in `<main>'
An error occurred while installing public_suffix (5.0.1), and Bundler cannot
continue.
In Gemfile:
capybara was resolved to 3.38.0, which depends on
addressable was resolved to 2.8.1, which depends on
public_suffix
It also seems the postgresql service is not installed (checked installed services starting with the "post" string):
[root#ip************ staging]# systemctl list-unit-files | grep post
postfix.service enabled
So, the question is, how do I make the eb deploy command run without failing? given that:
I see one error using eb deploy that highlights a failing rake db:migrate command but, when trying to running it manually after connecting through SSH it throws a different error; the one that mentions public_suffix. This seems unexpected.
The postgresql service should be installed in AWS's managed platform, as with previous versions
Adding a database to your Elastic Beanstalk environment
https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.managing.db.html
It looks like your app is trying to connect to a local database server:
Caused by:
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
What are the contents of your config/database.yml? Is it reading environment variables that elasticbeanstalk knows to pass in? Does your RDS instance have the right access rules set up to allow connections from EB?
Right now my DigitalOcean managed database backend connection limit is 22. Here are my sidekiq.yml and database.yml configurations:
# config/sidekiq.yml
development:
:concurrency: 18
production:
:concurrency: 18
.
# config/database.yml
default: &default
adapter: postgresql
encoding: unicode
pool: 18
I have a bunch or workers that interact with the database whenever I kick off a scheduled task. Since we're planning on allowing this scheduled task to kick off multiple times at the same time, we're running into a lot of connection database pool errors. So now I'm trying to figure out the best way to optimize this process or find another service that may be better for us.
For testing purposes, I created a sidekiq worker that looks like this:
class MySampleWorker
include Sidekiq::Worker
sidekiq_options queue: Rails.env.to_sym
def perform
User.first
end
end
If I call this sidekiq worker 20 times simultaneously, everything runs just smoothly. But if I call it 50 times simultaneously, then I get about 3-5 failed workers that ends up being retried.
My question is -- how would I scale something like this? In my case, I'm going to have to call the same workers multiple times, more and more as the demand grows, and that's obviously going to result in several failed workers. In some cases, these workers may take 5-10 minutes each - they're essentially running health check commands on remote systems and waiting on outputs in order to complete the worker.
Scaling in this manner seems like it would be disastrous. Rather than the workers failing, is there any way to just simply have them queued up and run when there's available space, as opposed to failing? If I understand the way this works, shouldn't database.yml be limiting the connection at 18 and, thus, it should never actually be trying to access the postgresql database beyond 18 connections at a time while the postgresql database has a connection limit of 22? I would think I should expect to see activerecord database connection timeout errors before I see any issues with postgresql database connection issues.
Here's the error I receive when one of the workers fail now:
2020-06-18T15:05:16.536Z pid=1152049 tid=gofhou0qp WARN: PG::ConnectionBad: FATAL: remaining connection slots are reserved for non-replication superuser connections
2020-06-18T15:05:16.536Z pid=1152049 tid=gofhou0qp WARN: /usr/local/rvm/gems/ruby-2.5.8/gems/pg-1.2.3/lib/pg.rb:58:in `initialize'
/usr/local/rvm/gems/ruby-2.5.8/gems/pg-1.2.3/lib/pg.rb:58:in `new'
/usr/local/rvm/gems/ruby-2.5.8/gems/pg-1.2.3/lib/pg.rb:58:in `connect'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:692:in `connect'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:223:in `initialize'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:48:in `new'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/postgresql_adapter.rb:48:in `postgresql_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:830:in `new_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:874:in `checkout_new_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:853:in `try_to_checkout_new_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:814:in `acquire_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:538:in `checkout'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:382:in `connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:1033:in `retrieve_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_handling.rb:118:in `retrieve_connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/connection_handling.rb:90:in `connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/delegation.rb:76:in `connection'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/query_methods.rb:934:in `build_arel'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/query_methods.rb:900:in `arel'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:560:in `block in exec_queries'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:584:in `skip_query_cache_if_necessary'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:547:in `exec_queries'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:422:in `load'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:200:in `records'
/usr/local/rvm/gems/ruby-2.5.8/gems/bullet-6.1.0/lib/bullet/active_record52.rb:46:in `records'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation.rb:195:in `to_ary'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/finder_methods.rb:532:in `find_nth_with_limit'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/finder_methods.rb:517:in `find_nth'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/relation/finder_methods.rb:125:in `first'
/usr/local/rvm/gems/ruby-2.5.8/gems/activerecord-5.2.4/lib/active_record/querying.rb:5:in `first'
/var/www/test-dev/app/workers/my_sample_worker.rb:12:in `perform'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:196:in `execute_job'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:163:in `block in process'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:111:in `local'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:43:in `block in call'
/usr/local/rvm/gems/ruby-2.5.8/gems/activesupport-5.2.4/lib/active_support/execution_wrapper.rb:87:in `wrap'
/usr/local/rvm/gems/ruby-2.5.8/gems/activesupport-5.2.4/lib/active_support/reloader.rb:73:in `block in wrap'
/usr/local/rvm/gems/ruby-2.5.8/gems/activesupport-5.2.4/lib/active_support/execution_wrapper.rb:87:in `wrap'
/usr/local/rvm/gems/ruby-2.5.8/gems/activesupport-5.2.4/lib/active_support/reloader.rb:72:in `wrap'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/rails.rb:42:in `call'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:257:in `stats'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:13:in `call'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/job_retry.rb:78:in `global'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/logger.rb:10:in `with'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/job_logger.rb:33:in `prepare'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:162:in `process'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:78:in `process_one'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/processor.rb:68:in `run'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:15:in `watchdog'
/usr/local/rvm/gems/ruby-2.5.8/gems/sidekiq-6.0.7/lib/sidekiq/util.rb:24:in `block in safe_thread'
Here's my Puma config:
threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
threads threads_count, threads_count
port ENV.fetch("PORT") { 3000 }
environment ENV.fetch("RAILS_ENV") { "development" }
As you scale up, you should just get a queue of jobs for workers to work on - if your concurrency is correctly set. Not failed jobs - your expectation there is correct.
However, the expectation that setting the database pool to 18 is maybe wrong. And, certainly only applies within each application server (do you have multiple?)
But this will depend on how you have Puma configured, post that config.
And see https://devcenter.heroku.com/articles/concurrency-and-database-connections for a good description of how the pool should be shared between workers and multiple application servers. And, https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server for best practise on setting worker and thread size.
I am trying to deploy my rails app with capistrano and I am getting the following error after the command:
cap production deploy
Output:
DEBUG[a3e9636c] Running /usr/bin/env [ -d ~/.rvm ] on handco-op.com
DEBUG[a3e9636c] Command: [ -d ~/.rvm ]
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing on host appname.com: Authentication failed for user deploy#appname.com
/home/deploy/.rvm/gems/ruby-2.1.2/gems/net-ssh-2.9.1/lib/net/ssh.rb:219:in `start'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/connection_pool.rb:50:in `call'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/connection_pool.rb:50:in `create_new_entry'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/connection_pool.rb:22:in `checkout'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:179:in `with_ssh'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:131:in `block in _execute'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:128:in `tap'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:128:in `_execute'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:62:in `test'
/home/deploy/.rvm/gems/ruby-2.1.2/bundler/gems/rvm-dc35ec5e7a74/lib/capistrano/tasks/rvm.rake:19:in `block (3 levels) in <top (required)>'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:54:in `instance_exec'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:54:in `run'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/runners/parallel.rb:13:in `block (2 levels) in execute'
Net::SSH::AuthenticationFailed: Authentication failed for user deploy#appname.com
/home/deploy/.rvm/gems/ruby-2.1.2/gems/net-ssh-2.9.1/lib/net/ssh.rb:219:in `start'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/connection_pool.rb:50:in `call'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/connection_pool.rb:50:in `create_new_entry'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/connection_pool.rb:22:in `checkout'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:179:in `with_ssh'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:131:in `block in _execute'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:128:in `tap'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:128:in `_execute'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:62:in `test'
/home/deploy/.rvm/gems/ruby-2.1.2/bundler/gems/rvm-dc35ec5e7a74/lib/capistrano/tasks/rvm.rake:19:in `block (3 levels) in <top (required)>'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:54:in `instance_exec'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/backends/netssh.rb:54:in `run'
/home/deploy/.rvm/gems/ruby-2.1.2/gems/sshkit-1.5.1/lib/sshkit/runners/parallel.rb:13:in `block (2 levels) in execute'
Tasks: TOP => rvm:hook
(See full trace by running task with --trace)
Just to be clear, I replaced my real server name with "appname" to stay secret;) Thanks for any advice! I hope it is a simple fix and I will keep looking into the error as well.
I was missing secret keys, had to set up nopassword ssh and a lot of other tweaks. It is a hard error to fix and takes a lot of things to get around. It was about three hours of debugging and fixing my permissions. Some tutorials setting this up actually can mess things up such as setting the users in the sudo group really makes it hard to define the user directly in: visudo I recommend not adding anything to the sudo group via:
sudo adduser user sudo
This was just my set up so you may need to add a user to sudo but I prefer giving ALL permissions to my users in visudo with:
visudo
user ALL = NOPASSWD:
Hope this someone get around this! it was a dang hard error to fix with a lot happening and was mostly my user errors when configuring capistrano. Good luck! Keep calm and code on!;)
This error in Capistrano (> 3) prevents me from deploying my Rails application. My Capistrano config works fine on another environment.
DEBUG [6c7d7bed] Command: if test ! -d /opt/my_app/releases/20140429175529; then echo "Directory does not exist '/opt/my_app/releases/20140429175529'" 1>&2; false; fi
cap aborted!
NoMethodError: undefined method `recv' for #<IO:fd 49>
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh/buffered_io.rb:65:in `fill'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/packet_stream.rb:86:in `next_packet'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:178:in `block in poll_message'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:173:in `loop'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:173:in `poll_message'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:210:in `block in wait'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:208:in `loop'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:208:in `wait'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh/transport/session.rb:87:in `initialize'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh.rb:200:in `new'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/net-ssh-2.8.0/lib/net/ssh.rb:200:in `start'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/backends/connection_pool.rb:24:in `create_or_reuse_connection'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:173:in `ssh'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:126:in `block in _execute'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:123:in `tap'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:123:in `_execute'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:66:in `execute'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/backends/abstract.rb:75:in `within'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/capistrano-3.2.1/lib/capistrano/tasks/deploy.rake:207:in `block (3 levels) in <top (required)>'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:54:in `instance_exec'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/backends/netssh.rb:54:in `run'
/Users/such/.rvm/gems/ruby-2.1.1#my_app/gems/sshkit-1.4.0/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
Tasks: TOP => deploy:set_current_revision
(See full trace by running task with --trace)
The deploy has failed with an error: #<NoMethodError: undefined method `recv' for #<IO:fd 49>>
I'm using a proxy to connect to my remote servers (on both environment). SSH connection is working perfectly.
set :ssh_options, proxy: Net::SSH::Proxy::Command.new('ssh user#gateway -W %h:%p')
Has anyone ever experienced this issue?
Workaround
My weak understanding of the error is that after the ssh negotiation, sometimes the socket returned to net-ssh is a file descriptor which is unexpected.
I have added:
gem 'sshkit', github: 'msimonin/sshkit', branch: 'gateway'
to my Gemfile so that the ssh connection is only established once.
Related configuration is:
SSHKit.config.backend.configure do |ssh|
ssh.gateway = 'user#gateway'
end
It works fine this way. I can't wait for the pull request to be merged.
The connection objects are always a file descriptor-backed IO instance if you're using proxy commands. The code lives in lib/net/ssh/proxy/command.rb and it injects the recv method right after the bit that popens the pipes and waits for something to become readable, so I'm not sure why it sometimes fails (I've seen it too). Based on the callstack, I think it has returned already, so recv should be there.
Have you opened an issue?
I made some changes in my local development version of my app.
Then I uploaded everything using git the usual thing.
To push my changes to the database I used:
heroku db:push
I get this error on my terminal:
Sending schema
/Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:175:in `async_exec': PG::Error: ERROR: permission denied for relation temp_listings (Sequel::DatabaseError)
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:175:in `block (2 levels) in execute'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/database/logging.rb:28:in `log_yield'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:175:in `block in execute'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:158:in `check_disconnect_errors'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:175:in `execute'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:240:in `block (2 levels) in execute'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/connection_pool/threaded.rb:84:in `hold'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/database/connecting.rb:226:in `synchronize'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:240:in `block in execute'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:261:in `check_database_errors'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:238:in `execute'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/dataset/actions.rb:541:in `execute'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/adapters/postgres.rb:319:in `fetch_rows'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/dataset/actions.rb:123:in `each'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/dataset/actions.rb:449:in `single_record'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/dataset/actions.rb:457:in `single_value'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/dataset/actions.rb:202:in `get'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/sequel-3.20.0/lib/sequel/dataset/actions.rb:96:in `count'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:570:in `block in fetch_local_tables_info'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:569:in `each'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:569:in `fetch_local_tables_info'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:551:in `local_tables_info'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:556:in `tables'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:439:in `push_schema'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:407:in `block in run'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:203:in `call'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:203:in `catch_errors'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/operation.rb:405:in `run'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/taps-0.3.24/lib/taps/cli.rb:172:in `clientxfer'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/heroku-2.25.0/lib/heroku/command/db.rb:194:in `taps_client'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/heroku-2.25.0/lib/heroku/command/db.rb:35:in `push'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/heroku-2.25.0/lib/heroku/command.rb:148:in `run'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/heroku-2.25.0/lib/heroku/cli.rb:9:in `start'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/gems/heroku-2.25.0/bin/heroku:16:in `<top (required)>'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/bin/heroku:19:in `load'
from /Users/AM/.rvm/gems/ruby-1.9.2-p290/bin/heroku:19:in `<main>'
Any idea how to fix this or what Im doing wrong?
I already tried running the following:
heroku pg:reset
then tried again
heroku db:push
that didnt work so I tried this
heroku run rake db:migrate
that didnt work either
so I created a new instance of my app and pushed it. Getting the same error
While convenient, using taps (what db:push/pull uses) may not replicate your data exactly and may run into errors for larger transfers. To import or export data from your production system, we highly recommend using pgbackups instead. http://devcenter.heroku.com/articles/pgbackups