I have installed the que gem and I would like to run the tasks I'm scheduling with active job, so far I have added in config/application.rb
config.active_job.queue_adapter = :que
and it stores the jobs in the database correctly but when I try to run
rake que:work
It starts but jobs are not working and I have reviewed run_at field, also tried running setting the env to RAIls_ENV=development. Tried with
que ./config/application.rb
and generates and error
I, [2016-06-15T17:30:27.592632 #30871] INFO -- : {"lib":"que","hostname":"boris-Satellite-U845","pid":30871,"thread":9851100,"event":"worker_count_change","value":"4"}
I, [2016-06-15T17:30:27.592851 #30871] INFO -- : {"lib":"que","hostname":"boris-Satellite-U845","pid":30871,"thread":9851100,"event":"mode_change","value":"async"}
/home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/activerecord-4.2.6/lib/active_record/connection_handling.rb:109:in `connection_pool': ActiveRecord::ConnectionNotEstablished (ActiveRecord::ConnectionNotEstablished)
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/adapters/active_record.rb:54:in `checkout_activerecord_adapter'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/adapters/active_record.rb:7:in `checkout'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/job.rb:85:in `work'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:80:in `block in work_loop'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:75:in `loop'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:75:in `work_loop'
from /home/boris/.rvm/gems/ruby-2.3.1#rig_cleanup/gems/que-0.11.5/lib/que/worker.rb:19:in `block in initialize'
A bit late, but Que requires you to be running a Postgres Server for it work.
I would imagine this is why you are having issues.
Related
Rails server fails to start in Discourse project either in development or production. Below are the logs when trying to start the server in dev mode. The application was installed and has been working, It's deployed on AWS in production mode and restarting the unicorn loads the application for some time and again the url stops responding with error messages.
Development logs from $rails s
rb t#ip-XXX-XX-XX-XX-app:/var/www/discourse# vi
config/environments/development.r
root#ip-172-31-25-46-app:/var/www/discourse# rails s
=> Booting Puma
=> Rails 5.1.4 application starting in production
=> Run `rails server -h` for more startup options
Exiting
bundler: failed to load command: script/rails (script/rails)
Redis::CommandError: ERR Error running script (call to f_b06356ba4628144e123b652c99605b873107c9be): #user_script:14: #user_script: 14: -MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis/client.rb:121:in `call'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:2399:in `block in _eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:58:in `block in synchronize'
/usr/local/lib/ruby/2.4.0/monitor.rb:214:in `mon_synchronize'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:58:in `synchronize'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:2398:in `_eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/redis-3.3.5/lib/redis.rb:2450:in `evalsha'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.1/lib/message_bus/backends/redis.rb:380:in `cached_eval'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.1/lib/message_bus/backends/redis.rb:140:in `publish'
/var/www/discourse/vendor/bundle/ruby/2.4.0/gems/message_bus-2.1.1/lib/message_bus.rb:248:in `publish'
/var/www/discourse/lib/distributed_cache.rb:72:in `publish'
**Production logs **
/var/www/discourse/lib/demon/base.rb:109:in `ensure_running'
/var/www/discourse/lib/demon/base.rb:34:in `block in ensure_running'
/var/www/discourse/lib/demon/base.rb:33:in `each'
/var/www/discourse/lib/demon/base.rb:33:in `ensure_running'
config/unicorn.conf.rb:145:in `master_sleep'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:284:in `join'
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
/var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `load'
/var/www/discourse/vendor/bundle/ruby/2.3.0/bin/unicorn:23:in `<main>'
E, [2018-01-04T08:43:37.949928 #60] ERROR -- : reaped #<Process::Status: pid 5870 exit 1> worker=unknown
Detected dead worker 5870, restarting...
Loading Sidekiq in process id 5883
Failed to report error: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error. 4 Redis::CommandError (MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.)
/var/www/discourse/vendor/bundle/ruby/2.3.0/gems/redis-3.3.0/lib/redis/client.rb:121:in `call' web-exception
Redis logs
47:M 17 Jan 09:38:01.070 # Can't save in background: fork: Cannot allocate memory
47:M 17 Jan 09:38:07.087 * 10000 changes in 60 seconds. Saving...
The issue has been fixed, Edited this file
/etc/sysctl.conf and added the line at last
vm.overcommit_memory=1
After this restarted sysctl.
$ sudo sysctl -p /etc/sysctl.conf
Redis doesn't need the amount of memory which OS thinks, the status 1 means always overcommit, never check.
More details can be found from Redis Docs.
I'm attempting to use Heroku's CI to run my Rails application's tests but it's running into a problem when attempting to load my structure.sql file.
-----> Preparing test database
Running: rake db:schema:load_if_ruby
db:schema:load_if_ruby completed (3.24s)
Running: rake db:structure:load_if_sql
psql:/app/db/structure.sql:28: ERROR: must be owner of extension plpgsql
rake aborted!
failed to execute:
psql -v ON_ERROR_STOP=1 -q -f /app/db/structure.sql d767koa0m1kne1
Please check the output above for any errors and make sure that `psql` is installed in your PATH and has proper permissions.
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/postgresql_database_tasks.rb:108:in `run_cmd'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/postgresql_database_tasks.rb:80:in `structure_load'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:223:in `structure_load'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:236:in `load_schema'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:255:in `block in load_schema_current'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:304:in `block in each_current_configuration'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:303:in `each'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:303:in `each_current_configuration'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/tasks/database_tasks.rb:254:in `load_schema_current'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/railties/databases.rake:290:in `block (3 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/activerecord-5.1.1/lib/active_record/railties/databases.rake:294:in `block (3 levels) in <top (required)>'
/app/vendor/bundle/ruby/2.4.0/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => db:structure:load
(See full trace by running task with --trace)
!
! Could not prepare database for test
!
The relevant line here is:
psql:/app/db/structure.sql:28: ERROR: must be owner of extension plpgsql
rake aborted!
Structure.sql contains this line:
COMMENT ON EXTENSION plpgsql IS 'PL/pgSQL procedural language';
Any ideas on how to get this working on Heroku's CI?
Ended up overriding db:structure:dump to remove the COMMENT ON ... statements:
namespace :db do
namespace :structure do
task dump: [:environment, :load_config] do
filename = ENV["SCHEMA"] || File.join(ActiveRecord::Tasks::DatabaseTasks.db_dir, "structure.sql")
sql = File.read(filename).each_line.grep_v(/\ACOMMENT ON EXTENSION.+/).join
File.write(filename, sql)
end
end
end
Another workaround would be to add something like
if Rails.env.development?
ActiveRecord::Tasks::DatabaseTasks.structure_load_flags = ["-v", "ON_ERROR_STOP=0"]
end
anywhere in the initialisation / tasks pipeline before the db:structure:load is executed.
If Kyle's solution isn't enough and the errors aren't caused only by comments on extensions, but actual extensions installations, you can still go the hard way and add this to an initializer:
# This is a temporary workaround for the Rails issue #29049.
# It could be safely removed when the PR #29110 got merged and released
# to use instead IGNORE_PG_LOAD_ERRORS=1.
module ActiveRecord
module Tasks
class PostgreSQLDatabaseTasks
ON_ERROR_STOP_1 = 'ON_ERROR_STOP=0'.freeze
end
end
end
Note: This isn't specific to Heroku but a broader Rails 5.1 issue
There are two solutions to this problem. First, as it was previously noted, is disabling the ON_ERROR_STOP feature. It'd help regardless of the environment. Custom rake task:
namespace :db do
namespace :structure do
# This little task is a workaround for a problem introduced in Rails5. Most specificaly here
# https://github.com/rails/rails/blob/5-1-stable/activerecord/lib/active_record/tasks/postgresql_database_tasks.rb#L77
# When psql encounters an error during loading of the structure it exits at once with error code 1.
# And this happens on heroku. It renders review apps and heroku CI unusable if you use a structure instead of a schema.
# Why?
# Our `db/structure.sql` contains entries like `CREATE EXTENSION` or `COMMENT ON EXTENSION`.
# Zylion of extensions on heroku are loaded in template0, so "our" db also has them, but because of that
# only a superuser (or owner of template0) has access to them - not our heroku db user. For that reason
# we can neither create an extension (it already exists, but that is not a problem, because dump contains IF NOT EXIST)
# nor comment on it (and comments don't have IF NOT EXIST directive). And that's an error which could be safely ignored
# but which stops loading of the rest of the structure.
desc "Disable exit-on-error behaviour when loading db structure in postgresql"
task disable_errors: :environment do
ActiveRecord::Tasks::DatabaseTasks.structure_load_flags = ["-v", "ON_ERROR_STOP=0"]
end
end
end
# And use it like so:
bin/rails db:structure:disable_errors db:structure:load
Another option, in my opinion, superior if it comes only to Heroku, would be using PostgreSQL in in-dyno plan (https://devcenter.heroku.com/articles/heroku-ci-in-dyno-databases), which is basically a DB instance sitting within dyno, thus we have full access to it. Also, the test suite should be significantly faster because we use a localhost connection, not over the wire. To enable it, change your app.json content to have entries like so:
{
"environments": {
"test": {
"addons": [
"heroku-postgresql:in-dyno"
]
}
}
}
I am currently running sidekiq 4.1.2. I've never managed to be able to run more than a handfull of jobs concurrently. Recently it's been looking like I've run into an issue described in the Sidekiq's Troubleshooting WIKI called Too many connections to MongoDB. Apparently, mongoid 3 doesn't properly disconnect workers. However, I am using mongoid 5.1.3.
My issue surfaces when a job, while a few other jobs are running, tries to hit the database with a query:
Timeout::Error: Timed out attempting to dequeue connection after 30 sec.
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:190:in `wait_for_next!'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:176:in `block in dequeue_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:190:in `wait_for_next!'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:176:in `block in dequeue_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:172:in `loop'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:172:in `dequeue_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:62:in `block in dequeue'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:61:in `synchronize'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool/queue.rb:61:in `dequeue'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool.rb:51:in `checkout'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/connection_pool.rb:107:in `with_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/server/context.rb:63:in `with_connection'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/operation/executable.rb:34:in `execute'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:80:in `send_initial_query'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:41:in `block in each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/retryable.rb:51:in `call'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/retryable.rb:51:in `read_with_retry'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongo-2.2.5/lib/mongo/collection/view/iterable.rb:39:in `each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/query_cache.rb:207:in `each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual/mongo.rb:121:in `each'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual/mongo.rb:295:in `map'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual/mongo.rb:295:in `map'
/home/me/applications/myapp/shared/bundle/ruby/2.2.0/gems/mongoid-5.1.3/lib/mongoid/contextual.rb:20:in `map'
/home/me/applications/myapp/releases/20160618143407/app/jobs/myjob.rb:8:in `block in perform'
After one job fails, the other jobs fail soon after. This most often happens after a few handfull of jobs have finished succesfully, which could indicate that those jobs don't disconnect from the database. Looking at top I can't see that mongo load cpu is too much.
At the same time this started to occur, I noticed that my sidetiq 0.7.0 enabled recurring jobs were not scheduled properly. One job has stopped being queued, and others are only queued once after restart.
According to my Sidekiq web interface I have 1 queue, called default, with 25 threads. Max. 12-15 of them get busy at the same time.
Any idea how to troubleshoot this issue?
The default max pool queue size is 5. Bumping max_pool_size up to e.g. 25 will enable more connections to your db.
production:
clients:
default:
options:
max_pool_size: 25
Change adding wait_queue_timeout attribute in mongoid.yml production config:
production:
clients:
default:
uri: mongodb://aaaaa.com:27017/mongo
options:
connect_timeout: 30
wait_queue_timeout: 30
I have a really simple job:
class MyJob
#queue = :high
def self.perform(user_id)
user = User.find(user_id)
MyMailer.send_email(user).deliver
end
end
If I run it manually MyJob.perform(some_id)it works perfect. However, when Rescue is executing it, it returns this error:
Exception
Errno::ENOENT Error
No such file or directory - getcwd
shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:221:in `expand_path' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:221:in `initialize' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:251:in `new' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/template/resolver.rb:251:in `instances' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/lookup_context.rb:16:in `<class:LookupContext>' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/lookup_context.rb:12:in `<module:ActionView>' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/action_view/lookup_context.rb:5:in `<top (required)>' shared/bundle/ruby/1.9.1/gems/actionpack-3.2.12/lib/abstract_controller/view_paths.rb:45:in `lookup_context' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:456:in `process' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:452:in `initialize' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:439:in `new' shared/bundle/ruby/1.9.1/gems/actionmailer-3.2.12/lib/action_mailer/base.rb:439:in `method_missing' releases/1111111111111/app/jobs/my_job.rb:6:in `perform'
Have any ideas why might be happening?
Thanks!
Yes Sky. You are right about it needing to be restarted.
Some people received this error after trying to run from an already deleted directory.
I received this error after switching databases and leaving the server running. The old server info was still showing up but I was getting this error. Restarted my rails server and everything works fine with the new db.
Basically it means that there is a significant state change on the server, and your environment needs to be reset/restarted.
I started having this same issue on my production environment. After some investigation I found that this was caused by my resque workers failing to be properly restarted on each capistrano deployment.
I am running systemu from within a delayed_job process. I had a job fail, and found this error in my delayed_jobs table:
execution expired
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:76:in `close'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:76:in `popen'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:76:in `block (2 levels) in systemu'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:168:in `quietly'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:75:in `block in systemu'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:236:in `call'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:236:in `block in tmpdir'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:221:in `loop'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:221:in `tmpdir'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:68:in `systemu'
/usr/local/rvm/gems/ruby-1.9.2-p320#global/gems/systemu-2.5.0/lib/systemu.rb:10:in `systemu'
It was working fine before, but this is a larger job I'm running. Is there some timeout that I need to set?
Thanks!
OK the problem was that the delayed_job timed out. I had to up the time limit with:
Delayed::Worker.max_run_time = 48.hours #2 days
in app/config/initializers/delayed_job_config.rb