I've managed to consistently reproduce a deadlock situation (I believe) despite following the following resources:
https://guides.rubyonrails.org/threading_and_code_execution.html#executor
https://github.com/rails/rails/issues/27025#issuecomment-260169414
The following code will deadlock:
Parallel
.map(domains, in_threads: 4) do |domain|
Rails
.application
.executor
.wrap { EcommEnrichmentUtils.qualified_domain(domain) }
end
.compact
end
The contents of EcommEnrichmentUtils essentially sends out some HTTP requests.
Output for /rails/locks
Thread 0 [0x1a518 sleep] No lock (yielded share)
Waiting in start_exclusive to "load"
may be pre-empted for: "load"
blocked by: 1, 2
/Users/amirsharif/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/monitor.rb:108:in `sleep'
/Users/amirsharif/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/monitor.rb:108:in `wait'
/Users/amirsharif/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/monitor.rb:108:in `wait_for_cond'
/Users/amirsharif/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/monitor.rb:108:in `wait'
/Users/amirsharif/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/monitor.rb:116:in `wait_while'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/concurrency/share_lock.rb:220:in `wait_for'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/concurrency/share_lock.rb:83:in `block (2 levels) in start_exclusive'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/concurrency/share_lock.rb:82:in `block in start_exclusive'
/Users/amirsharif/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
/Users/amirsharif/.rvm/rubies/ruby-2.7.4/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/concurrency/share_lock.rb:77:in `start_exclusive'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/concurrency/share_lock.rb:149:in `exclusive'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/dependencies/interlock.rb:13:in `loading'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/dependencies.rb:39:in `load_interlock'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/dependencies.rb:402:in `require_or_load'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/dependencies.rb:558:in `load_missing_constant'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/dependencies.rb:213:in `const_missing'
/Users/amirsharif/Projects/sendapostcardonline/app/services/domain_helper.rb:90:in `block (2 levels) in validated_url_without_subdomain'
/Users/amirsharif/Projects/sendapostcardonline/app/services/domain_helper.rb:89:in `each'
/Users/amirsharif/Projects/sendapostcardonline/app/services/domain_helper.rb:89:in `detect'
/Users/amirsharif/Projects/sendapostcardonline/app/services/domain_helper.rb:89:in `block in validated_url_without_subdomain'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:102:in `block (2 levels) in get'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:189:in `calc'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:102:in `block in get'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:204:in `block in synchronized'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:199:in `synchronize'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:199:in `synchronized'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:102:in `get'
/Users/amirsharif/Projects/sendapostcardonline/app/services/domain_helper.rb:68:in `validated_url_without_subdomain'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `bind_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `validate_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/_methods.rb:270:in `block in _on_method_added'
/Users/amirsharif/Projects/sendapostcardonline/app/typed_service_objects/ecomm_enrichment_utils.rb:22:in `qualified_domain'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `bind_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `validate_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/_methods.rb:270:in `block in _on_method_added'
/Users/amirsharif/Projects/sendapostcardonline/app/controllers/ecomm_analyze/collections_controller.rb:374:in `block (3 levels) in normalize_domains'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/Users/amirsharif/Projects/sendapostcardonline/app/controllers/ecomm_analyze/collections_controller.rb:374:in `block (2 levels) in normalize_domains'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:507:in `call_with_index'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:367:in `block (2 levels) in work_in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:518:in `with_instrumentation'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:366:in `block in work_in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:215:in `block (4 levels) in in_threads'
---
Thread 1 [0x1a52c sleep] Sharing
blocking: 0
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:199:in `synchronize'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:199:in `synchronized'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:102:in `get'
/Users/amirsharif/Projects/sendapostcardonline/app/services/domain_helper.rb:68:in `validated_url_without_subdomain'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation_2_7.rb:703:in `bind_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation_2_7.rb:703:in `block in create_validator_method_medium1'
/Users/amirsharif/Projects/sendapostcardonline/app/typed_service_objects/ecomm_enrichment_utils.rb:22:in `qualified_domain'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `bind_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `validate_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/_methods.rb:270:in `block in _on_method_added'
/Users/amirsharif/Projects/sendapostcardonline/app/controllers/ecomm_analyze/collections_controller.rb:374:in `block (3 levels) in normalize_domains'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/Users/amirsharif/Projects/sendapostcardonline/app/controllers/ecomm_analyze/collections_controller.rb:374:in `block (2 levels) in normalize_domains'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:507:in `call_with_index'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:367:in `block (2 levels) in work_in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:518:in `with_instrumentation'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:366:in `block in work_in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:215:in `block (4 levels) in in_threads'
---
Thread 2 [0x1a540 sleep] Sharing
blocking: 0
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:199:in `synchronize'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:199:in `synchronized'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/zache-0.12.0/lib/zache.rb:102:in `get'
/Users/amirsharif/Projects/sendapostcardonline/app/services/domain_helper.rb:68:in `validated_url_without_subdomain'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation_2_7.rb:703:in `bind_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation_2_7.rb:703:in `block in create_validator_method_medium1'
/Users/amirsharif/Projects/sendapostcardonline/app/typed_service_objects/ecomm_enrichment_utils.rb:22:in `qualified_domain'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `bind_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `validate_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/_methods.rb:270:in `block in _on_method_added'
/Users/amirsharif/Projects/sendapostcardonline/app/controllers/ecomm_analyze/collections_controller.rb:374:in `block (3 levels) in normalize_domains'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/Users/amirsharif/Projects/sendapostcardonline/app/controllers/ecomm_analyze/collections_controller.rb:374:in `block (2 levels) in normalize_domains'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:507:in `call_with_index'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:367:in `block (2 levels) in work_in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:518:in `with_instrumentation'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:366:in `block in work_in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:215:in `block (4 levels) in in_threads'
---
Thread 3 [0x15f7c sleep] No lock (yielded share)
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:217:in `value'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:217:in `map'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:217:in `block (2 levels) in in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:213:in `handle_interrupt'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:213:in `block in in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:211:in `handle_interrupt'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:211:in `in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:360:in `work_in_threads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/parallel-1.20.1/lib/parallel.rb:281:in `map'
/Users/amirsharif/Projects/sendapostcardonline/app/controllers/ecomm_analyze/collections_controller.rb:370:in `block in normalize_domains'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/Users/amirsharif/Projects/sendapostcardonline/app/controllers/ecomm_analyze/collections_controller.rb:368:in `normalize_domains'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `bind_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/call_validation.rb:161:in `validate_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/sorbet-runtime-0.5.9465/lib/types/private/methods/_methods.rb:270:in `block in _on_method_added'
/Users/amirsharif/Projects/sendapostcardonline/app/controllers/ecomm_analyze/collections_controller.rb:177:in `bulk_add_domains'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/abstract_controller/base.rb:228:in `process_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actiontext-6.1.4.1/lib/action_text/rendering.rb:20:in `with_renderer'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actiontext-6.1.4.1/lib/action_text/engine.rb:59:in `block (4 levels) in <class:Engine>'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:126:in `instance_exec'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:137:in `run_callbacks'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:203:in `block in instrument'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/notifications.rb:203:in `instrument'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/abstract_controller/base.rb:165:in `process'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionview-6.1.4.1/lib/action_view/rendering.rb:39:in `process'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_controller/metal.rb:190:in `dispatch'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_controller/metal.rb:254:in `dispatch'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/journey/router.rb:32:in `each'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/routing/route_set.rb:842:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/meta_request-0.7.3/lib/meta_request/middlewares/app_request_handler.rb:15:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/meta_request-0.7.3/lib/meta_request/middlewares/meta_request_handler.rb:15:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/flipper-0.22.1/lib/flipper/middleware/memoizer.rb:77:in `memoized_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/flipper-0.22.1/lib/flipper/middleware/memoizer.rb:42:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/warden-1.2.9/lib/warden/manager.rb:36:in `block in call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/warden-1.2.9/lib/warden/manager.rb:34:in `catch'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/warden-1.2.9/lib/warden/manager.rb:34:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/cookies.rb:689:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activerecord-6.1.4.1/lib/active_record/migration.rb:601:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rollbar-3.2.0/lib/rollbar/middleware/rails/rollbar.rb:25:in `block in call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rollbar-3.2.0/lib/rollbar.rb:145:in `scoped'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rollbar-3.2.0/lib/rollbar/middleware/rails/rollbar.rb:22:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rollbar-3.2.0/lib/rollbar/middleware/rails/show_exceptions.rb:22:in `call_with_rollbar'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/web-console-4.1.0/lib/web_console/middleware.rb:132:in `call_app'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/web-console-4.1.0/lib/web_console/middleware.rb:28:in `block in call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/web-console-4.1.0/lib/web_console/middleware.rb:17:in `catch'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/web-console-4.1.0/lib/web_console/middleware.rb:17:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-contrib-2.3.0/lib/rack/contrib/response_headers.rb:19:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/meta_request-0.7.3/lib/meta_request/middlewares/headers.rb:18:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/railties-6.1.4.1/lib/rails/rack/logger.rb:37:in `call_app'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/railties-6.1.4.1/lib/rails/rack/logger.rb:26:in `block in call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/tagged_logging.rb:37:in `tagged'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/tagged_logging.rb:99:in `tagged'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/railties-6.1.4.1/lib/rails/rack/logger.rb:26:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/request_store-1.5.0/lib/request_store/middleware.rb:19:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/activesupport-6.1.4.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-livereload-0.3.17/lib/rack/livereload.rb:23:in `_call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-livereload-0.3.17/lib/rack/livereload.rb:14:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/static.rb:24:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/static.rb:24:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/debug_locks.rb:41:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/actionpack-6.1.4.1/lib/action_dispatch/middleware/host_authorization.rb:98:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-mini-profiler-2.3.3/lib/mini_profiler/profiler.rb:393:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/webpacker-5.4.3/lib/webpacker/dev_server_proxy.rb:25:in `perform_request'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/rack-proxy-0.7.0/lib/rack/proxy.rb:63:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/railties-6.1.4.1/lib/rails/engine.rb:539:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/puma-5.5.2/lib/puma/configuration.rb:249:in `call'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/puma-5.5.2/lib/puma/request.rb:77:in `block in handle_request'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/puma-5.5.2/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/puma-5.5.2/lib/puma/request.rb:76:in `handle_request'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/puma-5.5.2/lib/puma/server.rb:447:in `process_client'
/Users/amirsharif/.rvm/gems/ruby-2.7.4/gems/puma-5.5.2/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
Unfortunately I don't know how to parse this or what a reasonable next step is.
Can anyone shed some light on this?
Okay, I figured it out.
It seems that this is an issue with the Zache gem.
See this line:
https://github.com/yegor256/zache/blob/master/lib/zache.rb#L206
This I believe introduces an external sleep.
I was able to fix my code by wrapping the method BEFORE called zache.get
Something like this:
ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
Rails
.application
.executor
.wrap do
zache.get(url, lifetime: 10.minutes) do
# do work here
end
end
end
Related
Example:
I'm changing something in controller, save it, and then refresh the page. Page is loading, and loading, and loading (same for server logs)... I can't kill server (CTRL + C doesn't work). I have to close tab, delete server pid and start server again.
Puma version: 6.0.0 (ruby 3.0.4-p208) ("Sunflower")
Ruby version: 3.0.4
Rails version: 6.1.6
EDIT 1:
MemTotal: 16351512 kB
MemFree: 6512996 kB
MemAvailable: 10926512 kB
This is what I got after visit localhost:3000/rails/locks
Thread 0 [0xe54c sleep] No lock (yielded share)
Waiting in start_exclusive to "unload"
may be pre-empted for: "load", "unload"
blocked by: 1
/usr/share/rvm/rubies/ruby-3.0.4/lib/ruby/3.0.0/monitor.rb:108:in `sleep'
/usr/share/rvm/rubies/ruby-3.0.4/lib/ruby/3.0.0/monitor.rb:108:in `wait'
/usr/share/rvm/rubies/ruby-3.0.4/lib/ruby/3.0.0/monitor.rb:108:in `wait_for_cond'
/usr/share/rvm/rubies/ruby-3.0.4/lib/ruby/3.0.0/monitor.rb:108:in `wait'
/usr/share/rvm/rubies/ruby-3.0.4/lib/ruby/3.0.0/monitor.rb:116:in `wait_while'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/concurrency/share_lock.rb:220:in `wait_for'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/concurrency/share_lock.rb:83:in `block (2 levels) in start_exclusive'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/concurrency/share_lock.rb:82:in `block in start_exclusive'
/usr/share/rvm/rubies/ruby-3.0.4/lib/ruby/3.0.0/monitor.rb:202:in `synchronize'
/usr/share/rvm/rubies/ruby-3.0.4/lib/ruby/3.0.0/monitor.rb:202:in `mon_synchronize'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/concurrency/share_lock.rb:77:in `start_exclusive'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/dependencies/interlock.rb:25:in `start_unloading'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/reloader.rb:100:in `require_unload_lock!'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/reloader.rb:119:in `class_unload!'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/railties-6.1.6/lib/rails/application/finisher.rb:244:in `block (2 levels) in <module:Finisher>'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:427:in `instance_exec'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:427:in `block in make_lambda'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:603:in `catch'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:603:in `block in default_terminator'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:199:in `block in halting'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:512:in `block in invoke_before'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:512:in `each'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:512:in `invoke_before'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/callbacks.rb:105:in `run_callbacks'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/execution_wrapper.rb:114:in `run!'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/reloader.rb:114:in `run!'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/execution_wrapper.rb:77:in `block in run!'
<internal:kernel>:90:in `tap'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/execution_wrapper.rb:74:in `run!'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/reloader.rb:63:in `run!'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/executor.rb:12:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/better_errors-2.9.1/lib/better_errors/middleware.rb:87:in `protected_app_call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/better_errors-2.9.1/lib/better_errors/middleware.rb:82:in `better_errors_call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/better_errors-2.9.1/lib/better_errors/middleware.rb:60:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/web-console-4.2.0/lib/web_console/middleware.rb:132:in `call_app'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/web-console-4.2.0/lib/web_console/middleware.rb:28:in `block in call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/web-console-4.2.0/lib/web_console/middleware.rb:17:in `catch'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/web-console-4.2.0/lib/web_console/middleware.rb:17:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/railties-6.1.6/lib/rails/rack/logger.rb:37:in `call_app'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/railties-6.1.6/lib/rails/rack/logger.rb:26:in `block in call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/tagged_logging.rb:99:in `block in tagged'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/tagged_logging.rb:37:in `tagged'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/activesupport-6.1.6/lib/active_support/tagged_logging.rb:99:in `tagged'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/railties-6.1.6/lib/rails/rack/logger.rb:26:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/debug_locks.rb:41:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/actionpack-6.1.6/lib/action_dispatch/middleware/host_authorization.rb:148:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/rack-mini-profiler-2.3.4/lib/mini_profiler/profiler.rb:393:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/webpacker-5.4.3/lib/webpacker/dev_server_proxy.rb:25:in `perform_request'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/rack-proxy-0.7.2/lib/rack/proxy.rb:67:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/railties-6.1.6/lib/rails/engine.rb:539:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/configuration.rb:268:in `call'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/request.rb:89:in `block in handle_request'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/thread_pool.rb:343:in `with_force_shutdown'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/request.rb:88:in `handle_request'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/server.rb:430:in `process_client'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/server.rb:233:in `block in run'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/thread_pool.rb:150:in `block in spawn_thread'
---
Thread 1 [0xe150 sleep] Sharing
blocking: 0
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/thread_pool.rb:136:in `sleep'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/thread_pool.rb:136:in `wait'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/thread_pool.rb:136:in `block (2 levels) in spawn_thread'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/thread_pool.rb:120:in `synchronize'
/home/maciej/.rvm/gems/ruby-3.0.4/gems/puma-6.0.0/lib/puma/thread_pool.rb:120:in `block in spawn_thread'
My console logs are normal. I mean... They appear very slowly, but they are correct
Getting this error even though email_verification does not exist in our codebase. Here's the stack trace:
NoMethodError: undefined method `email_verification' for UserMailer:Class
from action_mailer/base.rb:582:in `method_missing'
from action_mailer/delivery_job.rb:16:in `public_send'
from action_mailer/delivery_job.rb:16:in `perform'
from active_job/execution.rb:39:in `block in perform_now'
from active_support/callbacks.rb:109:in `block in run_callbacks'
from raven/integrations/rails/active_job.rb:18:in `capture_and_reraise_with_sentry'
from raven/integrations/rails/active_job.rb:12:in `block (2 levels) in included'
from active_support/callbacks.rb:118:in `instance_exec'
from active_support/callbacks.rb:118:in `block in run_callbacks'
from new_relic/agent/instrumentation/active_job.rb:53:in `perform'
from new_relic/agent/instrumentation/active_job.rb:23:in `block (4 levels) in <main>'
from active_support/callbacks.rb:118:in `instance_exec'
from active_support/callbacks.rb:118:in `block in run_callbacks'
from i18n.rb:313:in `with_locale'
from active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
from active_support/callbacks.rb:118:in `instance_exec'
from active_support/callbacks.rb:118:in `block in run_callbacks'
from active_job/logging.rb:26:in `block (4 levels) in <module:Logging>'
from active_support/notifications.rb:168:in `block in instrument'
from active_support/notifications/instrumenter.rb:23:in `instrument'
from active_support/notifications.rb:168:in `instrument'
from active_job/logging.rb:25:in `block (3 levels) in <module:Logging>'
from active_job/logging.rb:46:in `block in tag_logger'
from active_support/tagged_logging.rb:71:in `block in tagged'
from active_support/tagged_logging.rb:28:in `tagged'
from active_support/tagged_logging.rb:71:in `tagged'
from active_job/logging.rb:46:in `tag_logger'
from active_job/logging.rb:22:in `block (2 levels) in <module:Logging>'
from active_support/callbacks.rb:118:in `instance_exec'
from active_support/callbacks.rb:118:in `block in run_callbacks'
from active_support/callbacks.rb:136:in `run_callbacks'
from active_job/execution.rb:38:in `perform_now'
from active_job/execution.rb:24:in `block in execute'
from active_support/callbacks.rb:109:in `block in run_callbacks'
from active_job/railtie.rb:28:in `block (4 levels) in <class:Railtie>'
from active_support/execution_wrapper.rb:87:in `wrap'
from active_support/reloader.rb:73:in `block in wrap'
from active_support/execution_wrapper.rb:83:in `wrap'
from active_support/reloader.rb:72:in `wrap'
from active_job/railtie.rb:27:in `block (3 levels) in <class:Railtie>'
from active_support/callbacks.rb:118:in `instance_exec'
from active_support/callbacks.rb:118:in `block in run_callbacks'
from active_support/callbacks.rb:136:in `run_callbacks'
from active_job/execution.rb:22:in `execute'
from active_job/queue_adapters/sidekiq_adapter.rb:42:in `perform'
from sidekiq/processor.rb:192:in `execute_job'
from sidekiq/processor.rb:165:in `block (2 levels) in process'
from sidekiq/middleware/chain.rb:128:in `block in invoke'
from new_relic/agent/instrumentation/sidekiq.rb:33:in `block in call'
from new_relic/agent/instrumentation/controller_instrumentation.rb:376:in `perform_action_with_newrelic_trace'
from new_relic/agent/instrumentation/sidekiq.rb:29:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from raven/integrations/sidekiq.rb:9:in `call'
from sidekiq/middleware/chain.rb:130:in `block in invoke'
from sidekiq/middleware/chain.rb:133:in `invoke'
from sidekiq/processor.rb:164:in `block in process'
from sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
from sidekiq/job_retry.rb:109:in `local'
from sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
from sidekiq/rails.rb:43:in `block in call'
from active_support/execution_wrapper.rb:87:in `wrap'
from active_support/reloader.rb:73:in `block in wrap'
from active_support/execution_wrapper.rb:87:in `wrap'
from active_support/reloader.rb:72:in `wrap'
from sidekiq/rails.rb:42:in `call'
from sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
from sidekiq/processor.rb:250:in `stats'
from sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
from sidekiq/job_logger.rb:8:in `call'
from sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
from sidekiq/job_retry.rb:74:in `global'
from sidekiq/processor.rb:125:in `block in dispatch'
from sidekiq/logging.rb:48:in `with_context'
from sidekiq/logging.rb:42:in `with_job_hash_context'
from sidekiq/processor.rb:124:in `dispatch'
from sidekiq/processor.rb:163:in `process'
from sidekiq/processor.rb:83:in `process_one'
from sidekiq/processor.rb:71:in `run'
from sidekiq/util.rb:16:in `watchdog'
from sidekiq/util.rb:25:in `block in safe_thread'
Unresolved Rails issue: https://github.com/rails/rails/issues/40607
Upgrading from Rails 5.2.3 to 6.0.0 or 6.0.1, with Ruby 2.6.3, after bundling succeeds, the rails app:update command has been carefully run, and webpacker updated, a request to any page that requires no authentication gives:
Puma caught this error: undefined method `clear' for nil:NilClass (NoMethodError)
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:181:in `clear_cache'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/path_set.rb:16:in `each'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/path_set.rb:16:in `each'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/lookup_context.rb:79:in `block in clear'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/lookup_context.rb:78:in `each'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/lookup_context.rb:78:in `clear'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/cache_expiry.rb:40:in `clear_cache'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/cache_expiry.rb:30:in `block (2 levels) in clear_cache_if_necessary'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/file_update_checker.rb:83:in `execute'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/cache_expiry.rb:32:in `block in clear_cache_if_necessary'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/cache_expiry.rb:23:in `synchronize'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/cache_expiry.rb:23:in `clear_cache_if_necessary'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/cache_expiry.rb:11:in `before'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:429:in `block in make_lambda'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:201:in `block (2 levels) in halting'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:607:in `block (2 levels) in default_terminator'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:606:in `catch'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:606:in `block in default_terminator'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:202:in `block in halting'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:514:in `block in invoke_before'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:514:in `each'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:514:in `invoke_before'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:134:in `run_callbacks'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/execution_wrapper.rb:111:in `run!'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/execution_wrapper.rb:73:in `block in run!'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/execution_wrapper.rb:70:in `tap'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/execution_wrapper.rb:70:in `run!'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/executor.rb:12:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/host_authorization.rb:83:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/webpacker-4.2.0/lib/webpacker/dev_server_proxy.rb:23:in `perform_request'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/rack-proxy-0.6.5/lib/rack/proxy.rb:57:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/railties-6.0.1/lib/rails/engine.rb:526:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/railties-6.0.1/lib/rails/railtie.rb:190:in `public_send'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/railties-6.0.1/lib/rails/railtie.rb:190:in `method_missing'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/rack-2.0.7/lib/rack/deflater.rb:34:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/configuration.rb:228:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/server.rb:667:in `handle_request'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/server.rb:470:in `process_client'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/server.rb:328:in `block in run'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
A second request to the same page gives:
Error during failsafe response: undefined method `compute_if_absent' for nil:NilClass
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:199:in `block in query'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:196:in `map'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:196:in `query'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:189:in `_find_all'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:121:in `block in find_all'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:156:in `block in cached'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:66:in `cache'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:155:in `cached'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/template/resolver.rb:120:in `find_all'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/path_set.rb:77:in `block (2 levels) in _find_all'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/path_set.rb:76:in `each'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/path_set.rb:76:in `block in _find_all'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/path_set.rb:75:in `each'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/path_set.rb:75:in `_find_all'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/path_set.rb:55:in `find_all'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/path_set.rb:48:in `find'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/lookup_context.rb:129:in `find'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/renderer/template_renderer.rb:47:in `determine_template'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/renderer/template_renderer.rb:9:in `render'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/renderer/renderer.rb:61:in `render_template_to_object'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/renderer/renderer.rb:29:in `render_to_object'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/rendering.rb:118:in `block in _render_template'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/base.rb:304:in `in_rendering_context'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/rendering.rb:117:in `_render_template'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/streaming.rb:219:in `_render_template'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/rendering.rb:103:in `render_to_body'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/rendering.rb:52:in `render_to_body'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/renderers.rb:142:in `render_to_body'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/abstract_controller/rendering.rb:25:in `render'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/rendering.rb:36:in `render'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/instrumentation.rb:44:in `block (2 levels) in render'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/core_ext/benchmark.rb:14:in `block in ms'
/Users/Emma/.rvm/rubies/ruby-2.6.3/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/core_ext/benchmark.rb:14:in `ms'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/instrumentation.rb:44:in `block in render'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/instrumentation.rb:85:in `cleanup_view_runtime'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activerecord-6.0.1/lib/active_record/railties/controller_runtime.rb:34:in `cleanup_view_runtime'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/instrumentation.rb:43:in `render'
/Users/Emma/rails/consonance/app/controllers/exceptions_controller.rb:20:in `block (2 levels) in show'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/mime_responds.rb:214:in `respond_to'
/Users/Emma/rails/consonance/app/controllers/exceptions_controller.rb:14:in `show'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/abstract_controller/base.rb:196:in `process_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/callbacks.rb:135:in `run_callbacks'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/notifications.rb:180:in `block in instrument'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/notifications.rb:180:in `instrument'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activerecord-6.0.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/abstract_controller/base.rb:136:in `process'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionview-6.0.1/lib/action_view/rendering.rb:39:in `process'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal.rb:191:in `dispatch'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_controller/metal.rb:236:in `block in action'
/Users/Emma/rails/consonance/config/application.rb:172:in `block in <class:Application>'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/show_exceptions.rb:51:in `render_exception'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/show_exceptions.rb:36:in `rescue in call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/railties-6.0.1/lib/rails/rack/logger.rb:38:in `call_app'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/railties-6.0.1/lib/rails/rack/logger.rb:26:in `block in call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/tagged_logging.rb:80:in `block in tagged'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/tagged_logging.rb:28:in `tagged'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/tagged_logging.rb:80:in `tagged'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/railties-6.0.1/lib/rails/rack/logger.rb:26:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/request_store-1.4.1/lib/request_store/middleware.rb:19:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/skylight-core-4.0.2/lib/skylight/core/probes/action_dispatch/request_id.rb:12:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/rack-2.0.7/lib/rack/method_override.rb:22:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/rack-2.0.7/lib/rack/runtime.rb:22:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/activesupport-6.0.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/rack-2.0.7/lib/rack/sendfile.rb:111:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/actionpack-6.0.1/lib/action_dispatch/middleware/host_authorization.rb:83:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/webpacker-4.2.0/lib/webpacker/dev_server_proxy.rb:23:in `perform_request'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/rack-proxy-0.6.5/lib/rack/proxy.rb:57:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/railties-6.0.1/lib/rails/engine.rb:526:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/railties-6.0.1/lib/rails/railtie.rb:190:in `public_send'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/railties-6.0.1/lib/rails/railtie.rb:190:in `method_missing'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/rack-2.0.7/lib/rack/deflater.rb:34:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/configuration.rb:228:in `call'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/server.rb:667:in `handle_request'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/server.rb:470:in `process_client'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/server.rb:328:in `block in run'
/Users/Emma/.rvm/gems/ruby-2.6.3/gems/puma-4.3.0/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
This means that the #unbound_templates variable introduced here is nil. On a Rails console, Concurrent::Map.new works fine.
How can this instance variable, that gets set in the initializer, be nil?
I have the exact same error and I have had no progress debugging it thus far. My suspicion is that it is caused by a gem but I just don't know yet.
#snowangel if you want to compare Gemfiles I've posted mine here: https://gist.github.com/biscuitvile/7bf0d4423ce0b2aa79e0ccc08bff4295
NOTE: I got the same error on the second request
undefined method `compute_if_absent' for nil:NilClass
TLDR;
Configure database's prepared_statements only to true or false
EXPLANATION
In my case, I erroneously configured prepared_statements to nil.
In this line on ActiveRecord, it calls #find_by_statement_cache then calls compute_if_absent,
def cached_find_by_statement(key, &block)
cache = #find_by_statement_cache[connection.prepared_statements]
# THIS LINE BELOW
cache.compute_if_absent(key) { StatementCache.create(connection, &block) }
end
where it has the following value:
#find_by_statement_cache = { true => Concurrent::Map.new, false => Concurrent::Map.new }
And since I've configured it to nil, it renders an undefined method compute_if_absent error.
Now I am using Ruby on Rails framework. We use delayed_job to run a job to operate aws s3 file by aws-sdk. We deal with many files on aws s3.
When we run the job, we got this error:
e.message => Access Denied
e.backtrace.join("\n") =>
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-2.5.2/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-2.5.2/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-2.5.2/lib/aws-sdk-core/plugins/s3_accelerate.rb:33:in `call'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-2.5.2/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-2.5.2/lib/aws-sdk-core/plugins/response_paging.rb:26:in `call'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-2.5.2/lib/seahorse/client/plugins/response_target.rb:21:in `call'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-2.5.2/lib/seahorse/client/request.rb:70:in `send_request'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-core-2.5.2/lib/seahorse/client/base.rb:207:in `block (2 levels) in define_operation_methods'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-resources-2.5.2/lib/aws-sdk-resources/request.rb:24:in `call'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-resources-2.5.2/lib/aws-sdk-resources/operations.rb:139:in `all_batches'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-resources-2.5.2/lib/aws-sdk-resources/collection.rb:18:in `each'
/app/shared/bundle/ruby/2.3.0/gems/aws-sdk-resources-2.5.2/lib/aws-sdk-resources/collection.rb:18:in `each'
/app/releases/20161107014751/app/jobs/my_job.rb:35:in `perform'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/execution.rb:34:in `block in perform_now'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:126:in `call'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:455:in `call'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:448:in `block (2 levels) in around'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:286:in `block (2 levels) in halting'
/app/shared/bundle/ruby/2.3.0/gems/i18n-0.7.0/lib/i18n.rb:257:in `with_locale'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/translation.rb:7:in `block (2 levels) in <module:Translation>'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:391:in `instance_exec'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:391:in `block in make_lambda'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:285:in `block in halting'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:447:in `block in around'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:455:in `call'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:448:in `block (2 levels) in around'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:286:in `block (2 levels) in halting'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/logging.rb:24:in `block (4 levels) in <module:Logging>'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/notifications.rb:164:in `block in instrument'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/notifications.rb:164:in `instrument'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/logging.rb:23:in `block (3 levels) in <module:Logging>'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/logging.rb:44:in `block in tag_logger'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/tagged_logging.rb:70:in `block in tagged'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/tagged_logging.rb:26:in `tagged'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/tagged_logging.rb:70:in `tagged'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/logging.rb:44:in `tag_logger'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/logging.rb:20:in `block (2 levels) in <module:Logging>'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:391:in `instance_exec'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:391:in `block in make_lambda'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:285:in `block in halting'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:447:in `block in around'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:455:in `call'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:101:in `__run_callbacks__'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:750:in `_run_perform_callbacks'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:90:in `run_callbacks'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/execution.rb:33:in `perform_now'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/execution.rb:22:in `block in execute'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:126:in `call'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:506:in `block (2 levels) in compile'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:455:in `call'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:448:in `block (2 levels) in around'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:286:in `block (2 levels) in halting'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/railtie.rb:26:in `block (4 levels) in <class:Railtie>'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/execution_wrapper.rb:76:in `wrap'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/reloader.rb:68:in `block in wrap'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/execution_wrapper.rb:76:in `wrap'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/reloader.rb:67:in `wrap'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/railtie.rb:25:in `block (3 levels) in <class:Railtie>'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:391:in `instance_exec'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:391:in `block in make_lambda'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:285:in `block in halting'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:447:in `block in around'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:455:in `call'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:101:in `__run_callbacks__'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:750:in `_run_execute_callbacks'
/app/shared/bundle/ruby/2.3.0/gems/activesupport-5.0.0/lib/active_support/callbacks.rb:90:in `run_callbacks'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/execution.rb:20:in `execute'
/app/shared/bundle/ruby/2.3.0/gems/activejob-5.0.0/lib/active_job/queue_adapters/delayed_job_adapter.rb:36:in `perform'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/backend/base.rb:84:in `block in invoke_job'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:61:in `block in initialize'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:66:in `execute'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/backend/base.rb:81:in `invoke_job'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:230:in `block (2 levels) in run'
/usr/lib64/ruby/2.3.0/timeout.rb:91:in `block in timeout'
/usr/lib64/ruby/2.3.0/timeout.rb:101:in `timeout'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:230:in `block in run'
/usr/lib64/ruby/2.3.0/benchmark.rb:308:in `realtime'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:229:in `run'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:306:in `block in reserve_and_run_one_job'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:61:in `block in initialize'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:66:in `execute'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:306:in `reserve_and_run_one_job'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:213:in `block in work_off'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:212:in `times'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:212:in `work_off'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:175:in `block (4 levels) in start'
/usr/lib64/ruby/2.3.0/benchmark.rb:308:in `realtime'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:174:in `block (3 levels) in start'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:61:in `block in initialize'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:66:in `execute'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:173:in `block (2 levels) in start'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:172:in `loop'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:172:in `block in start'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:79:in `block (2 levels) in add'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:61:in `block in initialize'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:79:in `block in add'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:66:in `execute'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/lifecycle.rb:40:in `run_callbacks'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/worker.rb:171:in `start'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/command.rb:132:in `run'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/command.rb:120:in `block in run_process'
/app/shared/bundle/ruby/2.3.0/gems/daemons-1.2.3/lib/daemons/application.rb:265:in `block in start_proc'
/app/shared/bundle/ruby/2.3.0/gems/daemons-1.2.3/lib/daemons/daemonize.rb:84:in `call_as_daemon'
/app/shared/bundle/ruby/2.3.0/gems/daemons-1.2.3/lib/daemons/application.rb:269:in `start_proc'
/app/shared/bundle/ruby/2.3.0/gems/daemons-1.2.3/lib/daemons/application.rb:295:in `start'
/app/shared/bundle/ruby/2.3.0/gems/daemons-1.2.3/lib/daemons/controller.rb:56:in `run'
/app/shared/bundle/ruby/2.3.0/gems/daemons-1.2.3/lib/daemons.rb:193:in `block in run_proc'
/app/shared/bundle/ruby/2.3.0/gems/daemons-1.2.3/lib/daemons/cmdline.rb:88:in `catch_exceptions'
/app/shared/bundle/ruby/2.3.0/gems/daemons-1.2.3/lib/daemons.rb:192:in `run_proc'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/command.rb:118:in `run_process'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/command.rb:99:in `block in daemonize'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/command.rb:97:in `times'
/app/shared/bundle/ruby/2.3.0/gems/delayed_job-4.1.2/lib/delayed/command.rb:97:in `daemonize'
/app/current/bin/delayed_job:5:in `<main>'
I don't know whether it is the delayed_job problem or the aws-sdk problem, or s3 network problem.
For sure its related to AWS not with delayed job , you need to check your s3 creds or contact to them for more details
Reference
http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
I'm using Delayed Job gem to dispatch emails using Amazon SES. However it runs occasionally into execution expired error. Probably some connection issues to SES. I'm wondering how can I catch this exception and re-schedule the email manually? Due to internal reasons Delayed Job is configured to single max attempt
| execution expired
/usr/local/lib/ruby/2.0.0/net/smtp.rb:540:in `initialize'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:540:in `open'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:540:in `tcp_socket'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:550:in `block in do_start'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:549:in `do_start'
/usr/local/lib/ruby/2.0.0/net/smtp.rb:519:in `start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/network/delivery_methods/smtp.rb:112:in `deliver!'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:2129:in `do_delivery'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `block in deliver'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/actionmailer-4.0.0/lib/action_mailer/base.rb:456:in `block in deliver_mail'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications.rb:159:in `block in instrument'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications.rb:159:in `instrument'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/actionmailer-4.0.0/lib/action_mailer/base.rb:454:in `deliver_mail'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/mail-2.5.4/lib/mail/message.rb:232:in `deliver'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/performable_mailer.rb:6:in `perform'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/backend/base.rb:105:in `block in invoke_job'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/backend/base.rb:102:in `invoke_job'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:206:in `block (2 levels) in run'
/usr/local/lib/ruby/2.0.0/timeout.rb:66:in `timeout'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:206:in `block in run'
/usr/local/lib/ruby/2.0.0/benchmark.rb:296:in `realtime'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:205:in `run'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:267:in `block in reserve_and_run_one_job'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:267:in `reserve_and_run_one_job'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:189:in `block in work_off'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:188:in `times'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:188:in `work_off'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:153:in `block (4 levels) in start'
/usr/local/lib/ruby/2.0.0/benchmark.rb:296:in `realtime'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:152:in `block (3 levels) in start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:151:in `block (2 levels) in start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:150:in `loop'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:150:in `block in start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/plugins/clear_locks.rb:7:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/plugins/clear_locks.rb:7:in `block (2 levels) in <class:ClearLocks>'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `block (2 levels) in add'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:60:in `block in initialize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:78:in `block in add'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:65:in `execute'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/lifecycle.rb:38:in `run_callbacks'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/worker.rb:149:in `start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:104:in `run'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:92:in `block in run_process'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application.rb:255:in `block in start_proc'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/daemonize.rb:82:in `call_as_daemon'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application.rb:259:in `start_proc'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application.rb:296:in `start'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:159:in `block (2 levels) in start_all'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:158:in `fork'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:158:in `block in start_all'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:157:in `each'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/application_group.rb:157:in `start_all'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/controller.rb:80:in `run'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons.rb:197:in `block in run_proc'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `call'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons/cmdline.rb:109:in `catch_exceptions'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/daemons-1.1.9/lib/daemons.rb:196:in `run_proc'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:90:in `run_process'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:83:in `block in daemonize'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:81:in `times'
/home/ubuntu/www/tf/shared/bundle/ruby/2.0.0/gems/delayed_job-4.0.0/lib/delayed/command.rb:81:in `daemonize'
/home/ubuntu/www/tf/current/bin/delayed_job:5:in `<main>'
delayed_job provides some hooks, you can use the error hook to rescheduled the task.
def error(job, exception)
# if exception is the one occurring right now for expired execution
# reschedule the job
# be careful to avoid infinite rescheduling
end