I have been trying to get a resque task running. Resque seems to be enqueueing and running tasks; however, tasks seem to be "deadlocking" often upon execution.
A sample error is as follows: (taken from the Resque::Failure output)
{"failed_at"=>"2015/07/20 22:34:46 UTC",
"payload"=>{"class"=>"ImportHawaiiEventsOnline", "args"=>[]},
"exception"=>"fatal",
"error"=>"No live threads left. Deadlock?",
"backtrace"=>
["/usr/local/lib/ruby/2.1.0/open3.rb:262:in `value'",
"/usr/local/lib/ruby/2.1.0/open3.rb:262:in `block in capture3'",
"/usr/local/lib/ruby/2.1.0/open3.rb:199:in `popen_run'",
"/usr/local/lib/ruby/2.1.0/open3.rb:93:in `popen3'",
"/usr/local/lib/ruby/2.1.0/open3.rb:252:in `capture3'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:47:in `execute_open3'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:36:in `block (2 levels) in execute'",
"/usr/local/lib/ruby/2.1.0/timeout.rb:76:in `timeout'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:35:in `block in execute'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/logger.rb:32:in `call'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/logger.rb:32:in `block in benchmark'",
"/usr/local/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/logger.rb:32:in `benchmark'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/logger.rb:21:in `debug'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:34:in `execute'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/shell.rb:18:in `run'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/tool.rb:92:in `call'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/tool.rb:53:in `new'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:477:in `identify'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:198:in `validate!'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:113:in `block in create'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:112:in `tap'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:112:in `create'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:34:in `read'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:90:in `block in open'",
"/usr/local/lib/ruby/2.1.0/open-uri.rb:36:in `open'",
"/usr/local/lib/ruby/2.1.0/open-uri.rb:36:in `open'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/mini_magick-4.2.7/lib/mini_magick/image.rb:89:in `open'",
"/srv/releases/4/app/uploaders/photo_uploader.rb:57:in `get_geometry'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/processing.rb:84:in `block in process!'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/processing.rb:76:in `each'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/processing.rb:76:in `process!'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/callbacks.rb:18:in `block in with_callbacks'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/callbacks.rb:18:in `each'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/callbacks.rb:18:in `with_callbacks'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/cache.rb:122:in `cache!'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/uploader/download.rb:72:in `download!'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/mount.rb:356:in `remote_url='",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/mount.rb:187:in `remote_photo_url='",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/carrierwave-0.10.0/lib/carrierwave/orm/activerecord.rb:45:in `remote_photo_url='",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:45:in `public_send'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:45:in `_assign_attribute'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:32:in `block in assign_attributes'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:26:in `each'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/activerecord-4.1.1/lib/active_record/attribute_assignment.rb:26:in `assign_attributes'",
"/srv/releases/4/event_jobs/generic_import.rb:45:in `event_endpoint'",
"/srv/releases/4/event_jobs/generic_import.rb:113:in `save_event'",
"/srv/releases/4/event_jobs/hawaiieventonline/import.rb:26:in `block (2 levels) in get_parser'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `upto'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `each'",
"/srv/releases/4/event_jobs/hawaiieventonline/import.rb:24:in `each_with_index'",
"/srv/releases/4/event_jobs/hawaiieventonline/import.rb:24:in `block in get_parser'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `upto'",
"/srv/releases/4/vendor/bundle/ruby/2.1.0/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `each'",
"/srv/releases/4/event_jobs/hawaiieventonline/import.rb:21:in `get_parser'",
"/srv/releases/4/event_jobs/generic_import.rb:22:in `run_parse'",
"/srv/releases/4/app/jobs/import_hawaiieventsonline.rb:17:in `perform'"],
"worker"=>
"be920dfe9b17:5637:`(queues)`",
"queue"=>"`(queue)`"},
I was guessing it had to do with the mini_magick gem - though it seems odd that it would suggest a deadlock since I'm only running one thread. I would guess its possibly due the sheer amount of requests being made asynchronously - though I don't fully understand the mini_magick and carrierwave gems.
Does anyone have any suggestions on where to look to resolve this thread issue? I am running 17 queues with one worker thread handling all of these queues at the moment, and it seems to run fine - just that there may be an issue with saving to the cloud. Trying to search for anything related to mini_magick/carrierwave and a thread issue seems to not show anything.
Thanks in advance.
I'm curious, are you running Resque-Pool or any other forking/threading in your code? MiniMagick uses open3 to run shell commands, and it apparently has a reputation for not playing nice with threads. Luckily, MiniMagick allows you to swap out open3 for Posix-Spawn.
I was getting the same error using threaded workers (Resque-Pool jobs) and switching to Posix-Spawn appears to have resolved the error for me.
Related
I am using watir-webdriver gem to run over a website.
I tried to click a button to load more content in a webpage, for that I used the following code:
b = Watir::Browser.new
b.goto "https://www.example.com/"
b.button(:class => "ga_Finder_Load_More").click
This code is running fine in linux(ubuntu) and I can load the data in the webpage by clicking the button. But when I run the same code in window 10 I am getting the following error,
C:/Ruby200/lib/ruby/gems/2.0.0/gems/watir-webdriver-0.9.1/lib/watir-webdriver/elements/element.rb:508:in `assert_exists' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/watir-webdriver-0.9.1/lib/watir-webdriver/elements/element.rb:114:in `click' from leafly_v1.3.rb:70:in `block (2 levels) in <main>' from leafly_v1.3.rb:67:in `each' from leafly_v1.3.rb:67:in `block in <main>' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/nokogiri-1.6.6.4-x86-mingw32/lib/nokogiri/xml/node_set.rb:187:in `block in each' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/nokogiri-1.6.6.4-x86-mingw32/lib/nokogiri/xml/node_set.rb:186:in `upto' from C:/Ruby200/lib/ruby/gems/2.0.0/gems/nokogiri-1.6.6.4-x86-mingw32/lib/nokogiri/xml/node_set.rb:186:in `each' from leafly_v1.3.rb:62:in `<main>' C:\Ruby200>
Ruby version - 2.0.0
watir-webdriver - 0.9.1
Can anyone please help me to fix this or can any one suggest me alternate method to click a button.
Thanks.
It often happens between different operating systems and browsers that pages will load with slightly different timing. Try:
b.button(:class => "ga_Finder_Load_More").when_present.click
I'm using Rails and Rollbar to track errors and Resque to handle delayed tasks. One of my workers is failing with below error in the resque GUI:
Net::ReadTimeout
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue in rbuf_fill'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1412:in `block in transport_request'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `catch'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `transport_request'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1382:in `request'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1283:in `request_get'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:479:in `block in get_response'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:852:in `start'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:582:in `start'
/app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:477:in `get_response'
/app/app/services/get_json.rb:4:in `call'
/app/app/models/imdb.rb:9:in `data'
/app/app/models/customer.rb:92:in `imdb_data'
/app/app/models/customer.rb:60:in `get_external_data'
/app/app/models/customer.rb:13:in `scan'
/app/app/models/shop.rb:110:in `block in bulk_scan'
/app/app/models/shop.rb:101:in `each'
/app/app/models/shop.rb:101:in `bulk_scan'
/app/app/workers/bulk_scanner.rb:9:in `perform'
But it would be really helpful to see more robust information about the failure. E.g. what were the parameters. Usually I use Rollbar for this type of thing, but it doesn't seem to be logging this error for some reason.
How would I go about fixing this bug? Because I don't really know what's triggering it? (my worker is basically looping through a whole bunch of API calls with different users, and I don't know which one is failing)
You can create a custom logger, and use it to record useful information to trace your problem. In your worker, add debug statements that dump out variable contents, etc.
my_log = Logger.new("log/worker.log")
def do_stuff
my_log.info "#do_stuff"
end
Another option is to put a breakpoint in your code. You'll get an interactive shell that you can use to fiddle around.
debugger
We are currently upgrading our Rails app to Rails 4. In 3.2, our Cucumber (1.3.17) tests run fine (if lamentably slowly), with Capybara (2.4.4), Poltergeist (1.5.1), and PhantomJS (1.9.8) under the hood.
But in both 4.0.12 and 4.1.8, we get a stream closed (IOError) at a random point in the run:
stream closed (IOError)
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/formatter/pretty.rb:156:in `write'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/formatter/pretty.rb:156:in `puts'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/formatter/pretty.rb:156:in `step_name'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:181:in `block in send_to_all'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:179:in `each'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:179:in `send_to_all'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:173:in `broadcast'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:119:in `visit_step_name'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:112:in `block in visit_step_result'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:111:in `visit_step_result'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/step_invocation.rb:43:in `visit_step_result'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/step_invocation.rb:39:in `accept'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:106:in `block in visit_step'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:105:in `visit_step'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/step_collection.rb:19:in `block in accept'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/step_collection.rb:18:in `each'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/step_collection.rb:18:in `accept'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:100:in `block in visit_steps'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:99:in `visit_steps'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:15:in `block in execute'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime.rb:83:in `block (2 levels) in with_hooks'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime.rb:99:in `before_and_after'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime.rb:82:in `block in with_hooks'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime/support_code.rb:120:in `call'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime/support_code.rb:120:in `block (3 levels) in around'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/language_support/language_methods.rb:9:in `block in around'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/language_support/language_methods.rb:97:in `call'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/language_support/language_methods.rb:97:in `execute_around'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/language_support/language_methods.rb:8:in `around'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime/support_code.rb:119:in `block (2 levels) in around'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime/support_code.rb:123:in `call'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime/support_code.rb:123:in `around'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime.rb:94:in `around'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime.rb:81:in `with_hooks'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:13:in `execute'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/scenario.rb:32:in `block in accept'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/scenario.rb:79:in `with_visitor'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/scenario.rb:31:in `accept'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:58:in `block in visit_feature_element'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:57:in `visit_feature_element'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/feature.rb:38:in `block in accept'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/feature.rb:37:in `each'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/feature.rb:37:in `accept'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:27:in `block in visit_feature'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:26:in `visit_feature'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/features.rb:28:in `block in accept'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/features.rb:17:in `each'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/features.rb:17:in `each'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/features.rb:27:in `accept'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:21:in `block in visit_features'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:170:in `broadcast'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/ast/tree_walker.rb:20:in `visit_features'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/runtime.rb:49:in `run!'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/lib/cucumber/cli/main.rb:47:in `execute!'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/gems/cucumber-1.3.17/bin/cucumber:13:in `'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/bin/cucumber:23:in `load'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/bin/cucumber:23:in `'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/bin/ruby_executable_hooks:15:in `eval'
/var/lib/jenkins/.rvm/gems/ruby-2.1.5#tallama-integration/bin/ruby_executable_hooks:15:in `'
This error is not the same as a closed stream error, which signifies that you're attempting to touch a stream after it's been closed.
No, the only time a stream closed (IOError) is thrown in all of MRI is in rb_thread_fd_close. That code in turn is only called in rb_io_close – aka when closing a file descriptor; it iterates through the other live threads and sees if any of them is waiting for the closed file descriptor. If there’s anybody waiting, it raises the stream closed error in the waiting thread. That neatly explains why the stack trace is pointing to cucumber: that’s the thread that’s waiting to write out to STDOUT, from the formatter. But we don’t have any info on who closed the file descriptor.
Since I know I'm dealing with threads here, I start to suspect timeout is killing something when it shouldn't be, but I have no way of knowing what timeout is causing the mess; I removed the one in my code but then found 35 in various gems we're using, and the issue is still occurring.
We also suspected, at some point, that one of the phantomjs processes was being killed due to low memory, and we added instrumentation to puts the current process' memory before every scenario… and the IOErrors went away, almost certainly because of timing issues. When I removed the instrumentation the errors returned.
I've also attempted to update to tip-of-tree cucumber and capybara, but neither seemed to make a difference.
Is there some clever way to track this down that I've not yet tried? Have you seen this issue and know the one-line configuration change I need to make it better? Do you have a convenient patch that removes timeout from all Ruby everywhere?
Ideas to try...
Narrow the issue by trying Capybara + WebKit; Poltergeist is not thread safe (AFAIK).
Narrow the issue by seeing if it's a Rack timeout:
# config/initializers/timeout.rb
Rack::Timeout.timeout = 600
Rack::Timeout.wait_timeout = 600 # or 0 means never timeout
Temporarily turn off any tests that call to externals, such as system, sidekiq, eventmachine, delayed job, etc.
For your idea of a patch that removes timeout from all Ruby everywhere:
require 'timeout'
Thread.handle_interrupt(TimeoutError => :never) {
timeout(10){
# TimeoutError doesn't occur here
Thread.handle_interrupt(TimeoutError => :on_blocking) {
# possible to be killed by TimeoutError
# while blocking operation
}
# TimeoutError doesn't occur here
}
}
See Guarding from Timeout Error
Track the open file objects at any breakpoints you want:
# List all open File objects.
ObjectSpace.each_object(File) do |f|
puts "%s: %d" % [f.path, f.fileno] unless f.closed?
end
# List the "dangling" File object which we didn't store in a variable.
ObjectSpace.each_object(File) do |f|
unless f.closed?
printf "%s: %d\n", f.path, f.fileno unless f === filehandle
end
end
See Where does Ruby keep track of its open file descriptors
try this out:
In your code after opening the file for reading you must close it.
See this similar issue
I have a setup where a cron job periodically downloads an mp3 file from an external service (Twilio) and then uses Paperclip to upload the file to Amazon S3. The process is handled in the background by Resque.
Here is the code that handles the download from Twilio and subsequent attachment to Paperclip:
# Perform transfer from Twilio to S3
def self.perform(group_recording_id = nil)
gr = GroupRecording.find(group_recording_id)
# ...
recording = TwilioClient.account.recordings.get(gr.external_id)
if recording.present?
# ....
gr.audio_file = download_remote_file(gr.twilio_mp3_url)
gr.save
end
end
def download_remote_file(url)
io = open(URI.parse(url))
# overrides Paperclip::Upfile#original_filename
def io.original_filename
base_uri.path.split('/').last
end
io.original_filename.blank? ? nil : io
end
This fails with EOFErrors :
EOFError
end of file reached
/usr/local/lib/ruby/1.9.1/openssl/buffering.rb:145:in `sysread_nonblock'
/usr/local/lib/ruby/1.9.1/openssl/buffering.rb:145:in `read_nonblock'
/usr/local/lib/ruby/1.9.1/net/protocol.rb:135:in `rbuf_fill'
/usr/local/lib/ruby/1.9.1/net/protocol.rb:116:in `readuntil'
/usr/local/lib/ruby/1.9.1/net/protocol.rb:126:in `readline'
/usr/local/lib/ruby/1.9.1/net/http.rb:2219:in `read_status_line'
/usr/local/lib/ruby/1.9.1/net/http.rb:2208:in `read_new'
/usr/local/lib/ruby/1.9.1/net/http.rb:1191:in `transport_request'
/usr/local/lib/ruby/1.9.1/net/http.rb:1177:in `request'
/app/vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
/usr/local/lib/ruby/1.9.1/open-uri.rb:312:in `block in open_http'
/usr/local/lib/ruby/1.9.1/net/http.rb:627:in `start'
/usr/local/lib/ruby/1.9.1/open-uri.rb:306:in `open_http'
/usr/local/lib/ruby/1.9.1/open-uri.rb:769:in `buffer_open'
/usr/local/lib/ruby/1.9.1/open-uri.rb:203:in `block in open_loop'
/usr/local/lib/ruby/1.9.1/open-uri.rb:201:in `catch'
/usr/local/lib/ruby/1.9.1/open-uri.rb:201:in `open_loop'
/usr/local/lib/ruby/1.9.1/open-uri.rb:146:in `open_uri'
/usr/local/lib/ruby/1.9.1/open-uri.rb:671:in `open'
/usr/local/lib/ruby/1.9.1/open-uri.rb:33:in `open'
/app/app/models/group_recording.rb:112:in `download_remote_file'
/app/app/models/group_recording.rb:85:in `perform'
The above is the most common error, but I also encountered this:
Errno::ECONNRESET
Connection reset by peer
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/openssl/buffering.rb:145:in `sysread_nonblock'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/openssl/buffering.rb:145:in `read_nonblock'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/net/protocol.rb:135:in `rbuf_fill'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/net/protocol.rb:86:in `read'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/net/http.rb:2424:in `read_body_0'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/net/http.rb:2379:in `read_body'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:321:in `block (2 levels) in open_http'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/net/http.rb:1194:in `block in transport_request'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/net/http.rb:2342:in `reading_body'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/net/http.rb:1193:in `transport_request'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/net/http.rb:1177:in `request'
/app/vendor/bundle/ruby/1.9.1/gems/rest-client-1.6.7/lib/restclient/net_http_ext.rb:51:in `request'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:312:in `block in open_http'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/net/http.rb:627:in `start'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:306:in `open_http'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:769:in `buffer_open'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:203:in `block in open_loop'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:201:in `catch'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:201:in `open_loop'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:146:in `open_uri'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:671:in `open'
/app/vendor/ruby-1.9.2/lib/ruby/1.9.1/open-uri.rb:33:in `open'
/app/app/models/group_recording.rb:113:in `download_remote_file'
/app/app/models/group_recording.rb:86:in `perform'
The problem is intermittant and just started occurring in heroku's production environment recently without any change in the relevant code. The app is currently on ruby 1.9.3-p429 on heroku. The problem occurs locally as well, but somewhat less frequently. I tried the same and lower ruby versions (as low as 1.9.3-p194).
Has anybody come across a similar issue? Searching for the error online has turned up similar errors, but in vastly different contexts.
These are transient networking errors that you should expect in your code, and retry the operation (usually after a short sleep). The internet is not reliable!
You should log retries, and if you notice a spike or unusual pattern you may want to take it up with your service providers (heroku, twilio, etc) since there may be something on their end they can provide insight into.
There's nothing directly you can do to fix the problem from your code though (if it indeed did not change).
I am running into an error in utils.rb:333 when I process a job with delayed job that works perfectly when run synchronously. Basically what I am doing is making a get request in my code and when I do, I run into the error. This is the line of code that hits the error:
self.fb_friends = (JSON.parse self.fb_token.get('me/friends')).first[1] unless self.fb_friends
self is a ruby class and self.fb_token is created here
OAuth2::AccessToken.new client, fb_token_string
where client is the result of the call to OauthHelper::client and fb_token_string is a token string from facebook
here is my stack trace:
{You have a nil object when you didn't expect it!
You might have expected an instance of Array.
The error occurred while evaluating nil.delete
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/rack-1.2.3/lib/rack/utils.rb:333:in `delete'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/rack-1.2.3/lib/rack/utils.rb:326:in `[]='\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/faraday-0.6.1/lib/faraday/utils.rb:28:in `[]='\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/faraday-0.6.1/lib/faraday/utils.rb:85:in `block in merge_headers'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/faraday-0.6.1/lib/faraday/utils.rb:84:in `each'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/faraday-0.6.1/lib/faraday/utils.rb:84:in `merge_headers'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/faraday-0.6.1/lib/faraday/request.rb:69:in `to_env_hash'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/faraday-0.6.1/lib/faraday/request.rb:87:in `run'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/faraday-0.6.1/lib/faraday/request.rb:28:in `run'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/faraday-0.6.1/lib/faraday/connection.rb:170:in `run_request'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/oauth2-0.4.1/lib/oauth2/client.rb:59:in `request'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/oauth2-0.4.1/lib/oauth2/access_token.rb:34:in `request'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/oauth2-0.4.1/lib/oauth2/access_token.rb:38:in `get'\n
/Users/myuseraccount/Sites/tutorcloud/app/models/facebook_session.rb:69:in `get_fb_likes'\n
/Users/myuseraccount/Sites/tutorcloud/app/models/facebook_session.rb:305:in `save_to_user_info'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/performable_method.rb:20:in `perform'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/backend/base.rb:87:in `invoke_job'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:120:in `block (2 levels) in run'\n
/Users/myuseraccount/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/timeout.rb:57:in `timeout'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:120:in `block in run'\n
/Users/myuseraccount/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/benchmark.rb:309:in `realtime'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:119:in `run'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:177:in `reserve_and_run_one_job'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:104:in `block in work_off'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:103:in `times'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:103:in `work_off'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:78:in `block (2 levels) in start'\n
/Users/myuseraccount/.rvm/rubies/ruby-1.9.2-p0/lib/ruby/1.9.1/benchmark.rb:309:in `realtime'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:77:in `block in start'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:74:in `loop'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/worker.rb:74:in `start'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/command.rb:104:in `run'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/command.rb:83:in `block in run_process'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `call'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons/application.rb:249:in `block in start_proc'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons/daemonize.rb:197:in `call_as_daemon'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons/application.rb:253:in `start_proc'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons/application.rb:293:in `start'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons/controller.rb:70:in `run'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons.rb:195:in `block in run_proc'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `call'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons/cmdline.rb:109:in `catch_exceptions'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/daemons-1.1.4/lib/daemons.rb:194:in `run_proc'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/command.rb:81:in `run_process'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/command.rb:75:in `block in daemonize'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/command.rb:73:in `times'\n
/Users/myuseraccount/.rvm/gems/ruby-1.9.2-p0/gems/delayed_job-2.1.4/lib/delayed/command.rb:73:in `daemonize'\n
script/delayed_job:5:in `<main>'
I encountered the same error
Try to update Oauth2 to 0.5.0
When i update Omniauth to 0.3.0.rc2 and Oauth to 0.5.0 - problem has been resolved