PhantomJS, Capybara, Poltergeist failed to reach server - ruby-on-rails

I'm working with poltergeist for the first time, so I don't really know what I'm doing but I couldn't find any solution on the web. Please tell me if any information is missing.
Error message:
Capybara::Poltergeist::StatusFailError: Request failed to reach
server, check DNS and/or server status
this issue doesn't happen on production but only on development and staging environment.
This line of code is the one that is making the trouble
phantomjs_options: ['--ignore-ssl-errors=yes', '--ssl-protocol=any', '--load-images=no', '--proxy=localhost:9050', '--proxy-type=socks5']
without '--proxy=localhost:9050' everything's working perfectly on every environment but I don't want to delete it in case it's critical for the production.
I've also noticed that on staging/development there's no 9050 port listening but on production there is one
full config code part (capybara_drivers.rb):
Capybara.register_driver :polt do |app|
Capybara::Poltergeist::Driver.new(
app,
js_errors: false, # break on js error
timeout: 180, # maximum time in second for the server to produce a response
debug: false, # more verbose log
window_size: [1280, 800], # not responsive, used to simulate scroll when needed
inspector: false, # use debug breakpoint and chrome inspector,
phantomjs_options: ['--ignore-ssl-errors=yes', '--ssl-protocol=any', '--load-images=no', '--proxy=localhost:9050', '--proxy-type=socks5']
)
end

Sounds like your production environment needs to make outbound connections through a socks5 proxy, and your other environments don't. You'll need to make the configuration dependent on environment
Capybara.register_driver :polt do |app|
phantomjs_options = ['--ignore-ssl-errors=yes', '--ssl-protocol=any', '--load-images=no']
phantomjs_options.push('--proxy=localhost:9050', '--proxy-type=socks5') if Rails.env.production?
Capybara::Poltergeist::Driver.new(
app,
js_errors: false, # break on js error
timeout: 180, # maximum time in second for the server to produce a response
debug: false, # more verbose log
window_size: [1280, 800], # not responsive, used to simulate scroll when needed
inspector: false, # use debug breakpoint and chrome inspector,
phantomjs_options: phantomjs_options
)
end

Related

WebMock stopping selenium from deleting session

System
Ruby v2.6
WebMock gem v3.18.1
Selenium Webdriver gem v4.1.0
RSpec gem v3.11.0
Capybara gem v3.36.0
Summary
Whilst successfully running a suite of tests on GitLab CI[1], WebMock intercepts a request from Capybara's Selenium driver to delete a session[2].
743 examples, 0 failures, 30 pending
/builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/webmock-3.18.1/lib/webmock/http_lib_adapters/net_http.rb:104:in
`request': Real HTTP connections are disabled. Unregistered request:
DELETE
http://selenium__standalone-chrome:4444/wd/hub/session/0146cfc5158d585f445cfcdbda289733
Although the error appears after the tests have run I wonder if it occurs whilst the specs are running? This is due to seeing newlines appearing in the RSpec output:
......................................................................................................................................................................................................................................................................
...............................................................................................................................................................................................................................
.....................................................................................................................................................................................**...**............................................******
In spec/support/capybara.rb I have allowed connections to any URL with selenium, or session in it:
selenium_session_requests = %r{/((__.+__)|(hub/session.*))$}
allowed_connections = [
'0.0.0.0', '127.0.0.1', 'https://chromedriver.storage.googleapis.com',
'localhost', /selenium/, selenium_session_requests
].freeze
RSpec.configure do |config|
config.before(:suite) do
WebMock.enable!
end
config.after(:suite) do
WebMock.disable!
end
Capybara.app_host = "http://#{Capybara.server_host}:#{Capybara.server_port}"
Capybara.javascript_driver = ENV.fetch('CAPYBARA_JAVASCRIPT_DRIVER', :chrome_headless).to_sym
WebMock.disable_net_connect!(allow: allowed_connections, net_http_connect_on_start: allowed_connections)
end
I have previously confirmed in the before(:suite) block that a session URL, such as http://selenium__standalone-chrome:4444/wd/hub/session/0146cfc5158d585f445cfcdbda289733 would be allowed by WebMock.
Per the suggestion in [2] I stubbed the request - though I don't believe this is desired behaviour - but that still caused the same error.
Observed behaviour
WebMock does not allow the Selenium driver to exit the browser.
Desired behaviour
The browser is closed, and no errors are raised.
[1] In .gitlab-ci.yml I have a docker service to enable the use of Selenium:
image: ruby:2.6
services:
- postgres:10.1
- selenium/standalone-chrome:latest
- redis:latest
The URL for that service is exposed to my specs via an environment variable:
- export SELENIUM_REMOTE_URL="http://selenium__standalone-chrome:4444/wd/hub"
[2] Here is the full output from the WebMock error:
/builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/webmock-3.18.1/lib/webmock/http_lib_adapters/net_http.rb:104:in
request': Real HTTP connections are disabled. Unregistered request: DELETE http://selenium__standalone-chrome:4444/wd/hub/session/0146cfc5158d585f445cfcdbda289733 with headers {'Accept'=>'application/json', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Content-Type'=>'application/json; charset=UTF-8', 'User-Agent'=>'selenium/4.1.0 (ruby linux)'} (WebMock::NetConnectNotAllowedError) You can stub this request with the following snippet: stub_request(:delete, "http://selenium__standalone-chrome:4444/wd/hub/session/0146cfc5158d585f445cfcdbda289733"). with( headers: { 'Accept'=>'application/json', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'Content-Type'=>'application/json; charset=UTF-8', 'User-Agent'=>'selenium/4.1.0 (ruby linux)' }). to_return(status: 200, body: "", headers: {}) ============================================================ from /builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/selenium-webdriver-4.1.0/lib/selenium/webdriver/remote/http/default.rb:124:in response_for' from
/builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/selenium-webdriver-4.1.0/lib/selenium/webdriver/remote/http/default.rb:77:in
request' from /builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/selenium-webdriver-4.1.0/lib/selenium/webdriver/remote/http/common.rb:59:in call' from
/builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/selenium-webdriver-4.1.0/lib/selenium/webdriver/remote/bridge.rb:588:in
execute' from /builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/selenium-webdriver-4.1.0/lib/selenium/webdriver/remote/bridge.rb:188:in quit' from
/builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/selenium-webdriver-4.1.0/lib/selenium/webdriver/common/driver.rb:181:in
quit' from /builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/capybara-3.36.0/lib/capybara/selenium/driver.rb:293:in quit' from
/builds/abelsoninfo/aws/pips/pips-console/vendor/ruby/ruby/2.6.0/gems/capybara-3.36.0/lib/capybara/selenium/driver.rb:512:in
`block in setup_exit_handler'

Selenium WebDriver Chrome timeout and invalid URL

I am using Selenium Webdriver Chrome initialized in my Rails app as follows
host=XXX
port=XXX
Capybara.register_driver :selenium_chrome do |app|
options = Selenium::WebDriver::Chrome::Options.new
options.add_argument("--proxy-server=#{host}:#{port}")
options.add_argument("--headless") # Remove this option to run on Chrome browser
Capybara::Selenium::Driver.new( app,
browser: :chrome,
options: options
)
end
However, it is timing out always when I run a spec giving this error
Net::ReadTimeout upon running the command visit url
URI.parse(current_url) returns #<URI::Generic data:,> which looks incorrect and probably the reason why it is timing out. I looked into the Selenium Webdriver gem and added debugging to see how the request is fetched for this command but for some reason it is not stopping at the pry when the command is get_current_url
Why does current_url look incorrect and why would it not stop for the command get_current_url ?
EDIT: the url is obtained from here and returns the following locally
[6] pry(Selenium::WebDriver::Remote::Bridge)> Remote::OSS::Bridge.new(capabilities, bridge.session_id, **opts).url
=> "data:,"
Adding a pry to the URL method doesn't stop, so wondering how it is obtaining the value.
Ruby: ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-darwin20]
Selenium-Webdriver: 3.142.7

"Refused to connect" using ChromeDriver, Capybara & Docker Compose

I'm trying to make the move from PhantomJS to Headless Chrome and have run into a bit of a snag. For local testing, I'm using Docker Compose to get all dependent services up and running. To provision Google Chrome, I'm using an image that bundles both it and ChromeDriver together while serving it on port 4444. I then link it to the my app container as follows in this simplified docker-compose.yml file:
web:
image: web/chrome-headless
command: [js-specs]
stdin_open: true
tty: true
environment:
- RACK_ENV=test
- RAILS_ENV=test
links:
- "chromedriver:chromedriver"
chromedriver:
image: robcherry/docker-chromedriver:latest
ports:
- "4444"
cap_add:
- SYS_ADMIN
environment:
CHROMEDRIVER_WHITELISTED_IPS: ""
Then, I have a spec/spec_helper.rb file that bootstraps the testing environment and associated tooling. I define the :headless_chrome driver and point it to ChromeDriver's local binding; http://chromedriver:4444. I'm pretty sure the following is correct:
Capybara.javascript_driver = :headless_chrome
Capybara.register_driver :chrome do |app|
Capybara::Selenium::Driver.new(app, browser: :chrome)
end
Capybara.register_driver :headless_chrome do |app|
capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
chromeOptions: { args: %w[headless disable-gpu window-size=1440,900] },
)
Capybara::Selenium::Driver.new app,
browser: :chrome,
url: "http://chromedriver:4444/",
desired_capabilities: capabilities
end
We also use VCR, but I've configured it to ignore any connections to the port used by ChromeDriver:
VCR.configure do |c|
c.cassette_library_dir = 'spec/vcr_cassettes'
c.default_cassette_options = { record: :new_episodes }
c.ignore_localhost = true
c.allow_http_connections_when_no_cassette = false
c.configure_rspec_metadata!
c.ignore_hosts 'codeclimate.com'
c.hook_into :webmock, :excon
c.ignore_request do |request|
URI(request.uri).port == 4444
end
end
I start the services with Docker Compose, which triggers the test runner. The command is pretty much this:
$ bundle exec rspec --format progress --profile --tag 'broken' --tag 'js' --tag '~quarantined'
After a bit of waiting, I encounter the first failed test:
1) Beta parents code redemption: redeeming a code on the dashboard when the parent has reached the code redemption limit does not display an error message for cart codes
Failure/Error: fill_in "code", with: "BOOK-CODE"
Capybara::ElementNotFound:
Unable to find field "code"
# ./spec/features/beta_parents_code_redemption_spec.rb:104:in `block (4 levels) in <top (required)>'
All specs have the same error. So, I shell into the container to run the tests myself manually and capture the HTML it's testing against. I save it locally and open it up in my browser to be welcomed by the following Chrome error page. It would seem ChromeDriver isn't evaluating the spec's HTML because it can't reach it, so it attempts to run the tests against this error page.
Given the above information, what am I doing wrong here? I appreciate any and all help as moving away from PhantomJS would solve so many headaches for us.
Thank you so much in advance. Please, let me know if you need extra information.
The issue you're having is that Capybara, by default, starts the AUT bound to 127.0.0.1 and then tells the driver to have the browser request from the same. In your case however 127.0.0.1 isn't where the app is running (From the browsers perspective), since it's on a different container than the browser. To fix that, you need to set Capybara.server_host to whatever the external interface of the "web" container is (which is reachable from the "chromedriver" container). That will cause Capybara to bind the AUT to that interface and tell the driver to have the browser make requests to it.
In your case that probably means you can specify 'web'
Capybara.server_host = 'web'

How to setup error 404 page in Ruby on Rails system testing

Rails 5.1 introduced system testing that uses Capybara with Selenium to test the UI of Rails application.
I'm wondering to how to use this system testing to test the UI of error pages.
For standard controller tests, we can do something like below to assert response to be 404.
test 'should get not_found' do
get errors_not_found_url
assert_response :not_found
end
But for system tests, if I go to a 404 page, exception will be thrown in controller level and tests terminate immediately without rendering the page.
test '404 page should render with the correct title' do
# act.
visit NOT_FOUND_URL
# assert.
assert_equal("#{APP_NAME} - #{TITLE_404}", page.title)
end
Exception is thrown in controller level.
$ rails test test/system/error/error_page_test.rb
Run options: --seed 30076
# Running:
Puma starting in single mode...
* Version 3.9.1 (ruby 2.3.1-p112), codename: Private Caller
* Min threads: 0, max threads: 1
* Environment: test
* Listening on tcp://0.0.0.0:55237
Use Ctrl-C to stop
2017-07-09 11:10:45 +1200: Rack app error handling request { GET /books/12345678 }
#<ActionController::RoutingError: Could not find book '12345678' by id or name>
/myapp/app/controllers/books_controller.rb:7:in `index'
/Users/yze14/.rvm/gems/ruby-2.3.1/gems/actionpack-5.1.2/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
/Users/yze14/.rvm/gems/ruby-2.3.1/gems/actionpack-5.1.2/lib/abstract_controller/base.rb:186:in `process_action'
...
Under development/test environment, config.consider_all_requests_local can be set to false in order to show error page instead of stracktrace. But this doesn't swallow exception during system tests.
If you don't want Capybara to re-raise server exceptions in the tests you can set Capybara.raise_server_errors = false.
Secondly, you should check your Gemfile and make sure any gems like web-console,better-errrors, etc are only loaded in the development environment (not in the test environment)
Finally, you shouldn't be using assert_equal with title, you should be using the Capybara provided assert_title which includes waiting/retrying behavior and will reduce potential flakiness in tests.
assert_title("#{APP_NAME} - #{TITLE_404}")

websocket-rails / puma: "async response must have empty headers and body"

I'm using websocket_rails to provide an API for a JS client. Locally it works great, but the exact same setup in production will (seemingly randomly) decide to stop working.
My production.log yields RuntimeError (eventmachine not initialized: evma_install_oneshot_timer)
At first I thought this was the root issue, but my Puma error log yields this when restart the server and try again: RuntimeError: async response must have empty headers and body
I added some logging in the puma gem, and indeed, it's receiving rails session headers when doing GET /websocket
Sometimes there is no issue at all, and everything works fine for a few days, and then, not. And no matter what I do it just refuses to work again.
Thanks in advance. I've wasted days on this problem!
Puma config:
# Change to match your CPU core count
workers 1
# Min and Max threads per worker
threads 1, 6
app_dir = File.expand_path("../..", __FILE__)
shared_dir = "#{app_dir}/shared"
# Default to production
rails_env = ENV['RAILS_ENV'] || "production"
environment rails_env
# Set up socket location
bind "unix://#{shared_dir}/sockets/puma.sock"
# Logging
stdout_redirect "#{shared_dir}/log/puma.stdout.log", "#{shared_dir}/log/puma.stderr.log", true
# Set master PID and state locations
pidfile "#{shared_dir}/pids/puma.pid"
state_path "#{shared_dir}/pids/puma.state"
activate_control_app
on_worker_boot do
require "active_record"
ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
ActiveRecord::Base.establish_connection(YAML.load_file("#{app_dir}/config/database.yml")[rails_env])
end

Resources