I'm practicing webscraping using Watir, Mechanize and Nokigiri gems.
I'm running into an issue with my Watir script. My plan is to get a list of prices from flights via http://tripadvisor.com/. When I run the script, the Chrome browser opens as it should, the script proceeds to fill out the first parts of the form, origin and destination and then it halts. Here is the error message I'm getting:
This code has slept for the duration of the default timeout waiting for an Element to be present. If the test is still passing, consider using Element#exists? instead of rescuing UnknownObjectException
/home/jaffejoe/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/watir-6.2.0/lib/watir/elements/element.rb:515:in `rescue in wait_for_present': element located, but timed out after 30 seconds, waiting for true condition on #<Watir::Input: located: true; {:name=>"rt_leaveday", :tag_name=>"input"}> (Watir::Exception::UnknownObjectException)
from /home/jaffejoe/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/watir-6.2.0/lib/watir/elements/element.rb:505:in `wait_for_present'
from /home/jaffejoe/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/watir-6.2.0/lib/watir/elements/element.rb:522:in `wait_for_enabled'
from /home/jaffejoe/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/watir-6.2.0/lib/watir/elements/element.rb:534:in `wait_for_writable'
from /home/jaffejoe/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/watir-6.2.0/lib/watir/elements/element.rb:639:in `element_call'
from /home/jaffejoe/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/watir-6.2.0/lib/watir/elements/element.rb:303:in `send_keys'
from watir_test.rb:8:in `<main>' </
Here is my code:
require 'watir'
browser = Watir::Browser.new
browser.goto('https://tripadvisor.com/CheapFlightsHome')
browser.input(name: 'orig').send_keys('Boston, MA - Logan International Airport (BOS)')
browser.input(name: 'dest').send_keys('Milan, Italy - All Airports (MIL)')
browser.input(name: 'rt_leaveday').send_keys('1')
browser.input(name: 'rt_leavemonth').send_keys('06/2017')
browser.input(name: 'retday').send_keys('30')
browser.input(name: 'leavemonth').send_keys('06/2017')
browser.input(value: 'Search Flights').click
puts browser.url
browser.quit
It can't set value on the rt_leaveday or rt_leavemonth because they are hidden inputs. But you can execute a script to click on dateselector:
require 'watir'
browser = Watir::Browser.new
browser.goto('https://tripadvisor.com/CheapFlightsHome')
browser.text_field(name: 'orig').set('Boston, MA - Logan International Airport (BOS)')
browser.text_field(name: 'dest').set('Milan, Italy - All Airports (MIL)')
browser.execute_script('document.querySelector(".in_date").click()')
browser.execute_script('document.querySelector(".day_28").click()')
browser.execute_script('document.querySelector(".out_date").click()')
browser.execute_script('document.querySelector(".day_2").click()')
browser.span(id: "CHECK_FARES_BUTTON").fire_event :click
puts browser.url
browser.quit
=> https://www.tripadvisor.com/CheapFlightsSearchResults-g187849-a_airport0.BOS-a_airport1.MIL-a_cos.0-a_date0.20170328-a_date1.20170402-a_nearby0.no-a_nearby1.no-a_nonstop.no-a_pax0.a-a_travelers.1-Milan_Lombardy.html
Related
I have a static Watir script to fill a form which works fine locally. I'm planning to add a web UI in order to make it dynamic and user-accesible. User will fill in a form, and those values will be passed into the script to run.
Which is the best approach to do this? Using RoR into Heroku with a form in a view and calling the script in /lib/Rubyscript.rb?
Rubyscript.rb
require 'watir'
require 'pry'
class Script
def start
browser = Watir::Browser.new
browser.goto("url")
browser.input(id: "txtUsuarioLogin").send_keys(USER)
browser.input(id: "txtPasswordLogin").send_keys(PASSWORD)
browser.button(id: 'btnLoginUsuario').click
browser.goto("url")
browser.select_list(id: '1').click
browser.select_list(id: '2').select "4"
browser.select_list(id: '3').click
browser.select_list(id: '3').option(index: 1).click
browser.button(id: '4').click
sleep(5)
browser.quit
end
end
Script.new.start
I have a select2 v4 that loads options through AJAX.
I am running a Cucumber test where I need to select 2 options of the list, but I can't seem to make the list open up and load (which normally gets populated when I type 2 or characters).
I have tried:
As suggested here:
#session.execute_script("$('#publish_to').select2('open')")
and
#session.first(".input.publish_to .select2-container").click
and
#session.first("#publish_to").find(".select2-choice").click
which do not give me an error, but I am not getting the options to select, so I am assuming that the click is not really working. Things I have tried to select the options:
# This one cannot find the css:
#session.find(".select2-results__options", text: client.email).click
# This one gives me a Timeout error
#session.evaluate_script "$('#publish_to').val(#{client.id}).trigger('change')"
# This one gives me a Timeout error
#session.evaluate_script "$('.select2-search__field').trigger('keydown').val('#{client.email}').trigger('keyup')";
sleep 10
#session.find('.select2-search__option', text: client.email).click
Anything with trigger gives me a Timeout error, so I tried waiting for jQuery.active but I never got a true even waiting for 2 minutes:
counter = 0
timeout_in_sec = 120
while counter < timeout_in_sec && #session.evaluate_script('jQuery.active').zero?
sleep 1.second
counter+=1
end
I tried using the gem capybara-select2 running:
#session.select2 client.email, css: '#publish_to', search: true
but I get the error undefined methodselect2' for #and I haveWorld(CapybaraSelect2)in myenv.rb`
I am using Cucumber v3.1.2 with ruby gem 'cucumber-rails'
The poltergeist driver is roughly equivalent to a 7 year old version of Safari which means it doesn't support a lot of current JS/CSS. This means your issue could simply be that select2 is no longer compatible with Poltergeist (without a lot of polyfilling). You're going to be much better off updating to using a real browser (stable - chrome via selenium, etc) or one of the direct to Chrome drivers (highly beta) that have spun off Poltergeist (Apparition is one of them). Those will allow you to run with a visible browser (useful for debugging) or headless.
The following code uses Chrome via selenium and interacts with the select2 demo site to select an entry that is loaded via Ajax.
require "selenium/webdriver"
require "capybara/dsl"
sess = Capybara::Session.new(:selenium_chrome)
sess.visit("https://select2.org/data-sources/ajax")
sess.first('.select2-container', minimum: 1).click
sess.find('.select2-dropdown input.select2-search__field').send_keys("capy")
sleep 5 # just to watch the browser search
sess.find('.select2-results__option', text: 'teamcapybara/capybara').click
sess.assert_selector(:css, '.select2-selection__rendered', text: 'teamcapybara/capybara')
sleep 5 # just to see the effect
I'm noticing a strange issue that I haven't been able to solve for a few days.
I have a Rails 5 API server with system tests using RSpec and Capybara + Selenium-webdriver driving headless Chrome.
I'm using Capybara.app_host = 'http://localhost:4200' to make the tests hit a separate development server which is running an Ember front-end. The Ember front-end looks at the user agent to know to then send requests to the Rails API test database.
All the tests run fine except for ones which use RSpec file fixtures.
Here's one spec that is failing:
describe 'the affiliate program', :vcr, type: :system do
fixtures :all
before do
Capybara.session_name = :affiliate
visit('/')
signup_and_verify_email(signup_intent: :seller)
visit_affiliate_settings
end
it 'can use the affiliate page' do
affiliate_token = page.text[/Your affiliate token is \b(.+?)\b/i, 1]
expect(affiliate_token).to be_present
# When a referral signs up.
Capybara.session_name = :referral
visit("?client=#{affiliate_token}")
signup_and_verify_email(signup_intent: :member)
refresh
# It can track the referral.
Capybara.session_name = :affiliate
refresh
expect(page).to have_selector('.referral-row', count: 1)
# When a referral makes a purchase.
Capybara.session_name = :referral
find('[href="/videos"]').click
find('.price-area .coin-usd-amount', match: :first).click
find('.cart-dropdown-body .checkout-button').click
find('.checkout-button').click
wait_for { find('.countdown-timer') }
order = Order.last
order.force_complete_payment!
Rake::Task['affiliate_referral:update_amounts_earned'].invoke
# It can track the earnings.
Capybara.session_name = :affiliate
refresh
amount = (order.price * AffiliateReferral::COMMISSION_PERCENTAGE).floor.to_f
amount_in_dom = find('.referral-amount-earned', match: :first).text.gsub(/[^\d\.]/, '').to_f * 100
expect(amount).to equal(amount_in_dom)
end
end
This will fail maybe 99% of the time. There is the odd case where it passes. I can get my test suite to eventually pass by running it on a loop for a day.
I ended up upgrading all versions to the latest (Node 10, latest Ember, latest Rails) but the issue persists.
I can post a sample repo that reproduces the issue later. I just wanted to get this posted in case anyone has encountered the issue.
Here's a typical stack trace when the timeout happens:
1.1) Failure/Error: page.evaluate_script('window.location.reload()')
Net::ReadTimeout:
Net::ReadTimeout
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/webmock-3.3.0/lib/webmock/http_lib_adapters/net_http.rb:97:in `block in request'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/webmock-3.3.0/lib/webmock/http_lib_adapters/net_http.rb:110:in `block in request'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/webmock-3.3.0/lib/webmock/http_lib_adapters/net_http.rb:109:in `request'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/http/default.rb:121:in `response_for'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/http/default.rb:76:in `request'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/http/common.rb:62:in `call'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/bridge.rb:164:in `execute'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/oss/bridge.rb:584:in `execute'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/remote/oss/bridge.rb:267:in `execute_script'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/selenium-webdriver-3.14.0/lib/selenium/webdriver/common/driver.rb:211:in `execute_script'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/selenium/driver.rb:84:in `execute_script'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/selenium/driver.rb:88:in `evaluate_script'
# /home/mhluska/.rvm/gems/ruby-2.5.1/gems/capybara-3.8.2/lib/capybara/session.rb:575:in `evaluate_script'
# ./spec/support/selenium.rb:48:in `refresh'
# ./spec/support/pages.rb:70:in `signup_and_verify_email'
# ./spec/system/payment_spec.rb:43:in `block (3 levels) in <top (required)>'
I should point out it doesn't always happen with page.evaluate_script('window.location.reload()'). It can happen with something benign like visit('/').
Edit: I tried disabling Ember FastBoot (server-side rendering) using the DISABLE_FASTBOOT env variable and suddenly all tests pass. I'm thinking that somehow the RSpec fixtures are causing Ember FastBoot to not finish rendering in some cases. This certainly lines up with dropped connections I've occasionally seen in production logs.
I've been experimenting with the client code and it may be due to my use of FastBoot's deferRendering call.
Edit: I'm using the following versions:
ember-cli: 3.1.3
ember-data: 3.0.2
rails: 5.2.1
rspec: 3.8.0
capybara: 3.8.2
selenium-webdriver: 3.14.0
google chrome: 69.0.3497.100 (Official Build) (64-bit)
Edit: I'm using this somewhat flaky Node/Express library fastboot-app-server to do server-side rendering. I've discovered that it sometimes strips important response headers (Content-Type and Content-Encoding). I'm wondering if this is contributing to the issue.
Edit: I added a strict Content Security Policy to make sure there are no external requests running during the test suite that could be causing the Net::ReadTimeout.
I inspect the Chrome network tab at the point when it locks up and it seems to be loading nothing. Manually refreshing the browser allows the tests to pick up and continue running. How strange.
I've spent a couple weeks on this now and it may be time to give up on Selenium tests.
I upgraded to Chrome 70 and chromedriver 2.43. It didn't seem to make a difference.
I tried using the rspec-retry gem to force a refresh when the timeout occurs but the gem seems to fail to catch the timeout exception.
I've inspected the raw request to chromedriver where things hang. It looks like it's always POST http://127.0.0.1/session/<session id>/refresh. I tried refreshing in an alternate way: visit(page.current_path) which seems to fix things!
I finally got my test suite to pass by switching page.driver.browser.navigate.refresh to visit(page.current_path).
I know it's an ugly hack but it's the only thing I could find to get things working (see my various attempts in the question edits).
I looked at the request to chromedriver that was causing the timeouts each time: POST http://127.0.0.1/session/<session id>/refresh. I can only guess that it's some kind of issue with chromedriver. Perhaps incidentally, it only hangs when multiple chromedriver instances are active (which happens when multiple Capybara sessions are being used).
Edit: I needed to account for query params as well:
def refresh
query = URI.parse(page.current_url).query
path = page.current_path
path += "?#{query}" if query.present?
visit(path)
end
I tried just doing visit(page.current_url) but that was giving timeouts as well.
2 weeks ago I put up post about my watir script timing out on me, I was able to get a solution but I realized to late the results I was getting was different than the person that helped me out. Here is the original post: Ruby Watir Gem, Timing Out on Form Input
require 'watir'
browser = Watir::Browser.new
browser.goto('https://tripadvisor.com/CheapFlightsHome')
browser.text_field(name: 'orig').set('Boston, MA - Logan International Airport (BOS)')
browser.text_field(name: 'dest').set('Milan, Italy - All Airports (MIL)')
browser.execute_script('document.querySelector(".in_date").click()')
browser.execute_script('document.querySelector(".day_28").click()')
browser.execute_script('document.querySelector(".out_date").click()')
browser.execute_script('document.querySelector(".day_2").click()')
browser.span(id: "CHECK_FARES_BUTTON").fire_event :click
puts browser.url
browser.quit
The person who wrote that code got this as a result:
https://www.tripadvisor.com/CheapFlightsSearchResults-g187849-a_airport0.BOS-a_airport1.MIL-a_cos.0-a_date0.20170328-a_date1.20170402-a_nearby0.no-a_nearby1.no-a_nonstop.no-a_pax0.a-a_travelers.1-Milan_Lombardy.html
I have the same code in my script and for some reason I'm only getting:
https://www.tripadvisor.com/CheapFlightsHome
It seems as though the button click isn't happening for me, not sure. I tried both chrome and firefox.
First of all your click actually opens another two window and also the time you are clicking it's not exactly receiving the click, Please use this code, it will work for you and you will be left with your expected window
require 'watir'
caps = Selenium::WebDriver::Remote::Capabilities.firefox(marionette: false)
driver=Selenium::WebDriver.for :firefox, desired_capabilities: caps, profile: "default"
b=Watir::Browser.new driver
b.goto('https://tripadvisor.com/CheapFlightsHome')
b.text_field(name: 'orig').set('Boston, MA - Logan International Airport (BOS)')
b.text_field(name: 'dest').set('Milan, Italy - All Airports (MIL)')
b.execute_script('document.querySelector(".in_date").click()')
b.execute_script('document.querySelector(".day_28").click()')
b.execute_script('document.querySelector(".out_date").click()')
b.execute_script('document.querySelector(".day_2").click()')
begin
b.element(xpath: ".//*[#id='CHECK_FARES_BUTTON']").click
end until b.windows.count>1
b.windows[0].close
b.windows[1].close
puts b.url
b.quit
Firstly, I changed .fire_event :click to .click.
Then there was an error that appeared.
Please enter a valid airport code or city.
Secondly, I tried this:
browser.span(id: "CHECK_FARES_BUTTON").click
browser.span(id: "CHECK_FARES_BUTTON").click
puts browser.url
And it redirected me to:
https://www.tripadvisor.com/CheapFlightsSearchResults-g187849-a_airport0.BOS-a_airport1.MIL-a_cos.0-a_date0.20170401-a_date1.20170402-a_nearby0.no-a_nearby1.no-a_nonstop.no-a_pax0.a-a_travelers.1-Milan_Lombardy.html
To be honest I have no idea why it does not register the input during the first click...
I am using chrome.
Firstly it is showing error I have use #Watir.default_timeout = 900 also I try change the time in config file inside webrick still it doesn't work
The errors are
Net::ReadTimeout
Secondly for the next error I even tried changing port number still it doesn't work
Errno::ECONNREFUSED (Connection refused - connect(2) for "127.0.0.1" port 70 55):
I want to list all the seller name and price but it only list 2 sellers name and price I want them all
require 'selenium-webdriver'
require 'phantomjs'
require 'watir'
browser = Watir::Browser.new: chrome
browser.window.maximize
browser.goto "url"
browser.div(: class => 'sellCont').uls.each do |list |
puts list.lis.first.text# For dealer name
puts list.li(: class => 'price')# For price
end
browser.close
Are you getting all required data when you load given url first time without click on any link or button on browser?
As you said your example one-to-one as your code. It is really strange because it does not look like a valid ruby code. This code works in my console (irb)
require 'selenium-webdriver'
require 'watir-webdriver'
browser = Watir::Browser.new :chrome
browser.window.maximize
browser.goto "https://paytm.com/shop/p/gionee-e7-mini-black-MOBGIONEE-E7-MIHAPP44414CBBDB36C?psearch=organic%7Cundefined%7Cgionee%20e7%7Cgrid"
browser.div(:class => 'sellCont').uls.each do |list |
puts list.li.first.text# For dealer name
puts list.li(:class => 'price')# For price
end
browser.close
Pay attention that you don't use phantomjs (maybe you need it in general but not in the given example) so it was removed. And I require watir-webdriver not watir (I just take it from one of my project)