Trying to get the "what" form (aka job title), from indeed.com
Error when trying to run the program:
/var/lib/gems/2.3.0/gems/capybara-2.11.0/lib/capybara/node/finders.rb:44:in `block in find': Unable to find field "q" (Capybara::ElementNotFound)
Inspecting element via firefox from indeed.com yields: name="q"
<span class="inwrap">
<input class="input_text" maxlength="512" size="31" aria-labelledby="what_label_top hidden_colon what_label_bot" name="q" autocomplete="off" id="what">
</span>
<div style="width:250px"><!-- --></div>
Which matches the code in the scraper:
def perform_search
# For indeed
fill_in 'q', :with => #skillset
fill_in 'l', :with => #region
find('#fj').click
sleep(1)
end
The entire code can be found at:
https://github.com/jasnow/job-hunter/blob/master/scraper.rb
Now the problem here is the inability to locate name="q" Are there any other ways I could link to that form on indeed.com so I could initiate webscraping? I'm talking xpath or css perhaps.
Your code only allows the URL http://www.indeed.com , but that URL redirects to https://www.indeed.com and also hits http://indeed.com. Therefore your page load is being blocked. Change to config.allow_url("indeed.com") and it should be able to find the input.
Related
I am having trouble to attach files to inputs which use direct upload inside my system tests (Capybara). All tests worked before I switched to direct upload. I have also tried to manually submit appropriate forms via Browser and everything works there. Unfortunately, no luck with Capybara :/.
Inside view, I have following input:
<%= f.input :desktop_files, as: :file, input_html: { direct_upload: true, multiple: true } %>
and file is attached to input in system test by:
attach_file 'uploads_create_assets_former[desktop_files][]', "#{fixture_path}/files/image.jpg"
When I try to run test which uses similar piece of code, I get:
Selenium::WebDriver::Error::UnexpectedAlertOpenError: unexpected alert open: {Alert text : Error reading image.jpg}
(Session info: headless chrome=94.0.4606.81)
and when I check console inside browser opened by Capabyra, I can see following error:
FileReader error
My suspicion is that Capabyra/Selenium has problem to access attached file, but I don't know about any other way how to assign file to input. Maybe there is some Capybara magic which comes to play here :) -- hopefully, I am not only one who uses Rails direct upload and needs to test this piece of code with system tests...
I am using:
ruby (3.0.0)
rails (6.1.4.1)
selenium-webdriver (4.0.3)
capybara (3.35.3)
webdrivers (4.7.0)
and for capybara:
Capybara.register_driver :headless_chrome do |app|
options = Selenium::WebDriver::Chrome::Options.new(
args: %w[headless disable-gpu no-sandbox window-size=1440x768]
)
options.add_preference(:download, prompt_for_download: false,
default_directory: Rails.root.join('tmp/downloads').to_s)
options.add_preference(:browser, set_download_behavior: { behavior: 'allow' })
Capybara::Selenium::Driver.new(app, browser: :chrome, capabilities: options)
end
Edit:
Html code of form which should do upload looks like this:
<form class="formtastic uploads_create_assets_former" id="new_uploads_create_assets_former" enctype="multipart/form-data" action="/admin/upload/create" accept-charset="UTF-8" method="post">
<fieldset class="inputs">
<ol>
<li class="file input optional" id="uploads_create_assets_former_desktop_files_input"><label for="uploads_create_assets_former_desktop_files" class="label">Dateien (Computer)</label>
<input id="uploads_create_assets_former_desktop_files" multiple="multiple" data-direct-upload-url="http://127.0.0.1:49538/rails/active_storage/direct_uploads" type="file" name="uploads_create_assets_former[desktop_files][]" />
</li>
</ol>
</fieldset>
<fieldset class="actions">
<ol>
<li class="action input_action " id="uploads_create_assets_former_submit_action">
<input type="submit" name="commit" value="Nächster Schritt" data-disable-with="Nächster Schritt" />
</li>
</ol>
</fieldset>
</form>
I have not deviated in any way from Active Storage direct upload documented at https://edgeguides.rubyonrails.org/active_storage_overview.html#direct-uploads. Upload of files starts on former submission.
Another edit:
I have prepared minimalistic Rails app where you can try to play with my issue: https://github.com/martintomas/capybara-direct-upload. I have double checked that path is correct (otherwise Capybara::FileNotFound is raised), tried relative and absolute paths. I have also checked that anybody can read file:
-rw-r--r-- 1 martintomas staff 26436 Oct 22 12:51 image.jpg
Same problem happens when tests are run on my local machine or inside CI environment. To be honest, I have run out of ideas so I have decided to go for hacky solution now.
Hacky solution:
If you absolute trust active storage direct upload implementation and you don't have extra js code related to direct upload, you can turn it off inside system tests.
def attach_file(locator = nil, paths, make_visible: nil, **options)
turn_off_direct_upload # Capybara does not work with direct upload
super
end
def turn_off_direct_upload
page.execute_script 'document.querySelectorAll("input[data-direct-upload-url]:not([data-direct-upload-url=\"\"])").forEach((input) => { delete input.dataset.directUploadUrl } )'
end
I've read the similar questions to this but they don't solve my issue. I'm new with js testing so I think I might be doing something wrong.
The form produces this HTML
<form class="new_category_item_key" id="new_category_item_key" action="/guides/dungeon-boss/categories?category_id=heroes" accept-charset="UTF-8" data-remote="true" method="post"><input name="utf8" type="hidden" value="✓" /><input type="hidden" name="authenticity_token" value="b7wiF07zYh/Nl727M3y0Uk1TbroMJFuGqTK6fYNlNted/5G4Wmz4BZLb7IazzyP5md/wWRb1D28ePhrzt2uMSA==" />
<label for="category_item_key_name">Key name</label>
<input type="text" name="category_item_key[name]" id="category_item_key_name" />
<select name="category_item_key[key_type]" id="category_item_key_key_type"><option value="1">Value</option>
<option value="2">Text</option>
<option value="3">Image</option></select>
<input type="submit" name="commit" value="Add New Key" />
</form>
and I have the following integration test on the form
setup do
#user = users(:michael)
#user1 = users(:archer)
#guide = Guide.find(1)
#mod_relationship = game_mods_relationships(:mod1)
#category = Category.find(1)
Capybara.current_driver = Capybara.javascript_driver # :selenium by default
end
test "adding keys mod success then fail" do
log_in_as(#user)
get edit_guide_category_path(#guide, #category)
assert_template 'categories/edit'
assert_difference 'CategoryItemKey.count', 1 do
fill_in 'Key name', with: "diablo"
click_button "commit"
end
end
when I run the test I get the following error
Capybara::ElementNotFound: Capybara::ElementNotFound: Unable to find field "Key name"
Looking at the HTML I can see the field is there. If I try using the inputs id it still fails, if I remove the fill in line then it says it cant find the button to click which is also there. I assume its getting the right page because get edit_guide_category_path(#guide, #category) works for the other tests (but they are non js tests and don't use selenium).
Its probably something simple but I cant get it.
You're mixing up two different libraries -- You can't use get with Capybara, you use visit(url) to go to the page. You also shouldn't normally be asserting templates in a feature test, thats for lower level tests.
For a rails app I'm currently working on I recently changed the design somewhat so that the signout link no longer has the anchor text "Sign out" but instead a glyphicon from twitter bootstrap. the html for link now looks like this:
<a class="btn btn-primary btn-sm" data-method="delete" href="/users/sign_out" rel="nofollow">
<span class="glyphicon glyphicon-log-out"></span>
</a>
meanwhile my capybara test looks like this:
context "when not logged in" do
it 'cannot create wikis' do
#free_user = create(:user)
login_as(#free_user, :scope => :user)
click_link "Sign out"
visit root_path
expect(page).to_not have_link('Create wiki')
end
end
Now that the text "Sign out" is no longer there, I need a new way to identify the link. Checking the documentation for capybara (or rather this handy cheatsheet), it looks like I can supply either the text of the link or its id. So I tried giving it an id:
<a class="btn btn-primary btn-sm" data-method="delete" href="/users/sign_out" id="signout" rel="nofollow">
<span class="glyphicon glyphicon-log-out"></span>
</a>
So now it's got an id of "signout" however when I make this change to the test, it still won't pass.
1) Standard (free) User when not logged in cannot create wikis
Failure/Error: click_link "signout"
Capybara::ElementNotFound:
Unable to find link "signout"
# ./spec/features/standard_user_role_spec.rb:107:in `block (3 levels) in <top (required)>'
I tried making sure that I was still a logged in user in the test by creating and logging in the user as seen above and by adding a check that the html on the page contains Hello, since it says "Hello" and the name of the user when the user is logged in:
expect(page).to have_content('Hello')
This gave me another error that I don't understand:
1) Standard (free) User when not logged in cannot create wikis
Failure/Error: expect(page).to have_content('Hello')
Capybara::ElementNotFound:
Unable to find xpath "/html"
# ./spec/features/standard_user_role_spec.rb:107:in `block (3 levels) in <top (required)>'
So what could be going on here?
Full spec available here
You can re-add the text but hide it from all but screen readers:
<a class="btn btn-primary btn-sm" data-method="delete" href="/users/sign_out" id="signout" rel="nofollow">
<span class="sr-only">Sign out</span>
<span class="glyphicon glyphicon-log-out"></span>
</a>
This also improves accessibility somewhat.
context "when not logged in" do
it 'cannot create wikis' do
#free_user = create(:user)
login_as(#free_user, :scope => :user)
visit root_path
click_link "Sign out", visible: false
expect(page).to_not have_link('Create wiki')
end
end
Note that we explicitly tell Capybara to look for hidden text with the visible option.
For the applications, which use JS, that means the request is being sent using for example json format, and is begin triggered by click on the link. (suppose the HTML markup is the same). It is useful the following call:
find('Sign Out', visible: false).trigger('click')
or by id (if you'll add it):
find(:css, "#sign-out", visible: false).trigger('click')
I feel like I'm going insane here. Doing a routine capybara test of my Post model. Specifically, making sure the new page redirects to the newly created post after a valid submission. So in my spec/requests folder, I've got this:
describe "Valid post submission --" do
it "should log in a user and redirect to new post" do
# Sign in works find, so I won't reproduce it
# Make a blogpost
Post.destroy_all
fill_in :name_here, with: "Post Name Here"
fill_in :content_here, with: "This is the content of a really short post."
screenshot_and_open_image
click_on "Post It"
screenshot_and_open_image
puts current_path
page.should have_selector 'h2', text: "prohibited this post"
page.should have_selector 'h1', text: "Post Name Here"
page.should have_selector '.alert', text: "Post was successfully created."
page.should have_selector 'title', text: full_title('Post Name Here')
end
The screenshots and puts are there to clarify what's going on. There are basically two different cases. In
I just leave the fields be and test for the default IDs ("post_name"
and "post_content"); obviously the above test is for case two:
I change the field ids to "name_here" and "content_here" to test if
the label names or anything else is interfering. That's the one the spec of which is
above.
In case 2 the post is getting rejected because Capybara can't find either field.
In case 1, only the content would be empty, but the name field would be filled with the content.
The precise error in case 1 is:
2) Posts Invalid post - should refresh with error if content empty
Screenshot: /Users/myusername/rails/appname/tmp/capybara/screenshot_2013-03-14-22-37-36.634.png
Failure/Error: fill_in :post_name, with: "Post Name Here"
Capybara::ElementNotFound:
cannot fill in, no text field, text area or password field with id, name, or label 'post_name' found
# (eval):2:in `fill_in'
# ./spec/requests/posts_spec.rb:43:in `block (3 levels) in <top (required)>'
Case 2 doesn't throw an error about not finding the appropriate content, I believe because the label has that id as well.
This is ridiculous, because I took a snapshot of the HTML durin the test, just before it errors. Here's case 2 -- both cases look 90% the same here, apart from that ID difference:
<form accept-charset="UTF-8" action="/posts" class="new_post" id="new_post" method="post">
<div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="✓"></div>
<div class="field">
<label for="post_name">Name</label>
<br><input id="name_here" name="post[name]" size="30" type="text">
</div>
<div class="field text-area">
<label for="post_content">Content</label>
<br><textarea cols="50" id="content_here" name="post[content]" rows="20"></textarea>
</div>
<div class="actions btn-group">
<input class="btn" name="commit" type="submit" value="Post It">
</div>
</form>
Note the clear existence of the post_content field. In cases where both fail (Case 1), there are both content_here and name_here fields. So the fields are there. And capybara is generally working (this sort of thing works find in other parts of my app). And the problem isn't a conflict with the label name, because putting a different id on the input fields doesn't help capybara find them.
By the way, this all works perfectly in reality.
Any idea at all what's going on here? I'm super frustrated/clueless.
UPDATE -- Following user2172816's advice, I changed my HAML so the HTML looks like this:
<form accept-charset="UTF-8" action="/posts" class="new_post" id="new_post" method="post">
<div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="✓"></div>
<div class="field">
<label for="post_name">Name</label>
<br><input id="name" name="post[name]" size="30" type="text">
</div>
<div class="field text-area">
<label for="post_content">Content</label>
<br><textarea cols="50" id="content" name="post[content]" rows="20"></textarea>
</div>
<div class="actions btn-group">
<input class="btn" name="commit" type="submit" value="Post It">
</div>
</form>
The test now looks like this:
# Make a blogpost
Post.destroy_all
fill_in "Name", with: "Post Name Here"
fill_in "Content", with: "This is the content of a really short post."
page.should have_selector 'h2', text: "prohibited this post"
page.should have_selector 'h1', text: "Post Name Here"
page.should have_selector '.alert', text: "Post was successfully created."
page.should have_selector 'title', text: full_title('Post Name Here')
But it's still erroring (in new parts of the same spec, too!):
1) Posts Invalid post - should refresh with error if content empty
Screenshot: /Users/username/rails/appname/tmp/capybara/screenshot_2013-03-14-23-54-38.072.png
Failure/Error: fill_in :name, with: "Post Name Here"
Capybara::ElementNotFound:
cannot fill in, no text field, text area or password field with id, name, or label 'name' found
# (eval):2:in `fill_in'
# ./spec/requests/posts_spec.rb:43:in `block (3 levels) in <top (required)>'
2) Posts Invalid post - should refresh with error if name empty
Screenshot: /Users/username/rails/appname/tmp/capybara/screenshot_2013-03-14-23-54-38.274.png
Failure/Error: fill_in :name, with: ""
Capybara::ElementNotFound:
cannot fill in, no text field, text area or password field with id, name, or label 'name' found
# (eval):2:in `fill_in'
# ./spec/requests/posts_spec.rb:33:in `block (3 levels) in <top (required)>'
3) Posts Valid post submission -- should log in a user and redirect to new post
Screenshot: /Users/username/rails/appname/tmp/capybara/screenshot_2013-03-14-23-54-38.459.png
Failure/Error: fill_in :name, with: "Post Name Here"
Capybara::ElementNotFound:
cannot fill in, no text field, text area or password field with id, name, or label 'name' found
# (eval):2:in `fill_in'
# ./spec/requests/posts_spec.rb:67:in `block (3 levels) in <top (required)>'
The attribute "for" of label should be the "id" of the input element, i.e.
<label for="name_here">Name</label>
<br><input id="name_here" name="post[name]" size="30" type="text">
And in the spec you should call:
fill_in 'Name', with: "Post Name Here"
Using capybara, I am trying to fill in a date. There is a javascript date picker on the form. I have tried the following ways to choose it:
find('input[type=date]').focus()
find(:xpath, ".//*[#id='Sdate']")
fill_in('Sdate', :with => '25/07/2555')
However for all of these attempts, I got the exact same error message:
Selenium::WebDriver::Error::UnknownError: '[JavaScript Error: "a is null" {file: "file:///var/folders/ky/r1ft86wd5x9cjq0f3f6tq0600000gn/T/webdriver-profile20120725-41019-m72bh5/extensions/fxdriver#googlecode.com/components/command_processor.js" line: 5813}]' when calling method: [nsICommandProcessor::execute]
The html I am trying to fill in:
<input id="Sdate" type="text" readonly="" value="" size="10" name="Sdate">
<img align="absmiddle" onclick="return showCalendar('Sdate', 'dd/mm/yy');" style="cursor:pointer" src="../images/calendar_s.gif">
I am using Selenium webdriver on Firefox with Capybara 1.1.2 in a Rails 3.2.6 task.
What am I missing?
I have the same setup and I was able to select box with this line:
select 'May', from: 'date_select_1'
where 'May' is a value which you want select from list and the *data_select_1* is ID of HTML tag from which you want select the value.