I am experiencing flaky action cable tests on capybara backed by cuprite(headless mode). Basically, I am creating a post using action cable and setting it on React using Mobx. I ran the test 150 times in a loop and it failed 30 times. What can cause this inconsistent failure?
If I make the driver go to another page or reload the same page the post appears as expected.
The settings are as follow:
spec/rails_helper.rb
require 'rails_helper.rb'
Dir[File.join(__dir__, "system/support/**/*.rb")].sort.each { |file| require file }
module CupriteHelpers
# some test helpers
end
RSpec.configure do |config|
config.include CupriteHelpers, type: :system
end
spec/system/support/capybara.rb
Capybara.default_max_wait_time = 10
Capybara.default_normalize_ws = true
Capybara.save_path = ENV.fetch("CAPYBARA_ARTIFACTS", "./tmp/capybara")
Capybara.singleton_class.prepend(Module.new do
attr_accessor :last_used_session
def using_session(name, &block)
self.last_used_session = name
super
ensure
self.last_used_session = nil
end
end)
spec/system/support/curprite.rb
Capybara.register_driver(:cuprite) do |app|
Capybara::Cuprite::Driver.new(
app,
**{
window_size: [1440, 900],
browser_options: {},
process_timeout: 60,
timeout: 15,
inspector: ENV['INSPECTOR'] == 'true',
headless: !ENV['HEADLESS'].in?(%w[n 0 no false]),
slowmo: (0.2 if ENV['HEADLESS'].in?(%w[n 0 no false])),
js_errors: false,
logger: FerrumLogger.new
}
)
end
Capybara.default_driver = Capybara.javascript_driver = :cuprite
spec/system/post_spec.rb
it 'can create a new post and the creator is the user do
click_button 'Add New Post +'
expect(page).to have_css('#post-2')
# rest of the tests but the line above fails
end
Related
I have a Rails 6 application which I test with rspec, Capybara and Chrome headless on a remote VM. With the new webdrivers gem, not that ancient poltergeist thing.
It has an user manager mini-app written in Vue 2.something that behaves in some stupefying ways:
Excerpt from Vue application
{
el: "#app",
data: {
initial_load_completed: false,
users: []
},
created: function(){
this.loadUsers();
},
methods: {
loadUsers: function(){ /* straightforward JSON load from server into .users and set .initial_load_completed to true */ }
/* lots of other code */
},
computed: {
hasUsers: function(){
return this.users.length > 0;
}
/* lots of other code */
}
}
View excerpt
<div id="app">
<!-- loads of other code -->
<div v-if="!initial_load_completed && !hasUsers">Loading your users, please wait...</div>
<div v-if="initial_load_completed && !hasUsers">There are no users for your account right now...</div>
<!-- lots of other code -->
</div>
The application works perfectly in prod and dev, on chrome, safari, tablets, iphones, even on my 3 year old smart TV Trashroid, even on IE. But under rspec tests it does things such as this:
This example with those 2 divs showing/hiding based on users loaded is just a small thing that's wrong in this picture, many other controls were supposed to not show with an empty users array. And this is a happy happy joy joy case, about 50% of example runs it just doesn't output anything at all, #app is blank... randomly.
In my test.log I see how the Vue app hits the JSON endpoint of my back-end and how it renders data with a 200.
For the life of me I can't imagine how initial_load_completed can be true and false at the same time.
What I've tried?
Rebooted the machine (heh). Then reinstalled all software to latest versions.
Then spent about 2 days trying to get chrome to work on a "virtual" display to which I would connect to see what's going on... after some 218 iterations fixing various deps/errors and configurations and code and signs and more errors and so on I just gave up.
Driver definition:
Webdrivers.logger.level = :DEBUG
default_chrome_args = [ '--disable-extensions', '--no-sandbox', '--disable-dev-shm-usage', '--remote-debugging-port=9222', '--remote-debugging-address=0.0.0.0' ]
Capybara.register_driver :headless_chrome do |app|
capabilities = Selenium::WebDriver::Remote::Capabilities.chrome( loggingPrefs: { browser: 'ALL' }, chromeOptions: {
'args' => default_chrome_args + ['--headless', '--disable-gpu', '--window-size=1920,1600' ]
})
Capybara::Selenium::Driver.new app, browser: :chrome, desired_capabilities: capabilities
end
CSP's are disabled, tried with and without them anyway.
Yesterday I tried logging JS errors:
config.after(:each, type: [ :feature, :system ], js: true) do
errors = page.driver.browser.manage.logs.get(:browser)
if errors.present?
aggregate_failures 'javascript errrors' do
errors.each do |error|
expect(error.level).not_to eq('SEVERE'), error.message
next unless error.level == 'WARNING'
STDERR.puts 'WARN: javascript warning'
STDERR.puts error.message
end
end
end
end
... no luck.
config.after(:each, type: [ :feature, :system ], js: true) do
errors = page.driver.browser.manage.logs.get(:browser)
errors.each do |error|
STDERR.puts error.message
end
end
... also nada just like several other few variations of this code.
Can't even seem to get the examples to "puts :whatever" to stdout but that's another story.
Can someone kind at heart pretty please help a poor dumb me not lose all hair?
Something that is not clear from the code samples in your question, is whether you are actually applying the driver you are defining.
System tests will use the default driver, so you must set it explicitly:
RSpec.configure do |config|
config.before(:each, type: :system) {
driven_by :headless_chrome
}
end
It can also be applied on a per-scenario basis on feature tests:
RSpec.feature 'Balance' do
scenario 'check the balance', driver: :headless_chrome do
...
end
end
I'm using Capybara with WebMock and a proxy (Sinatra) to test a remote app.
I'm not stubbing the requests but using WebMock to assert requests. I've assigned the proxy to capybara.app and added to the chrome driver so the requests will be forwarded to the proxy.
My problem is that sometimes I have pending requests which will raise the following error:
Failure/Error: raise "Requests did not finish in 60 seconds"
I wonder how can I disable this error?
Also how can I change the hard coded timeout which is 60 (which will block the continuation of the test anyway)
Timeout.timeout(60) { sleep(0.01) while #middleware.pending_requests? }
capybara.rb:
require 'capybara/rspec'
require 'capybara'
require 'capybara/dsl'
require_relative 'sinatra_proxy'
require 'selenium/webdriver'
require 'selenium/webdriver/remote/http/curb' if !isWindows
Capybara.server_port = 9980
Capybara.register_driver :selenium_chrome do |app|
http_client = isWindows ? nil : Selenium::WebDriver::Remote::Http::Curb.new
options = {
http_client: http_client,
browser: :chrome,
switches: [
"--proxy-server=0.0.0.0:9980",
"--disable-web-security",
'--user-agent="Chrome under Selenium for Capybara"',
"--start-maximized",
'--no-sandbox',
]
}
Capybara::Selenium::Driver.new app, options
end
Capybara.default_driver = :selenium_chrome
Capybara.app = SinatraProxy.new
Capybara.server_host = '0.0.0.0'
Capybara.default_max_wait_time = 8
Sinatra proxy:
require "sinatra"
require 'net/http'
require 'json'
file = File.read 'config.json'
config_json = JSON.parse(file)
HOST = 'remote_app'
PORT = '80'
HEADERS_NOT_TO_PROXY = %w(transfer-encoding)
class SinatraProxy < Sinatra::Base
# configure :development do
# register Sinatra::Reloader
# end
def request_headers
request.env.select {|k,v| k.start_with? 'HTTP_'}
.collect {|pair| [pair[0].sub(/^HTTP_/, ''), pair[1]]}
.to_h # Ruby 2.1
.merge('CONTENT-TYPE' => request.env['CONTENT_TYPE'] || 'application/json')
end
proxy = lambda do
# puts "REQUEST HEADERS #{request_headers}"
uri = URI.parse(request.url)
http = Net::HTTP.new(HOST, PORT)
response = http.send_request(
request.request_method.upcase,
uri.request_uri,
request.body.read,
request_headers)
response_headers = {}
response.to_hash.each{|k,v| response_headers[k]=v.join unless HEADERS_NOT_TO_PROXY.include?(k) }
status response.code
headers response_headers
headers 'Access-Control-Allow-Origin' => '*'
# puts "RESPONSE HEADERS #{response_headers}, BODY: #{response.body}"
body response.body
end
get '/*', &proxy
post '/*', &proxy
patch '/*', &proxy
put '/*', &proxy
delete '/*', &proxy
options "*" do
headers 'Access-Control-Allow-Origin' => '*',
'Access-Control-Allow-Methods' => 'HEAD,GET,PUT,DELETE,OPTIONS'
halt 200
end
end
You can't change the timeout for waiting during reset, however you shouldn't be setting Capybara.app to your proxy app to start with. Capybara.app is meant for your AUT (application under test) where any requests that hang for more than 60 seconds after being told to quit is definitely an error. Since your proxy isn't your AUT just run it separately and tell Capybara not to run a server/app.
Capybara.run_server = false
Thread.new do
Rack::Handler::Puma.run(SinatraProxy.new, Host: '0.0.0.0', Port: 9980) # You might need/want other options here or to use thin/webrick/etc
end
I have been trying to create the following test :
Edit a model (client side), check if the view is updated and if the model changed in database.
there is the code :
test 'a' do
user = User.joins(:organization_users).find_by organization_users: { role: OrganizationUser.roles.values_at(:ORGANIZER, :ADMINISTRATOR) }
sign_in_user user
criterion = create(:criterion, scoring_id: #scoring.id, name: "Test criterion name",
description: "Test description")
step = create(:step, criterion_id: criterion.id)
visit "scorings/" + (#scoring.id).to_s + "/criteria"
find("#criteria > div > div > a > i").click()
fill_in 'name', with: 'New name'
fill_in 'description', with: 'New description'
find('#criterion-modal > div:nth-child(2) > form > div:nth-child(4) > input').click()
criterion = criterion.reload
assert criterion.name == 'New name'
end
`
Driver :
Capybara.register_driver :poltergeist do |app|
Capybara::Poltergeist::Driver.new app , { phantomjs: Phantomjs.path }
end
Capybara.javascript_driver = :poltergeist
Capybara.current_driver = Capybara.javascript_driver
Teardown :
teardown do
DatabaseCleaner.clean
ActiveRecord::Base.connection.close
Capybara.reset_sessions!
end
As you can see at the end of the test i reload the criterion, but when i do that the teardown function is called. After that the Database is cleaned and i get the error "cant find criterion id:1". I'm only using minitest , factory girl and Capybara. So what i want to understand is why Teardown is called since its not the end of the test and how can i fix that ?
Thank you.
You don't show what you have setup for your teardown method, nor do you specify what driver you are using with Capybara. However, since the test code and the teardown are run in the same thread there really is no way for the teardown to run before the test has ended. What is possible (when using a JS capable driver, where clicks are processed asynchronously) is for the teardown to run before a click is processed/handled by the app code. That would mean the "cant find criterion id:1" would actually be coming from your controller code. The reason for this is that you're not actually checking for anything on the page to change after clicking so the test just keeps on moving, finishes (failing the assertion), the teardown cleans and the controller action can't find the record. Something like
assert_text 'Criterion updated' # if a message is displayed on successful update
or
assert_current_path("scorings/#{#scoring.id}") # whatever path it redirects to after updating
after the click and before your reload
On a side note - Using long selectors like '#criterion-modal > div:nth-child(2) > form > div:nth-child(4) > input' will lead to really brittle tests -- It would be much nicer to use simpler selectors or the capybara click_button type helpers if possible
I have a problem, I can run a test that uses vcr on its own and it works, it creates the cassette and it uses that on the next test. Great.
The problem is when I run all my tests together this particular test fails, because webmock disables http connections, I have seen this example on the Github repo page that explains how to expect real and not stubbed requests
My question is how Do I say: Allow Http connections for requests UNLESS there is a cassette. It should also CREATE the cassette when HTTP connections are allowed.
The VCR Settings
require 'vcr'
VCR.configure do | c |
if !ARGV.first.nil?
c.default_cassette_options = { :record => :new_episodes, :erb => true }
c.filter_sensitive_data('<BLACKBIRD_API_KEY>') {YAML.load(File.read('config/application.yml'))['BLACKBIRD_API_KEY'].to_s}
c.filter_sensitive_data('<BLACKBIRD_API_URL>') {YAML.load(File.read('config/application.yml'))['BLACKBIRD_API_URL'].to_s}
c.debug_logger = File.open(ARGV.first, 'w')
c.cassette_library_dir = 'spec/vcr'
c.hook_into :webmock
end
end
the above if statement exists because not EVERY test creates a cassette. So we want them to run when a cassette isn't needed.
The Test
require 'spec_helper'
describe Xaaron::Publishers::Users do
context "publish created users" do
before(:each) do
Xaaron.configuration.reset
no_user_member_roles_relation
Xaaron.configuration.publish_to_black_bird = true
Xaaron.configuration.black_bird_api_url = YAML.load(File.read('config/application.yml'))['BLACKBIRD_API_URL']
Xaaron.configuration.black_bird_api_key =YAML.load(File.read('config/application.yml'))['BLACKBIRD_API_KEY']
end
it "should publish to blackbird" do
VCR.use_cassette 'publisher/create_user_response' do
expect(
Xaaron::Publishers::Users.publish_new_user({user: {
first_name: 'adsadsad', user_name: 'sasdasdasdsa' ,
email: 'asdassad#sample.com', auth_token: 'asdsadasdasdsa'
}}).code
).to eql 200
end
end
end
end
Runs fine on its own, creates the cassette, fails when run with all other tests due to webmock.
The Failure
Failure/Error: Xaaron::Publishers::Users.publish_new_user({user: {
WebMock::NetConnectNotAllowedError:
Real HTTP connections are disabled. Unregistered request: GET some_site_url_here with headers {'Http-Authorization'=>'api_key_here', 'User-Agent'=>'Typhoeus - https://github.com/typhoeus/typhoeus'}
You can stub this request with the following snippet:
stub_request(:get, "some site url here").
with(:headers => {'Http-Authorization'=>'some api key here', 'User-Agent'=>'Typhoeus - https://github.com/typhoeus/typhoeus'}).
to_return(:status => 200, :body => "", :headers => {})
I'm using Capybara 2.x for some integration tests for a large Rails/AngularJS app and I've come across a test in which I need to put a sleep to get it working.
My test:
describe "#delete", js: true do
it "deletes a costing" do
costing = Costing.make!
visit "/api#/costings"
page.should have_content("General")
click_link "Delete" # Automatically skips the confirm box when in capybara
sleep 0.4
page.should_not have_content("General")
end
end
The code it tests is using ng-table which takes a split second to update, without that sleep it will fail. Capybara used to have a wait_until method for this but it's been taken out. I found this website: http://www.elabs.se/blog/53-why-wait_until-was-removed-from-capybara but cannot get any of the recommended alternatives working for this problem.
Here is the code I'm testing.
# --------------------------------------------------------------------------------
# Delete
# --------------------------------------------------------------------------------
$scope.destroy = (id) ->
Costing.delete (id: id), (response) -> # Success
$scope.tableParams.reload()
flash("notice", "Costing deleted", 2000)
This updates the ng-table (tableParams variable) which is this code
$scope.tableParams = new ngTableParams({
page: 1,
count: 10,
sorting: {name: 'asc'}
},{
total: 0,
getData: ($defer, params) ->
Costing.query {}, (data) ->
# Once successfully returned from the server with my data process it.
params.total(data.length)
# Filter
filteredData = (if params.filter then $filter('filter')(data, params.filter()) else data)
# Sort
orderedData = (if params.sorting then $filter('orderBy')(filteredData, params.orderBy()) else data)
# Paginate
$defer.resolve(orderedData.slice((params.page() - 1) * params.count(), params.page() * params.count()))
})
Try bumping the Capybara.default_wait_time up to 3 seconds or 4.
If that fails, try changing the spec to look for the flash notice message before it checks to see if the item has been removed from the page. (Assuming the flash message gets rendered in the HTML body)
describe "#delete", js: true do
it "deletes a costing" do
costing = Costing.make!
visit "/api#/costings"
page.should have_content("General")
click_link "Delete"
page.should have_content("Costing deleted")
page.should_not have_content("General")
end
end
Edit - removed explanation because it was incorrect.