I've recently moved mi CI server (Teamcity) to another powerful machine with same configuration and pretty similar OS.
Since then some of my integration specs have started to fail. My setup is pretty standard, Rails 3 + capybara + poltergeist + phantomjs.
Failures are deterministic, they always happen and they are always related to some missing nodes in the DOM. Also, failures happens across different projects with similar setup so it's not something related to project configuration. This is happening with both capybara 1.x and capybara 2.
This is the simplest failing spec. Note that this spec runs with no need of javascript, so the issue is also present in rack only specs.
scenario 'require an unsubscription' do
visit unsubscribe_index_path
within main_content do
choose list.description
fill_in 'Email', :with => subscriber.email
click_button 'Unsubscribe'
end
save_page # <--- Added to debug output
# !!! HERE is the first failing assertion
page.should have_content('You should have received a confirmation message')
# Analytics event recorded
# !!! this also is failing
page.should have_event('Unsubscription', 'Sent', list_name)
# If I comment previous two lines the spec passes on CI machine
# this means that the form is submitted with success since email is triggered
# from controller code
last_email_sent.should have_subject 'Unsubscribe request received'
last_email_sent.should deliver_to subscriber.email
end
What I've tried:
run the specs on different machines, they works on every dev machine and also in a staging server. I can only reproduce the failure on the CI machine even outside of CI environment (i.e. by running the specs via command line)
Increased Capybara.default_wait_time to a ridiculous 20
Tried with a brutal sleep before the page.should have_content line
upgrade RVM, ruby, capybara, poltergeist on their latest versions on the CI machine.
upgrade teamcity to its latest version
The strangest thing I found is when I've added a save_page call just before the failing line. If i run the spec on my machine and then on the CI where the server is failing and comparing those two files the result is this:
$ diff capybara-201309071*.html
26a27,29
> <script type='text/javascript'>
> _gaq.push(["_trackEvent","Unsubscription","Sent","listname"]);
> </script>
90a94,96
> <div class="alert-message message notice">
> <p>You should have received a confirmation message</p>
> </div>
Which are the two missing pieces which make the spec failing, so the form is submitted, controller action is run successfully but there are two missing pieces of dom. How that is possible? And why this is happening only on one machine?
For the records, those two pieces of DOM are added with standard rails tools one with
redirect_to unsubscribe_index_path, notice: ...
and the other with the analytical gem
I've found the issue, the two failing projects I'm using dalli_store as session store and I've put the config.cache_store = :dalli_store line in config/application.rb instead of config/environments/production.rb.
On the old CI server there was a memcached daemon running hence all specs was running fine.
In the new server since it's just a CI server and it doesn't run any production or staging code memcached is not present thus any session write (such as flash messages) was silently discarded and this is the reason why all that kind of specs was failing.
I solved by putting the config.cache line in the appropriate environment file, but I'm still wondering why dalli gem doesn't raise any warning when no memcached is available. While the choice of not failing on missing cache daemon is reasonable since the application should work with no cached data, it could be a performance killer in production code and it might go unnoticed if no warning is given.
Related
I maintain several extension for the Spree/Solidus Rails platform(s), and for some reason on one extension in particular I’m having a Capybara issue I can’t seem to track down.
Once I build a test environment, the first test of the first run through the specs always fails, with the following:
Capybara::Poltergeist::StatusFailError:
Request to 'http://127.0.0.1:52234/products' failed to reach server,
check DNS and/or server status - Timed out with no open resource requests
All subsequent specs pass. If I run bundle exec rspec spec again, then all specs pass.
I have tried increasing the Capybara timeout values to super high numbers:
RSpec.configure do |config|
config.include Spree::TestingSupport::CapybaraHelpers, type: :feature
Capybara.register_driver(:poltergeist) do |app|
Capybara::Poltergeist::Driver.new app, timeout: 90
end
Capybara.javascript_driver = :poltergeist
Capybara.default_max_wait_time = 90
end
But it seems to have no effect.
All my Travis builds fail on the first spec of the first run (and pass on everything else), which is making it hard to maintain the project, as all PRs look red.
Any ideas what might be going on here?
Most likely this is failing when Rails processes the asset pipeline at the first request. Try precompiling the test mode assets before running the test or increase the timeout in the driver registration even more. The Capybara.default_max_wait_time shouldn't affect this at all.
I'm trying to set up my ember-cli / rails app with integration testing. After fussing with Ember's built in testing library, I've switched over to using RSpec (which I was using for backend anyway), and Capybara. While I can finally fill in forms correctly, my post request to sign_in always fails. I think the issue is Capybara is posting the request to a different database environment or something! If I check at the rails console, the user is certainly there, and I create a user as part of the RSpec test anyway.
Has anyone managed to set up Ember/Rails/Capybara/RSpec?
This is my spec:
describe "the signin process", :type => :feature, :js => true do
it "signs me in" do
visit '/'
FactoryGirl.create :user, email: "user#example.com", password: 'password'
within("#tufts-nav") do
fill_in 'email', :with => 'test#test.com'
fill_in 'password', :with => 'password'
end
click_button 'Sign In'
# here authentication fails mysteriously
expect(page).to have_content 'Jobs'
end
end
Simple/dumb solution
Have RSpec build ember into rails' public/ before your feature specs.
# build ember, hijack rails public/ directory to host ember app
# this way no need to change settings or run any servers
# assumes rails API root path is not used (since ember now hosted from it)
RSpec.configure do |config|
public_path = Rails.root.join('public')
config.before(:context, type: :feature) do
Dir.chdir 'frontend' do
builder = spawn("ember build --environment=ci -output-path=#{public_path}")
_pid, status = Process.wait2(builder)
fail "non-zero exit status #{status}" unless status == 0
end
end
config.after(:context, type: :feature) do
`git clean -fd #{public_path}`
`git checkout #{public_path}`
end
end
Configuring
Our ember-cli app is in rails-root/frontend, you may need to change the name or path to point at yours
You might want to experiment with the environment part, e.g. using production instead. I do this because my production env is hard-coded to target an API we host on heroku but I want the tests to be self-contained, i.e. run against the rails app capybara hosts.
Git is needed for cleanup. If you don't have that you could build to another path and use mv to swap out the rails public/ dir then put it back afterwards.
You may prefer not to have a global install of ember-cli used to build your project (for versioning reasons). If you want to use the project local one point the spawn command at node_modules/ember-cli/bin/ember instead of just ember.
Otherwise, ember-cli-rails
If you're treating the ember app as a component of your rails app, and want to write tests at the rails level (rspec, capybara, etc) then ember-cli-rails is likely a good choice.
This gem handles building the ember app and serving it from the urls you mount it at in your rails routes.
This is transparent to capybara: it sends a request to a ruby webserver and is given back html that calls out to css and js, just like rails normally does.
Be aware there are some issues with assets from ember-cli getting served by rails with the right paths at the moment, which made me switch away to get something deployed quickly. If you're using the rails asset pipeline for css, images and so on, then you shouldn't have a problem. For me, it affected images and webfonts in the ember-cli app.
Apart from that there will need to be a server for the api and a server for the ember front-end (proxying to the rails api), and capybara will need telling to connect to the ember front-end.
This Rakefile and this post seem like a start.
I've configured my CI server with Jenkins in order to run my test after each push to Git. I have some integration tests and there is always one that fails. The output given is that capybara can't find the element but if I run the test in local it works.
I've research a bit and I found headless gem but it doesn't work. I also tried to set Capybara.default_wait_time = 5 but nothing.
Does anyone know what kind of configuration I should set in order to get my jenkins green?
Thank you in advance
The failure to find may be element within the environment. Using the 'save_and_open' method will show the exact state of the page that is trying to be rendered in the CI server environment. You can add it to a test as simply as:
it 'should register successfully' do
visit registration_page
save_and_open_page
fill_in 'username', :with => 'mrUser'
end
This assumes that you can view the page on the CI server or alternatively with the headless version the file should be saved so that you review later.
I'm a rails dev working on a rails 4.0.4 app. As of yesterday (possibly before, yesterday was the first I noticed it, because I usually use CI), my archlinux dev machine doesn't run rspec features marked with js: true metadata tags, it just returns passes for all of them. e.g.:
$ be rspec spec/features/activity_spec.rb
..........
Finished in 0.38366 seconds
10 examples, 0 failures
There are no such problems for any other specs so far as I can tell, just those that use selenium. It does not spawn a browser (we use firefox, though I tried chrome with chromedriver). It seems to not even call the Procs created in spec_helper, as raising an exception doesn't happen:
Capybara.server do |app, port|
raise "Hell"
require 'rack/handler/thin'
Rack::Handler::Thin.run(app, :Port => port)
end
# use BROWSER=safari,chrome,etc
browser = (ENV["BROWSER"] || "firefox").to_sym
Capybara.register_driver :selenium do |app|
raise "Hell"
Capybara::Selenium::Driver.new(app, browser: browser)
end
Updating to the most recent capybara/selenium-webdriver/rspec does not change anything, nor does checking out old tags from my repo, which used previous versions of gems/ruby.
The rest of my team (all running OSX) have no problems with the exact same branch/set of gems/ruby version (these same specs ran previously on 2.1.1 and 1.9.3 on the same machine).
All of this screams "OS specific problem" at me. Any suggestions on what to try (other than switching to OSX - have had enough of that from my colleagues) would be appreciated. Cheers.
This was due to setting RETRIES=0 when using rspec-retry. Oops.
I am looking into using capybara-webkit to do somewhat close-to-reality tests of app. This is absolutely neccessary as the app features a very rich JS-based UI and the Rails part is mostly API calls.
The question is: is there any tools to integrate into testing pipeline which could instrument Javascript code and report its coverage? The key here is the ability to integrate into testing workflow (just like rcov/simplecov) easily – I don't like the idea do it myself with jscoverage or analogue :)
Many thanks in advance.
This has now been added to JSCover (in trunk) - the related thread at JSCover is here.
I managed to get JSCover working in the Rails + Capybara pipeline, but
it did take quite a bit of hacking to get it to work
These changes are now in JSCover's trunk and will be part of version 1.0.5. There's working examples (including a Selenium IDE recorded example) and documentation in there too.
There is some additional work needed to get the branch detection to
work since that uses objects that cannot be easily serialized to JSON
This is a function to do this which is used in the new code.
Anyway, the end result works nicely
I agree. This makes JSCover useable by higher level tools that don't work well with iFrames or multiple windows which are avoided by this approach. It also means code coverage can be added to existing Selenium tests with two adjustments:
Make the tests run through the JSCover proxy
Save the coverage report at the end of the test suite
See JSCover's documentation for more information. Version 1.0.5 containing these changes should be released in a few days.
Update: Starting from JSCover version 1.05 the hacks I outlined in my previous answer are no longer needed. I've updated my answer to reflect this.
I managed to get JSCover working in the Rails + Capybara pipeline, but it did take some hacking to get it to work. I built a little rake task that:
uses the rails asset pipeline to generate the scripts
calls the java jar to instrument all the files and generate an empty report into a temp dir
patches the jscover.js script to operate in "report mode" (simply add jscoverage_isReport=true at the end)
copies the result to /public/assets so the tests pick it up without needing any changes and so the coverage report can be opened automatically in the browser
Then I added a setup task to clear out the browser's localStorage at the start of the tests and a teardown task that writes out the completed report at the end.
def setup
unless $startup_once
$startup_once=true
puts 'Clearing localStorage'
visit('/')
page.execute_script('localStorage.removeItem("jscover");')
end
end
def teardown
out=page.evaluate_script("typeof(_$jscoverage)!='undefined' && jscoverage_serializeCoverageToJSON()")
unless out.blank? then
File.open(File.join(Rails.root,"public/assets/jscoverage.json"), 'w') {|f| f.write(out) }
end
end
Anyway, the end result works nicely, the advantage of doing it this way is that it also works on headless browsers so it can also be included in CI.
*** Update 2: Here is a rake task that automates the steps, drop this in /lib/tasks
# Coverage testing for JavaScript
#
# Usage:
# Download JSCover from: http://tntim96.github.io/JSCover/ and move it to
# ~/Applications/JSCover-1
# First instumentalize the javascript files:
# rake assets:coverage
# Then run browser tests
# rake test
# See the results in the browser
# http://localhost:3000/assets/jscoverage.html
# Don't forget to clean up instrumentalization afterwards:
# rake assets:clobber
# Also don't forget to re-do instrumentalization after changing a JS file
namespace :assets do
desc 'Instrument all the assets named in config.assets.precompile'
task :coverage do
Rake::Task["assets:coverage:primary"].execute
end
namespace :coverage do
def jscoverage_loc;Dir.home+'/Applications/JSCover-1/';end
def internal_instrumentalize
config = Rails.application.config
target=File.join(Rails.public_path,config.assets.prefix)
environment = Sprockets::Environment.new
environment.append_path 'app/assets/javascripts'
`rm -rf #{tmp=File.join(Rails.root,'tmp','jscover')}`
`mkdir #{tmp}`
`rm -rf #{target}`
`mkdir #{target}`
print 'Generating assets'
require File.join(Rails.root,'config','initializers','assets.rb')
(%w{application.js}+config.assets.precompile.select{|f| f.is_a?(String) && f =~ /\.js$/}).each do |f|
print '.';File.open(File.join(target,f), 'w') {|ff| ff.write(environment[f].to_s) }
end
puts "\nInstrumentalizing…"
`java -Dfile.encoding=UTF-8 -jar #{jscoverage_loc}target/dist/JSCover-all.jar -fs #{target} #{tmp} #{'--no-branch' unless ENV['C1']} --local-storage`
puts 'Copying into place…'
`cp -R #{tmp}/ #{target}`
`rm -rf #{tmp}`
File.open("#{target}/jscoverage.js",'a'){|f| f.puts 'jscoverage_isReport = true' }
end
task :primary => %w(assets:environment) do
unless Dir.exist?(jscoverage_loc)
abort "Cannot find JSCover! Download from: http://tntim96.github.io/JSCover/ and put in #{jscoverage_loc}"
end
internal_instrumentalize
end
end
end