Poltergeist/PhantomJS crashing in Gitlab CI - ruby-on-rails

Question

Background:
My testing suite did not handle any JS functionality until recently. I started out
only on my machine with a selenium-webdriver javascript driver and firefox 47.0.1
with no issues. I then migrated over to poltergeist supporting PhantomJS as
my javascript driver. I had to make a few changes to some broken tests, but have
very much enjoyed the changeover and am 'sold' on PhantomJS/poltergeist.
Expected Behavior:
I anticipated the tests to pass on my Gitlab-ci just as they had on my machine
(hey, we can dream), with maybe a few configuration tweaks needed.
Versions:
Ruby - 2.3.1
Rails - 5.0.0.1
poltergeist - 1.12.0
rspec - 3.5.0
rspec-rails - 3.5.1
capybara - 2.11.0
cliver - 0.3.2
factory_girl_rails - 4.7.0
phantomjs - 2.1.1 (on both my machine and CI server)
Capybara Config:
require 'capybara/rspec'
require 'capybara/poltergeist'
Capybara.default_driver = :rack_test
Capybara.register_driver :poltergeist do |app|
options = {
:js_errors => false,
:timeout => 360,
:phantomjs_options => ['--load-images=no', '--ignore-ssl-errors=yes', '--disk-cache=false']
}
Capybara::Poltergeist::Driver.new(app, options)
end
Capybara.javascript_driver = :poltergeist
DatabaseCleaner Config
config.before(:suite) do
DatabaseCleaner.clean_with(:truncation)
end
config.before(:each) do
DatabaseCleaner.strategy = Capybara.current_driver == :rack_test ? :transaction : :truncation
DatabaseCleaner.clean
DatabaseCleaner.start
end
config.after(:each) do |example|
puts 'RAM USAGE: ' + `pmap #{Process.pid} | tail -1`[10,40].strip
if defined?(page)
Capybara.reset_sessions!
page.driver.restart
end
DatabaseCleaner.clean
end
Actual Behavior/Backtraces:
1. Failing Tests, PhantomJS Crash
To get a full stacktrace on my failing tests before I ate up all of my memory I
used the rspec --fail-fast option to kick back results for me on my first error.
This is what I returned:
ArgumentError:
The detector #<struct Cliver::Detector command_arg=nil, version_pattern=nil> failed to detect theversion of the executable at '/usr/bin/phantomjs'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:191:in `detect_version'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:87:in `block in installed_versions'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:214:in `block in find_executables'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:208:in `map'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:208:in `find_executables'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:86:in `each'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:86:in `installed_versions'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:108:in `each'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:108:in `detect!'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver/dependency.rb:97:in `detect'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/cliver-0.3.2/lib/cliver.rb:35:in `detect'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/client.rb:47:in `initialize'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/client.rb:14:in `new'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/client.rb:14:in `start'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/driver.rb:44:in `client'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/driver.rb:25:in `browser'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/poltergeist-1.12.0/lib/capybara/poltergeist/driver.rb:183:in `reset!'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara/session.rb:110:in `reset!'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara.rb:335:in `block in reset_sessions!'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara.rb:335:in `reverse_each'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara.rb:335:in `reset_sessions!'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/capybara-2.11.0/lib/capybara/rspec.rb:21:in `block (2 levels) in <top (required)>'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:357:in `run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:507:in `block in run_owned_hooks_for'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:506:in `each'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:506:in `run_owned_hooks_for'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:593:in `block in run_example_hooks_for'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:592:in `each'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:592:in `run_example_hooks_for'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:463:in `run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:503:in `run_after_example'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:269:in `block in run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:496:in `block in with_around_and_singleton_context_hooks'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:453:in `block in with_around_example_hooks'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:464:in `block in run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:604:in `block in run_around_example_hooks_for'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:338:in `call'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-rails-3.5.1/lib/rspec/rails/adapters.rb:127:in `block (2 levels) in <module:MinitestLifecycleAdapter>'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:375:in `execute_with'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:606:in `block (2 levels) in run_around_example_hooks_for'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:338:in `call'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec/retry.rb:98:in `block in run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec/retry.rb:88:in `loop'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec/retry.rb:88:in `run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-retry-0.4.5/lib/rspec/retry.rb:22:in `block (2 levels) in setup'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:375:in `execute_with'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:606:in `block (2 levels) in run_around_example_hooks_for'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:338:in `call'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:607:in `run_around_example_hooks_for'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:464:in `run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:453:in `with_around_example_hooks'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:496:in `with_around_and_singleton_context_hooks'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:251:in `run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:627:in `block in run_examples'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:623:in `map'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:623:in `run_examples'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/example_group.rb:589:in `run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:113:in `block (3 levels) in run_specs'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:113:in `map'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:113:in `block (2 levels) in run_specs'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/configuration.rb:1835:in `with_suite_hooks'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:112:in `block in run_specs'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/reporter.rb:77:in `report'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:111:in `run_specs'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:87:in `run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:71:in `run'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/lib/rspec/core/runner.rb:45:in `invoke'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/gems/rspec-core-3.5.4/exe/rspec:4:in `<top (required)>'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/bin/rspec:23:in `load'
# /home/gitlab-runner/builds/wow_such_build/0/schwad_project/redacted/vendor/bundle/ruby/2.3.0/bin/rspec:23:in `<main>'
#
# Showing full backtrace because every line was filtered out.
# See docs for RSpec::Configuration#backtrace_exclusion_patterns and
# RSpec::Configuration#backtrace_inclusion_patterns for more information.
Also, in my gitlab-ci.yml file, if I call phantomjs --version in before_script I receive:
QXcbConnection: Could not connect to display
PhantomJS has crashed. Please read the bug reporting guide at
<http://phantomjs.org/bug-reporting.html> and file a bug report.
Any help on this would be much appreciated.
2. Memory Blowup SOLVED: see update 2
The tests would eat up the memory, even after we increased memory capacity on the
servers, and crash the test suite before it would finish. To track memory usage
in my RSpec configuration after each test I ran:
puts 'enter code hereRAM USAGE: ' +pmap #{Process.pid} | tail -1[10,40].strip
The memory would be minimally affected under the model tests, but grow rapidly
under the integration tests. Like so:
Model Test-
RAM USAGE: 567800K
Model Test-
RAM USAGE: 567800K
Model Test-
RAM USAGE: 567800K
Model Test-
RAM USAGE: 568220K
Model Test-
RAM USAGE: 568220K
Model Test-
RAM USAGE: 568360K
Model Test-
RAM USAGE: 568500K
Model Test-
RAM USAGE: 568652K
Model Test-
RAM USAGE: 568788K
First Feature Test:
RAM USAGE: 1100628K
And blowing up from there.
To handle this I have made sure to use appropriate DatabaseCleaner strategies,
also cycled through trying tools such as Capybara.reset_sessions!,
Capybara.current_session.driver.quit, page.driver.quit, page.driver.restart
with no real joy.
UPDATE:
5:06 PM UTC, 15/12/16
1. Crashes
Gitlab support has suggested investigating whether our packages and dependencies are resulting in a non-headless situation with PhantomJS. We are investigating this now.
2. Memory
I've run a memory tracker against a few different iterations here, and it seems to be simply enabling our js: true feature tests with a javascript driver to be eating up the memory, going from 1G for about 1,750 tests to surpassing our 4G limit on 1,800-1,850 tests. This seems to occur, I believe, with either selenium-webdriver OR poltergeist as the js driver.
UPDATE 2:
11:45 PM UTC, 16/12/16
1. Crashing
Still awaiting proper installation of packages on our server. Will update and award answer if this is the issue.
2. Memory
On the branch where we were migrating over to the Javascript-driver enabled batch of tests, part of the configuration to speed up the tests implemented a DeferredGarbageCollector which had settings that gobbled up the memory. PhantomJS and Poltergeist are not leaking. SOLVED

Answer 1

http://github.com/ariya/phantomjs/issues/14240 is the cause of your PhantomJS crashing - The version of PhantomJS shipped by Ubuntu requires X to run.
Download the official release build and use that to cure your crashing issue.
You may want to add the memory issue as a separate question.