I use the capybara-webkit gem to scrape data from certain pages in my Rails application. I've noticed, what seems to be "random" / "sporadic", that the application will crash with the following error:
Capybara::Webkit::ConnectionError: /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/bin/webkit_server failed to start.
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/server.rb:56:in `parse_port'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/server.rb:42:in `discover_port'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/server.rb:26:in `start'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/connection.rb:67:in `start_server'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/connection.rb:17:in `initialize'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/driver.rb:16:in `new'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit/driver.rb:16:in `initialize'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit.rb:15:in `new'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-webkit-1.11.1/lib/capybara/webkit.rb:15:in `block in <top (required)>'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-2.7.1/lib/capybara/session.rb:85:in `driver'
from /home/daveomcd/.rvm/gems/ruby-2.3.1/gems/capybara-2.7.1/lib/capybara/session.rb:233:in `visit'
It happens even after it's already connected and accessed a website multiple times before. Here's a code snippet of what I'm using currently...
if site.url.present?
begin
# Visit the URL
session = Capybara::Session.new(:webkit)
session.visit(site.url) # here is where the error occurs...
document = Nokogiri::HTML.parse(session.body)
# Load configuration options for Development Group
roster_table_selector = site.development_group.table_selector
header_row_selector = site.development_group.table_header_selector
row_selector = site.development_group.table_row_selector
row_offset = site.development_group.table_row_selector_offset
header_format_type = site.config_header_format_type
# Get the Table and Header Row for processing
roster_table = document.css(roster_table_selector)
header_row = roster_table.css(header_row_selector)
header_hash = retrieve_headers(header_row, header_format_type)
my_object = process_rows(roster_table, header_hash, site, row_selector, row_offset)
rescue ::Capybara::Webkit::ConnectionError => e
raise e
rescue OpenURI::HTTPError => e
if e.message == '404 Not Found'
raise "404 Page not found..."
else
raise e
end
end
end
I've even thought perhaps I don't find out why it's happening necessarily - but just recover when it does. So I was going to do a "retry" in the rescue block for the error but it appears the server is just down - so I get the same result when retrying. Perhaps someone knows of a way I can check if the server is down and restart it then perform a retry? Thanks for the help!
So after further investigating it appears that I was generating a new Capybara::Session for each iteration of my loop. I moved it outside of the loop and also added Capybara.reset_sessions! at the end of my loop. Not sure if that helps with anything -- but the issue seems to have been resolved. I'll monitor it for the next hour or so. Below is an example of my ActiveJob code now...
class ScrapeJob < ActiveJob::Base
queue_as :default
include Capybara::DSL
def perform(*args)
session = Capybara::Session.new(:webkit)
Site.where(config_enabled: 1).order(:code).each do |site|
process_roster(site, session)
Capybara.reset_sessions!
end
end
def process_roster(site, session)
if site.roster_url.present?
begin
# Visit the Roster URL
session.visit(site.roster_url)
document = Nokogiri::HTML.parse(session.body)
# processing code...
# pass the session that was created as the final parameter..
my_object = process_rows( ..., session)
rescue ::Capybara::Webkit::ConnectionError => e
raise e
rescue OpenURI::HTTPError => e
if e.message == '404 Not Found'
raise "404 Page not found..."
else
raise e
end
end
end
end
end
I'm working on integrating Stripe's webhooks into a Rails app using https://github.com/integrallis/stripe_event. I'm struggling to get my code to working according to the example in the gem's docs whereby an initializer is used to dictate which code responds to a particular event. It seems that Rails isn't (auto)loading my module in the initializer.
I'm configuring the autoload path properly:
# config/application.rb
config.autoload_paths += %W(#{config.root}/lib)
The stripe initializer:
#config/initializers/stripe.rb
stripe_config = YAML.load_file(Rails.root.join('config', 'stripe.yml'))[Rails.env]
Stripe.api_key = stripe_config["secret_key"]
STRIPE_PUBLIC_KEY = stripe_config["publishable_key"]
StripeEvent.setup do
# Not sure if I need this to load my module
require 'stripe_event_handlers' # => true
subscribe 'customer.subscription.created' do |event|
StripeEventHanders.handle_customer_subscription_created(event) # Define subscriber behavior
end
end
Here's my custom module (though I've tried it as a class too):
#lib/stripe_event_handlers.rb
module StripeEventHandlers
def handle_customer_subscription_created(event) # Define subscriber behavior
puts event
end
end
This is my test:
require 'test_helper'
# --- Run this in the console to get event response for mocking ---
#serialized_object = YAML::dump(Stripe::Event.retrieve('evt_0Cizt88YP0nCle'))
#filename = Rails.root.join('test/fixtures/stripe_objects', 'customer_subscription_created.yml')
#File.open(filename, 'w') {|f| f.write(serialized_object) }
class StripeEvent::WebhookControllerTest < ActionController::TestCase
def test_mock_event
event_id = 'evt_0Cizt88YP0nCle'
event = YAML.load_file(Rails.root.join('test/fixtures/stripe_objects', 'customer_subscription_created.yml'))
Stripe::Event.expects(:retrieve).with(event_id).returns(event)
assert_equal Stripe::Event.retrieve(event_id), event
end
def test_customer_subscription_created_webhook
event_id = 'evt_0Cizt88YP0nCle'
event = YAML.load_file(Rails.root.join('test/fixtures/stripe_objects', 'customer_subscription_created.yml'))
Stripe::Event.expects(:retrieve).at_most(2).with(event_id).returns(event)
# This should be a raw post request but that doesn't seem to come through
# on the stripe_event / rails side in the params hash. For testing
# purposes, we can just use a get request as the route doesn't specify an
# HTTP method.
get :event, :use_route => :stripe_event, :id => event_id
assert_response :success
end
end
And here's my test result failure:
StripeEvent::WebhookControllerTest
ERROR (0:00:00.043) test_customer_subscription_created_webhook
uninitialized constant StripeEventHanders
# config/initializers/stripe.rb:10:in `block (2 levels) in <top (required)>'
PASS (0:00:00.053) test_mock_event
Finished in 0.055477 seconds.
2 tests, 1 passed, 0 failures, 1 errors, 0 skips, 2 assertions
You are just missing the letter l in StripeEventHandlers.
subscribe 'customer.subscription.created' do |event|
StripeEventHanders.handle_customer_subscription_created(event)
end
Also, handle_customer_subscription_created should be defined as a class method:
module StripeEventHandlers
def self.handle_customer_subscription_created(event) # Define subscriber behavior
puts event
end
end
I want Rails to raise an exception when an I18n translation is missing in the testing environment (instead of rendering text 'translation missing'). Is there a simple way to achieve this?
As of Rails 4.1.0, there's now a better solution than the 4 year-old answers to this question: add the following line to your config file:
config.action_view.raise_on_missing_translations = true
I like to set this in the test environment only, but you might also want to set it in development. I would strongly advise against setting it to true in production.
To raise exceptions, you can define a class to handle localization errors.
class TestExceptionLocalizationHandler
def call(exception, locale, key, options)
raise exception.to_exception
end
end
Then you attach it to the desired test cases with
I18n.exception_handler = TestExceptionLocalizationHandler.new
This way you get exceptions raised. I don't know how to raise failures (with flunk) to get better results.
Or you can just add those lines to your config/test.rb
config.action_view.raise_on_missing_translations = true
config.i18n.exception_handler = Proc.new { |exception| raise exception.to_exception }
Rails 4.1+
To raise i18n translation missing exceptions you need two things:
1) An initializer config/initializers/i18n_force_exceptions.rb:
module I18n
class ForceMissingTranslationsHandler < ExceptionHandler
def call(exception, locale, key, options)
if Rails.env.test?
raise exception.to_exception
else
super
end
end
end
end
I18n.exception_handler = I18n::ForceMissingTranslationsHandler.new
2) A config setting in config/environments/test.rb (and other environments as needed):
config.action_view.raise_on_missing_translations = true
Note: The config setting is needed in addition to the exception handler because rails wraps calls to I18n.translate in it's view and helpers preventing exceptions from triggering.
I've created this initializer to raise an exception - args are passed so you will know which i18n key is missing!
# only for test
if Rails.env.test?
# raises exception when there is a wrong/no i18n key
module I18n
class JustRaiseExceptionHandler < ExceptionHandler
def call(exception, locale, key, options)
if exception.is_a?(MissingTranslation)
raise exception.to_exception
else
super
end
end
end
end
I18n.exception_handler = I18n::JustRaiseExceptionHandler.new
end
Source
If you're using rails between 4.0.0 to 4.1.0 you should monkey patch this way:
module ActionView::Helpers::TranslationHelper
def t_with_raise(*args)
value = t_without_raise(*args)
if value.to_s.match(/title="translation missing: (.+)"/)
raise "Translation missing: #{$1}"
else
value
end
end
alias_method :translate_with_raise, :t_with_raise
alias_method_chain :t, :raise
alias_method_chain :translate, :raise
end
When I run 'script/server' everything works fine, but when I run my unit tests (rake test:units), I get the error below, and am not sure how to solve this.
Error
NameError: undefined local variable or method `logger' for #<GiveawayEligibleMemberTest:0x10477dff8>
/Users/kamilski81/Sites/pe/vitality_mall/vendor/rails/actionpack/lib/action_controller/test_process.rb:471:in `method_missing'
/Users/kamilski81/Sites/pe/vitality_mall/lib/update_giveaway_eligible_members.rb:17:in `is_valid_checksum?'
/Users/kamilski81/Sites/pe/vitality_mall/test/unit/giveaway_eligible_member_test.rb:26:in `test_that_checksum_is_valid'
/Users/kamilski81/Sites/pe/vitality_mall/vendor/rails/activesupport/lib/active_support/testing/setup_and_teardown.rb:60:in `__send__'
/Users/kamilski81/Sites/pe/vitality_mall/vendor/rails/activesupport/lib/active_support/testing/setup_and_teardown.rb:60:in `run'
I tried putting:
class Test::Unit::TestCase
RAILS_DEFAULT_LOGGER = Logger.new(STDOUT)
RAILS_DEFAULT_LOGGER.level = Logger::WARN
logger = Logger.new(STDOUT)
logger.level = Logger::WARN
end
Here is the code that is using my logger:
def is_valid_checksum?(csv_arr)
expected_row_count = csv_arr[0][3].to_i
logger.debug "Expected record count: #{expected_row_count}"
actual_row_count = csv_arr.nitems - 1
logger.debug "Actual record count: #{actual_row_count}"
checksum_valid = false
if expected_row_count == actual_row_count
logger.debug "Checksum is valid"
checksum_valid = true
end
return checksum_valid
end
But this still does not solve the error
You can use the Rails logger outside of models and controllers:
Rails.logger.info "..."
Source
The logger method isn't available to test cases instance methods.
You have defined a local variable in your class definition but this isn't enough. The logger variable falls out of scope once the class is initialized. You may be looking for a class variable (##logger). But a cleaner solution would be wrapping it in a method like this.
class Test::Unit::TestCase
def logger
RAILS_DEFAULT_LOGGER ||= Logger.new(STDOUT)
end
end
Notice this code will use the default logger if it is available (which it should be). If this isn't desired, you can make your own just as easily.
def logger
#logger ||= Logger.new(STDOUT)
end
you should use RAILS_DEFAULT_LOGGER.debug the constant in your test case is_valid_checksum? not the variable logger(class level variable ) which cannot be used in a instance method.
I would suggest to wrap the code in a instance method something like
def logger
logger = Logger.new(STDOUT)
logger.level = Logger::WARN
logger
end
and then use logger
I have a Rails application that has an action invoked frequently enough to be inconvenient when I am developing, as it results in a lot of extra log output I don't care about. How can I get rails not to log anything (controller, action, parameters, completion time, etc.) for just this one action? I'd like to conditionalize it on RAILS_ENV as well, so logs in production are complete.
Thanks!
You can silence the Rails logger object:
def action
Rails.logger.silence do
# Things within this block will not be logged...
end
end
Use lograge gem.
Gemfile:
gem 'lograge'
config/application.rb:
config.lograge.enabled = true
config.lograge.ignore_actions = ['StatusController#nginx', ...]
The following works with at least Rails 3.1.0:
Make a custom logger that can be silenced:
# selective_logger.rb
class SelectiveLogger < Rails::Rack::Logger
def initialize app, opts = {}
#app = app
#opts = opts
#opts[:silenced] ||= []
end
def call env
if #opts[:silenced].include?(env['PATH_INFO']) || #opts[:silenced].any? {|silencer| silencer.is_a?( Regexp) && silencer.match( env['PATH_INFO']) }
Rails.logger.silence do
#app.call env
end
else
super env
end
end
end
Tell Rails to use it:
# application.rb
config.middleware.swap Rails::Rack::Logger, SelectiveLogger, :silenced => ["/remote/every_minute", %r"^/assets/"]
The example above shows silencing asset serving requests, which in the development environment means less ( and sometimes no) scrolling back is required to see the actual request.
The answer turns out to be a lot harder than I expected, since rails really does provide no hook to do this. Instead, you need to wrap some of the guts of ActionController::Base. In the common base class for my controllers, I do
def silent?(action)
false
end
# this knows more than I'd like about the internals of process, but
# the other options require knowing even more. It would have been
# nice to be able to use logger.silence, but there isn't a good
# method to hook that around, due to the way benchmarking logs.
def log_processing_with_silence_logs
if logger && silent?(action_name) then
#old_logger_level, logger.level = logger.level, Logger::ERROR
end
log_processing_without_silence_logs
end
def process_with_silence_logs(request, response, method = :perform_action, *arguments)
ret = process_without_silence_logs(request, response, method, *arguments)
if logger && silent?(action_name) then
logger.level = #old_logger_level
end
ret
end
alias_method_chain :log_processing, :silence_logs
alias_method_chain :process, :silence_logs
then, in the controller with the method I want to suppress logging on:
def silent?(action)
RAILS_ENV == "development" && ['my_noisy_action'].include?(action)
end
You can add the gem to the Gemfile silencer.
gem 'silencer', '>= 1.0.1'
And in your config/initializers/silencer.rb :
require 'silencer/logger'
Rails.application.configure do
config.middleware.swap Rails::Rack::Logger, Silencer::Logger, silence: ['/api/notifications']
end
The following works with Rails 2.3.14:
Make a custom logger that can be silenced:
#selective_logger.rb
require "active_support"
class SelectiveLogger < ActiveSupport::BufferedLogger
attr_accessor :silent
def initialize path_to_log_file
super path_to_log_file
end
def add severity, message = nil, progname = nil, &block
super unless #silent
end
end
Tell Rails to use it:
#environment.rb
config.logger = SelectiveLogger.new config.log_path
Intercept the log output at the beginning of each action and (re)configure the logger depending on whether the action should be silent or not:
#application_controller.rb
# This method is invoked in order to log the lines that begin "Processing..."
# for each new request.
def log_processing
logger.silent = %w"ping time_zone_table".include? params[:action]
super
end
With Rails 5 it gets more complicated request processing is logged in several classes. Firstly we need to override call_app in Logger class, let's call this file lib/logger.rb:
# original class:
# https://github.com/rails/rails/blob/master/railties/lib/rails/rack/logger.rb
require 'rails/rack/logger'
module Rails
module Rack
class Logger < ActiveSupport::LogSubscriber
def call_app(request, env) # :doc:
unless Rails.configuration.logger_exclude.call(request.filtered_path)
instrumenter = ActiveSupport::Notifications.instrumenter
instrumenter.start "request.action_dispatch", request: request
logger.info { started_request_message(request) }
end
status, headers, body = #app.call(env)
body = ::Rack::BodyProxy.new(body) { finish(request) }
[status, headers, body]
rescue Exception
finish(request)
raise
ensure
ActiveSupport::LogSubscriber.flush_all!
end
end
end
end
Then follow with lib/silent_log_subscriber.rb:
require 'active_support/log_subscriber'
require 'action_view/log_subscriber'
require 'action_controller/log_subscriber'
# original class:
# https://github.com/rails/rails/blob/master/actionpack/lib/action_controller/log_subscriber.rb
class SilentLogSubscriber < ActiveSupport::LogSubscriber
def start_processing(event)
return unless logger.info?
payload = event.payload
return if Rails.configuration.logger_exclude.call(payload[:path])
params = payload[:params].except(*ActionController::LogSubscriber::INTERNAL_PARAMS)
format = payload[:format]
format = format.to_s.upcase if format.is_a?(Symbol)
info "Processing by #{payload[:controller]}##{payload[:action]} as #{format}"
info " Parameters: #{params.inspect}" unless params.empty?
end
def process_action(event)
return if Rails.configuration.logger_exclude.call(event.payload[:path])
info do
payload = event.payload
additions = ActionController::Base.log_process_action(payload)
status = payload[:status]
if status.nil? && payload[:exception].present?
exception_class_name = payload[:exception].first
status = ActionDispatch::ExceptionWrapper.status_code_for_exception(exception_class_name)
end
additions << "Allocations: #{event.allocations}" if event.respond_to? :allocations
message = +"Completed #{status} #{Rack::Utils::HTTP_STATUS_CODES[status]} in #{event.duration.round}ms"
message << " (#{additions.join(" | ")})" unless additions.empty?
message << "\n\n" if defined?(Rails.env) && Rails.env.development?
message
end
end
def self.setup
# unsubscribe default processors
ActiveSupport::LogSubscriber.log_subscribers.each do |subscriber|
case subscriber
when ActionView::LogSubscriber
self.unsubscribe(:action_view, subscriber)
when ActionController::LogSubscriber
self.unsubscribe(:action_controller, subscriber)
end
end
end
def self.unsubscribe(component, subscriber)
events = subscriber.public_methods(false).reject { |method| method.to_s == 'call' }
events.each do |event|
ActiveSupport::Notifications.notifier.listeners_for("#{event}.#{component}").each do |listener|
if listener.instance_variable_get('#delegate') == subscriber
ActiveSupport::Notifications.unsubscribe listener
end
end
end
end
end
# subscribe this class
SilentLogSubscriber.attach_to :action_controller
SilentLogSubscriber.setup
Make sure to load modified modules e.g. in config/application.rb after loading rails:
require_relative '../lib/logger'
require_relative '../lib/silent_log_subscriber'
Finally configure excluded paths:
Rails.application.configure do
config.logger_exclude = ->(path) { path == "/health" }
end
As we're modifying core code of Rails it's always good idea to check original classes in Rails version you're using.
If this looks like too many modifications, you can simply use lograge gem which does pretty much the same with few other modifications. Although the Rack::Loggger code has changed since Rails 3, so you might be loosing some functionality.
#neil-stockbridge 's answer not worked for Rails 6.0, I edit some to make it work
# selective_logger.rb
class SelectiveLogger
def initialize app, opts = {}
#app = app
#opts = opts
#opts[:silenced] ||= []
end
def call env
if #opts[:silenced].include?(env['PATH_INFO']) || #opts[:silenced].any? {|silencer| silencer.is_a?( Regexp) && silencer.match( env['PATH_INFO']) }
Rails.logger.silence do
#app.call env
end
else
#app.call env
end
end
end
Test rails app to use it:
# application.rb
config.middleware.swap Rails::Rack::Logger, SelectiveLogger, :silenced => ["/remote/every_minute", %r"^/assets/"]
Sprockets-rails gem starting from version 3.1.0 introduces implementation of quiet assets. Unfortunately it's not flexible at this moment, but can be extended easy enough.
Create config/initializers/custom_quiet_assets.rb file:
class CustomQuietAssets < ::Sprockets::Rails::QuietAssets
def initialize(app)
super
#assets_regex = %r(\A/{0,2}#{quiet_paths})
end
def quiet_paths
[
::Rails.application.config.assets.prefix, # remove if you don't need to quiet assets
'/ping',
].join('|')
end
end
Add it to middleware in config/application.rb:
# NOTE: that config.assets.quiet must be set to false (its default value).
initializer :quiet_assets do |app|
app.middleware.insert_before ::Rails::Rack::Logger, CustomQuietAssets
end
Tested with Rails 4.2
Rails 6. I had to put this in config/application.rb, inside my app's class definition:
require 'silencer/logger'
initializer 'my_app_name.silence_health_check_request_logging' do |app|
app.config.middleware.swap(
Rails::Rack::Logger,
Silencer::Logger,
app.config.log_tags,
silence: %w[/my_health_check_path /my_other_health_check_path],
)
end
That leaves the log_tags config intact and modifies the middleware before it gets frozen. I would like to put it in config/initializers/ somewhere tucked away but haven't figured out how to do that yet.