Whats the best way to test concurrency locally? i.e. i want to test 10 concurrent hits. I am aware of services like Blitz. However, I am trying to find a simpler way of doing it locally to test against race conditions.
Any ideas? Via Curl maybe?
Check out Apache Bench (ab). Basic usage is dead simple:
ab -n 100 -c 10 http://your.application
For locally testing race conditions in the tests you can use helpers like this
# call block in a forked process
def fork_with_new_connection(config, object = nil, options={})
raise ArgumentError, "Missing block" unless block_given?
options = {
:stop => true, :index => 0
}.merge(options)
fork do
# stop the process after fork
Signal.trap('STOP') if options[:stop]
begin
ActiveRecord::Base.establish_connection(config)
yield(object)
ensure
ActiveRecord::Base.remove_connection
end
end
end
# call multiply times blocks
def multi_times_call_in_fork(count=3, &block)
raise ArgumentError, "Missing block" unless block_given?
config = ActiveRecord::Base.remove_connection
pids = []
count.times do |index|
pids << fork_with_new_connection(config, nil, :index=>index, &block)
end
# continue forked processes
Process.kill("CONT", *pids)
Process.waitall
ActiveRecord::Base.establish_connection(config)
end
# example
multi_times_call_in_fork(5) do
# do something with race conditions
# add asserts
end
Related
I'm using rb-notify to monitor a directory within my Rspec tests for a Rails project. If the file isn't found then the inotify continues in an endless loop. Is there a way to tell it to give up after 10 seconds.
#notifier = INotify::Notifier.new
#notifier.watch(DownloadHelpers::PATH.to_s, :create) do |file|
puts "Check: #{file.name}"
if file.name.eql?(File.basename(#filename.to_s))
#pdf = PdfReader.new(#filename)
# Check PDF
expect(#pdf.page_count).to eq(1)
expect(#pdf.pages[1]).to have_text("CONTRACTORS")
#notifier.stop # Stops the infinite loop watch
end
end
visit "/#/contractors_report"
within "#heading" do
expect(page).to have_text("Contractors")
end
click_on("pdf-btn")
#notifier.run # Make sure we finish our thread
See the advanced usage section on inotify's readme on GitHub:
# Wait 10 seconds for an event then give up
if IO.select([notifier.to_io], [], [], 10)
notifier.process
end
I have a Rails application that runs some background jobs via ActiveJob and Sidekiq. The sidekiq logs in both the terminal and the log file show the following:
2016-10-18T06:17:01.911Z 3252 TID-oukzs4q3k ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-97318b38b1391672d21feb93 INFO: start
Is there some way to show the class names of the jobs here similar to how logs work for a regular Sidekiq Worker?
Update:
Here is how a Sidekiq worker logs:
2016-10-18T11:05:39.690Z 13678 TID-or4o9w2o4 ClientJob JID-b3c71c9c63fe0c6d29fd2f21 INFO: start
Update 2:
My sidekiq version is 3.4.2
I'd like to replace ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper with Client Job
So I managed to do this by removing Sidekiq::Middleware::Server::Logging from the middleware configuration and adding a modified class that displays the arguments in the logs. The arguments themself contain the job and action names as well.
For latest version, currently 4.2.3, in sidekiq.rb
require 'sidekiq'
require 'sidekiq/middleware/server/logging'
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def log_context(worker, item)
klass = item['wrapped'.freeze] || worker.class.to_s
"#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}"
end
end
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.remove Sidekiq::Middleware::Server::Logging
chain.add ParamsLogging
end
end
For version 3.4.2, or similar, override the call method instead:
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def call(worker, item, queue)
klass = item['wrapped'.freeze] || worker.class.to_s
Sidekiq::Logging.with_context("#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}") do
begin
start = Time.now
logger.info { "start" }
yield
logger.info { "done: #{elapsed(start)} sec" }
rescue Exception
logger.info { "fail: #{elapsed(start)} sec" }
raise
end
end
end
end
You must be running some ancient version. Upgrade.
Sorry, looks like that's a Rails 5+ feature only. You'll need to upgrade Rails. https://github.com/rails/rails/commit/8d2b1406bc201d8705e931b6f043441930f2e8ac
I have spent all day on this and I cannot figure out how to use multithreading in rails, without causing the following error:
Uncaught exception: Circular dependency detected while autoloading constant Docket::System::ExecutableFactory
/Users/myuser/.rvm/gems/ruby-2.1.2#core/gems/activesupport-4.1.5/lib/active_support/dependencies.rb:478:in `load_missing_constant'
/Users/myuser/.rvm/gems/ruby-2.1.2#core/gems/activesupport-4.1.5/lib/active_support/dependencies.rb:180:in `const_missing'
The code works without threads, but as soon as I introduce threads, that is when I get the "circular dependency" error.
Below is the code in question:
def run
records.in_groups_of(4) do |records|
workers = records.map do |record|
Thread.new do
begin
options[:record] = record
county = create_county options
county.prepare
if county.ready
county.crawl
end
rescue ThreadError => e
puts "THREAD ERROR: #{e}"
end
end
end
workers.map(&:join)
end
end
...
def crawl
cmd = "#{executable_factory.path} #{crawler_path}"
Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
stdout_is = stdout.read
stderr_is = stderr.read
end
end
Docket::System::ExecutableFactory is one of my classes, but I have it properly nested. Directory structure:
\docket
\system
executable_factory.rb
And I have it properly namedspace:
module Docket
module System
class ExecutableFactory
...
end
end
end
So why does this error occur with threads and how can I fix it?
What is the best way to test something that requires background jobs with Cucumber? I need to run DelayedJob and Sneakers workers in background while tests are running.
You can run any application in the background:
#pid = Process.spawn "C:/Apps/whatever.exe"
Process.detach(#pid)
And even kill it after tests are done:
Process.kill('KILL', #pid) unless #pid.nil?
You can create your own step definition in features/step_definitions/whatever_steps.rb (hopefully with a better name)
When /^I wait for background jobs to complete$/ do
Delayed::Worker.new.work_off
end
That can be extended for any other scripts you'd like to run with that step. Then in the test, it goes something like:
Then I should see the text "..."
When I wait for background jobs to complete
And I refresh the page
Then I should see the text "..."
If anyone has similar problem I ended up writing this (thanks to Square blog post):
require "timeout"
class CucumberExternalWorker
attr_accessor :worker_pid, :start_command
def initialize(start_command)
raise ArgumentError, "start_command was expected" if start_command.nil?
self.start_command = start_command
end
def start
puts "Trying to start #{start_command}..."
self.worker_pid = fork do
start_child
end
at_exit do
stop_child
end
end
private
def start_child
exec({ "RAILS_ENV" => Rails.env }, start_command)
end
def stop_child
puts "Trying to stop #{start_command}, pid: #{worker_pid}"
# send TERM and wait for exit
Process.kill("TERM", worker_pid)
begin
Timeout.timeout(10) do
Process.waitpid(worker_pid)
puts "Process #{start_command} stopped successfully"
end
rescue Timeout::Error
# Kill process if could not exit in 10 seconds
puts "Sending KILL signal to #{start_command}, pid: #{worker_pid}"
Process.kill("KILL", worker_pid)
end
end
end
This can be called as following (added it to env.rb for cucumber):
# start delayed job
$delayed_job_worker = CucumberExternalWorker.new("rake jobs:work")
$delayed_job_worker.start
Using the console, how can I tell if sidekiq is connected to a redis server? I want to be able to do something like this:
if (sidekiq is connected to redis) # psuedo code
MrWorker.perform_async('do_work', user.id)
else
MrWorker.new.perform('do_work', user.id)
end
You can use Redis info provided by Sidekiq:
redis_info = Sidekiq.redis { |conn| conn.info }
redis_info['connected_clients'] # => "16"
Took it from Sidekiq's Sinatra status app.
I make this method to Rails whit the obove answer, return true if connected and false if not.
def redis_connected?
!!Sidekiq.redis(&:info) rescue false
end
It sounds like you want to know if there is a Sidekiq process up and running to process jobs at a given point in time. With Sidekiq 3.0, you can do this:
require 'sidekiq/api'
ps = Sidekiq::ProcessSet.new
if ps.size > 0
MyWorker.perform_async(1,2,3)
else
MyWorker.new.perform(1,2,3)
end
Sidekiq::ProcessSet gives you almost real-time (updated every 5 sec) info about any running Sidekiq processes.
jumping off #overallduka's answer, for those using the okcomputer gem, this is the custom check i set up:
class SidekiqCheck < OkComputer::Check
def check
if sidekiq_accessible?
mark_message "ok"
else
mark_failure
end
end
private
def sidekiq_accessible?
begin
Sidekiq.redis { |conn| conn.info }
rescue Redis::CannotConnectError
end.present?
end
end
OkComputer::Registry.register "sidekiq", SidekiqCheck.new
begin
MrWorker.perform_async('do_work', user.id)
rescue Redis::CannotConnectError => e
MrWorker.new.perform('do_work', user.id)
end