In my Ruby on Rails project I have 3 workers. They are in /lib/workers/worker_a.rb, /lib/workers/worker_b.rb, and /lib/workers/worker_c.rb.
Worker A is defined like
class WorkerA
def run!
logger.info "Starting worker A"
loop do
begin
do workerAWork
sleep 1.5 if workerAWork
rescue => e
logger.error e
sleep 3
end
end
end
end
WorkerA.new.run! if __FILE__==$0
Worker B is defined very similarly:
class WorkerB
def run!
logger.info "Starting Worker B"
loop do
begin
do workerBWork
sleep 1.5 if workerBWork
rescue => e
logger.error e
sleep 3
end
end
end
end
WorkerA.new.run! if __FILE__==$0
Worker C again is defined like above.
All 3 workers are doing what the app needs them to do. However, during deployment using Docker, I would need to open 3 terminals and start each worker using bin/rails runner ./lib/workers/worker_a.rb,bin/rails runner ./lib/workers/worker_b.rb, bin/rails runner ./lib/workers/worker_c.rb.
If I create a shell script that looks like
bin/rails runner ./lib/workers/worker_a.rb
bin/rails runner ./lib/workers/worker_b.rb
bin/rails runner ./lib/workers/worker_c.rb
Only the first worker will start and the terminal will hang there as there is a loop. The loop will prevent workers B and C from starting.
Is there a way to write the script such that they can be started one after another?
Thanks!
Related
I have a Rails application that runs some background jobs via ActiveJob and Sidekiq. The sidekiq logs in both the terminal and the log file show the following:
2016-10-18T06:17:01.911Z 3252 TID-oukzs4q3k ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-97318b38b1391672d21feb93 INFO: start
Is there some way to show the class names of the jobs here similar to how logs work for a regular Sidekiq Worker?
Update:
Here is how a Sidekiq worker logs:
2016-10-18T11:05:39.690Z 13678 TID-or4o9w2o4 ClientJob JID-b3c71c9c63fe0c6d29fd2f21 INFO: start
Update 2:
My sidekiq version is 3.4.2
I'd like to replace ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper with Client Job
So I managed to do this by removing Sidekiq::Middleware::Server::Logging from the middleware configuration and adding a modified class that displays the arguments in the logs. The arguments themself contain the job and action names as well.
For latest version, currently 4.2.3, in sidekiq.rb
require 'sidekiq'
require 'sidekiq/middleware/server/logging'
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def log_context(worker, item)
klass = item['wrapped'.freeze] || worker.class.to_s
"#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}"
end
end
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.remove Sidekiq::Middleware::Server::Logging
chain.add ParamsLogging
end
end
For version 3.4.2, or similar, override the call method instead:
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def call(worker, item, queue)
klass = item['wrapped'.freeze] || worker.class.to_s
Sidekiq::Logging.with_context("#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}") do
begin
start = Time.now
logger.info { "start" }
yield
logger.info { "done: #{elapsed(start)} sec" }
rescue Exception
logger.info { "fail: #{elapsed(start)} sec" }
raise
end
end
end
end
You must be running some ancient version. Upgrade.
Sorry, looks like that's a Rails 5+ feature only. You'll need to upgrade Rails. https://github.com/rails/rails/commit/8d2b1406bc201d8705e931b6f043441930f2e8ac
def process_order
OrderProcess.delay(run_at:1.minutes.from_now).processing_order(params[:id])
redirect_to '/'
end
when process_order is executed
it will execute delayed job
class OrderProcess
def self.processing_order(order_id)
system("rails generate controller welcome index")
end
end
when we remove delay(run_at:1.minutes.from_now) and process_order then rails commands in system command are getting executed but when we run through delayed job rails commands are not getting executed
thanks in advance
What is the best way to test something that requires background jobs with Cucumber? I need to run DelayedJob and Sneakers workers in background while tests are running.
You can run any application in the background:
#pid = Process.spawn "C:/Apps/whatever.exe"
Process.detach(#pid)
And even kill it after tests are done:
Process.kill('KILL', #pid) unless #pid.nil?
You can create your own step definition in features/step_definitions/whatever_steps.rb (hopefully with a better name)
When /^I wait for background jobs to complete$/ do
Delayed::Worker.new.work_off
end
That can be extended for any other scripts you'd like to run with that step. Then in the test, it goes something like:
Then I should see the text "..."
When I wait for background jobs to complete
And I refresh the page
Then I should see the text "..."
If anyone has similar problem I ended up writing this (thanks to Square blog post):
require "timeout"
class CucumberExternalWorker
attr_accessor :worker_pid, :start_command
def initialize(start_command)
raise ArgumentError, "start_command was expected" if start_command.nil?
self.start_command = start_command
end
def start
puts "Trying to start #{start_command}..."
self.worker_pid = fork do
start_child
end
at_exit do
stop_child
end
end
private
def start_child
exec({ "RAILS_ENV" => Rails.env }, start_command)
end
def stop_child
puts "Trying to stop #{start_command}, pid: #{worker_pid}"
# send TERM and wait for exit
Process.kill("TERM", worker_pid)
begin
Timeout.timeout(10) do
Process.waitpid(worker_pid)
puts "Process #{start_command} stopped successfully"
end
rescue Timeout::Error
# Kill process if could not exit in 10 seconds
puts "Sending KILL signal to #{start_command}, pid: #{worker_pid}"
Process.kill("KILL", worker_pid)
end
end
end
This can be called as following (added it to env.rb for cucumber):
# start delayed job
$delayed_job_worker = CucumberExternalWorker.new("rake jobs:work")
$delayed_job_worker.start
A JRuby/Rails application with a rufus-scheduler job that runs daily. However, when the application runs into a Java Heap Space (Tomcat) the job stops running silently.
This apparently occurs only in Threadsafe mode.
Abstracted config/initializer/update_scheduler.rb
scheduler = Rufus::Scheduler.start_new
#Runs everyday at 7:30 AM - expected after all ETL/MV refreshes are executed
scheduler.cron("30 7 * * *", :tags => 'auto_update') do
begin
# do stuff
Rails.logger.info "log info and duration"
rescue Exception => s_e
Rails.logger.error "Scheduler Failed"
end
Rails.logger.flush
end
class << scheduler
def lwarn (&block)
Rails.logger.error("Scheduler Error: " + block.call)
end
end
MyApp::Application.config.scheduler = scheduler
I am using rescue to handle all the heavy lifting background tasks,
In my library/parsers/file.rb I have
Resque.enqueue(Hello)
This will redirect app/workers/file.rb where I have
class Hello
def self.perform(page)
.......
.......
end
rescue Exception => e
log "error: #{e}"
end
end
my lib/tasks/resque.rake file is
require "resque/tasks"
task "resque:setup" => :environment
I am able to queue the jobs buts when i try to execute the job using
rake resque:work QUEUE=*
it is throwing an error by saying
argument error
wrong number of arguments (0 for 1)
what am I doing wrong in this?
pjumble is exactly right, you're not passing the page.
Resque.enqueue(Hello, page_id)
enqueue takes the Job followed by the args which go into the perform action. If you had:
class Hello
def self.perform(page_number, page_foo, page_bar)
...
end
end
Then you would do this:
Resque.enqueue(Hello, page_number, page_foo, page_bar)