Let's say we have test job:
class TestJob < ActiveJob::Base
queue_as :default
def perform(video)
video.process
end
end
After it we run bundle exec sidekiq start
And run in new terminal window in rails console
Video.first.pending!; TestJob.perform_later(Video.first)
Job is running, ffmpeg is on background, everything is fine, but according to official sidekiq wiki docs I try:
require 'sidekiq/api'
Sidekiq::Queue.new
=> #<Sidekiq::Queue:0x00000006406978 #name="default", #rname="queue:default">
Sidekiq::Queue.new.each {|job| puts job}
=> nil
Sidekiq::Queue.new.size
=> 0
ss = Sidekiq::ScheduledSet.new
=> #<Sidekiq::ScheduledSet:0x000000064a4330 #_size=0, #name="schedule">
ss.size
=> 0
Why there is no jobs? The job is running successfully ( I see it also in first window where sidekiq starts, but I can't see and delete it in rails console )
I am using Ubuntu 14 if it helps
With best regards, Ruslan.
UPD:
It looks that
ps = Sidekiq::ProcessSet.new; ps.each(&:quiet!)
works
, but it doesn't stop my ffmpeg process this part of code internally in process:
cmd = "ffmpeg -i #{input_file.shellescape} #{options} -threads 0 -y #{self.path + outfile}"
pid = spawn(cmd, :out => output_file, :err => output_file)
Process.wait(pid)
How to stop it?
If a job is processing, it's not enqueued anymore.
I use sidekiq web UI and various queues for my jobs. If I realize that particular job is failing I can clear out the queue of subsequent jobs of the same class.
Related
I have a Rails application that runs jobs on the background using the Resque adapter. I have noticed that once a couple of days my workers disappear (just stop), my jobs get stuck in the queue, and I have to restart the workers anew every time they stop.
I check using ps -e -o pid,command | grep [r]esque and launch workers in the background using
(RAILS_ENV=production PIDFILE=./resque.pid BACKGROUND=yes bundle exec rake resque:workers QUEUE='*' COUNT='12') 2>&1 | tee -a log/resque.log.
Then I stopped redis-server using /etc/init.d/redis-server stop and again checked the worker processes. They disappeared.
This gives a reason to think that worker processes stop maybe because the redis server restarting because of some reason.
Is there any Rails/Ruby way solution to this problem? What comes to my mind is writing a simple Ruby code that would watch the worker processes with the period, say, 5 seconds, and restart them if they stop.
UPDATE:
I don't want to use tools such as Monit, God, eye, and etc. They are not reliable. Then I will need to watch them too. Something like to install God to manage Resque workers, then install Monit to watch God, ...
UPDTAE
This is what I am using and it is really working. I manually stoped redis-server and then started it again. This script successfully launched the workers.
require 'logger'
module Watch
def self.workers_dead?
processes = `ps -e -o pid,command | grep [r]esque`
return true if processes.empty?
false
end
def self.check(time_interval)
logger = Logger.new('watch.log', 'daily')
logger.info("Starting watch")
while(true) do
if workers_dead?
logger.warn("Workers are dead")
restart_workers(logger)
end
sleep(time_interval)
end
end
def self.restart_workers(logger)
logger.info("Restarting workers...")
`cd /var/www/agts-api && (RAILS_ENV=production PIDFILE=./resque.pid BACKGROUND=yes rake resque:workers QUEUE='*' COUNT='12') 2>&1 | tee -a log/resque.log`
end
end
Process.daemon(true,true)
pid_file = File.dirname(__FILE__) + "#{__FILE__}.pid"
File.open(pid_file, 'w') { |f| f.write Process.pid }
Watch.check 10
You can use process monitoring tools such as monit, god, eye etc. These tools can check for resque PID and memory usage at time interval specified by you. You also have options to restart background processes if the memory limit exceeds your specified expectations. Personally, I use eye gem.
You could do it much simpler. Start Resque in foreground. When it exits, start it again. No pid files, no monitoring, no sleep.
require 'logger'
class Restarter
def initialize(cmd:, logger: Logger.new(STDOUT))
#cmd = cmd
#logger = logger
end
def start
loop do
#logger.info("Starting #{#cmd}")
system(#cmd)
#logger.warn("Process exited: #{#cmd}")
end
end
end
restarter = Restarter.new(
cmd: 'cd /var/www/agts-api && (RAILS_ENV=production rake resque:workers QUEUE='*' COUNT='12') 2>&1 | tee -a log/resque.log',
logger: Logger.new('watch.log', 'daily')
)
restarter.start
I have Capistrano deploying my app to a Ubuntu remote server on a cloud host. It works except that Sidekiq does not get restarted. After a deploy new Sidekiq jobs are stuck in the queue until it does finally get restarted. I currently manually SSH into the machine and run sudo initctl stop/start workers which works. I am not super strong at all with Capistrano and me research so far has failed to find me a solution to this. I am hoping I am missing something obvious to someone more familiar than me. Here is the relevant portion of my /config/deploy.rb file:
namespace :deploy do
namespace :sidekiq do
task :quiet do
on roles(:app) do
puts capture("pgrep -f 'workers' | xargs kill -USR1")
end
end
task :restart do
on roles(:app) do
execute :sudo, :initctl, :stop, :workers
execute :sudo, :initctl, :start, :workers
end
end
end
after 'deploy:starting', 'sidekiq:quiet'
after 'deploy:reverted', 'sidekiq:restart'
after 'deploy:published', 'sidekiq:restart'
end
UPDATE
From my reply logs:
DEBUG [268bc235] Running /usr/bin/env kill -0 $( cat /home/ubuntu/staging/shared/tmp/pids/sidekiq-0.pid ) as ubuntu#159.203.8.242
DEBUG [268bc235] Command: cd /home/ubuntu/staging/releases/20160806065537 && ( export RBENV_ROOT="$HOME/.rbenv" RBENV_VERSION="2.2.3" ; /usr/bin/env kill -0 $( cat /home/ubuntu/staging/shared/tmp/pids/sidekiq-0.pid ) )
DEBUG [268bc235] Finished in 0.471 seconds with exit status 1 (failed).
I don't believe you need those configs in your deploy.rb if you have the capistrano-sidekiq gem installed and called in your Capfile.
Make sure you have require 'capistrano/sidekiq' in your Capfile or it won't know to call the default tasks.
I have the following setup, an ubuntu server with nginx - passenger, postgres db, redis and sidekiq.
i have set up an upstart job to keep an eye on the sidekiq process.
description "Sidekiq Background Worker"
# no "start on", we don't want to automatically start
stop on (stopping workers or runlevel [06])
# change to match your deployment user
setuid deploy
setgid deploy
respawn
respawn limit 3 30
# TERM and USR1 are sent by sidekiqctl when stopping sidekiq. Without declaring these as normal exit codes, it just respawns.
normal exit 0 TERM USR1
script
# this script runs in /bin/sh by default
# respawn as bash so we can source in rbenv
exec /bin/bash <<EOT
# use syslog for logging
exec &> /dev/kmsg
# pull in system rbenv
export HOME=/home/deploy
export RAILS_ENV=production
source /home/deploy/.rvm/scripts/rvm
cd /home/deploy/myapp/current
bundle exec sidekiq -c 25 -L log/sidekiq.log -P tmp/pids/sidekiq.pid -q default -q payment -e production
EOT
end script
This actually work i can start, stop and restart the process. i can see the process over at the sidekiq web http monitor. but when i run this workers workers they get processed. Here are both workers:
class EventFinishedWorker
include Sidekiq::Worker
def perform(event_id)
event = Event.find(event_id)
score_a = event.team_A_score
score_b = event.team_B_score
event.bets.each do |bet|
if (score_a == bet.team_a_score) && (score_b == bet.team_b_score)
bet.guessed = true
bet.save
end
end
end
end
and
class PayPlayersBetsWorker
include Sidekiq::Worker
sidekiq_options :queue => :payment
def perform(event_id)
event = Event.find(event_id)
bets = event.bets.where(guessed: true)
total_winning_bets = bets.count
unless total_winning_bets == 0
pot = event.pot
amount_to_pay = pot / total_winning_bets
unless bets.empty?
bets.each do |bet|
account = bet.user.account
user = User.find_by_id(bet.user.id)
bet.payed = true
bet.save
account.wincoins += amount_to_pay
account.accum_wincoins.increment(amount_to_pay)
account.save
user.set_score(user.current_score)
end
end
end
end
end
2014-05-27T18:48:34Z 5985 TID-7agy8 EventFinishedWorker JID-33ef5d7e7de51189d698c1e7 INFO: start
2014-05-27T18:48:34Z 5985 TID-5s4rg PayPlayersBetsWorker JID-e8473aa1bc59f0b958d23509 INFO: start
2014-05-27T18:48:34Z 5985 TID-5s4rg PayPlayersBetsWorker JID-e8473aa1bc59f0b958d23509 INFO: done: 0.07 sec
2014-05-27T18:48:35Z 5985 TID-7agy8 EventFinishedWorker JID-33ef5d7e7de51189d698c1e7 INFO: done: 0.112 sec
I get no errors running both workers processes, but ...
cheking my data, EventFinishedWorker, does its job. no problems whatsoever. the second job PayPlayersBetsWorker doesnt work. but when i go into the console and do something like.
worker = PayPlayersBetsWorker.new
worker.perform(1)
it works it runs the jobs flawlessly. i have detected also that if i run sidekiq from the console directly not using upstart it works too.
bundle exec sidekiq -P tmp/pids/sidekiq.pid -q default -q payment -e production
this works. the problem seems to be running sidekiq with upstart. help would be appreciated :D
It looks like you have a race condition between the two jobs. EFW isn't finished before PPBW queries, doesn't find anything and immediately exits.
I have a Rails app deployed on Heroku with the Heroku scheduler add-on successfully working for daily jobs.
Now I want a weekly job, but the scheduler add-on does not allow me a weekly option.
Any suggestions on how I could accomplish this:
I've tried using rufus-scheduler in the past but it caused me some problems, so that's not an option. See here for detail.
I'm thinking of something along the lines of checking for current day within a daily job. Has anyone tried it and have feedback or know of issues with the approach?
Other ideas much appreciated.
One approach would be the one of your 2nd bullet point:
activate the Heroku cron add-on, and add a cron.rake task in app/lib/tasks
Activate the Heroku scheduler add-on, and add a scheduler.rake task in app/lib/tasks
task :your_weekly_task=> :environment do
if Time.now.friday? # previous answer: Date.today.wday == 5
#do something
end
end
You even have the luxury of defining the day you want your task to run ;o)
(5 is for Friday)
EDIT: by the way, Cron is deprecated and Heroku recommends switching to their Scheduler add-on. This doesn't change the approach for a weekly job though.
EDIT2: adjustments to my answer based on feedback from sunkencity.
An alternate option using only shell code. Setup the Heroku scheduler hourly, and do a comparison against the date command:
# setting the schedular to run hourly at *:30 is equivalent to the
# crondate: 30 8 * * 1
if [ "$(date +%H)" = 08 ] && [ "$(date +%d)" = 01 ]; then YOUR_COMMAND ; fi
I've used this code to mimic cron in my local time zone:
nz_hour="$(TZ=NZ date +%H)" ; nz_day="$(TZ=NZ date +%d)" ; if [ "$nz_hour" = 08 ] && [ "$nz_day" = 01 ]; then YOUR_COMMAND ; fi
It's not ideal, but I've taken to adding a RUN_IF environment variable to rake tasks run through heroku:scheduler which lets me weekly and monthly schedules for jobs.
# lib/tasks/scheduler.rake
def run?
eval ENV.fetch('RUN_IF', 'true')
end
def scheduled
if run?
yield
else
puts "RUN_IF #{ENV['RUN_IF'].inspect} eval'd to false: aborting job."
end
end
# lib/tasks/job.rake
task :job do
scheduled do
# ...
end
end
If a rake task is run without a RUN_IF variable it will run. Otherwise, the job will be aborted unless the value of RUN_IF evals to a falsey value.
$ rake job # => runs always
$ rake job RUN_IF='Date.today.monday?' # => only runs on Mondays
$ rake job RUN_IF='Date.today.day == 1' # => only runs on the 1st of the month
$ rake job RUN_IF='false' # => never runs (not practical, just demonstration)
Similar to other ideas above, but I prefer moving the scheduling details out of the application code.
As discussed over here, and using the above logic from Rob, here are the bash scripts broken down by a day of the week, once a month, and on a specific date.
Run a task every Monday:
if [ "$(date +%u)" = 1 ]; then MY_COMMAND; fi
Run a task every 1st day in a month:
if [ "$(date +%d)" = 01 ]; then MY_COMMAND; fi
You could also run a job every year on December 24th:
if [ "$(date +%m)" = 12 ] && [ "$(date +%d)" = 24 ]; then MY_COMMAND; fi
If you don't want or cannot do this with Rake, another option is to do this with Ruby from bash:
#!/bin/bash
cmd="echo your schedule job here"
# Only run on even days
ruby -e 'if Time.now.utc.day.even?; puts "Not today!"; exit 1; end' && $cmd "$#"
This works even for non-ruby projects (as in my case: Clojure).
I use a scheduler (Rufus scheduler) to launch a process called "ar_sendmail" (from ARmailer), every minute.
The process should NOT be launched when there is already such a process running in order not to eat up memory.
How do I check to see if this process is already running? What goes after the unless below?
scheduler = Rufus::Scheduler.start_new
scheduler.every '1m' do
unless #[what goes here?]
fork { exec "ar_sendmail -o" }
Process.wait
end
end
end
unless `ps aux | grep ar_sendmai[l]` != ""
unless `pgrep -f ar_sendmail`.split("\n") != [Process.pid.to_s]
This looks neater I think, and uses built-in Ruby module. Send a 0 kill signal (i.e. don't kill):
# Check if a process is running
def running?(pid)
Process.kill(0, pid)
true
rescue Errno::ESRCH
false
rescue Errno::EPERM
true
end
Slightly amended from Quick dev tips You might not want to rescue EPERM, meaning "it's running, but you're not allowed to kill it".