Sidekiq/Redis queuing a job that doesn't exist - ruby-on-rails

I built a simple test job for Sidekiq and added it to my schedule.yml file for Sidekiq Cron.
Here's my test job:
module Slack
class TestJob < ApplicationJob
queue_as :default
def perform(*args)
begin
SLACK_NOTIFIER.post(attachments: {"pretext": "test", "text": "hello"})
rescue Exception => error
puts error
end
end
end
end
The SLACK_NOTIFIER here is a simple API client for Slack that I initialize on startup.
And in my schedule.yml:
test_job:
cron: "* * * * *"
class: "Slack::TestJob"
queue: default
description: "Test"
So I wanted to have it run every minute, and it worked exactly as I expected.
However, I've now deleted the job file and removed the job from schedule.yml, and it still tries to run the job every minute. I've gone into my sidekiq dashboard, and I see a bunch of retries for that job. No matter how many times I kill them all, they just keep coming.
I've tried shutting down both the redis server and sidekiq several times. I've tried turning off my computer (after killing the servers, of course). It still keeps scheduling these jobs and it's interrupting my other jobs because it raises the following exception:
NameError: uninitialized constant Slack::TestJob
I've done a project-wide search for "TestJob", but get no results.
I only had the redis server open with this job for roughly 10 minutes...
Is there maybe something lingering in the redis database? I've looked into the redis-cli documentation, but I don't think any of it helps me.

Try calling $ FLUSHALL in the redis cli...
Other than that...
The sidekiq-cron documentation seems to expect that you check for the existence of schedule.yml explicitly...
#initializers/sidekiq.rb
schedule_file = "config/schedule.yml"
if File.exist?(schedule_file) && Sidekiq.server?
Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file)
end

The default for Sidekiq is to retry a failed job 25 times. Because the job doesn't exist, it fails... every time; Sidekiq only knows that the job fails (because attempting to perform a job that doesn't exist would raise an exception), so it flags it for retry... 25 times. Because you were scheduling that job every minute, you probably had a TON of these failed jobs queued up for retry. You can:
Wait ~3 weeks to hit the maximum number of retries
If you have a UI page set up for Sidekiq, you can see and clear these jobs in the Retries tab
Dig through the Redis CLI documentation for a way to identify these specific jobs and delete them (or flush the whole thing if you're not worried about losing other jobs)

Related

- Rails & cron & whenever / a few minutes delay when the server is heavily loaded

I use a gem called whenever to manage my cron jobs.
In cronfile, I have every 1 minute cron job which call a task XXXX. My config/schedule.rb is like this:
every '* * * * *' do
rake "XXXXXXXX"
end
This cron job is working fine with make slight delay. Task XXXX starts to run its first line a few seconds after process is created. Since this task finishes in less than 1 minute, I should never have multiple processes at the same time.
However, the server is heavily loaded, this delay will become a few minutes.
This leads that many undone processes remain in my process list beacause cron job creates a process every minute.
This will cause the server to become heavier, if worst comes to worst, the server is completely dead.
why does it happen? How can I prevent cronjob from to delay calling a task?
you can add a dependent task to check whether the server is running or not, fail fast (a.k.a. fail early), for example i want to verify that my rails server is already started on port 3000 before call the rake :test
# Rakefile
task :check_localhost do
pid = system("lsof -i tcp:3000 -t")
fail unless pid # or you can use `abort('message')`
end
task :test => :check_localhost do
puts "****** THIS IS TEST ******"
end

Rails - Old cron job keeps running, can't delete it

So I'm using Rails and I have a few Sidekiq workers, but none are enabled. I'm using the sidekiq-cron gem, which requires you to put files in app/workers/, configure a sidekiq scheduler in config/sidekiq_schedule.yml, and also add a few lines in config/initializers/sidekiq.rb. However, I've commented everything out from sidekiq_schedule.yml and also commented the following lines out from sidekiq.rb:
# Sidekiq scheduler.
# schedule_file = 'config/sidekiq_schedule.yml'
# if File.exists?(schedule_file) && Sidekiq.server?
# Sidekiq::Cron::Job.load_from_hash! YAML.load_file(schedule_file)
# end
However, if I launch Sidekiq, every minute (which is the old schedule), I see this in the prompt:
2018-01-19T02:54:04.156Z 22197 TID-ovsidcme8 ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-8609429b89db2a91793509ea INFO: start
2018-01-19T02:54:04.164Z 22197 TID-ovsidcme8 ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-8609429b89db2a91793509ea INFO: fail: 0.008 sec
and it fails because it's trying to launch code a job that's not supposed to be launching.
I've went to the rails console prompt (rails -c) and tried to find the job, but nothing's in there:
irb(main):001:0> Sidekiq::Cron::Job.all
=> []
so I'm not quite sure why it's constantly trying to launch a job. If I go to the rails interface on my application, I don't see anything in the queue, nothing being processed, busy, retries, enqueued, nothing.
Any suggestions would be greatly appreciated. I've been trying to hunt this down for like the last hour and have no success. I even removed ALL of the workers from the workers directory, and yet it's still trying to launch one of them.
Because you have already load jobs, I think that those jobs configuration are still in REDIS. Checking this assumption by opening a new terminal tab with redis-cli:
KEYS '*cron*'
If there are those keys on REDIS, clear them will fix your issue.
Since you mentioned a cron job in your title but not in the question, I'm assuming there's a cronjob running the background sidekiq task.
Try running crontab - l in Terminal to see all your cron jobs. If you see something like "* * * * *", that means there's a job that is running every minute.
Then, use crontab - r to clear your cron tab and delete all scheduled tasks.

Rails delayed_job start and stop

I'm using delayed_job on my project, and until now calling rake jobs:work in the console it worked.
But now I'm trying to stop the processing of jobs after a concret event, and continue processing after other event.
To simulate this behaviour I've created a script config/initializers/delayed_jobs.rb
puts "START DELAYED JOB PRE"
`script/delayed_job start`
puts "START DELAYED JOB POST"
And only the first puts is called, the server has mired in the instruction, and the web is not being showed.
How to call stop and start delayed_jobs getting the correct behaviour?
You can move it into an own thread:
puts "START DELAYED JOB PRE"
Thread.new do
`script/delayed_job start`
end
puts "START DELAYED JOB POST"
I would suggest using external daemon that would be responsible for keeping your workers up. I had quite good experience with resque-pool that is designed for resque queue. Usually such daemon would allow you pausing workers just by sending signal to a master process. It also makes deployment restarts and management quite easy.

"Whenever" gem running cron jobs on Heroku

I created an app that uses the whenever gem. The gem creates cron jobs. I got it working locally but can't seem to get it working on heroku cedar. What's the command to do this?
running:
heroku run whenever --update-crontab job1
doesn't work
Short answer: use the scheduler add-on: http://addons.heroku.com/scheduler
Long answer: When you do heroku run, we
spin up a dyno
put your code on it
execute your command, wait for it to finish
throw the dyno away
Any changes you made to crontab would be immediately thrown away. Everything is ephemeral, you cannot edit files on heroku, just push new code.
You need to add Heroku Scheduler addon.
You can add it directly from your dashboard or using following commands:
install the add-on:
heroku addons:create scheduler:standard
Create a rake task in lib/tasks
# lib/tasks/scheduler.rake
task :send_reminders => :environment do
User.send_reminders
end
Schedule job
Visit Heroku Dashboard
Open your app
Select Scheduler from add-ons list
Click Add Job, enter a task and select frequency.
e.g. Add rake send_reminders, select "Daily" and "00:00" to send reminders every day at midnight.
The other answers specify you should use the Heroku Scheduler add-on, and it is able to run a background tasks indeed, but it doesn't support the flexibility of cron.
There's another add-on, called Cron To Go, that is able to run your jobs on one-off dynos with cron's flexibility. You can also specify a timezone for your job and get notifications (email or webhook) when job fail, succeed or start.
(Full disclosure - I work for the company that created and operates Cron To Go)
If you want to:
Use Heroku Scheduler
Run tasks every minute (not 10 min)
Don't care about dyno hours
This was my solution hack to run jobs every minute - assuming the task completes in under 60 seconds.
task start_my_service: :environment do
1.upto(9) do |iteration|
start_time = DateTime.now
Services::MyService.call
end_time = DateTime.now
wait_time = 60 - ((end_time - start_time) * 24 * 60 * 60).to_i
sleep wait_time if wait_time > 0
end
end
Heroku doesn't support cron jobs. And there are two drawbacks to the Heroku Scheduler :
you cannot choose an arbitrary interval or time at which to run jobs (it's either every 10 mins, 1 hour or daily).
your jobs are not defined in code, hence not in your versioning system and not easy to keep track of or modify.
Heroku does provide an alternative : custom clock processes. But the clock process requires its own dyno, and "Since dynos are restarted at least once a day some logic will need to exist on startup of the clock process to ensure that a job interval wasn’t skipped during the dyno restart".
Simple scheduler is a gem made specifically made for scheduling on Heroku, but seems a bit hackish.
I ended up using sidekiq-cron. Only drawback : if sidekiq is down right when a job is scheduled to run, the job won't run.

Getting delayed_job to just work

I followed the railscast which uses CollectiveIdea's fork. I'm not able to get it to work. I created a new file in my /lib folder and included this
class Device
def deliver
#my long running method
end
handle_asynchronously :deliver
end
device = Device.new
device.deliver
I do a script/delayed_job and that forks an app instance. Now,
There's no job activity going on. Nothing in the delayed_jobs table and nothing in the logs. Am I missing something here?
How do I set the interval for which the method should be run? (Ex every 30 seconds)
I'm testing this in the development mode (Rails 2.3.2), and soon will be moving this into production.
Thanks !
Do you see a process for the script/delayed_job that you ran? Do a ps aux | grep delayed_job and see if there is a process running.
AFAIK, you cannot set any time intervals using Delayed Job.
As a first step to diagnose the problem:
Stop your job workers
Launch a delayed job
Check whether it is present in the database.

Resources