I am new to this and is little confused about how Delayed Job works ?
I know it creates a table and puts the jobs in the table and then I need to run
rake jobs:work
to start the background process. Now my question is
Does DJ script checks the table every minute and when the time matches job_at time, it runs that job ?
How it is different than cron (whenever gem) if the script is just checking the table every min ?
Thanks
Does DJ script checks the table every minute and when the time matches job_at time, it runs that job ?
When you run rake jobs:work DelayedJob will poll the delayed_jobs table, performing jobs matching the job_at column value if it's been set. This part you're correct about.
How it is different than cron (whenever gem) if the script is just checking the table every min ?
whenever is a gem that helps you configure a crontab. It has nothing directly to do with performing tasks on your server on a periodic basis.
You could setup a cron to run whatever tasks exist in the queue every minute, but leaving a delayed_job daemon running has multiple benefits.
Even if the cron ran every minute, delayed_job's daemon will see and perform any jobs queued within that 1-minute window between cron runs
Every time the cron would run, it will rebuild a new Rails environment in which to perform the jobs. This is a waste of time and resources when the daemon can just sit there immediately ready to perform a newly queued job.
If you want to configure delayed_job through a cron every minute you can add something like this to your crontab
* * * * * RAILS_ENV=production script/delayed_job start --exit-on-complete
Every minute, delayed_job will spin up, perform whatever jobs are ready for it or which it must retry from a previously failed run, and then quit. I don't recommend this though. Setting up a delayed_job as a daemon is the right way to go.
Does DJ script checks the table every minute and when the time matches
job_at time, it runs that job ?
yes. It checks the database every 5 seconds.
How it is different than cron (whenever gem) if the script is just
checking the table every min ?
In the context of background jobs, they are not that different. Their main difference is how they usually run the jobs.
DJ | Crontab
uses additional database | you should either set up a rake task
table but that's it. easier | or a runner which can be called on the
to code compared to crontab | crontab
------------------------------|------------------------------------------
requires you to run a worker | requires you to setup your cron which
that will poll the database | you can easily do using the whenever gem
------------------------------|------------------------------------------
since this uses a table, it | you have to setup some sort of logging so
is easier to debug errors | that you have an idea what caused the error
when they happen |
------------------------------|------------------------------------------
the worker should always be | as long as your crontab is set up properly,
running to perform the job | you should have no issues
------------------------------|------------------------------------------
harder to setup recurring | easy to setup recurring tasks
tasks |
------------------------------|------------------------------------------
Related
I use a gem called whenever to manage my cron jobs.
In cronfile, I have every 1 minute cron job which call a task XXXX. My config/schedule.rb is like this：
every '* * * * *' do
rake "XXXXXXXX"
end
This cron job is working fine with make slight delay. Task XXXX starts to run its first line a few seconds after process is created. Since this task finishes in less than 1 minute, I should never have multiple processes at the same time.
However, the server is heavily loaded, this delay will become a few minutes.
This leads that many undone processes remain in my process list beacause cron job creates a process every minute.
This will cause the server to become heavier, if worst comes to worst, the server is completely dead.
why does it happen? How can I prevent cronjob from to delay calling a task?
you can add a dependent task to check whether the server is running or not, fail fast (a.k.a. fail early), for example i want to verify that my rails server is already started on port 3000 before call the rake :test
# Rakefile
task :check_localhost do
pid = system("lsof -i tcp:3000 -t")
fail unless pid # or you can use `abort('message')`
end
task :test => :check_localhost do
puts "****** THIS IS TEST ******"
end
So I'm using Rails and I have a few Sidekiq workers, but none are enabled. I'm using the sidekiq-cron gem, which requires you to put files in app/workers/, configure a sidekiq scheduler in config/sidekiq_schedule.yml, and also add a few lines in config/initializers/sidekiq.rb. However, I've commented everything out from sidekiq_schedule.yml and also commented the following lines out from sidekiq.rb:
# Sidekiq scheduler.
# schedule_file = 'config/sidekiq_schedule.yml'
# if File.exists?(schedule_file) && Sidekiq.server?
# Sidekiq::Cron::Job.load_from_hash! YAML.load_file(schedule_file)
# end
However, if I launch Sidekiq, every minute (which is the old schedule), I see this in the prompt:
2018-01-19T02:54:04.156Z 22197 TID-ovsidcme8 ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-8609429b89db2a91793509ea INFO: start
2018-01-19T02:54:04.164Z 22197 TID-ovsidcme8 ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-8609429b89db2a91793509ea INFO: fail: 0.008 sec
and it fails because it's trying to launch code a job that's not supposed to be launching.
I've went to the rails console prompt (rails -c) and tried to find the job, but nothing's in there:
irb(main):001:0> Sidekiq::Cron::Job.all
=> []
so I'm not quite sure why it's constantly trying to launch a job. If I go to the rails interface on my application, I don't see anything in the queue, nothing being processed, busy, retries, enqueued, nothing.
Any suggestions would be greatly appreciated. I've been trying to hunt this down for like the last hour and have no success. I even removed ALL of the workers from the workers directory, and yet it's still trying to launch one of them.
Because you have already load jobs, I think that those jobs configuration are still in REDIS. Checking this assumption by opening a new terminal tab with redis-cli:
KEYS '*cron*'
If there are those keys on REDIS, clear them will fix your issue.
Since you mentioned a cron job in your title but not in the question, I'm assuming there's a cronjob running the background sidekiq task.
Try running crontab - l in Terminal to see all your cron jobs. If you see something like "* * * * *", that means there's a job that is running every minute.
Then, use crontab - r to clear your cron tab and delete all scheduled tasks.
I am on Rails 4 using the Resque Scheduler gem.
I am also using the sitemap generator gem in order to dynamically generate my sitemap.
I am having trouble figuring out the best way to schedule a rake task with resque scheduler. The sitemap generator recommends whenever, but I am assuming resque scheduler can accomplish the same thing (don't want to install another gem if I don't have to).
Does anyone know how to set this up?
I would like to run rake sitemap:refresh:no_ping every 5 hours.
I was thinking I would just schedule a background job and run it from there:
# resque_schedule.yml
update_sitemap:
every: 5h
class: "SitemapUpdater"
description: "This job refreshes the sitemap"
# sitemap_updater.rb
class SitemapUpdater
#queue = :sitemap_queue
def self.perform
# run rake task here
end
end
... however, I'm not sure if this is a good practice. Any advice would be much appreciated.
I don't see a problem with your approach, you just must be aware that the scheduler is reset during every deployment, so if you do frequent deploys, your scheduled jobs might be run later or even not run at all, as documented:
IMPORTANT: Rufus every syntax will calculate jobs scheduling time starting from the moment of deploy, resulting in resetting schedule time on every deploy, so it's probably a good idea to use it only for frequent jobs (like every 10-30 minutes), otherwise - when you use something like every 20h and deploy once-twice per day - it will schedule the job for 20 hours from deploy, resulting in a job to never be run.
You might also run the rake from system cron itself, which is an even more lightweight solution as it requires no scheduler gems at all, just the rake task, and will be scheduled reliably in time.
See e.g. this answer for setting up the "every 5 hours" frequency in crontab and you might also need to study RVM wrappers if you use RVM for your ruby project (you must call rake using the RVM wrappers in such case, e.g. call /home/deploy/.rvm/wrappers/ruby-2.3.0#mygemset/rake instead of just rake).
I created an app that uses the whenever gem. The gem creates cron jobs. I got it working locally but can't seem to get it working on heroku cedar. What's the command to do this?
running:
heroku run whenever --update-crontab job1
doesn't work
Short answer: use the scheduler add-on: http://addons.heroku.com/scheduler
Long answer: When you do heroku run, we
spin up a dyno
put your code on it
execute your command, wait for it to finish
throw the dyno away
Any changes you made to crontab would be immediately thrown away. Everything is ephemeral, you cannot edit files on heroku, just push new code.
You need to add Heroku Scheduler addon.
You can add it directly from your dashboard or using following commands:
install the add-on:
heroku addons:create scheduler:standard
Create a rake task in lib/tasks
# lib/tasks/scheduler.rake
task :send_reminders => :environment do
User.send_reminders
end
Schedule job
Visit Heroku Dashboard
Open your app
Select Scheduler from add-ons list
Click Add Job, enter a task and select frequency.
e.g. Add rake send_reminders, select "Daily" and "00:00" to send reminders every day at midnight.
The other answers specify you should use the Heroku Scheduler add-on, and it is able to run a background tasks indeed, but it doesn't support the flexibility of cron.
There's another add-on, called Cron To Go, that is able to run your jobs on one-off dynos with cron's flexibility. You can also specify a timezone for your job and get notifications (email or webhook) when job fail, succeed or start.
(Full disclosure - I work for the company that created and operates Cron To Go)
If you want to:
Use Heroku Scheduler
Run tasks every minute (not 10 min)
Don't care about dyno hours
This was my solution hack to run jobs every minute - assuming the task completes in under 60 seconds.
task start_my_service: :environment do
1.upto(9) do |iteration|
start_time = DateTime.now
Services::MyService.call
end_time = DateTime.now
wait_time = 60 - ((end_time - start_time) * 24 * 60 * 60).to_i
sleep wait_time if wait_time > 0
end
end
Heroku doesn't support cron jobs. And there are two drawbacks to the Heroku Scheduler :
you cannot choose an arbitrary interval or time at which to run jobs (it's either every 10 mins, 1 hour or daily).
your jobs are not defined in code, hence not in your versioning system and not easy to keep track of or modify.
Heroku does provide an alternative : custom clock processes. But the clock process requires its own dyno, and "Since dynos are restarted at least once a day some logic will need to exist on startup of the clock process to ensure that a job interval wasn’t skipped during the dyno restart".
Simple scheduler is a gem made specifically made for scheduling on Heroku, but seems a bit hackish.
I ended up using sidekiq-cron. Only drawback : if sidekiq is down right when a job is scheduled to run, the job won't run.
I have a rails app with the whenever gem installed to setup cron jobs which invoke various rake tasks. For reasons unbeknownst to me, each rake task gets invoked twice at precisely the same time. So my db backup task backs up the db twice at 4:00am.
Inspecting crontab reveals correct syntax for all of the cron jobs, so I don't think this is an issue with the whenever gem not correctly configuring the cron jobs. Also confusing is that in both staging and production environments and can invoke tasks on the command line and they only run once.
Any thoughts on what would cause this? I'm at a complete loss troubleshooting wise.
The number of cron jobs that run depends on the number of application instances running in the server box. Are you have two instances of rails application running in the same server box?