I want run an scheduler at every saturday at 11.45pm, But if use whenever gem it will not support in heroku, How can we write Heroku Scheduler with Custom Intervals?
Answer:
Simple Scheduler is a scheduling add-on that is designed to be used with Sidekiq and Heroku Scheduler.It gives you the ability to schedule tasks at any interval without adding a clock process. Heroku Scheduler only allows you to schedule tasks every 10 minutes, every hour, or every day.
STEP 1: gem "simple_scheduler"
STEP 2: bundle
STEP 3: Create the file config/simple_scheduler.yml in your Rails project:
# Global configuration options. These can also be set on each task.
queue_ahead: 360 # Number of minutes to queue jobs into the future
tz: nil # The application time zone will be used by default if not set
# Runs once every 2 minutes
simple_task:
class: "SomeActiveJob"
every: "2.minutes"
# Runs once every day at 4:00 AM. The job will expire after 23 hours, which means the
# job will not run if 23 hours passes (server downtime) before the job is actually run
overnight_task:
class: "SomeSidekiqWorker"
every: "1.day"
at: "4:00"
expires_after: "23.hours"
# Runs once every half hour, starting on the 30 min mark
half_hour_task:
class: "HalfHourTask"
every: "30.minutes"
at: "*:30"
# Runs once every week on Saturdays at 12:00 AM
weekly_task:
class: "WeeklyJob"
every: "1.week"
at: "Sat 0:00"
tz: "America/New_York"
STEP 4: Add the rake task to Heroku Scheduler and set it to run every 10 minutes: rake simple_scheduler
For more information you can follow below link,
https://github.com/simplymadeapps/simple_scheduler
This can be easily achieved. Say we have a task named example_task, we can do this:
example_task:
class: "ExampleJob"
every: "1.week"
at: "Sat 23:45"
Note that ExampleJob has to extend ActiveJob::Base and implement the perform method which will hold the logic for your task.
Related
I'm using sidekiq cron to run some jobs. I have a parent job which only runs once, and that parent job starts 7 million child jobs. However, in my sidekiq dashboard, it says over 42 million jobs enqueued. I checked those enqueued jobs, they are my child jobs. I'm trying to figure out why so many more jobs than expected are enqueued. I checked the log in sidekiq, one thing I noticed is, "Cron Jobs - add job with name: new_topic_post_job" shows up many times in the log. new_topic_post is the name of the parent job in schedule.yml. Following lines also show up many times
2019-04-18T17:01:22.558Z 12605 TID-osb3infd0 WARN: Processing recovered job from queue queue:low (queue:low_i-03933b94d1503fec0.nodemodo.com_4): "{\"retry\":false,\"queue\":\"low\",\"backtrace\":true,\"class\":\"WeeklyNewTopicPostCron\",\"args\":[],\"jid\":\"f37382211fcbd4b335ce6c85\",\"created_at\":1555606809.2025042,\"locale\":\"en\",\"enqueued_at\":1555606809.202564}"
2019-04-18T17:01:22.559Z 12605 TID-osb2wh8to WeeklyNewTopicPostCron JID-f37382211fcbd4b335ce6c85 INFO: start
WeeklyNewTopicPostCron is the name of the parent job class. Wondering does this mean my parent job runs multiple times instead of only 1? If so, what's the cause? I'm pretty sure the time in the cron job is right, I set it to "0 17 * * 4" which means it only runs once a week. Also I set retry to false for parent job and 3 for child jobs. So even all child jobs fail, we should still only have 21 million jobs. Following is my cron job setting in schedule.yml
new_topic_post_job:
cron: "0 17 * * 4"
class: "WeeklyNewTopicPostCron"
queue: low
and this is WeeklyNewTopicPostCron:
class WeeklyNewTopicPostCron
include Sidekiq::Worker
sidekiq_options queue: :low, retry: false, backtrace: true
def perform
processed_user_ids = Set.new
TopicFollower.select("id, user_id").find_in_batches(batch_size: 1000000) do |topic_followers|
new_user_ids = []
topic_followers.map(&:user_id).each { |user_id| new_user_ids << user_id if processed_user_ids.add?(user_id) }
batch_size = 1000
offset = 0
loop do
batched_user_ids_for_redis = new_user_ids[offset, batch_size]
Sidekiq::Client.push_bulk('class' => NewTopicPostSender,
'args' => batched_user_ids_for_redis.map { |user_id| [user_id, 7] }) if batched_user_ids_for_redis.present?
break if batched_user_ids_for_redis.size < batch_size
offset += batch_size
end
end
end
end
Most probably your parent sidekiq job is causing the sidekiq process to crash, which then results in a worker restart. On restart sidekiq probably tries to recover the interrupted job and starts processing it again (from the beginning). Some details here:
https://github.com/mperham/sidekiq/wiki/Reliability#recovering-jobs
This probably happens multiple times before the parent job eventually finishes, and hence the extremely high number of child jobs are created. You can easily verify this by checking the process id of the sidekiq process while this job is being run and it most probably will keep changing after a while:
ps aux | grep sidekiq
It could be that you have some monit configuration to restart sidekiq in case memory usage goes too high.Or it might be that this query is causing the process to crash:
TopicFollower.select("id, user_id").find_in_batches(batch_size: 1000000)
Try reducing the batch_size. 1million feels like too high a number. But my best guess is that the sidekiq process dies while processing the long running parent process.
I am trying to run a cron job in every 10 seconds that runs a piece of code. I have used an approach which requires running a code and making it sleep for 10 seconds, but it seems to make drastically degrading the app performance. I am using whenever gem, which run every minute and sleeps for 10 seconds. How can I achieve the same w/o using sleep method. Following is my code.
every 1.minute do
runner "DailyNotificationChecker.send_notifications"
end
class DailyNotificationChecker
def self.send_notifications
puts "Triggered send_notifications"
expiry_time = Time.now + 57
while (Time.now < expiry_time)
if RUN_SCHEDULER == "true" || RUN_SCHEDULER == true
process_notes
end
sleep 10 #seconds
end
def self.process_notes
notes = nil
time = Benchmark.measure do
Note.uncached do
notes = Note.where(status: false)
notes.update_all(status: true)
end
end
puts "time #{time}"
end
end
Objective of my code is to change the boolean status of objects to true which gets checked every 10 seconds. This table has 2 million records.
I suggest using a Sidekiq background jobs for this. With the sidekiq-scheduler gem you can run ordinary sidekiq jobs schedules in whatever internal you need. Bonus points for having a web-interface to handle and monitor the jobs via the Sidekiq gem.
You would use the clockwork gem. It runs in a separate process. The configuration is pretty simple.
require 'clockwork'
include Clockwork
every(10.seconds, 'frequent.job') { DailyNotificationChecker.process_notes }
I am new to Rails, I am using whenever to gem to run cron job.In my application, I have some jobs that can be performed by drivers.The jobs has start and end time and it can be assigned to drivers.By cron job I need to check some conditions, like a
1. job is assigned and start within 30 minutes.
2. job is unassigned and start within 30 minutes.
3. job is assigned and start within 15 minutes.
I added the conditions to .rake file and mentioned the jobs in schedule file, below I paste my code snippets.
schedule.rb
every 1.minutes do
rake "jobs:unassigned_job_start_in_30"
end
every 1.minutes do
rake "jobs:assigned_job_start_in_30"
end
every 1.minutes do
rake "jobs:assigned_job_start_in_15"
end
and my rake file
auto_expire.rake
desc "assigned job starts in 30 mins."
task :assigned_job_start_in_30 => :environment do
start_range = Time.now.utc.to_time.to_i+(CONFIG[:job_starts_in_time_30]*60)
end_range = start_range + (1*60)
jobs = Job.where("start_at >= :start_range AND start_at <= :end_range AND status =:status", {start_range: start_range.to_s, end_range: end_range.to_s, status: 2000} )
jobs.each do |job|
job.add_event_log(nil, 2054)
create_and_send_notification(2054, job.id, 0, nil)
end
end
desc "assigned job starts in 15 mins."
task :assigned_job_start_in_15 => :environment do
start_range = Time.now.utc.to_time.to_i+ (CONFIG[:job_starts_in_time_15]*60)
end_range = start_range + (1*60)
jobs = Job.where("start_at >= :start_range AND start_at <= :end_range AND status =:status", {start_range: start_range.to_s, end_range: end_range.to_s, status: 2000} )
jobs.each do |job|
job.add_event_log(nil, 2055)
create_and_send_notification(2055, job.id, 0, nil)
end
end
As the cron jobs performed with 1 minutes duration, so my machine goes hang.Please suggest me what is the best way to check the conditions and performed cron jobs.
The cron jobs should run in the background, so your machine shouldn't hang while running it, there might be other reasons this is happening.
An alternative, is to just have one single task that runs, that checks and creates the tasks for a separate system that runs it. One example would be Sidekiq.
I am using Rails ActiveJob with Sidekiq.
I have a Job that is supposed to execute after 5 seconds.
UserArrivalJob.set(wait: 5.seconds).perform_later(user, planet)
Only after 5 seconds the job still hasnt ran.
When i look in the sidekiq web interface after those 5 seconds the job is there and it says: Not yet enqueued. After about another 6 till 10 seconds the job gets enqeued and is immediatly executed.
How come that there is this delay?
When i use perform now this delay is not there.
Here is my Job:
class UserArrivalJob < ActiveJob::Base
queue_as :default
def perform(user, planet)
user.planet = planet
user.save
end
end
Read here. Basically I think your sidekiq poller runs every 10 seconds and it picks the job when it pools.
bcd was right. I set the sidekiq configuration to run the poller every 2 seconds.
environments/development.rb / environments/production.rb
Sidekiq.configure_server do |config|
config.average_scheduled_poll_interval = 2
end
I can't figure why my job is not triggering nightly. In Jenkins have 2 jobs set up intended to function in the following way
Job 1: DataCheck
Runs every hour runs a check to see if data is consistent.
Job 2: MoveDataToProduction
Every Day at 8pm moves data to production as long as the most recent DataCheck job has passed
I'm using a BuildResultTrigger in MoveDataToProduction , with the Job to monitor DataCheck. The Job Build result is SUCCESS. The schedule is H 20 * * *.
Now, I can see the BuildResultTrigger running a check at 8pm, but even if the DataCheck job last run is success the MoveDataToProduction does not run. Here's all I see:
Polling started on Nov 19, 2014 8:00:00 PM
Polling for the job MoveDataToProduction
Recording context. Check changes in next poll.
Polling complete. Took 0 ms.
No changes.
Will the SUCCESS of DataCheck only be picked up if it changed from a failure state or something? Ideally, my DataCheck should never fail, but I still want the MoveDataToProduction to trigger.