I am trying to add couple of scheduled workers to my rails application. These workers will be crawling different sites in given intervals.
I want to test these workers but not able to do it. I am starting redis and my application. What should I do to see whether my scheduled jobs are working or not?
Here is my crawler class:
class AyedasCrawler
include Sidekiq::Worker
and my sidekiq.yml is:
:schedule:
ayedas_crawler:
cron: '0 * * * * *' # Runs once per minute
class: AyedasCrawler
start the sidekiq worker and the scheduler processes by running
bundle exec sidekiq or sidekiq from your app root in the command-line.
sidekiq-scheduler provides an extension to the Sidekiq web interface that adds a Recurring Jobs page.
There are two ways to do this:
In your routes.rb file, just below the require 'sidekiq/web', add require 'sidekiq-scheduler/web'
In your config.ru, just below the require 'sidekiq/web', add
require 'sidekiq-scheduler/web'
run Sidekiq::Web
On the browser, goto ==> http://localhost:{port}/sidekiq/recurring-jobs. where {port} is the port your application is running in.
You will see the list of scheduled jobs for your application and some other details about it.
Read more in the official documentation
You need to run Sidekiq process as well.
bundle exec sidekiq
It will start both worker/s and the scheduler
If you wish to test it using rspec, you can to the following:
it 'spawns scheduled workers' do
Sidekiq::Cron::Job.load_from_hash YAML.load_file('config/sidekiq.yml')[: schedule]
Sidekiq::Cron::Job.all.each(&:enque!)
expect(AyedasCrawler.jobs.size).to be(1)
end
It loads the YAML configuration, enqueues all the jobs, and asserts if the job has been enqued.
Using this method you can validate if your schedule YAML is correct. It will NOT test CRON syntax and scheduled intervals.
I'm also using https://github.com/philostler/rspec-sidekiq to allow sidekiq testing without jobs actually being executed.
Related
I am on Rails 4 using the Resque Scheduler gem.
I am also using the sitemap generator gem in order to dynamically generate my sitemap.
I am having trouble figuring out the best way to schedule a rake task with resque scheduler. The sitemap generator recommends whenever, but I am assuming resque scheduler can accomplish the same thing (don't want to install another gem if I don't have to).
Does anyone know how to set this up?
I would like to run rake sitemap:refresh:no_ping every 5 hours.
I was thinking I would just schedule a background job and run it from there:
# resque_schedule.yml
update_sitemap:
every: 5h
class: "SitemapUpdater"
description: "This job refreshes the sitemap"
# sitemap_updater.rb
class SitemapUpdater
#queue = :sitemap_queue
def self.perform
# run rake task here
end
end
... however, I'm not sure if this is a good practice. Any advice would be much appreciated.
I don't see a problem with your approach, you just must be aware that the scheduler is reset during every deployment, so if you do frequent deploys, your scheduled jobs might be run later or even not run at all, as documented:
IMPORTANT: Rufus every syntax will calculate jobs scheduling time starting from the moment of deploy, resulting in resetting schedule time on every deploy, so it's probably a good idea to use it only for frequent jobs (like every 10-30 minutes), otherwise - when you use something like every 20h and deploy once-twice per day - it will schedule the job for 20 hours from deploy, resulting in a job to never be run.
You might also run the rake from system cron itself, which is an even more lightweight solution as it requires no scheduler gems at all, just the rake task, and will be scheduled reliably in time.
See e.g. this answer for setting up the "every 5 hours" frequency in crontab and you might also need to study RVM wrappers if you use RVM for your ruby project (you must call rake using the RVM wrappers in such case, e.g. call /home/deploy/.rvm/wrappers/ruby-2.3.0#mygemset/rake instead of just rake).
I'm trying to use cron in my application to send mails every week but I think it doesn't work on Windows.
Does anybody knows any equivalent to cron solution that works on Windows?
Windows equivalent of Unix's cron is a "Task Scheduler". You can configure your periodical task there.
Purely Ruby solution
If you want a purely Ruby solution look into:
rufus-scheduler - it's Windows cron gem.
crono - it's a in-Rails cron scheduler, so it should work anywhere.
Web services - there are plenty of free online services that would make a request to a given URL in specific time periods. This is basically a poor man's cronjob.
I recommend taking a look at Resque and the extension Resque-scheduler gems. You will need to have a resque scheduler process running with bundle exec rake resque:scheduler and at least one worker process running with QUEUE=* bundle exec rake resque:work.
If you want these services to run in the background as a windows service, you can do it with srvany.exe as described in this SO question.
The above assumes you are ok with installing Redis - a key-value store that is very popular among the Rails community as it can be easily used to support other Rails components such as caching and ActionCable, and it is awesome by itself for many multi-process use cases.
Resque is a queue system on top of Redis that allows you to define jobs that can be executed asynchronously in the background. When you run QUEUE=* bundle exec rake resque:work, a worker process runs constantly and polls the queue. Once a job is enqueued, an available worker pops it from the queue and starts working on it. This architecture is quite scalable, as you can have multiple workers listening to the queues if you'd like.
To define a job, you do this:
class MyWeeklyEmailSenderJob
def self.perform
# Your code to send weekly emails
end
end
While you can enqueue this job to the queue yourself from anywhere (e.g. from a controller as a response to an action), in your case you want it to automatically be placed into the queue once a week. This is what Resque-scheduler is for. It allows you to configure a file such as app/config/resque_schedule.yml in which you can define which jobs should be enqueued in which time interval. For example:
send_weekly_emails:
cron: 0 8 * * Mon
class: MyWeeklyEmailSenderJob
queue: email_sender_queue
description: "Send weekly emails"
Remember that a scheduling process has to run in order for this to work with bundle exec rake resque:scheduler.
thanks guys , actually i tried rufus scheduler gem and it worked for me , i guess it's the best and easier solution
Running: sidekiq 2.1.17, rails 3.2.21
I'm trying to use clockwork to schedule some recurring tasks to be done by sidekiq workers. Running foreman in development, everything runs perfectly as scheduled.
When I deploy to heroku, however, I get the "Triggering 'NameWorker.perform_async'" message in the logs at the appropriate times but then the respective jobs don't actually run.
When I instead call NameWorker.perform_async in a controller action, the job runs as it should. The jobs I'm trying to schedule just contain puts statements to verify that they're working. Anyone have any ideas about what I'm doing wrong? Thanks in advance.
Solved by adding the Sidekiq.configure_client logic from the sidekiq.rb file into clock.rb. This logic was already there and in unicorn but sidekiq was not being initialized in production for the purposes of the clockwork process.
Cronjob vs Scheduler for Heroku.
Sounds like you just need a periodic task to run once per day. The currently recommended way to do that at Heroku is to use the scheduler add-on:
Scheduler is an add-on for running jobs on your app at scheduled time intervals, much like cron in a traditional server environment.
A dashboard allows you to configure jobs to run every 10 minutes, every hour, or every day, at a specified time. When invoked, these jobs will run as one-off processes and show up in your logs as a process named run.N.
The basic process is pretty simple:
Set up a Rake task to do whatever it is you need to do:
desc "This task does ..."
task :your_task => :environment do
# Do something interesting...
end
Add the scheduler add-on in the usual manner:
$ heroku addons:add scheduler:standard
Then add your new Rake task through the Heroku dashboard.
If you want to run your periodic task outside Heroku then set up a cron job to run the appropriate Rake task.
You probably want to put this task into a Cron job which handles recurring processes or tasks.
Here is an excellent screencast on the subject courtesy of Ryan Bates:
http://railscasts.com/episodes/164-cron-in-ruby-revised
If you need to set up recurring jobs on Heroku, you need only add your tasks to lib/tasks/scheduler.rake and configure them using the Schedular Add-on
http://devcenter.heroku.com/articles/scheduler
I created an app that uses the whenever gem. The gem creates cron jobs. I got it working locally but can't seem to get it working on heroku cedar. What's the command to do this?
running:
heroku run whenever --update-crontab job1
doesn't work
Short answer: use the scheduler add-on: http://addons.heroku.com/scheduler
Long answer: When you do heroku run, we
spin up a dyno
put your code on it
execute your command, wait for it to finish
throw the dyno away
Any changes you made to crontab would be immediately thrown away. Everything is ephemeral, you cannot edit files on heroku, just push new code.
You need to add Heroku Scheduler addon.
You can add it directly from your dashboard or using following commands:
install the add-on:
heroku addons:create scheduler:standard
Create a rake task in lib/tasks
# lib/tasks/scheduler.rake
task :send_reminders => :environment do
User.send_reminders
end
Schedule job
Visit Heroku Dashboard
Open your app
Select Scheduler from add-ons list
Click Add Job, enter a task and select frequency.
e.g. Add rake send_reminders, select "Daily" and "00:00" to send reminders every day at midnight.
The other answers specify you should use the Heroku Scheduler add-on, and it is able to run a background tasks indeed, but it doesn't support the flexibility of cron.
There's another add-on, called Cron To Go, that is able to run your jobs on one-off dynos with cron's flexibility. You can also specify a timezone for your job and get notifications (email or webhook) when job fail, succeed or start.
(Full disclosure - I work for the company that created and operates Cron To Go)
If you want to:
Use Heroku Scheduler
Run tasks every minute (not 10 min)
Don't care about dyno hours
This was my solution hack to run jobs every minute - assuming the task completes in under 60 seconds.
task start_my_service: :environment do
1.upto(9) do |iteration|
start_time = DateTime.now
Services::MyService.call
end_time = DateTime.now
wait_time = 60 - ((end_time - start_time) * 24 * 60 * 60).to_i
sleep wait_time if wait_time > 0
end
end
Heroku doesn't support cron jobs. And there are two drawbacks to the Heroku Scheduler :
you cannot choose an arbitrary interval or time at which to run jobs (it's either every 10 mins, 1 hour or daily).
your jobs are not defined in code, hence not in your versioning system and not easy to keep track of or modify.
Heroku does provide an alternative : custom clock processes. But the clock process requires its own dyno, and "Since dynos are restarted at least once a day some logic will need to exist on startup of the clock process to ensure that a job interval wasn’t skipped during the dyno restart".
Simple scheduler is a gem made specifically made for scheduling on Heroku, but seems a bit hackish.
I ended up using sidekiq-cron. Only drawback : if sidekiq is down right when a job is scheduled to run, the job won't run.