Running: sidekiq 2.1.17, rails 3.2.21
I'm trying to use clockwork to schedule some recurring tasks to be done by sidekiq workers. Running foreman in development, everything runs perfectly as scheduled.
When I deploy to heroku, however, I get the "Triggering 'NameWorker.perform_async'" message in the logs at the appropriate times but then the respective jobs don't actually run.
When I instead call NameWorker.perform_async in a controller action, the job runs as it should. The jobs I'm trying to schedule just contain puts statements to verify that they're working. Anyone have any ideas about what I'm doing wrong? Thanks in advance.
Solved by adding the Sidekiq.configure_client logic from the sidekiq.rb file into clock.rb. This logic was already there and in unicorn but sidekiq was not being initialized in production for the purposes of the clockwork process.
Related
I am trying to add couple of scheduled workers to my rails application. These workers will be crawling different sites in given intervals.
I want to test these workers but not able to do it. I am starting redis and my application. What should I do to see whether my scheduled jobs are working or not?
Here is my crawler class:
class AyedasCrawler
include Sidekiq::Worker
and my sidekiq.yml is:
:schedule:
ayedas_crawler:
cron: '0 * * * * *' # Runs once per minute
class: AyedasCrawler
start the sidekiq worker and the scheduler processes by running
bundle exec sidekiq or sidekiq from your app root in the command-line.
sidekiq-scheduler provides an extension to the Sidekiq web interface that adds a Recurring Jobs page.
There are two ways to do this:
In your routes.rb file, just below the require 'sidekiq/web', add require 'sidekiq-scheduler/web'
In your config.ru, just below the require 'sidekiq/web', add
require 'sidekiq-scheduler/web'
run Sidekiq::Web
On the browser, goto ==> http://localhost:{port}/sidekiq/recurring-jobs. where {port} is the port your application is running in.
You will see the list of scheduled jobs for your application and some other details about it.
Read more in the official documentation
You need to run Sidekiq process as well.
bundle exec sidekiq
It will start both worker/s and the scheduler
If you wish to test it using rspec, you can to the following:
it 'spawns scheduled workers' do
Sidekiq::Cron::Job.load_from_hash YAML.load_file('config/sidekiq.yml')[: schedule]
Sidekiq::Cron::Job.all.each(&:enque!)
expect(AyedasCrawler.jobs.size).to be(1)
end
It loads the YAML configuration, enqueues all the jobs, and asserts if the job has been enqued.
Using this method you can validate if your schedule YAML is correct. It will NOT test CRON syntax and scheduled intervals.
I'm also using https://github.com/philostler/rspec-sidekiq to allow sidekiq testing without jobs actually being executed.
I'm trying to use cron in my application to send mails every week but I think it doesn't work on Windows.
Does anybody knows any equivalent to cron solution that works on Windows?
Windows equivalent of Unix's cron is a "Task Scheduler". You can configure your periodical task there.
Purely Ruby solution
If you want a purely Ruby solution look into:
rufus-scheduler - it's Windows cron gem.
crono - it's a in-Rails cron scheduler, so it should work anywhere.
Web services - there are plenty of free online services that would make a request to a given URL in specific time periods. This is basically a poor man's cronjob.
I recommend taking a look at Resque and the extension Resque-scheduler gems. You will need to have a resque scheduler process running with bundle exec rake resque:scheduler and at least one worker process running with QUEUE=* bundle exec rake resque:work.
If you want these services to run in the background as a windows service, you can do it with srvany.exe as described in this SO question.
The above assumes you are ok with installing Redis - a key-value store that is very popular among the Rails community as it can be easily used to support other Rails components such as caching and ActionCable, and it is awesome by itself for many multi-process use cases.
Resque is a queue system on top of Redis that allows you to define jobs that can be executed asynchronously in the background. When you run QUEUE=* bundle exec rake resque:work, a worker process runs constantly and polls the queue. Once a job is enqueued, an available worker pops it from the queue and starts working on it. This architecture is quite scalable, as you can have multiple workers listening to the queues if you'd like.
To define a job, you do this:
class MyWeeklyEmailSenderJob
def self.perform
# Your code to send weekly emails
end
end
While you can enqueue this job to the queue yourself from anywhere (e.g. from a controller as a response to an action), in your case you want it to automatically be placed into the queue once a week. This is what Resque-scheduler is for. It allows you to configure a file such as app/config/resque_schedule.yml in which you can define which jobs should be enqueued in which time interval. For example:
send_weekly_emails:
cron: 0 8 * * Mon
class: MyWeeklyEmailSenderJob
queue: email_sender_queue
description: "Send weekly emails"
Remember that a scheduling process has to run in order for this to work with bundle exec rake resque:scheduler.
thanks guys , actually i tried rufus scheduler gem and it worked for me , i guess it's the best and easier solution
When I start sidekiq in my development environment (Rails 3.2), I use the following command:
bundle exec sidekiq
When I do this, sidekiq will execute all jobs that have been queued up when it was not running. e.g. If I have created a bunch of new user accounts during testing, it will try and send welcome emails to all of the fake accounts (my emails are sent from a sidekiq job).
Is there a way to start sidekiq and tell it to delete all pending jobs? That way I can turn it back on without worrying that it will try and run a bunch of jobs that don't need to run (since this is my dev environment).
I have looked in documentation, but can't find an answer, hopefully it's something simple I overlooked...
redis-cli flushall && bundle exec sidekiq
I found a solution: Using the sidekiq monitoring UI that comes with sidekiq (https://github.com/mperham/sidekiq/wiki/Monitoring), I'm able to view all queues (even when sidekiq is not running). Deleting the queue will remove all of the jobs in it, which solves the problem.
I had developed an application that has delayed job and cron job and I need to deploy it to Heroku. I realized my delayed job need Heroku adds-on, "Heroku Scheduler Standard" and it costs $34.50.
1) Does that mean I need to buy that so that my delayed and cron job will run automatically?I wonder if delayed job and cron job can be done by using Heroku schedular then why we still need javan/whenever and collectiveidea/delayed_job? Can I use these in Heroku preferably in free condition?
2) My cron job does not worked in Heroku. How do I run my whenever gem in heroku?
There appear to be a couple of thoughts intermixed in your question. I'll do my best to separate them:
a) DelayedJob on Heroku
One way to process jobs queued in DelayedJob (i.e. records sitting in the delayed_jobs table in your database) is to run the following rake task:
$ rake jobs:work
On heroku, this rake task is commonly run via a 'worker' process.
By default for rails apps, your heroku app should already come with a slider for 'workers' (initially set to 0) that will run the rake task above. Merely sliding this to '1' (which will cost you around $34.00 per month) will launch the DelayedJob dequeue process in the background on a dedicated heroku instance, with your codebase, independent of your web dynos.
Note that DelayedJob does NOT require Heroku's "Scheduler" add-on.
Also note that heroku charges by hours of processing per month. So, if you slide your 'worker' to '1' for one hour a month, and then slide it back to '0' again when you're done, you will be paying far, far less than the ~$34 charge. There is a way to slide workers on and off programatically too.
b) Cron on Heroku
As it stands, managing your own, customized cron file is not possible on heroku or other cloud-based app service providers. Everything is ephemeral, and deployed files cannot be altered by you or your code.
Heroku's Scheduler is the appropriate add-on in this case to replace cron. You should be able to set tasks in your heroku scheduler to do what your cron tasks are currently doing.
Last I checked, heroku's scheduler was free (see: https://addons.heroku.com/scheduler), so I'm not sure why you state it is $34.50.
Hope this helps.
I have a rails app with the whenever gem installed to setup cron jobs which invoke various rake tasks. For reasons unbeknownst to me, each rake task gets invoked twice at precisely the same time. So my db backup task backs up the db twice at 4:00am.
Inspecting crontab reveals correct syntax for all of the cron jobs, so I don't think this is an issue with the whenever gem not correctly configuring the cron jobs. Also confusing is that in both staging and production environments and can invoke tasks on the command line and they only run once.
Any thoughts on what would cause this? I'm at a complete loss troubleshooting wise.
The number of cron jobs that run depends on the number of application instances running in the server box. Are you have two instances of rails application running in the same server box?