I am looking for the best way to run a very long task in Heroku.
I use Ruby On Rails for my web application and I have a very long task that I want to run it every week on Sunday during the night. It takes around 15~20 minutes. I already have Rufus-Scheduler, but I am not sure it is the most effective solution.
I also find something about Backgrounding Tasks in Heroku with Delayed Job. But is it the best way to handle it ?
Thanks.
This is what I use for a job that I run every night: https://devcenter.heroku.com/articles/scheduler
It works really well if your job is configured as a rake task. The guide at the link shows you how to configure everything and even addresses long-running jobs.
Heroku does not recommend to run long-running jobs with Heroku Scheduler.
Heroku says,
Scheduled jobs are meant to execute short running tasks or enqueue longer running tasks into a background job queue. Anything that takes longer than a couple of minutes to complete should use a worker dyno to run.
So, in my opinion, the best approach would be to use Heroku Scheduler to run rake task for every kind of job (short or long) but if a task takes longer than a couple of minutes then I would simply create a Background Job within that rake task. That way the scheduler will never run longer than a couple of minutes.
Related
I'm a little confused about this. I have a couple of tasks that I would like to run asynchronously, for example my inventory sync integration. For this I have implemented delayed job, but I realize that I need to run rake jobs:work on Heroku for this. I can use the Heroku scheduler to run this rake task every 10 minutes. My question is; if I create rake tasks to run i.e. my inventory sync method, do I still need delayed job? My understanding is that heroku scheduler kicks off 'one off dynos'.
Instead of using delayed job, could I not just kick off the sync method directly since a separate dyno is used anyway? What is the added value of delayed job here?
Heroku's Scheduler replaces what cron would handle on a typical server. Delayed Job or Sidekiq are for processing jobs asynchronously from your app, not a timed schedule.
The reason you use a worker & run these jobs on the back-end is so that your server can return a response as soon as is possible rather than making the user wait for some potentially unnecessarily long running process to finish (lots of queries, outbound e-mail, external API requests, etc.).
Ex, scheduler can run analytics or updates from a script every hour or day, but delayed job can not.
I am not sure if this question is of the correct format for SO.
I have a rails app with deployment on Heroku. In development I am using the "crono" gem to send a simple email out every week to remind users of various things. I can see how to use this in production with Heroku. There is almost nothing on this in a google search. Therefore my question is: "what is the best way to implement a simple weekly job in a rails app that is deployed on Heroku. Heroku has "scheduler" but this is a "best effort" add on that can claim to be reliable (according to their documentation)
Thanks
There's two ways to achieve what you want:
1. Use the Heroku scheduler (I would)
Honestly it's just so simple to set up, I would use this. It's also extremely cheap because you only pay for the dyno while the job is running. After it's run Heroku destroys the dyno (it's a one off dyno)
The best way to implement it is to have you background jobs callable by a rake task, and simply have the scheduler call that.
If you have a time interval Heroku doesn't support, simply handle that in your code. For example if you want to send e-mails once a week, have a timestamp to record when the last email was sent. Run the scheduler once a day and just check to see if it's ok to send another email, if not do nothing.
2. Use some kind of scheduler gem
There's a bunch of them out there. For example, rufus.
In this case, you'd write your rufus jobs. You would then need to have a worker dyno always running for rufus, this is much more expensive.
You'd tell the dyno to run rufus and keep running rufus by specifying the command rufus needs in your procfile. Something like:
scheduler: rake rufus:scheduler # Add rake task for rufus scheduler process
(credit for the above snippet How to avoid Rufus scheduler being called as many times as there are no.of dynos?)
Is there any way to find out whether particular rake task is currently running from Rails controller for example? I have an extensive rake task, which takes 5-6 hours to finish.
I need to see status of that rake task from frontend web interface, like:
Task "some operation" is running...
Also it would be nice to be able to hard stop / run that rake task from within frontend web interface.
If found Railscast devoted to it, but the method described there allows only to run rake task from controller, but not to stop/see its status.
If your job is taking 5-6 hours to complete, you must run it in background. To run long running jobs in background you can use one of these:
Resque
Sidekiq
And for tracking status of your jobs, you can use corresponding status tracking gems:
ResqueStatus
SidekiqStatus
Personally, I prefer sidekiq for its efficiency.
NOTE: All the above mentioned gems has good enough docs. Kindly refer to those.
Ok, so this is probably evil, however.. here's the question! I want to run a pretty lightweight app on a shared environment (site5). Ideally I would like to use delayed_job for the ease of queueing the mails (~200+ every so often). However, being a shared environment they don't want background processes running all the time (fair enough).
So, my plan, such as it is, is to queue the mails using delayed job, and then every hour or something, spin up a cron job, send a few emails (10 or something small) and then kill the process. And repeat.
Q) Is there a rake jobs:works:1 equivalent task it'd be easy to setup? - pointer would be handy.
I'm quite open to "this is a terrible idea, don't even go there" being the answer.. in which case I might look at another queuing strategy... (or heroku hire-fire perhaps..)
You can get delayed job to process only a certain number of jobs by doing:
Delayed::Worker.new.work_off(10)
You could fire a script to do that from cron or use "rails runner":
rails runner -e production 'Delayed::Worker.new.work_off(10)'
I guess the main issue on whether it is a good idea or not is working out what small value is actually high enough to make sure you process all your jobs in a reasonable time-frame. Also, you've got the overhead of firing up the rails environment every time you want to process, or even check whether you should process, any jobs. That might cause problems in a shared environment if they are particularly strict on spikes of memory or CPU usage.
Why not skip the 'workers' (which are just daemons which look for work else sleep) and have your cron fire a custom rake task of 10.times { MailerJob.first.perform }
You'd just need to require you're app in the line before that so its loaded ofc.
If I need to create recurring tasks in delayed jobs, what is a clean solution? I have an import task that I want to run every 5 minutes, but I don't want to fire up rails/rake in order to tell it to create a Delayed job that can be picked up. If Rails is already running on a system, perhaps I can just create an HTTP request that will make the rails app fire off a DJ? I could put that cron task in a ruby script which runs every 5 minutes, making requests to a server, but without firing up rails. What do you think?
This fork of delayed_job has recurrence built right in (and there may be other forks that do the same):
http://github.com/andrewroth/delayed_job
The cron approach still seems to be quite clean. It need only involve running a rake invocation, not a full Rails server. Rake doesn't need Rails to be running to work.
However if you really don't like that approach then you could arrange for the recurring jobs to be re-queue themselves when they are being processed setting a run_at time to 5 minutes (or whatever) in the future. Obviously you'd need to prime the queue the first time and make sure the delayed_job server stays running
About that... I have the same desire,
I'm planning to have a cron to fire up a curl request at a specific route at my site every 5 minutes, so it runs a action with the result, and I'm gonna be pretty sure it only ran once.
My purposes includes: Awarding Badges and compiling some Averages