I need to build a background job that goes through a list of RSS feeds and analyze them say every 10 minutes.
I have been using delayed_job for handling background jobs and I liked it a lot. I believe though that it's not built for recurring background jobs. I guess I can auto-schedule background job at the end of everyone (maybe with begin..rescue just to ensure it gets executes). Or preschedule say a month of advance worth of jobs and have another one that reschedule the every month..etc
This raised some concerned to me as I started asking myself: what if the server goes down in the middle of execution and the jobs didn't get scheduled?
I have also looked at Daemons gems which seemed the like it runs simple Ruby scripts with start/stop commands. I like the way delayed_job schedules and handles retries.
What do you recommend using in this case? What do you think the best way to design such a system with recurring background jobs? Also do you know a way I can monitor that background process and get notified if it stops?
I just implemented delayed_job for a similar task (using :run_at => 2.days.from_now) and found it to be a perfect fit. The easiest way to handle your concern about a process failing is to make the first step of the job to create the next job. Also, you can create a has_many relationship to the delayed_job model which would allow you to access the :last_error. Or, look at the "Hooks" section of readme and it has a perfect example for failure.
I think that this was a similar question: A cron job for rails: best practices? - not only are there answers, but also links to railscasts about background jobs in rails.
I used cron + delayed_job, but scheduled tasks were supposed to run few times a day, mostly just once.
Take a look at SimpleWorker. It's an elastic scheduling and background processing worker queue. It's cloud based and has persistence and redundancy so you don't need to worry if your servers go down or are restarted.
Very flexible in terms of scheduling, provides great introspection of jobs in the queue as well as notifications on status and errors.
Full disclosure: I work at SimpleWorker.
Related
I'm writting webtool using Ruby+Rails. I decided to ask about already existing solution before inventing mine.
Task: I need a pool of background jobs, that will run periodicaly (user sets his interval). There are plenty solutions like Resque or Sidekiq, but they provide "one-time" jobs. User can create new task, that gets into "pool".
Are there any solutions for this?
One simple solution might be to use cron to enqueue background jobs.
Another option might be to use an extension to Sidekiq that handles periodical jobs: sidekiq-scheduler for example.
My application (in Rails) should provide two main tasks:
create activity (for example post to Twitter) in given time in the future
periodically crawler some site or download Tweets
I'm thinking about using DelayedJob gem or Whenever gem for cron tasks. Which is better in these situations?
Thanks for any advice.
create activity (for example post to Twitter) in given time in the future
If this is in response to an event/user action, then a background task would be ideal, as it's not a regular schedule. Ruby toolbox seems to be favouring Resque and Sidekiq over delayed job though, so have a look at those before settling.
periodically crawler some site or download Tweets
Converse to the above, if this is a regularly scheduled event then cron jobs are ideal. Nothing wrong with using whenever for this, but make sure you have some form of monitoring on the jobs to alert you if something goes wrong.
For creating the activity, I would definitely use a background job to do that (i.e. PostToTwitterJob). I have a preference for Sidekiq instead DelayedJob, as delayed_job does not seem to be active anymore. If you want the job to be performed at a specific time in the future, check out sidekiq's documentation.
As for the crawling, I would use both. Your crawler would be a background job (i.e. TweetsCrawlerJob), and you could launch it every hour with whenever.
I'm working on the billing system for my app, but I'm not sure how to go about setting up scripts that run daily, one to send out payment reminders (email), and another to downgrade subscriptions that have not been renewed.
Any tips on how to do this?
Any gotchas I need to watch out for?
Any gems that I might be able to use?
I've completed most of the purchasing side already, so am not looking for gems like paypal_recurring, or stripe - just need to handle the payment reminders and dealing of accounts that have expired.
I've done this a number of ways and there are a bunch of ways to do it. I think the current best practice is to use Resque to queue the jobs and Rescue Scheduler to schedule them.
Resque is a solid 'job scheduling' app that can handle all kinds of tasks for you. Resque Scheduler can be fed full cron expressions (in addition to other methods of scheduling tasks) to manage the jobs.
One of the advantages of this approach is that you get the Resque Web app that gives you a web app to use for monitoring the jobs (or launching them for one-off job runs).
I've used this approach with Heroku and it works well and has been reliable.
Whenever at https://github.com/javan/whenever is a nice way to define scheduled tasks using cron. Or you can use cron directly: http://www.ameravant.com/posts/recurring-tasks-in-ruby-on-rails-using-runner-and-cron-jobs
I am trying to find out the best way to run scripts in the background. I have been looking around and found plenty of options, but many/most seem to have become inactive in the past few years. Let me describe my needs.
The rails app is basically a front-end to configure when and how these scripts will be run. The scripts run and generate reports and send email alerts. So the user must be able to configure the start times and how often these scripts will run dynamically. The scripts themselves should have access to the rails environment in order to save the resulting reports in the DB.
Just trying to figure out the best method from the myriad of options.
I think you're looking for a background job queuing system.
For that, you're either looking for resque or delayed_job. Both support scheduling tasks at some point in the future -- delayed_job does this natively, whereas resque has a plugin for it called resque_scheduler.
You would enqueue jobs in the background with parameters that you specify, and then at the time you selected they'll be executed. You can set jobs to recur indefinitely or a fixed number of times (at least with resque-scheduler, not sure about delayed_job).
delayed_job is easier to set up since it saves everything in the database. resque is more robust but requires you to have redis in your stack -- but if you do already it's pretty much the ideal solution for your problem.
I recently learned about Sidekiq, and I think it is really great.
There's also a RailsCast about it - Sidekiq.
Take a look at the gem whenever at https://github.com/javan/whenever.
It allows you to schedule tasks like cron jobs.
Works very well under linux, and the last commit was 14 days ago. A friend of mine used it in a project and was pretty satisfied with it.
edit: take a look at the gem delayed_job as well, it is good for executing long tasks in the background. Useful when creating a cron job only to start other tasks.
What is the preferred way to create a background task for a Rails application? I've heard of Starling/Workling and the good ol' script/runner, but I am curious which is becoming the defacto way to manage this need?
Thanks!
Clarification: I like the idea of Rake in Background, but the problem is, I need something that is running constantly or every 10 hours. I am not going to have the luxury of sitting on a web request, it will need to be started by the server asynchronous to the activities occurring on my site.
Ryan Bates created three great screencasts that might really help you:
Rake in Background
Starling and Workling
Custom Daemon
He talks about the various pros and cons for using each one. This should help you get started.
It depends on your needs.
Try out delayed_job, which was created by Tobi delayed_job (last updated 2011), a Shopify founder.
There are forks by DHH deleayed_job (last updated 2008), and collectiveidea delayed_job (last updated 20 days ago as of 6/28/2018).
I usually rely on cronjob scheduling as it gives the flexibility without having to write separate code to schedule it. Anything that can be executed from shell, can be scheduled! Be it any script (ruby / rake task / py / bash / any other you like), cronjob scheduling can be easily achieved.
If running on windows, one can use scheduled tasks
Hope this helps.
async_observer is the best. It doesn't do all kinds of dumb busy wait stuff or lose jobs on worker crashes like starling, no DB polling, etc... and it integrates into rails remarkably well.
I push tons of jobs through it and it pretty much doesn't care.
Most of the plugins that have been mentioned will do the job, but if all you need is a Rake task run on a set schedule, then there's really no need to start throwing more architecture at it.
Just add a cron job which executes
"cd /path/to/rails/app; RAILS_ENV=production rake run:my:task"
Why reinvent the wheel, when Unix like operating systems have been running tasks on a schedule for decades?
I have used the daemons plugin in the past.
While I don't know if it is becoming a standard, I have had great success with BackgroundRB. I have several workers, some are long running tasks triggered by a user action while others are started on a schedule.
Have a look at Taskr. It's basically like cron, but with a RESTful web interface. You can use it to schedule tasks to periodically connect to your Rails app and trigger arbitrary code (via the Taskr4rails plugin). It's meant to fit nicely into a system built around RESTful services, plus it can notify you if a task returns an error, fails to run, etc.