I'm writting webtool using Ruby+Rails. I decided to ask about already existing solution before inventing mine.
Task: I need a pool of background jobs, that will run periodicaly (user sets his interval). There are plenty solutions like Resque or Sidekiq, but they provide "one-time" jobs. User can create new task, that gets into "pool".
Are there any solutions for this?
One simple solution might be to use cron to enqueue background jobs.
Another option might be to use an extension to Sidekiq that handles periodical jobs: sidekiq-scheduler for example.
Related
My application (in Rails) should provide two main tasks:
create activity (for example post to Twitter) in given time in the future
periodically crawler some site or download Tweets
I'm thinking about using DelayedJob gem or Whenever gem for cron tasks. Which is better in these situations?
Thanks for any advice.
create activity (for example post to Twitter) in given time in the future
If this is in response to an event/user action, then a background task would be ideal, as it's not a regular schedule. Ruby toolbox seems to be favouring Resque and Sidekiq over delayed job though, so have a look at those before settling.
periodically crawler some site or download Tweets
Converse to the above, if this is a regularly scheduled event then cron jobs are ideal. Nothing wrong with using whenever for this, but make sure you have some form of monitoring on the jobs to alert you if something goes wrong.
For creating the activity, I would definitely use a background job to do that (i.e. PostToTwitterJob). I have a preference for Sidekiq instead DelayedJob, as delayed_job does not seem to be active anymore. If you want the job to be performed at a specific time in the future, check out sidekiq's documentation.
As for the crawling, I would use both. Your crawler would be a background job (i.e. TweetsCrawlerJob), and you could launch it every hour with whenever.
I'm working on the billing system for my app, but I'm not sure how to go about setting up scripts that run daily, one to send out payment reminders (email), and another to downgrade subscriptions that have not been renewed.
Any tips on how to do this?
Any gotchas I need to watch out for?
Any gems that I might be able to use?
I've completed most of the purchasing side already, so am not looking for gems like paypal_recurring, or stripe - just need to handle the payment reminders and dealing of accounts that have expired.
I've done this a number of ways and there are a bunch of ways to do it. I think the current best practice is to use Resque to queue the jobs and Rescue Scheduler to schedule them.
Resque is a solid 'job scheduling' app that can handle all kinds of tasks for you. Resque Scheduler can be fed full cron expressions (in addition to other methods of scheduling tasks) to manage the jobs.
One of the advantages of this approach is that you get the Resque Web app that gives you a web app to use for monitoring the jobs (or launching them for one-off job runs).
I've used this approach with Heroku and it works well and has been reliable.
Whenever at https://github.com/javan/whenever is a nice way to define scheduled tasks using cron. Or you can use cron directly: http://www.ameravant.com/posts/recurring-tasks-in-ruby-on-rails-using-runner-and-cron-jobs
I am trying to find out the best way to run scripts in the background. I have been looking around and found plenty of options, but many/most seem to have become inactive in the past few years. Let me describe my needs.
The rails app is basically a front-end to configure when and how these scripts will be run. The scripts run and generate reports and send email alerts. So the user must be able to configure the start times and how often these scripts will run dynamically. The scripts themselves should have access to the rails environment in order to save the resulting reports in the DB.
Just trying to figure out the best method from the myriad of options.
I think you're looking for a background job queuing system.
For that, you're either looking for resque or delayed_job. Both support scheduling tasks at some point in the future -- delayed_job does this natively, whereas resque has a plugin for it called resque_scheduler.
You would enqueue jobs in the background with parameters that you specify, and then at the time you selected they'll be executed. You can set jobs to recur indefinitely or a fixed number of times (at least with resque-scheduler, not sure about delayed_job).
delayed_job is easier to set up since it saves everything in the database. resque is more robust but requires you to have redis in your stack -- but if you do already it's pretty much the ideal solution for your problem.
I recently learned about Sidekiq, and I think it is really great.
There's also a RailsCast about it - Sidekiq.
Take a look at the gem whenever at https://github.com/javan/whenever.
It allows you to schedule tasks like cron jobs.
Works very well under linux, and the last commit was 14 days ago. A friend of mine used it in a project and was pretty satisfied with it.
edit: take a look at the gem delayed_job as well, it is good for executing long tasks in the background. Useful when creating a cron job only to start other tasks.
I have a web app (ASP.NET). There are some scheduled tasks and background tasks that need to be run regularly (for example email queue, search indexer...). My question is should I create a windows service to handle all those jobs, or separate ones for each job? What is best practice?
Thank you.
I think it would be better to create separate services for each job. This will help if any functionality related to a job causes problem and stops the service it won't affect the other job.
I need to build a background job that goes through a list of RSS feeds and analyze them say every 10 minutes.
I have been using delayed_job for handling background jobs and I liked it a lot. I believe though that it's not built for recurring background jobs. I guess I can auto-schedule background job at the end of everyone (maybe with begin..rescue just to ensure it gets executes). Or preschedule say a month of advance worth of jobs and have another one that reschedule the every month..etc
This raised some concerned to me as I started asking myself: what if the server goes down in the middle of execution and the jobs didn't get scheduled?
I have also looked at Daemons gems which seemed the like it runs simple Ruby scripts with start/stop commands. I like the way delayed_job schedules and handles retries.
What do you recommend using in this case? What do you think the best way to design such a system with recurring background jobs? Also do you know a way I can monitor that background process and get notified if it stops?
I just implemented delayed_job for a similar task (using :run_at => 2.days.from_now) and found it to be a perfect fit. The easiest way to handle your concern about a process failing is to make the first step of the job to create the next job. Also, you can create a has_many relationship to the delayed_job model which would allow you to access the :last_error. Or, look at the "Hooks" section of readme and it has a perfect example for failure.
I think that this was a similar question: A cron job for rails: best practices? - not only are there answers, but also links to railscasts about background jobs in rails.
I used cron + delayed_job, but scheduled tasks were supposed to run few times a day, mostly just once.
Take a look at SimpleWorker. It's an elastic scheduling and background processing worker queue. It's cloud based and has persistence and redundancy so you don't need to worry if your servers go down or are restarted.
Very flexible in terms of scheduling, provides great introspection of jobs in the queue as well as notifications on status and errors.
Full disclosure: I work at SimpleWorker.