My application (in Rails) should provide two main tasks:
create activity (for example post to Twitter) in given time in the future
periodically crawler some site or download Tweets
I'm thinking about using DelayedJob gem or Whenever gem for cron tasks. Which is better in these situations?
Thanks for any advice.
create activity (for example post to Twitter) in given time in the future
If this is in response to an event/user action, then a background task would be ideal, as it's not a regular schedule. Ruby toolbox seems to be favouring Resque and Sidekiq over delayed job though, so have a look at those before settling.
periodically crawler some site or download Tweets
Converse to the above, if this is a regularly scheduled event then cron jobs are ideal. Nothing wrong with using whenever for this, but make sure you have some form of monitoring on the jobs to alert you if something goes wrong.
For creating the activity, I would definitely use a background job to do that (i.e. PostToTwitterJob). I have a preference for Sidekiq instead DelayedJob, as delayed_job does not seem to be active anymore. If you want the job to be performed at a specific time in the future, check out sidekiq's documentation.
As for the crawling, I would use both. Your crawler would be a background job (i.e. TweetsCrawlerJob), and you could launch it every hour with whenever.
Related
Is there a way to automate and schedule an API request with Rails? I'd like to make a request and save that information into my database, however I'd probably only need it to make a request every few hours for the most up to date info. It's a good amount of data, so I'd like to have it stored as opposed to making a request every time a user visits.
Is there some kind of rake task that I can set up to do this for me on a schedule (or an alternative for what I'm hoping to accomplish)?
Thank you!
The clockwork or whenever gems are made for running scheduled tasks.
If you're hosted on Heroku, you can also use the Heroku Scheduler add-on to execute tasks in your Rails app every 10 minutes, every hour or daily.
The Above solutions are good and may suit your needs. There are more sophisticated queuing and scheduling solutions based on Redis that support multiple retries, have monitoring interfaces etc.
I have had good experiences with Resque and https://github.com/resque/resque-scheduler
If you can be sure your code is threadsafe there is also Sidekiq https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs
I'd check out Sidekiq. It uses redis and therefore supports retries like errata mentioned.
https://github.com/mperham/sidekiq/wiki/Scheduled-Jobs
I'm planing in using a rake task to develop a long running background process for my rails application. Are rake tasks appropriate for this kind of processes? Ideally, I would like wrap it inside a linux daemon to be able to start and end the process easily.
If it's not the best option, which are the alternatives? I'm trying to avoid using a cron-based solution to avoid having to worry about the schedule and the posibility of having diferent running instances of the same process overlapping between them.
Thanks!
You can try delayed job with this extension.
class MyJob
include Delayed::ScheduledJob
run_every 1.day
def display_name
"MyJob"
end
def perform
# code to run ...
end
end
Or manually enqueue another job with Time.now + 5.minutes for example after current job is finished inside perform method.
Have you looked at the delayed_job gem?
https://github.com/collectiveidea/delayed_job
From their documentation:
Delayed::Job (or DJ) encapsulates the common pattern of asynchronously executing longer tasks in the background.
It is a direct extraction from Shopify where the job table is responsible for a multitude of core tasks. Amongst those tasks are:
sending massive newsletters
image resizing
http downloads
updating smart collections
updating solr, our search server, after product changes
batch imports
spam checks
it might depend on the kind of background jobs you need to run.
Basically if you need some sort of post processing on data the users enter, like rendering images for posts, do some async integration with third party resources, etc. then you better off with using Sidekiq (yeah, it's better than DelayedJob as people suggested)
But if you need to run something on schedule, like some say night downloads, cleaning up blocked users and stuff, then writing a rake task and kick it in with a cron task might be a perfectly useful option, coz you could use those tasks from CLI whenever you need to run them on demand
I have a ruby on rails app in which I'm trying to find a way to run some code every few seconds.
I've found lots of info and ideas using cron, or cron-like implementations, but these are only accurate down to the minute, and/or require external tools. I want to kick the task off every 15 seconds or so, and I want it to be entirely self contained within the application (if the app stops, the tasks stop, and no external setup).
This is being used for background generation of cache data. Every few seconds, the task will assemble some data, and then store it in a cache which gets used by all the client requests. The task is pretty slow, so it needs to run in the background and not block client requests.
I'm fairly new to ruby, but have a strong perl background, and the way I'd solve this there would be to create an interval timer & handler which forks, runs the code, and then exits when done.
It might be even nicer to just simulate a client request and have the rails controller fork itself. This way I could kick off the task by hitting the URI for it (though since the task will be running every few seconds, I doubt I'll ever need to, but might have future use). Though it would be trivial to just have the controller call whatever method is being called by the periodic task scheduler (once I have one).
I'd suggest the whenever gem https://github.com/javan/whenever
It allows you to specify a schedule like:
every 15.minutes do
MyClass.do_stuff
end
There's no scheduling cron jobs or monkeying with external services.
Generally speaking, there's no built in way that I know of to create a periodic task within the application. Rails is built on Rack and it expects to receive http requests, do something, and then return. So you just have to manage the periodic task externally yourself.
I think given the frequency that you need to run the task, a decent solution could be just to write yourself a simple rake task that loops forever, and to kick it off at the same time that you start your application, using something like Foreman. Foreman is often used like this to manage starting up/shutting down background workers along with their apps. On production, you may want to use something else to manage the processes, like Monit.
You can either write you own method, something like
class MyWorker
def self.work
#do you work
sleep 15
end
end
run it with rails runner MyWorker.work
There will be a separate process running in the background
Or you can use something like Resque, but that's a different approach. It works like that: something adds a task to the queue, meanwhile a worker is fetching whatever job it is in the queue, and tries to finish it.
So that depends on your own need.
I know it is an old question. But maybe for someone this answer could be helpful. There is a gem called crono.
Crono is a time-based background job scheduler daemon (just like Cron) for Ruby on Rails.
Crono is pure Ruby. It doesn't use Unix Cron and other platform-dependent things. So you can use it on all platforms supported by Ruby. It persists job states to your database using Active Record. You have full control of jobs performing process. It's Ruby, so you can understand and modify it to fit your needs.
The awesome thing with crono is that its code is self explained. In order to do a task periodically you can just do:
Crono.perform(YourJob).every 2.days
Maybe you can also do:
Crono.perform(YourJob).every 30.seconds
Anyway you really can do a lot of things. Another example could be:
Crono.perform(TestJob).every 1.week, on: :monday, at: "15:30"
I suggest this gem instead of whenever because whenever uses Unix Cron table which not always is available.
Throwing out a solution just because it looks somewhat elegant and answers the question without any extra gems. In my scenario I wanted to run some code, but only after all my Sidekiq workers were done doing their thing.
First I defined a method to check if any workers were working...
def workers_working?
workers = Sidekiq::Workers.new.map do |_process_id, _thread_id, work|
work
end
workers.size > 0
end
Then we just call the method with a loop which sleeps between calls.
sleep 5 while workers_working?
Use something like delayed job, and requeue it every so often?
Use thin or other server which uses eventmachine, then just use timers that are part of eventmachine. Example: in config/application.rb
EM.add_periodic_timer(2) do
do_this_every_2_sec
end
I am trying to find out the best way to run scripts in the background. I have been looking around and found plenty of options, but many/most seem to have become inactive in the past few years. Let me describe my needs.
The rails app is basically a front-end to configure when and how these scripts will be run. The scripts run and generate reports and send email alerts. So the user must be able to configure the start times and how often these scripts will run dynamically. The scripts themselves should have access to the rails environment in order to save the resulting reports in the DB.
Just trying to figure out the best method from the myriad of options.
I think you're looking for a background job queuing system.
For that, you're either looking for resque or delayed_job. Both support scheduling tasks at some point in the future -- delayed_job does this natively, whereas resque has a plugin for it called resque_scheduler.
You would enqueue jobs in the background with parameters that you specify, and then at the time you selected they'll be executed. You can set jobs to recur indefinitely or a fixed number of times (at least with resque-scheduler, not sure about delayed_job).
delayed_job is easier to set up since it saves everything in the database. resque is more robust but requires you to have redis in your stack -- but if you do already it's pretty much the ideal solution for your problem.
I recently learned about Sidekiq, and I think it is really great.
There's also a RailsCast about it - Sidekiq.
Take a look at the gem whenever at https://github.com/javan/whenever.
It allows you to schedule tasks like cron jobs.
Works very well under linux, and the last commit was 14 days ago. A friend of mine used it in a project and was pretty satisfied with it.
edit: take a look at the gem delayed_job as well, it is good for executing long tasks in the background. Useful when creating a cron job only to start other tasks.
I need to build a background job that goes through a list of RSS feeds and analyze them say every 10 minutes.
I have been using delayed_job for handling background jobs and I liked it a lot. I believe though that it's not built for recurring background jobs. I guess I can auto-schedule background job at the end of everyone (maybe with begin..rescue just to ensure it gets executes). Or preschedule say a month of advance worth of jobs and have another one that reschedule the every month..etc
This raised some concerned to me as I started asking myself: what if the server goes down in the middle of execution and the jobs didn't get scheduled?
I have also looked at Daemons gems which seemed the like it runs simple Ruby scripts with start/stop commands. I like the way delayed_job schedules and handles retries.
What do you recommend using in this case? What do you think the best way to design such a system with recurring background jobs? Also do you know a way I can monitor that background process and get notified if it stops?
I just implemented delayed_job for a similar task (using :run_at => 2.days.from_now) and found it to be a perfect fit. The easiest way to handle your concern about a process failing is to make the first step of the job to create the next job. Also, you can create a has_many relationship to the delayed_job model which would allow you to access the :last_error. Or, look at the "Hooks" section of readme and it has a perfect example for failure.
I think that this was a similar question: A cron job for rails: best practices? - not only are there answers, but also links to railscasts about background jobs in rails.
I used cron + delayed_job, but scheduled tasks were supposed to run few times a day, mostly just once.
Take a look at SimpleWorker. It's an elastic scheduling and background processing worker queue. It's cloud based and has persistence and redundancy so you don't need to worry if your servers go down or are restarted.
Very flexible in terms of scheduling, provides great introspection of jobs in the queue as well as notifications on status and errors.
Full disclosure: I work at SimpleWorker.