I have an Ruby on Rails application which allows the user the execute certain jobs. These jobs are handled and executed by Sidekiq.
Now I want to give the user the possibility to schedule these jobs and like cron jobs allow them to be recurring on certain dates and time. The question is how to do this?
All Gems I have come across so far, like whenever, rufus, sidetiq or simple cron jobs only allow for the job to be hardcoded in a file. But what I want is a job that is set by the user and stored in the database.
The obvious way would be to create a cron that checks the database every minute for jobs that are due, executes them and sets the date and time to the next scheduled point. But this way also involved a lot of coding and seems to be a little complicated. Also I see it as a fake cron, because there is simply one cron job that always checks the database.
What I would really like is a solution where the user actually creates a cron job, but I didn't come across anything so far that does this. Also it doesn't have to be a Gem. A more simpler way of just doing this would be enough. And I'd like to think that I can't be the only one who wants to do something like this.
Related
I am using Sidekiq to schedule some tasks based on a schedule that the user provides. However, if the user changes the schedule, I want to be able to simply update the old schedule with the new one.
Suggestion one
I saw a suggestion to just find the old job with Sidekiq::ScheduledSet.new.find_job(job_id), but I am trying to avoid having to create a new model just to simply store the job ID and the task.
Suggestion two
Another suggestion I saw was to just have the worker check if the time of the task matches the current time, but that won't work because if the server is offline, it won't process jobs when it returns back online because the time of those delayed jobs won't match the current time.
If I could assign my own job ID, like a hex version of the job name or a padded version of the task ID, then I could easily avoid having to create a new model to store the job IDs. So when the user reschedules a task, then it would be a lot easier.
Other thoughts
Maybe if I could check the job's at attribute and match that with the task, that may work, but I'm not sure how to access that attribute from within the worker without knowing the job ID.
Edit
I just tried to pull the current job's at attribute, but it looks like once the job kicks off, it doesn't exist anymore in Sidekiq::ScheduledSet, so there's no matching this job's time with Task's time from what it seems like.
I am using Sidekiq to schedule some tasks based on a schedule that the user provides...
There's an extension for that. Sidekiq-Scheduler gives you a cron-like schedule configuration file. Then you can alter the schedule as you see fit. This seems like the best option as it avoids having to write your own scheduler interface.
Can I assign my own randomized Job ID to Sidekiq?
Yes, though it's undocumented. You can give Sidekiq::Client.push a jid attribute.
Sidekiq::Client.push('class' => MyWorker, 'args' => [1, 2, 3], 'jid' => ... )
This is not a good way to solve your problem. It's relying on an undocumented feature. And it invites collisions with normal Sidekiq IDs.
Maybe if I could check the job's at attribute and match that with the task, that may work, but I'm not sure how to access that attribute from within the worker without knowing the job ID.
This sounds very error prone. You'd have to store the timestamp in a model anyway. Better to store the job ID in the first place.
I am trying to avoid having to create a new model just to simply store the job ID and the task.
Storing things in models is what Rails does really well. This would seem to be the way to go. It will take a trivial amount of coding, database storage, and processing. You should have a model, view, and controller for your scheduled jobs anyway else how will you create scheduled jobs and view your schedule?
However, the Sidekiq docs notes that find_job is "a slow, inefficient operation. Do not use under normal conditions. Sidekiq Pro contains a faster version." This is because it has to iterate through all jobs.
I had a case where I had to reschedule jobs based on updates from the User. It is actually pretty slow and complicated.
It's simpler to not reschedule, but instead make the old queued tasks no-ops (no operations) and then queue up the new tasks.
This is basically defined by the logic within the task. You'd have to know that the user updated their schedule somehow and check that within the old jobs and based on some if-check, not go through with the job.
I am trying to find out the best way to run scripts in the background. I have been looking around and found plenty of options, but many/most seem to have become inactive in the past few years. Let me describe my needs.
The rails app is basically a front-end to configure when and how these scripts will be run. The scripts run and generate reports and send email alerts. So the user must be able to configure the start times and how often these scripts will run dynamically. The scripts themselves should have access to the rails environment in order to save the resulting reports in the DB.
Just trying to figure out the best method from the myriad of options.
I think you're looking for a background job queuing system.
For that, you're either looking for resque or delayed_job. Both support scheduling tasks at some point in the future -- delayed_job does this natively, whereas resque has a plugin for it called resque_scheduler.
You would enqueue jobs in the background with parameters that you specify, and then at the time you selected they'll be executed. You can set jobs to recur indefinitely or a fixed number of times (at least with resque-scheduler, not sure about delayed_job).
delayed_job is easier to set up since it saves everything in the database. resque is more robust but requires you to have redis in your stack -- but if you do already it's pretty much the ideal solution for your problem.
I recently learned about Sidekiq, and I think it is really great.
There's also a RailsCast about it - Sidekiq.
Take a look at the gem whenever at https://github.com/javan/whenever.
It allows you to schedule tasks like cron jobs.
Works very well under linux, and the last commit was 14 days ago. A friend of mine used it in a project and was pretty satisfied with it.
edit: take a look at the gem delayed_job as well, it is good for executing long tasks in the background. Useful when creating a cron job only to start other tasks.
I have a rails app with a Location model, which has a rating, and a rating_count field. Now, I need to reset every Location's rating and rating_countattributes to 0 at a specific time everyday, lets say 12:00:00 UTC.
How would I accomplish this? I'm using the default sqlite3 databases.
The best option is to use cron. You can find tons of documentation out there!Although if you are running a Rails app you should check out whenever a pretty neat gem for managing cron jobs for your app!
The easiest thing is to write a rake task that does that job, and then use whatever scheduling system your host uses (cron).
An alternative is to use delayed_job which allows to push work to a background process. While delayed-job is not exactly suited for something like this, it is perfectably capable of doing this. If your rails process starts, you add a new job, to run at 12:00. And the running of the job reschedules the job.
The nice thing of delayed-job is that your code runs in the context of a rails-process, so you can use methods you already have. Also nice: jobs are stored in the database, so you can have an overview.
If you're on a *nix box, write a script to do the updates (eg. in PHP, or Perl) and simply add it to crontab. Check out cron.
Is there a way to setup a callback in ROR that would trigger at a specific time?
Lets say I'm running a contest that expries at a certain time. Lets say Monday July 28th at 9:00. I'd like to set up an observer that ran a function at Monday July 28th at 9:00. Does rails have a method to do this?
There's a run_at field in Delayed Job. You have to have a worker process in the background always running and looking for jobs that a set to run, but if your application is doing this a lot, it might be easier than always writing new cron jobs.
So, you could have a method in your Contest model that gets called in a after_create callback that sets up a delayed job to send out an email to a random winner at the date that's specified.
If it's a one time, or very infrequent deal, though, I'll agree about using whenever
You'd be better off writing a ruby script that runs via crontab at the exact time you need it to.
I agree about crontab, but I like whenever. It has a nice integration into cron that you can store with your repo and it integrates nicely with capistrano.
I'm using Delayed Job to manage background work.
However I have some tasks that need to be executed at regular interval. Every hour, every day or every week for example.
For now, when I execute the task, I create a new one to be executed in one day/week/month.
However I don't really like it. If for any reason, the task isn't completely executed, we don't create the next one and we might lose the execution of the task.
How do you manage that kind of things (with delayed job) in your rails apps to be sure your regular tasks list remains correct ?
If you have access to Cron, I highly recommend Whenever
http://github.com/javan/whenever
You specify what you want to run and at what frequency in dead simple ruby, and whenever supplies rake tasks to convert this into a crontab and to update your system's crontab.
If you don't have access to frequent cron (like I don't, since we're on Heroku), then DJ is the way to go.
You have a couple options.
Do what you're doing. DJ will retry each task a certain number of times, so you have some leniency there
Put the code that creates the next DJ job in an ensure block, to make sure it gets created even after an exception or other bad event
Create another DJ that runs periodically, checks to make sure the appropriate DJs exist, and creates them if they don't. Of course, this is just as error prone as the other options, since the monitor and the actual DJ are both running in the same env, but it's something.
Is there any particular reason why you wouldn't use cron for this type of things?
Or maybe something more rubyish like rufus-scheduler, which is quite easy to use and very reliable.
If you don't need queuing, these tools are a way to go, I think.