I am using Sidekiq to schedule some tasks based on a schedule that the user provides. However, if the user changes the schedule, I want to be able to simply update the old schedule with the new one.
Suggestion one
I saw a suggestion to just find the old job with Sidekiq::ScheduledSet.new.find_job(job_id), but I am trying to avoid having to create a new model just to simply store the job ID and the task.
Suggestion two
Another suggestion I saw was to just have the worker check if the time of the task matches the current time, but that won't work because if the server is offline, it won't process jobs when it returns back online because the time of those delayed jobs won't match the current time.
If I could assign my own job ID, like a hex version of the job name or a padded version of the task ID, then I could easily avoid having to create a new model to store the job IDs. So when the user reschedules a task, then it would be a lot easier.
Other thoughts
Maybe if I could check the job's at attribute and match that with the task, that may work, but I'm not sure how to access that attribute from within the worker without knowing the job ID.
Edit
I just tried to pull the current job's at attribute, but it looks like once the job kicks off, it doesn't exist anymore in Sidekiq::ScheduledSet, so there's no matching this job's time with Task's time from what it seems like.
I am using Sidekiq to schedule some tasks based on a schedule that the user provides...
There's an extension for that. Sidekiq-Scheduler gives you a cron-like schedule configuration file. Then you can alter the schedule as you see fit. This seems like the best option as it avoids having to write your own scheduler interface.
Can I assign my own randomized Job ID to Sidekiq?
Yes, though it's undocumented. You can give Sidekiq::Client.push a jid attribute.
Sidekiq::Client.push('class' => MyWorker, 'args' => [1, 2, 3], 'jid' => ... )
This is not a good way to solve your problem. It's relying on an undocumented feature. And it invites collisions with normal Sidekiq IDs.
Maybe if I could check the job's at attribute and match that with the task, that may work, but I'm not sure how to access that attribute from within the worker without knowing the job ID.
This sounds very error prone. You'd have to store the timestamp in a model anyway. Better to store the job ID in the first place.
I am trying to avoid having to create a new model just to simply store the job ID and the task.
Storing things in models is what Rails does really well. This would seem to be the way to go. It will take a trivial amount of coding, database storage, and processing. You should have a model, view, and controller for your scheduled jobs anyway else how will you create scheduled jobs and view your schedule?
However, the Sidekiq docs notes that find_job is "a slow, inefficient operation. Do not use under normal conditions. Sidekiq Pro contains a faster version." This is because it has to iterate through all jobs.
I had a case where I had to reschedule jobs based on updates from the User. It is actually pretty slow and complicated.
It's simpler to not reschedule, but instead make the old queued tasks no-ops (no operations) and then queue up the new tasks.
This is basically defined by the logic within the task. You'd have to know that the user updated their schedule somehow and check that within the old jobs and based on some if-check, not go through with the job.
Related
I am working on implementing a requirement for a web app:
Users should be able to schedule a job in the future
Users should be able to view a list of scheduled jobs
Users should be able to edit the time of scheduled jobs
All of the above can easily be done with Sidekiq using something like:
ss = Sidekiq::ScheduledSet.new
job = ss.find_job(jid)
job.reschedule(at)
However, I would have to store in the db at least the jid and the scheduled time so that the user can determine if he / she wants to reschedule it.
My concern is that the above is not efficient. First, I would have to duplicate storage in redis and the db, additionally, as Mike Perham put it:
You can do this but it won't be efficient. It's a linear scan for find
a scheduled job by JID.
See https://stackoverflow.com/a/24189112/1755697
If I need to store details in the db and in redis (sidekiq storage), I am wondering if it may be simpler to go with Delayed Job which stores job in the db, thus not duplicating storage.
The main concern about Delayed Job is that it is not yet Rails 5.0 ready and it has not been updated for a while.
I would appreciate any suggestion on which system to chose or how to use Sidekiq in a more efficient way to meet the above mentioned requirements.
I'm playing with Rails 4.2 app which uses ActiveJob backed by resque/sidekiq for email scheduling. When a user creates newsletter campaign a new job is created and scheduled on certain date. That's all great but what happens when the user changes the delivery date.
In this case every job could check if it should be delivered or not thus invalid jobs would be ignored and only the last one would be executed. This could work but if a user would make 1k edits that would push 1k-1 invalid jobs into queue - not good. I believe that the existing job should be updated or replaced with a new one. As far as I know searching through the Redis queue for the job_id is slow.
What would be the proper way for rescheduling ActiveJobs in Rails (with resque/sidekiq)?
There is none, jobs are not meant to be rescheduled. You have answered your own question:
In this case every job could check if it should be delivered or not thus invalid jobs would be ignored and only the last one would be executed.
The alternative is to re-architect how you send campaigns: store the delivery date in the database and have cron check every minute for campaigns which need delivery now and create the Sidekiq job right then.
I have an Ruby on Rails application which allows the user the execute certain jobs. These jobs are handled and executed by Sidekiq.
Now I want to give the user the possibility to schedule these jobs and like cron jobs allow them to be recurring on certain dates and time. The question is how to do this?
All Gems I have come across so far, like whenever, rufus, sidetiq or simple cron jobs only allow for the job to be hardcoded in a file. But what I want is a job that is set by the user and stored in the database.
The obvious way would be to create a cron that checks the database every minute for jobs that are due, executes them and sets the date and time to the next scheduled point. But this way also involved a lot of coding and seems to be a little complicated. Also I see it as a fake cron, because there is simply one cron job that always checks the database.
What I would really like is a solution where the user actually creates a cron job, but I didn't come across anything so far that does this. Also it doesn't have to be a Gem. A more simpler way of just doing this would be enough. And I'd like to think that I can't be the only one who wants to do something like this.
I have a 'main' rails app that needs to offload some work to some named delayed-job queues. I'd like to keep all the logic for processing those jobs in a separate app, which would run on a separate server from the main web app servers. It would connect to the same database, but only have a small fraction of the models defined, just what it will need to process these jobs.
When I queue up a job, is there any way to ONLY queue up the id of the record to act on? Combined with the queue name, I think that should be enough for the worker to know what to do. Or, if a class needs to be provided, can I have a dummy stripped down version of the class in the main app, with the actual implementation logic in a version of the class in the separate 'worker' app?
Actually, what you're describing above is correct:
You want to queue an id, and sometimes other required information to identify the record you want to pull from the database, and NOT an object, especially if it's an ActiveRecord::Base object.
Queuing an object is problematic and can lead to issues. Check my answer at:
https://stackoverflow.com/a/21751030/226255
You can test if your second application is pulling correctly by doing the following in console: (make sure to push a Delayed::Job to queue first)
Delayed::Job.first.invoke_job
I'm using Delayed Job to manage background work.
However I have some tasks that need to be executed at regular interval. Every hour, every day or every week for example.
For now, when I execute the task, I create a new one to be executed in one day/week/month.
However I don't really like it. If for any reason, the task isn't completely executed, we don't create the next one and we might lose the execution of the task.
How do you manage that kind of things (with delayed job) in your rails apps to be sure your regular tasks list remains correct ?
If you have access to Cron, I highly recommend Whenever
http://github.com/javan/whenever
You specify what you want to run and at what frequency in dead simple ruby, and whenever supplies rake tasks to convert this into a crontab and to update your system's crontab.
If you don't have access to frequent cron (like I don't, since we're on Heroku), then DJ is the way to go.
You have a couple options.
Do what you're doing. DJ will retry each task a certain number of times, so you have some leniency there
Put the code that creates the next DJ job in an ensure block, to make sure it gets created even after an exception or other bad event
Create another DJ that runs periodically, checks to make sure the appropriate DJs exist, and creates them if they don't. Of course, this is just as error prone as the other options, since the monitor and the actual DJ are both running in the same env, but it's something.
Is there any particular reason why you wouldn't use cron for this type of things?
Or maybe something more rubyish like rufus-scheduler, which is quite easy to use and very reliable.
If you don't need queuing, these tools are a way to go, I think.