I am using multiple heroku servers that share the same DB. I would like to have each server only process delayed jobs for the server that created the delayed job entry.
For example:
Server A only processes queue "server_a"
Server A only processes queue "server_b"
etc...
This is accomplishable by using Delayed Job 3 (https://github.com/collectiveidea/delayed_job)
However, for this to work I would need to manually assign a queue name for each delayed job created, which can be a pain. (for example: object.delay(:queue => 'tracking').method)
Instead I would like to be able to assign a "default queue" for all new jobs. Ideally, I put something like this in a delayed_job_config.rb & it works:
DEFAULT_QUEUE_NAME = ENV['APP_NAME']
... the idea being that I do nothing to existing delayed jobs & they automatically get assigned a queue with the same name as the app server.
I am looking for suggestions on how to accomplish this -- or if you want to give it a stab, throw some code my way.
Thanks in advance!
In config/initializers/delayed_job.rb
Delayed::Worker.default_queue_name = `hostname`.chomp
Related
I have a use case where user schedules a 'command' from the web interface. The user also specifies the date and time the command needs to be triggred.
This is sequence of steps:
1.User schedules a command 'Restart Device' at May 31, 3pm.
2.This is saved in a database table called Command.
3.Now there needs to be a background job that needs to be triggered at this specified time to do something (make an api call, send email etc.)
4.Once job is executed, It is removed or marked done, until a new command is issued.
There could be multpile users concurrently performing the above sequence of steps.
Is delayed_job a good choice for above? I couldnt find an example as how to implement above using delayed job.
EDIT: the reason I was looking at delayed_job is because eventually I would need to leverage existing relational database
I would advise to use Sidekiq. With it you can use scheduled jobs to tell sidekiq when to perform the jobs.
Example :
MyWorker.perform_at(3.hours.from_now, 'mike', 1)
EDIT : worker example
#app/workers/restart_device_worker.rb
class RestartDeviceWorker
include Sidekiq::Worker
def perform(params)
# Do the job
# ...
# update in DB
end
end
see doc: https://blog.codeship.com/how-to-use-rails-active-job/
https://guides.rubyonrails.org/active_job_basics.html
If you are using Rails 5 then you have best option of ActiveJob(inbuilt feature)
Use ActiveJob
"Active Job – Make work happen later. Active Job is a framework for declaring jobs and making them run on a variety of queuing backends. These jobs can be everything from regularly scheduled clean-ups, to billing charges, to mailings. Anything that can be chopped up into small units of work and run in parallel, really."
Active Job has built-in adapters for multiple queuing backends (Sidekiq, Resque, Delayed Job and others). You just need to tell them.
Scenario: I want to delete my story after 24 hours(1 day). Then we do create a job named "StoriesCleanupJob". Call this job at the time of the creation of story like below
StoriesCleanupJob.set(wait: 1.day).perform_later(story)
It will call the Job after 1 day.
class StoriesCleanupJob < ApplicationJob
queue_as :default
def perform(story)
if story.destroy
#put your own conditions like update the status and all, whatever you want to perform.
end
end
end
I have 20 Delayed job workers
I had search so many links and post for synchronous processing for that job but nothing works for me
Also assign QUEUE for perticular worker like this
script/delayed_job --queue=order_item_kit start -i 0
order.delay(:queue => "order_item_kit").update_order
But this will assign to this queue and other worker also works for it.
I am searching for one worker will run one job at a time.
for this particular case
Please give suggestion to work for this case, one method calls only one worker as a time.
Thanks in advance.
I can solve this by doing that way
When it assign next same job to another worker, we can add run_at one second ahead, so multiple workers does not save object same time and this fix our issue
latest_job = Delayed::Job.where(queue: "order_item_kit").last
run_at = latest_job ? latest_job.run_at + 1.seconds : 1.seconds.from_now
kit_sku.delay(:run_at => run_at,:queue =>"order_item_kit").update_ordere_item
According to the DJ documentation
Processes started without specifying a queue will run jobs from any queue.
You should run your workers with explicit queue name:
RAILS_ENV=production script/delayed_job --queue=tracking start
And with some other queue name for all other ('default') workers.
To define that name set Delayed::Worker.default_queue_name in initializer.
I've a job, and when the job is run, at the very bottom of it, and I want to enqueue the same job again to run after 1 hour, but with different arguments.
What I've achieved so far:
class SimpleJob
#queue = :normal
def self.perform(start)
puts "Right now, start = #{start}"
start += 12
time = some_request_external_api
self.set(wait: time).perform_later(start)
end
end
I'm using resque gem, and running the job through QUEUE=* rake resque:work. Surely, it prints Right now, start = 12 in beginning, but after that, nothing happens. How exactly can I achieve this functionality?
Rather than enqueuing the job again within itself, you could either use a scheduler, such as clockwork.
Or, if what you are trying to accomplish is in response to some event on another service, maybe you could look into it's documentation and see if it provides webhook functionality.
These would send post requests to your desired action whenever any action occurs on their side.
Sounds like you are looking for ways to setup recurring job. If thats the case, take a look at this:
https://github.com/resque/resque-scheduler
Currently I am working with queue job on the ruby on rail with the Sidekiq. I have 2 jobs that are depend to each other and I want 1st job to finish first before starting the 2nd job, so is there any way to make it with Sidekiq.
Yes, you can use the YourSidekiqJob.new.perform(parameters_to_the_job) pattern. This will run your jobs in order, synchronously.
However, there are 2 things to consider here:
What happens if the first job fails?
How long does the each job run?
For #2, the pattern blocks execution for the length of time each job takes to run. If the jobs are extremely short in runtime, why use the jobs in the first place? If they're long, are you expecting the user to wait until they're done?
Alternatively, you can schedule the running of the second job as the last line in the body of the first one. You still need to account for the failure mode of job #1 or #2. Also, you need to consider that the job won't necessarily run when it's scheduled to run, due to the state of the queue at schedule time. How does this affect your business logic?
Hope this helps
--edit according to last comment
class SecondJob < SidekiqJob
def perform(params)
data = SomeData.find
return unless data.ready?
# do whatever you need to do with the ready data
end
end
I am on Heroku and I got an error because my redis db got too full. The my sidekiq processes stopped working. It was like that for a day until I realized it. Now I have 600+ jobs that I have tried to process but they are just breaking everything now. How can I sound off the alarms when sidekiq can't process jobs or when the Enqueue starts to fill up?
You could set a rake task on a schedule to check Sidekiq stats, and then take the appropriate action ( like send an email ).
I've created my own module with helper methods for Sidekiq that serves a number of purposes, e.g deleting jobs, checking queues, retriving jobs by certain criteria etc. https://gist.github.com/blotto/10324119
For your purpose, grab the sidekiq stats as such :
def sidekiq_stats()
summary = Hash.new
stats = Sidekiq::Stats.new
summary = { processed: stats.processed,
failed: stats.failed,
enqueued: stats.enqueued,
queues: stats.queues}
end
And then evaluate the enqueued value, set a tolerance on what you think is too high, and then let loose the hounds.
If you're using zabbix for monitoring, you could use sidekiq_queue_zabbix template at https://github.com/hungntit/sidekiq_queue_zabbix. This template supports showing graph and sending alert when sidekiq queue size is higher than one specified limit number