After running SomeJob.perform_later I can see that job was enqueued in ActiveJob::Base.queue_adapter.enqueued_jobs.
How can I remove the job from the queue if I already have saved job_id?
Basically I want to remove job from the queue.
Calling perform_later will enqueue the job into whichever backend you use. The ActiveJob interface doesn't provide a way to remove jobs. If you are using Sidekiq as your backend this documentation explains how to remove a job from a queue.
Related
How can I manage to execute job after the first job that has executed is done in sidekiq. For example:
I triggered the first job for this morning
GoodWorker.perform_async(params) #=> JID-eetc
while it is still in progress I've executed again a job in the same worker dynamically
GoodWorker.perform_ascyn(params) #=> JID-eetc2
and etc.
What's going on now is Sidekiq processing the jobs all of the time,
is there a way performing the job one at a time?
Short answer: no.
Long answer: You can use a mutex to guarantee that only one instance of a worker is executing at a time. If you're running on a cluster, you'll need to use Redis or some other medium to maintain the mutex. Otherwise, you might try putting these jobs in their own queue, and firing up a separate instance of Sidekiq that only monitors that queue, with a concurrency of one.
Can you not setup Sidekiq to only have one thread? Then only one job will be executed at a time.
I have an Job in my rails app that extends from ActiveJob::Base, and there are two ways that it is called:
ResqueScheduler on a daily basis, where it is picked up by a Resque worker and processed
Within the app via a #perform_now call, so it is processed by the app.
Is there a way for me to determine within the job whether it is being processed by a Resque worker?
You should see a reference to ActiveJob in your server log.
[ActiveJob] [<YOUR-JOB-CLASS>] [b8ed2ecb-10ce-433d-8ddb-33a04c8021cd] Performing <YOUR-JOB-CLASS> from Resque(default)...
This isn't definitive but it's a strong indicator in to the context your code is being executed.
You could use defined?(Rails) to determine if you are inside the Rails app or if you are in the Resque context.
What is the best approach to make sure specific background jobs (DelayedJob, or Resque) are executed sequentially, instead of in parallel? I guess one option is to have a dedicated queue and assign one worker only to the queue. Is there a better approach?
I did some background job functions using ActiveJob & Resque before.
I think do it by checking and setting status(e.g. pending, in-progress ..)
for each job.
And passing it for background jobs one by one.
I have resque and resque-scheduler workers with 2 different queues,
they do the same task, fetch links for a certain website and save that links.
What will happen if resque-scheduler and resque workers work in parallel and do the same task (fetching links for the same website)? How can i handle such situations?
Either you have not clarified your setup or there are some big issues there. Resque and Resque-scheduler were meant to be run together. Resue-scheduler is only supposed to schedule tasks in the future. Such tasks are still executed by Resque workers. Please read this section on their homepage on github: https://github.com/resque/resque-scheduler#delayed-jobs. To quote them,
This will store the job ... in the resque delayed queue at
which time the scheduler process will pull it from the delayed queue
and put it in the appropriate work queue for the given job and it will
be processed as soon as a worker is available (just like any other
resque job).
So, there you go. Keep running your resque workers and schedulers together forever. To answer the other part of your question, if you schedule some task through scheduler and the same task is also queued for resque to pick up directly, the net outcome depends on the task execution logic. "Fetching something from a website" sounds a harmless thing to do twice. But if you update some transaction table to make payments to your vendors based on the result of the fetch, you are in deep trouble.
I am using Resque to enqueue jobs.
I start a worker and the jobs are processed.
My jobs extend a gem that implements job hooks like before_enqueue, after_enqueue, before_perform, after_perform and sends stuff to statsd. Those work. However, before_dequeue and after_dequeue do not seem to be called. Is there a reason why?
Also, my understanding of Resque isn't all quite there. I would call Resque.enqueue to queue up a job class, and then if I start a Resque worker, it will automatically pop a task from the queue and then perform the task. Where does dequeue come into play? I notice that dequeue destroys the the task, when does the dequeue step happen in the Resque worker workflow?
I want to hook into after_dequeue because I want to log the time that a task stays in the queue, so I need to hook into before_enqueue and after_dequeue.
So dequeue is used by the client to manually dequeue jobs from Redis/Resque. To calculate the time a job spends in the queue, I will have to capture the time in after_enqueue and before_perform. When Resque pops a job off the queue, there is no hook that we can hook into.