Does the :async queue adapter actually do anything?
:inline, which is what is default in Rails 4, processes jobs built with ActiveJob, uh... inline, in the current execution thread. Async, shouldn't. It should use the ConnectionPool to not run it in the current thread, and that's ideally what would be happening. It'd run perform outside of the current execution thread.
But nothing executes it.
I've pored through the docs, and the only thing I can fathom is is that :async, unlike :inline, doesn't execute tasks, and expects you to build a system around execution locally. I have to manually perform perform on all jobs in order to get them to execute locally. When I set the adapter to :inline, it works just fine without having to execute.
Is there some configuration issue I'm missing that's preventing async from working correctly (like ActionCable?).
Does it not work if executed from a rake task (or the console?).
It works fine with :sidekiq/:resque, but I don't want to be running these locally all the time.
Rails by default comes with an "immediate runner" queuing implementation. That means that each job that has been enqueued will run immediately.
This is kind of what's cueing me in there being something wrong. I have jobs that are sitting in a queue somewhere that just don't run. What could be stopping this?
This is what I discovered. With the advent of concurrent-ruby, rake tasks aren't set up to handle this.
If you read in the documentation, it says with :async, it's cleared out of memory when the process ends.
Rails itself only provides an in-process queuing system, which only
keeps the jobs in RAM. If the process crashes or the machine is reset,
then all outstanding jobs are lost with the default async back-end.
Rake processes end when they're over. So, if you're doing any sort of data changes, rake tasks won't be open long enough to run a job, which is why they run :inline just fine, but not :async.
So, I haven't figured out a solution to keep rake tasks open long enough to run something :async (and keep the app :async the entire time). I have to switch it to :inline to run tasks, and then back to :async when I'm done for the rest of my jobs. It's why it works fine with :sidekiq or :resque, because those applications keep the job information in memory and do not release when the rake task is over.
In order for rake tasks to work with :async locally, there's not much you can do other than run tasks as :inline if you're local until rake (as a task runner) understands how to stay open while asynchronous tasks have been launched (or not). As a development only feature, this isn't really high priority, but, if you're bashing your head on the table not understanding why :async by default tasks that run jobs won't actually run, that's why.
Here's what you can put at the end of a rake task to wait for the AsyncAdapter to finish processing before exiting the rake task:
if Rails.configuration.active_job.queue_adapter == :async
executor =
ActiveJob::Base._queue_adapter.instance_eval do
#scheduler.instance_eval { #async_executor }
end
sleep(0.1) while executor.scheduled_task_count > executor.completed_task_count
end
Related
I am currently using the delayed_job_active_record gem to run some scheduled tasks on a long run basis. The processes run in the background on a separate worker dyno on heroku and rarely go wrong but in some cases I would like to be able to stop a process mid run. I have been running the processes locally and because of the setup I have, the scheduled tasks only kick off the process which is essentially a very long loop.
Using
bin/delayed_job stop
only stops the jobs but since the process has started, it doesn't top this.
Because of this, I can't seem to stop the process once it has got going without restarting the entire dyno. This seems a bit excessive but is my only option at the moment.
Any help is greatly appreciated
I don't think there's anyway to interrupt it without essentially killing the process like you are doing. I would usually delete the job record in the database and then terminate the worker running it so it doesn't just retry the job (if you've got retries enabled for that job).
Another option... Since you know it's long running and, I imagine, has multiple steps... Modularize the operation and/or add periodic checks for a 'cancelled' flag you put somewhere in the model(s). If you detect the cancelled request, you can then give up and do any cleanup needed. This is probably preferred anyway so you can manage what happens when it's aborted more explicitly.
I have a rails module that processes some active record objects, only about 15-20 at a time, that I need to start off every two minutes.
I have tried to offload it to sidekiq (and sidekiq-cron), which works, but with the concurrency, created many race conditions and duplicate data.
I really just need a simple rake task cron for rails or maybe sinatra (as I would create a new sinatra app just to complete these tasks)
I either need to force sidekiq to process in a single thread or
have a "cron" job run a rake task or even the module directly
def self.process_events
events = StripeEvent.where(processed: false)
events = StripeServices.arrange_processing_order events
events.each do |event_obj|
StripeServices.new(event_obj).process_event_obj
end
end
thanks for any point in the right direction.
edited
sorry I wasnt very clear. pushing my module to sidekiq caused concurrency issues that I wasnt ready for (my bit of code is not threadsafe), and with the restrictions that Heroku places on "crons", whats the best way to run a rake task every 2 min?
If Sinatra can do it, I would prefer it, but I cant find the solution for that same problem.
It's not clear what are you asking. You already tried option 1, you can try option 2 (create the task and cron it, it's pritty easy) and you'll know better than anyone if it's better.
Anyway, I guess that both methods will have concurrency problems if one task takes more than 2 minutes.
You can add extra flags to prevent two task to process the same ServiceEvent (maybe add a boolean "processing" and set it to true when a task takes it).
Or maybe you can have a lock file to prevent a task to run if another one is already running (you create a file with a specific location and name when the task starts and delete it when it finishes processing, you can check if the file exists before starting a new task).
I am looking for the best way to run a very long task in Heroku.
I use Ruby On Rails for my web application and I have a very long task that I want to run it every week on Sunday during the night. It takes around 15~20 minutes. I already have Rufus-Scheduler, but I am not sure it is the most effective solution.
I also find something about Backgrounding Tasks in Heroku with Delayed Job. But is it the best way to handle it ?
Thanks.
This is what I use for a job that I run every night: https://devcenter.heroku.com/articles/scheduler
It works really well if your job is configured as a rake task. The guide at the link shows you how to configure everything and even addresses long-running jobs.
Heroku does not recommend to run long-running jobs with Heroku Scheduler.
Heroku says,
Scheduled jobs are meant to execute short running tasks or enqueue longer running tasks into a background job queue. Anything that takes longer than a couple of minutes to complete should use a worker dyno to run.
So, in my opinion, the best approach would be to use Heroku Scheduler to run rake task for every kind of job (short or long) but if a task takes longer than a couple of minutes then I would simply create a Background Job within that rake task. That way the scheduler will never run longer than a couple of minutes.
I have a class (/lib/updater.rb) that do a large updating of the database (calling external server, calculations,...). Normally this task is called by the cron of the server (rake /lib/tasks/launch_updater.rake that start the updater.rb), but I would like to give the opportunity to start it manually from the client too.
At this moment, from the client, the user can click on a button and launch it in this manner:
# the controller
Thread.new {
Updater.start
}
It is a good solution or is better to launch directly from a rake task?
# something like this from the controller
Rake::Task[params[:task]].reenable
Rake::Task[params[:task]].invoke
The task should be no-blocker (the user should navigate normally on the app without waiting the end of the task).
Which is better and why?
Working a little on my question I found the following notes:
when using a Thread, the CPU used is the same at the CPU of the app (even if you have a multi-core server, the CPU is the same). If you want to use Thread, the Thread task should not be "heavy" or you can go in CPU problems (slow app processing).
when you start a Rake task from the terminal or from a server cron, this should take the CPU with the lighter running process. But if you start a task from the application I think the CPU is even here the same of the app.
the better solution to work with heavy charge task is to use a delayed service, in this manner the job task should take another CPU than the app's CPU without make problems at the performance of the app:
Sidekiq
Delayed Job
...
Ok, so this is probably evil, however.. here's the question! I want to run a pretty lightweight app on a shared environment (site5). Ideally I would like to use delayed_job for the ease of queueing the mails (~200+ every so often). However, being a shared environment they don't want background processes running all the time (fair enough).
So, my plan, such as it is, is to queue the mails using delayed job, and then every hour or something, spin up a cron job, send a few emails (10 or something small) and then kill the process. And repeat.
Q) Is there a rake jobs:works:1 equivalent task it'd be easy to setup? - pointer would be handy.
I'm quite open to "this is a terrible idea, don't even go there" being the answer.. in which case I might look at another queuing strategy... (or heroku hire-fire perhaps..)
You can get delayed job to process only a certain number of jobs by doing:
Delayed::Worker.new.work_off(10)
You could fire a script to do that from cron or use "rails runner":
rails runner -e production 'Delayed::Worker.new.work_off(10)'
I guess the main issue on whether it is a good idea or not is working out what small value is actually high enough to make sure you process all your jobs in a reasonable time-frame. Also, you've got the overhead of firing up the rails environment every time you want to process, or even check whether you should process, any jobs. That might cause problems in a shared environment if they are particularly strict on spikes of memory or CPU usage.
Why not skip the 'workers' (which are just daemons which look for work else sleep) and have your cron fire a custom rake task of 10.times { MailerJob.first.perform }
You'd just need to require you're app in the line before that so its loaded ofc.