I've got an app that could benefit from delayed_job and some background processing. The thing is, I don't really need/want delayed_job workers running all the time.
The app runs in a shared hosting environment and in multiple locations (for different users). Plus, the app doesn't get a large amount of usage.
Is there a way to start and stop processing jobs (either with the script or rake task) from my app only after certain actions/events?
You could call out to system:
system "cd #{Rails.root} && rake delayed_job:start RAILS_ENV=production"
You could just change delayed_job to check less often too. Instead of the 5 second default, set it to 15 minutes or something.
Yes, you can, but I'm not sure what the benefit will be. You say you don't want workers running all the time - what are your concerns? Memory usage? Database connections?
To keep the impact of delayed_job low on your system, I'd run only one worker, and configure it to sleep most of the time.
Delayed::Worker::sleep_delay = 60 * 5 # in your initializer.rb
A single worker will only wake up and check the db for new jobs every 5 minutes. Running this way keeps you from 'customizing' too much.
But if you really want to start a Delayed::Worker programatically, look in that class for work_off, and implement your own script/run_jobs_and_exit script. It should probably look much like script/delayed_job does - 3 lines.
I found this because I was looking for a way to run some background jobs without spending all the money to run them all the time when they weren't needed. Someone made a hack using google app engine to run the background jobs:
http://viatropos.com/blog/how-to-run-background-jobs-on-heroku-for-free/
It's a little outdated though. There is an interesting comment in the thread:
"When I need to send an e-mail, copy a file, etc I basically add it to the queue. At the end of every request it checks if there is anything in the queue. If so then it uses the Heroku API to set the worker to 1. At the end of a worker getting a task done it checks to see if there is anything left in the queue. If not then it sets the workers back to 0. The end result is the background worker will just work for a few seconds here and there. I can do all the background processing that I need and the bill at the end of the month rarely ever reaches 1 hour total worth of work. Even if it does no problem, I'll pay $0.05 for background processing. :)"
If you go to stop a worker, you are given the PID. You can simply kill -9 PID if all else fails.
Related
I am currently using the delayed_job_active_record gem to run some scheduled tasks on a long run basis. The processes run in the background on a separate worker dyno on heroku and rarely go wrong but in some cases I would like to be able to stop a process mid run. I have been running the processes locally and because of the setup I have, the scheduled tasks only kick off the process which is essentially a very long loop.
Using
bin/delayed_job stop
only stops the jobs but since the process has started, it doesn't top this.
Because of this, I can't seem to stop the process once it has got going without restarting the entire dyno. This seems a bit excessive but is my only option at the moment.
Any help is greatly appreciated
I don't think there's anyway to interrupt it without essentially killing the process like you are doing. I would usually delete the job record in the database and then terminate the worker running it so it doesn't just retry the job (if you've got retries enabled for that job).
Another option... Since you know it's long running and, I imagine, has multiple steps... Modularize the operation and/or add periodic checks for a 'cancelled' flag you put somewhere in the model(s). If you detect the cancelled request, you can then give up and do any cleanup needed. This is probably preferred anyway so you can manage what happens when it's aborted more explicitly.
I want to use Heroku but the fact they restart dynos every 24 hours at random times is making things a bit difficult.
I have a series of jobs dealing with payment processing that are very important, and I want them backed by the database so they're 100% reliable. For this reason, I chose DJ which is slow.
Because I chose DJ, it means that I also can't just push 5,000,000 events to the database at once (1 per each email send).
Because of THAT, I have longer running jobs (send 200,000 text messages over a few hours).
With these longer running jobs, it's more challenging to get them working if they're cut off right in the middle.
It appears heroku sends SIGTERM and then expects the process to shut down within 30 seconds. This is not going to happen for my longer jobs.
Now I'm not sure how to handle them... the only way I can think is to update the database immediately after sending texts for instance (for example, a sms_sent_at column), but that just means I'm destroying database performance instead of sending a single update query for every batch.
This would be a lot better if I could schedule restarts, at least then I could do it at night when I'm 99% likely not going to be running any jobs that don't take longer than 30 seconds to shut down.
Or.. another way, can I 'listen' for SIGTERM within a long running DJ and at least abort the loop early so it can resume later?
Manual restarts will reset the 24 hr clock - heroku ps:restart at your preferred time ought to give you the control you are looking for.
More info can be found here: Dynos and the Dyno Manager
Here's the proper answer, you listen for SIGTERM (I'm using DJ here) and then gracefully rescue. It's important that the jobs are idempotent.
Long running delayed_job jobs stay locked after a restart on Heroku
class WithdrawPaymentsJob
def perform
begin
term_now = false
old_term_handler = trap('TERM') { term_now = true; old_term_handler.call }
loop do
puts 'doing long running job'
sleep 1
if term_now
raise 'Gracefully terminating job early...'
end
end
ensure
trap('TERM', old_term_handler)
end
end
end
Here's how you solve it with Que:
if Que.worker_count.zero?
raise 'Gracefully terminating job early...'
end
Ok, so this is probably evil, however.. here's the question! I want to run a pretty lightweight app on a shared environment (site5). Ideally I would like to use delayed_job for the ease of queueing the mails (~200+ every so often). However, being a shared environment they don't want background processes running all the time (fair enough).
So, my plan, such as it is, is to queue the mails using delayed job, and then every hour or something, spin up a cron job, send a few emails (10 or something small) and then kill the process. And repeat.
Q) Is there a rake jobs:works:1 equivalent task it'd be easy to setup? - pointer would be handy.
I'm quite open to "this is a terrible idea, don't even go there" being the answer.. in which case I might look at another queuing strategy... (or heroku hire-fire perhaps..)
You can get delayed job to process only a certain number of jobs by doing:
Delayed::Worker.new.work_off(10)
You could fire a script to do that from cron or use "rails runner":
rails runner -e production 'Delayed::Worker.new.work_off(10)'
I guess the main issue on whether it is a good idea or not is working out what small value is actually high enough to make sure you process all your jobs in a reasonable time-frame. Also, you've got the overhead of firing up the rails environment every time you want to process, or even check whether you should process, any jobs. That might cause problems in a shared environment if they are particularly strict on spikes of memory or CPU usage.
Why not skip the 'workers' (which are just daemons which look for work else sleep) and have your cron fire a custom rake task of 10.times { MailerJob.first.perform }
You'd just need to require you're app in the line before that so its loaded ofc.
With a Rails stack, how can I create a background process that handles events by spawning threads that are worked in real time?
The workers on Heroku pick up jobs every 5 seconds. I need real time. Ideally I'd like to get this working on Heroku, but if I need to, I will move away from it.
This has a long list of background workers: Background Job Manager for Rails 3 but it is not clear if your question heroku specific or not
I think you are looking for something like "run_later" which instead of queueing a job actually returns the request and runs a block in a separate process.
Here is a link to the Rails 3+ version, you can follow the fork network to find many other implementations:
https://github.com/Zelnox/run_later
(I don't use Heroku so I don't know if it runs on it)
Heroku runs rake jobs:work, so you can replace that with your own rake task, either running delayed_job with a shorter than 5 second timeout, or just performing your own task. Probably a good idea to keep a sleep statement in there.
The new cedar stack will run anything you want, so it might be worth checking that out.
With regards to run_later, or spawning from the current request, this does work but if the background process doesn't complete within the 30 second request timeout then heroku will kill it.
I think you need delay_job. please checkout this gem
I have a rails 3 app that writes certain events to a queue.
Now on the server I want to create a service that polls the queue every x seconds, and performs other tasks on a scheduled basis.
Other than creating a ruby script and running it via a cron job, are there other alternatives that are stable?
Although spinning up a persistent Rails-based task is an option, you may want to look at more orderly systems like delayed_job or Starling to manage your workload.
I'd advise against running something in cron since the expense of spinning up a whole Rails stack can be significant. Running it every few seconds isn't practical as the ramp-up time on Rails is usually 5-15 seconds depending on your hardware. Doing this a few times a day is usually no big deal, though.
A simple alternative is to create a work loop in a script you can engage with runner:
interval = 15.minutes
next_time = Time.now + interval
while (true)
if (stuff_to_do?)
do_stuff
end
# Figure out how much time is left before the next iteration
delay = next_time.to_i - Time.now.to_i
if (delay > 0)
# If ahead of schedule, take a break
sleep(delay)
end
end
The downside to this is that the Rails stack will remain in memory as long as this background process is running, but this is a trade-off between huge CPU hits and a memory hit.
You have several options for that, including DelayedJob and Resque.
Resque relies on Redis and is the solution I use all the time (and am very happy with).
I would recommend Ryan Bates' railscast on this subject which talks about beanstalkd and the stalker wrapper for it:
http://railscasts.com/episodes/243-beanstalkd-and-stalker
To add to the possibilities here, Using a more heavy-duty queuing system like AMQP (RabbitMQ) is made easy by the 'minion' gem. Similar to beanstalkd:
https://github.com/orionz/minion
#Blankman, you should check out http://www.simpleworker.com, it's made for things like this and takes the burden of running/scheduling/monitoring your processes off of you. And it's very stable.