The documentation states that a delayed job worker can be invoked using a rake task like so: rake jobs:work, or QUEUE=queue1 rake jobs:work if you want it to work on a specific queue.
I have a couple of questions about this way to invoke jobs:
Is there a way to pass other parameters like sleep-delay or read-ahead (like you would do if you start the worker using the script: delayed_job start --sleep-delay 30 --read-ahead 500 --queue=queue1)?
Is there any gain in processing speed if you launch 2 workers on the same queue using the rake task?
In answer to 1. - yes you can set sleep delay and read ahead from the command line. You do it via environment variables:
QUEUE=queue1 SLEEP_DELAY=1 rake jobs:work
for example. See this commit.
rake jobs:work is just a means to an end to put up another worker, for development purposes or to work off a big queue (you have rake jobs:workoff for this though) so all benefits and disclaimers of multiple workers apply,
two jobs process in parallel so if you've got the cpu power your queue will be worked quicker
I don't know about the question #1 though, it's possible rake jobs wasn't intended to be used outside of development
Related
I have ROR application and 1 delay_job process ran using rake job:work.
ROR application add Job in multiple queue.
Lets say we have queue 1 and queue 2.
My Question is task in queue 1 and task in queue 2 will be executed concurrently?
Currently in my application after running rake job:work process only 1 thread is spawn which executes queue1 task and then queue2 task.
If i have to execute in parallel, i have to run two rake task of job:work.
Is it correct behavior or it can be run concurrently in 1 rake task of job:work.
And what is worker in Delay Job. Is delay Job interchangeably used with worker
Thanks
Priyanka
No, one worker cannot run two jobs concurrently, you need more than one process running for that.
In the example you are describing, you are starting a worker that is running in the foreground (rake job:work), but what you could do instead, is to start them as background workers, by running bin/delayed_job instead (script/delayed_job for earlier versions). That command has multiple options that you can use to specify how you want delayed_job to function.
One of the options is -n or --number_of_workers=workers. That means you can start two workers by running the following command:
bundle exec bin/delayed_job --number_of_workers=2 start
It is also possible to dedicate certain workers to only run jobs from a specific queue, or only high priority jobs.
I have a task that takes over 45 minutes. It runs successfully with
$ heroku run rake:sales
I also doublechecked my settings in scheduler based on this question. Everything looks fine.
In order to prevent run-away jobs, jobs that run longer than their frequency will be terminated. For example, a job that runs every 10 minutes will be terminated after running for 10 minutes.
What's happening is that your rake task is running for the first 10 minutes, but Heroku aborts it after that elapses. They suggest using a background job queue for long-running tasks.
Source:
https://devcenter.heroku.com/articles/scheduler#long-running-jobs
Apart from the long-running issue that #KKobayashi has alluded to, you may not have the correct rake file created for the scheduler to run:
Heroku Scheduler:
For Rails, the convention is to set up rake tasks. To create your
scheduled tasks in Rails, copy the code below into
lib/tasks/scheduler.rake and customize it to fit your needs
Have you tried putting your tasks into a a scheduler.rake file?
It could be that you're scheduling the task for an app other than the one you intend to schedule it for.
To check, open the scheduler (heroku addons:open scheduler) and check the url. If you see another app's name in the url, you need to add the scheduler addon again i.e.:
heroku addons:create scheduler:standard
Now open it again (heroku addons:open scheduler)
I have a few long-running restarts of processes in my deploy.rb like:
rake assets:precompile
script/delayed_job restart
rake sunspot:solr:stop, rake sunspot:solr:start
All of these processes have to occur, but not necessarily one after another.
I was wondering if I can run the assets:precompile and the delayed_job restart simultaneously, as they don't need to happen one after another, and I could speed up my deploy time by doing them asynchronously.
I've run some Google searches but I can't find anything about it.
This is not a feature that capistrano supports.
I have been looking around for a solution and found something on the Capistrano google groups. The suggestion was to use Capistrano to run a ruby script that runs the jobs in parallel using Ruby's own threading support.
If you read the post one of the authors does ask why do these tasks need to run in parallel because you can introduce race conditions and other non-deterministic behaviour which can make the deployment process more brittle.
In Resque, if I have the following:
bundle exec rake resque:work QUEUE=critical,high,normal,low,aggregate
How do I indicate that I only want one, and ONLY ONE worker, for a specific queue (i.e. the aggregate queue)?
I dont think that is possible
Reason if you look the current code over here
Resque poll all the queue
value = #redis.blpop(*(queue_names + [1])) until value
where queue_names is your critical,high,normal,low,aggregate
so the point over here Irrespective you if you use single resque work or mutilple workers
using resque:workers each of the resque work poll all the available queue and once the
message is consumed from any on the queue the work start acting upon it
If you want to assign only one work for the above queue it would be better running two rake
task like this
bundle exec rake resque:worker COUNT=4 QUEUE=critical,high,normal,low
bundle exec rake resque:work QUEUE=aggregate
This way you can assign a single resque worker for the aggregate queue
Hope this help
I'm using Heroku and would like to have a script (on another server) called via cron that periodically requests an action on my Heroku app. This action will in turn run some processing that may take 1 or 2 minutes to complete. I believe Heroku has 30 second request limit, I was thinking could call a Rake task from my controller action instead.
Would this work? I'm curious if anyone has tried this yet.
Thanks.
The rake task would work as long as you don't use a HTTP request as proxy to initiate the task. In fact, if the task is forked from a HTTP Request, the timeout will be the same of the HTTP request.
You should a different method to start the task. Either a crontab (on Heroku side) or a Worker as good solutions.
I'd recommend using a background job on a worker for this. Your periodic process would then just have to start the worker and it wouldn't matter how long the process took.
I've just created a gem to solve exactly this problem. It allows you to queue up any rake task as a delayed_job e.g.
rake delay:db:seed
which will execute
rake db:seed
as a delayed_job. You can find it at http://rubygems.org/gems/delayed_task or http://blog.opsb.co.uk/long-running-rake-tasks-on-heroku.