In Resque, if I have the following:
bundle exec rake resque:work QUEUE=critical,high,normal,low,aggregate
How do I indicate that I only want one, and ONLY ONE worker, for a specific queue (i.e. the aggregate queue)?
I dont think that is possible
Reason if you look the current code over here
Resque poll all the queue
value = #redis.blpop(*(queue_names + [1])) until value
where queue_names is your critical,high,normal,low,aggregate
so the point over here Irrespective you if you use single resque work or mutilple workers
using resque:workers each of the resque work poll all the available queue and once the
message is consumed from any on the queue the work start acting upon it
If you want to assign only one work for the above queue it would be better running two rake
task like this
bundle exec rake resque:worker COUNT=4 QUEUE=critical,high,normal,low
bundle exec rake resque:work QUEUE=aggregate
This way you can assign a single resque worker for the aggregate queue
Hope this help
Related
I have ROR application and 1 delay_job process ran using rake job:work.
ROR application add Job in multiple queue.
Lets say we have queue 1 and queue 2.
My Question is task in queue 1 and task in queue 2 will be executed concurrently?
Currently in my application after running rake job:work process only 1 thread is spawn which executes queue1 task and then queue2 task.
If i have to execute in parallel, i have to run two rake task of job:work.
Is it correct behavior or it can be run concurrently in 1 rake task of job:work.
And what is worker in Delay Job. Is delay Job interchangeably used with worker
Thanks
Priyanka
No, one worker cannot run two jobs concurrently, you need more than one process running for that.
In the example you are describing, you are starting a worker that is running in the foreground (rake job:work), but what you could do instead, is to start them as background workers, by running bin/delayed_job instead (script/delayed_job for earlier versions). That command has multiple options that you can use to specify how you want delayed_job to function.
One of the options is -n or --number_of_workers=workers. That means you can start two workers by running the following command:
bundle exec bin/delayed_job --number_of_workers=2 start
It is also possible to dedicate certain workers to only run jobs from a specific queue, or only high priority jobs.
I'm trying to use cron in my application to send mails every week but I think it doesn't work on Windows.
Does anybody knows any equivalent to cron solution that works on Windows?
Windows equivalent of Unix's cron is a "Task Scheduler". You can configure your periodical task there.
Purely Ruby solution
If you want a purely Ruby solution look into:
rufus-scheduler - it's Windows cron gem.
crono - it's a in-Rails cron scheduler, so it should work anywhere.
Web services - there are plenty of free online services that would make a request to a given URL in specific time periods. This is basically a poor man's cronjob.
I recommend taking a look at Resque and the extension Resque-scheduler gems. You will need to have a resque scheduler process running with bundle exec rake resque:scheduler and at least one worker process running with QUEUE=* bundle exec rake resque:work.
If you want these services to run in the background as a windows service, you can do it with srvany.exe as described in this SO question.
The above assumes you are ok with installing Redis - a key-value store that is very popular among the Rails community as it can be easily used to support other Rails components such as caching and ActionCable, and it is awesome by itself for many multi-process use cases.
Resque is a queue system on top of Redis that allows you to define jobs that can be executed asynchronously in the background. When you run QUEUE=* bundle exec rake resque:work, a worker process runs constantly and polls the queue. Once a job is enqueued, an available worker pops it from the queue and starts working on it. This architecture is quite scalable, as you can have multiple workers listening to the queues if you'd like.
To define a job, you do this:
class MyWeeklyEmailSenderJob
def self.perform
# Your code to send weekly emails
end
end
While you can enqueue this job to the queue yourself from anywhere (e.g. from a controller as a response to an action), in your case you want it to automatically be placed into the queue once a week. This is what Resque-scheduler is for. It allows you to configure a file such as app/config/resque_schedule.yml in which you can define which jobs should be enqueued in which time interval. For example:
send_weekly_emails:
cron: 0 8 * * Mon
class: MyWeeklyEmailSenderJob
queue: email_sender_queue
description: "Send weekly emails"
Remember that a scheduling process has to run in order for this to work with bundle exec rake resque:scheduler.
thanks guys , actually i tried rufus scheduler gem and it worked for me , i guess it's the best and easier solution
I have a task that takes over 45 minutes. It runs successfully with
$ heroku run rake:sales
I also doublechecked my settings in scheduler based on this question. Everything looks fine.
In order to prevent run-away jobs, jobs that run longer than their frequency will be terminated. For example, a job that runs every 10 minutes will be terminated after running for 10 minutes.
What's happening is that your rake task is running for the first 10 minutes, but Heroku aborts it after that elapses. They suggest using a background job queue for long-running tasks.
Source:
https://devcenter.heroku.com/articles/scheduler#long-running-jobs
Apart from the long-running issue that #KKobayashi has alluded to, you may not have the correct rake file created for the scheduler to run:
Heroku Scheduler:
For Rails, the convention is to set up rake tasks. To create your
scheduled tasks in Rails, copy the code below into
lib/tasks/scheduler.rake and customize it to fit your needs
Have you tried putting your tasks into a a scheduler.rake file?
It could be that you're scheduling the task for an app other than the one you intend to schedule it for.
To check, open the scheduler (heroku addons:open scheduler) and check the url. If you see another app's name in the url, you need to add the scheduler addon again i.e.:
heroku addons:create scheduler:standard
Now open it again (heroku addons:open scheduler)
The documentation states that a delayed job worker can be invoked using a rake task like so: rake jobs:work, or QUEUE=queue1 rake jobs:work if you want it to work on a specific queue.
I have a couple of questions about this way to invoke jobs:
Is there a way to pass other parameters like sleep-delay or read-ahead (like you would do if you start the worker using the script: delayed_job start --sleep-delay 30 --read-ahead 500 --queue=queue1)?
Is there any gain in processing speed if you launch 2 workers on the same queue using the rake task?
In answer to 1. - yes you can set sleep delay and read ahead from the command line. You do it via environment variables:
QUEUE=queue1 SLEEP_DELAY=1 rake jobs:work
for example. See this commit.
rake jobs:work is just a means to an end to put up another worker, for development purposes or to work off a big queue (you have rake jobs:workoff for this though) so all benefits and disclaimers of multiple workers apply,
two jobs process in parallel so if you've got the cpu power your queue will be worked quicker
I don't know about the question #1 though, it's possible rake jobs wasn't intended to be used outside of development
I'm using Heroku and would like to have a script (on another server) called via cron that periodically requests an action on my Heroku app. This action will in turn run some processing that may take 1 or 2 minutes to complete. I believe Heroku has 30 second request limit, I was thinking could call a Rake task from my controller action instead.
Would this work? I'm curious if anyone has tried this yet.
Thanks.
The rake task would work as long as you don't use a HTTP request as proxy to initiate the task. In fact, if the task is forked from a HTTP Request, the timeout will be the same of the HTTP request.
You should a different method to start the task. Either a crontab (on Heroku side) or a Worker as good solutions.
I'd recommend using a background job on a worker for this. Your periodic process would then just have to start the worker and it wouldn't matter how long the process took.
I've just created a gem to solve exactly this problem. It allows you to queue up any rake task as a delayed_job e.g.
rake delay:db:seed
which will execute
rake db:seed
as a delayed_job. You can find it at http://rubygems.org/gems/delayed_task or http://blog.opsb.co.uk/long-running-rake-tasks-on-heroku.