How many workers are initialized by default in Resque queues when the following command is executed?
QUEUE=(QUEUE NAME) rake environment resque:work
rake resque:work starts exactly one worker. If you want to start multiple workers (for example four workers) you need to run:
COUNT=4 rake resque:workers
When you start resque:workers without a COUNT or with a COUNT of 0 then no workers will be started.
You might want to have a look at the implementation of the resque's rake tasks.
Related
My application is processing jobs with Resque.enqueue
I am starting the worker with rake resque:work QUEUE='converter'
To my understanding, in order to start 2 workers to process 2 requests for the same queue concurrently, I can start another worker with rake resque:work QUEUE='converter' in another terminal.
Is there a simple option to start 2 workers to work concurrently on the same queue without using the resque-pool gem, and without having to type in rake resque:work QUEUE='converter' twice?
It's advised within the same Resque codebase to only use COUNT=2 while on development environment. To manage more than one Resque worker, you'd need something like https://github.com/nevans/resque-pool
You can specify the worker count when running the command, like so:
$ COUNT=2 QUEUE='converter' rake resque:workers
In development mode, we use rake jobs:work. In the same way, inorder to test in the production mode, we use RAILS_ENV=production rake jobs:work.
As entire my application is on Apache Nginx server, is there any option like any gem / code that runs background and how it is used to run the jobs without running this command?
Delayed job is great if you don't have Redis, if you are already using Redis I would recommend Sidekiq over delayed job. The main difference is delayed job is an SQL based job worker and Sidekiq uses Redis.
Check out the Sidekiq: Getting Started guide for more information about using Sidekiq.
Delayed also comes with a script to run jobs in the background.
From the README: Running Jobs
script/delayed_job can be used to manage a background process which will
start working off jobs.
To do so, add gem "daemons" to your Gemfile and make sure you've run rails
generate delayed_job.
You can then do the following:
RAILS_ENV=production script/delayed_job start
RAILS_ENV=production script/delayed_job stop
# Runs two workers in separate processes.
RAILS_ENV=production script/delayed_job -n 2 start
RAILS_ENV=production script/delayed_job stop
# Set the --queue or --queues option to work from a particular queue.
RAILS_ENV=production script/delayed_job --queue=tracking start
RAILS_ENV=production script/delayed_job --queues=mailers,tasks start
# Use the --pool option to specify a worker pool. You can use this option multiple times to start different numbers of workers for different queues.
# The following command will start 1 worker for the tracking queue,
# 2 workers for the mailers and tasks queues, and 2 workers for any jobs:
RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start
# Runs all available jobs and then exits
RAILS_ENV=production script/delayed_job start --exit-on-complete
# or to run in the foreground
RAILS_ENV=production script/delayed_job run --exit-on-complete
Rails 4: replace script/delayed_job with bin/delayed_job
Workers can be running on any computer, as long as they have access to the
database and their clock is in sync. Keep in mind that each worker will check
the database at least every 5 seconds.
You can also invoke rake jobs:work which will start working off jobs. You can
cancel the rake task with CTRL-C.
If you want to just run all available jobs and exit you can use rake jobs:workoff
Work off queues by setting the QUEUE or QUEUES environment variable.
QUEUE=tracking rake jobs:work
QUEUES=mailers,tasks rake jobs:work
A couple more things:
You should always specify which queues to run.
You'll also need to ensure the script is run when your app is deployed. (You can do this with Capistrano, Mina, Foreman, upstart and many other ways.)
Miad is correct, Sidekiq is likely what you are looking for, unless you are literally talking about using the delayed job gem, which is another queue adapter. Sidekiq is basically a queue adapter that connects Rails' ActiveJob with Redis. You can create Jobs with ActiveJob and kick them off by calling the perform method from your Job class. This will queue them in a Sidekiq queue, pass them to redis, and they will be performed asynchronously. Your code might look something like this:
in app/jobs/your_job.rb
class YourJob < ActiveJob::Base
#specify the name of your queue
queue_as :the_queue
# you must define perform, this is where the async magic happens
def perform(something)
do_stuff_to(something)
end
end
in app/models/place_where_job_is_kicked_off.rb
class PlaceWhereJobIsKickedOff
def do_the_jobs
Something.all.each do |something|
# enqueue your jobs to be performed as soon as the queueing system is free. The queue size is set in your sidekiq.yml
# each job will be enqueued and run asynchronously, so watch out for race conditions.
YourJob.perfom_later(something)
end
end
end
in app/config/enviroments/production.rb
Rails.application.configure do
#other production configs...
#set the ActiveJob queue adapter to sidekiq
config.active_job.queue_adapter = :sidekiq
#other production configs...
end
in app/config/sidekiq.yml
:verbose: true
:pidfile: tmp/pids/sidekiq.pid
:logfile: log/sidekiq.log
# 5 jobs can run asynchronously simultaneously
:concurrency: 5
:queues:
# queue names go here [name, size]
- [the_queue, 5]
Make sure that your have Redis installed and running on your production server, and sidekiq is running. After adding the sidekiq gem to gemfile, and bundle installing, run:
sidekiq -C path/to/sidekiq.yml -d (-e environment)
this will start sidekiq as daemon process.
I think what you are looking for is sidekiq gem. it is used in order to run jobs asynchronously.
http://sidekiq.org
I've seen some documentation about resque, and all the tips tell to run the queues with this command:
rake resque:work QUEUE='*'
What is this command really doing? Is it running all the queues in just one job?
I didn't find documentation talking about how to run the queues one by one. But, is there any performance difference considering that I will run all of them in the same server?
Next command
rake resque:work QUEUE='*'
creates a single process that includes all your queues, so definitely, it won't have a good performance. If you have multiples queues, it will work as if they were sequential.
If you want to have each queue running in a different process, you should execute your queues one by one:
rake resque:work QUEUE=queue_one &
rake resque:work QUEUE=queue_two &
rake resque:work QUEUE=queue_three &
I have multiple jobs running in multiple queues with Resque. My problem is that when one queue is running lots of jobs, it holds up my other queues.
How can I get Resque to run parallel queues but still limit each queue to one job at a time?
You can launch different workers for each of your queue.
In order to do this you have to specify the name of the queue to process on the QUEUE env variable.
So if you have 3 queues you'd do the following:
QUEUE=queue1 bundle exec rake resque:work
QUEUE=queue2 bundle exec rake resque:work
QUEUE=queue3 bundle exec rake resque:work
I have these Resque workers defined in my Procfile
resque1: env QUEUE=mailer rake resque:work
resque2: env QUEUE=calculation rake resque:work
But seems non of them gets created, is it because the process type is wrong? If yes, how can I create two resque workers for my rails app?
[Problem Solved]
The reason been heroku only give one free dyno and the rails app already used one. Once I scale the web dyno to 0, and scale my resque dyno to 1 instance, the queued job gets executed immediately.
Thanks!