I have 5 Resque workers setup like this:
QUEUE=* rake environment resque:work
QUEUE=* rake environment resque:work
QUEUE=* rake environment resque:work
when I run a heavy job like this:
100.times do
Resque.enqueue(DoTheJob)
end
First worker gets about 80 of the job and other workers share the rest...
In my case, I may have 40 concurrent and really heavy jobs -video transcoding-. They will be triggered consecutively and I want jobs to be seperated equally or at least fairly to my existing workers (they can be up to 30).
Is there an option or something like that?
How can I achieve this?
Thank you
Well, when I try the code above with a simplest Resque worker class structure like this:
class MyWorker
#queue=:test
def self.perform(data)
puts "Testing...."
end
end
100.times do
Resque.enqueue(MyWorker)
end
All the jobs are queued to the same worker.
BUT.
When I put a few seconds sleep to the worker, I see that jobs are distrubeted pretty fairly.
class MyWorker #queue=:test
def self.perform(data)
puts "Testing...."
sleep 3
end
end
I don't know what is Resque's algorythm for this purpose but looks like there's no problem with it.
Related
Task:
Delete an item in Async manner [HomeWork]
I have already configured Active jobs with delayed_jobs in my rails application but I am still confused about performing Async task in rails project.
Let's take an example:
I have some item to delete from the database but I want to do it in Async manner. I also read about perform_later or perform_now method in delayed_job blogs. Here is my code which is working fine:
Controller class
def destroy
PostJob.perform_now(params[:id])
respond_to do |format|
format.xml { head :ok }
format.js { render 'posts.js.erb' }
end
end
Job class
class PostJob < ActiveJob::Base
queue_as :default
def perform(id)
#post = Post.find(id)
#post.destroy
end
end
According to official doc of delayed_jobs I can add handle_asynchronously in the end of method to run in async manner. How can I implement in this case?
My Question:
When I am looking at destroy method it is not deleting the element in Async way. However every steps written in destroy method is in Synchronous. Am I wrong?
If it's not then How can I implement destroy method to delete post in async manner?
Backgrounding task and cron job are same thing?
Edit -1
After giving suggestion by A Fader Darkly, I changed perform_now to perform_later which is working perfectly for Async process but it is not deleting the entry from table (code is fine because it works when i user perform_now).
Also when I am running job manually by following command, Everything works fine:
rake jobs:work
Is there any way to execute delay_job task as soon as the queue get some new data?
If you change your destroy method to call:
PostJob.perform_later(params[:id])
it should happen asynchronously. If not, you have some more set-up to do.
For your questions:
Yes you are right, but what you say is a tautology. Everything in that method is synchronous - the job queue isn't used because of the perform_now. Thus destroy isn't deleting in an async way.
See above.
Cron jobs work on the operating system level and are scheduled regularly for particular times. You could have a cron job working every minute, for example, or every day, or week (on a particular day at a particular time). They run from a schedule file called a crontab.
'Backgrounding' a task simply stops it from taking over the IO of your terminal session. So you can carry on using the terminal while the process runs in the background. Generally this is done on an ad-hoc basis, so you don't have to wait for a heavy operation to complete before going on to do different tasks.
EDIT
Based on edits to the question, it sounds like Delayed Job daemon needs to be started. From the instructions:
Note: For Rails 4 replace script/delayed_job with bin/delayed_job
When running a queue locally, omit the 'RAILS_ENV=production' part of commands.
Running Jobs
script/delayed_job can be used to manage a background process which will start working off jobs.
To do so, add gem "daemons" to your Gemfile and make sure you've run rails generate delayed_job.
You can then do the following:
RAILS_ENV=production script/delayed_job start
RAILS_ENV=production script/delayed_job stop
Runs two workers in separate processes.
RAILS_ENV=production script/delayed_job -n 2 start
RAILS_ENV=production script/delayed_job stop
Set the --queue or --queues option to work from a particular queue.
RAILS_ENV=production script/delayed_job --queue=tracking start
RAILS_ENV=production script/delayed_job --queues=mailers,tasks start
Use the --pool option to specify a worker pool. You can use this option multiple times to start different numbers of workers for different queues.
The following command will start 1 worker for the tracking queue,
2 workers for the mailers and tasks queues, and 2 workers for any jobs:
RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start
Runs all available jobs and then exits
RAILS_ENV=production script/delayed_job start --exit-on-complete
or to run in the foreground
RAILS_ENV=production script/delayed_job run --exit-on-complete
I am a beginner when it comes to Rails. I am trying to follow this example:
http://ryanselk.com/2014/09/25/using-background-jobs-in-rails-42-with-active-job/
It says:
"Jobs can be added to the job queue from anywhere. We can add a job to the queue by: ResizeImage.perform_later 'http://example.com/ex.png' "
[UPDATE] Sorry, I am stupid. I came up with this task:
namespace :simple do
# call from command line:
# rake simple:resize_images
desc "Resize images"
task resize_images: :environment do
Dir.foreach('storage') do |next_image|
puts next_image
next if next_image == '.' or next_image == '..'
ResizeImage.perform_later next_image
end
end
end
but now I do:
rake simple:resize_images
and I get:
zacek2_phpP9JGif.jpg
rake aborted!
NameError: uninitialized constant ResizeImage
I've tried:
require ResizeImage
but that did not fix the problem.
I am afraid I don't understand how loading works in Rails. How do I load ResizeImage?
Do I set it up as a cron job?
No.
From the rails guides:
Active Job is a framework for declaring jobs and making them run on a variety of queueing backends.
Active Job is an interface to queueing backends such as sidekiq, delayed_job or resque. It's simply a way for you to write background jobs where you don't have to care about which of the queueing backends will be used.
How do I start ActiveJob?
So ActiveJob doesn't run background jobs on it's own. You're still missing one of the backends. Say you have decided to use delayed_job: Get it installed and then start it via:
script/delayed_job start
I don't understand where "anywhere" is.
That means anywhere in your code, you could write something like:
user.rb
def send_registration_email
UserRegistraionMailJob.perform_later self
end
The documentation states that a delayed job worker can be invoked using a rake task like so: rake jobs:work, or QUEUE=queue1 rake jobs:work if you want it to work on a specific queue.
I have a couple of questions about this way to invoke jobs:
Is there a way to pass other parameters like sleep-delay or read-ahead (like you would do if you start the worker using the script: delayed_job start --sleep-delay 30 --read-ahead 500 --queue=queue1)?
Is there any gain in processing speed if you launch 2 workers on the same queue using the rake task?
In answer to 1. - yes you can set sleep delay and read ahead from the command line. You do it via environment variables:
QUEUE=queue1 SLEEP_DELAY=1 rake jobs:work
for example. See this commit.
rake jobs:work is just a means to an end to put up another worker, for development purposes or to work off a big queue (you have rake jobs:workoff for this though) so all benefits and disclaimers of multiple workers apply,
two jobs process in parallel so if you've got the cpu power your queue will be worked quicker
I don't know about the question #1 though, it's possible rake jobs wasn't intended to be used outside of development
I've got Resque workers that typically shouldn't take longer than about 1-5 minutes to run, but frequently those workers will get "stuck" and go idle, clogging up workers and doing nothing.
So I'd like to regularly check for workers that have been running longer than X time and purge them. But I need to do this automatically, so I don't have to personally go in and manually clear them (Resque.workers.each {|w| w.unregister_worker}) every few hours.
This needs to work on Heroku.
Put this into a rake task:
allocated_time = 60 * 60 # 1 hour
Resque::WorkerRegistry.working.each do |worker|
if (worker.started <=> Time.now - allocated_time) < 1
worker.unregister
end
end
Use heroku scheduler, you can set it to minimum of 10 minutes if that suites.
For Resque v1,
# lib/tasks/clear_stale_workers.rake
namespace :clear do
desc 'Clearing stuck workers ...'
task :stale_workers => :environment do
Resque.workers.each do |w|
w.unregister_worker unless w.started > 1.hour.ago
end
end
end
From the command line, rake clear:stale_workers
On Heroku, set the set the scheduler to run this Rake task.
This worked for me to remove the specific workers running stale jobs. You could add it to a rake task.
Resque::Worker.working.each{|w| w.done_working }
I am using this plugin for scheduled job.But it is not working.
I am confused about some points,Should I need to create the Job class and set their name in to schedule file?When testing it then,Should I run the rescue scheduler and Resque worker both or only one of them.
Thanks in advance.
My Resque Scheduler config... you will mostly need all these pieces:
YML file (config/resque_scheduler.yml):
every_1_minute:
cron: "* * * * *"
class: EveryMinute
queue: some_queue
description: Tasks to perform every minute
config/initializers/resque.rb:
require 'resque_scheduler'
Resque.schedule = YAML.load_file(File.join(Rails.root, 'config/resque_scheduler.yml'))
Ruby class (lib/every_minute.rb or somewhere in the load path):
class EveryMinute
def self.perform
puts "Hello every minute!"
end
end
You need to run
rake resque:scheduler
rake resque:work
The resque:scheduler process periodically queues up jobs, hence the scheduling. And the workers will just do the jobs blindly. This is why you need BOTH to successfully schedule and run jobs periodically.