I have a rails app using an engine where Sidekiq workers are defined. The worker's perform_async is invoked in a controller within the engine. The worker's perform does the work on arguments passed in through that controller. The worker specific queue is defined in the worker class too. However, when a request comes in to that controller, it gets pushed to the redis server from 'perform_async', to the right queue. A bundle exec sidekiq starts up Sidekiq. However, the worker's perform never gets executed. Checking the sidekiq UI, I can see that the job is in the right queue.
This is how my worker looks like
require 'pando'
class PandoWorker
include Sidekiq::Worker
sidekiq_options :queue => :pando, :backtrace=> true
def perform(*args)
puts "in here"
puts args
end
end
So in this case the sidekiq UI shows that the args are queued in 'pando'. The sidekiq process never processes from that queue or even the default.
You have to tell sidekiq process which queue to look at:
bundle exec sidekiq -q pando
Otherwise process is only watching 'default' queue.
Related
I'm using delayed_job library as adapter for Active Jobs in Rails:
config.active_job.queue_adapter = :delayed_job
My delayed_job worker is configured to sleep for 60 seconds before checking the queue for new jobs:
Delayed::Worker.sleep_delay = 60
In my Rails code I add a new job to the queue like this:
MyJob.perform_later(data)
However, the job will not be picked up by the delayed_job worker immediately, even if there are no jobs in the queue, because of the sleep_delay. Is there a way to tell the delayed_job worker to wake up and start processing the job queue if it's sleeping?
There is a MyJob.perform_now method, but it blocks the thread, so it's not what I want because I want to execute a job asynchronously.
Looking at the delayed_job code and it appears that there's no way to directly control or communicate with the workers after they are daemonized.
I think the best you can do would be to start a separate worker with a small sleep_delay that only reads a specific queue, then use that queue for these jobs. A separate command is necessary because you can't start a worker pool where the workers have different sleep delays:
Start your main worker: bin/delayed_job start
Start your fast worker: bin/delayed_job start --sleep-delay=5 --queue=fast --identifier=999 (the identifier is necessary to differentiate the workers daemons)
Update your job to use that queue:
class MyJob < ApplicationJob
queue_as :fast
def perform...
end
Notes:
When you want to stop the workers you'll also have to do it separately: bin/delayed_job stop and bin/delayed_job stop --identifier=999.
This introduces some potential parallelism and extra load on the server when both workers are working at the same time.
The main worker will process jobs from the fast queue too, it's just a matter of which worker grabs the job first. If you don't want that you need to setup the main worker to only read from the other queue(s), by default that's only 'default', so: bin/delayed_job start --queue=default.
I'm trying to send email campaigns in a rails app with the Heya gem and GoodJob. The example in the Heya readme as well as the Heya example app uses Sidekiq as the Active Job backend.
I'm confused about how to actually send the Heya campaigns with GoodJob.
The docs for Heya show this example of starting Sidekick: bundle exec sidekiq -q default -q heya
I assume that there is a Job queue somewhere in the gem called "Heya", but I can't find this in the source code. Do I need to create one?
Do I need to create a job that runs the Heya scheduler? While the example app uses Sidekiq, I also don't see any custom jobs in that app.
I have the following setup for GoodJob and it appears to be running fine with good_job start which should run all of the jobs and queues, but I've also tried good_job start --queues=heya,default.
Here is the relevant code:
Profile.dev
web: bin/rails server -p 3000
css: bin/rails tailwindcss:watch
worker: bundle exec good_job start
config/initializers/heya.rb
Heya.configure do |config|
config.user_type = "User"
config.campaigns.priority = [
"WelcomeCampaign",
]
end
app/jobs/application_job.rb
class ApplicationJob < ActiveJob::Base
# Automatically retry jobs that encountered a deadlock
# retry_on ActiveRecord::Deadlocked
# Most jobs are safe to ignore if the underlying records are no longer available
# discard_on ActiveJob::DeserializationError
end
app/campaigns/application_campaign.rb
class ApplicationCampaign < Heya::Campaigns::Base
segment :email_subscriber?
default from: "#{I18n.t('settings.site_name')} <#{I18n.t('settings.newsletter_email')}>"
end
app/campaigns/welcome_campaign.rb
class WelcomeCampaign < ApplicationCampaign
default wait: 5.minutes,
layout: "newsletter"
step :intro, wait: 0.minutes,
subject: "Welcome to #{I18n.t('settings.site_name')}"
end
I also have a layout and views for the campaign similar to the Heya example app, and I'm using Mailcatcher to see if any email is being sent.
What am I missing to send these emails with Heya and GoodJob?
Note that I'm subscribing the users on signups like this:
class User < ApplicationRecord
after_create_commit :add_user_to_newsletters
private
def add_user_to_newsletters
WelcomeCampaign.add(self)
EvergreenCampaign.add(self)
self.update(email_subscriber: true)
end
end
And the default segment in campaigns/application_campaign.rb is segment :email_subscriber?
If I run User.last.email_subscriber? in the console to check this it returns true.
I feel like I'm missing something about how Heya connects to Active Job that is not obvious in the Heya docs.
Also, not sure if this is related, but I added this to config/puma.rb
# https://github.com/bensheldon/good_job#execute-jobs-async--in-process
before_fork do
GoodJob.shutdown
end
on_worker_boot do
GoodJob.restart
end
on_worker_shutdown do
GoodJob.shutdown
end
MAIN_PID = Process.pid
at_exit do
GoodJob.shutdown if Process.pid == MAIN_PID
end
preload_app!
Are you running the heya scheduler periodically? $ rails heya:scheduler
It looks like you could create your own background job to be run using GoodJob Cron, by executing Heya::Campaigns::Scheduler.new.run to run the scheduler and enqueue the emails.
Reading the "Running the Scheduler" part of the README explains what's happening:
To start queuing emails, run the scheduler task periodically:
rails heya:scheduler
Heya uses ActiveJob to send emails in the background. Make sure your
ActiveJob backend is configured to process the heya queue. By default, GoodJob runs from all queues "*".
You can change Heya's default queue using the queue option:
# app/campaigns/application_campaign.rb
class ApplicationCampaign < Heya::Campaigns::Base
default queue: "custom"
end
Task:
Delete an item in Async manner [HomeWork]
I have already configured Active jobs with delayed_jobs in my rails application but I am still confused about performing Async task in rails project.
Let's take an example:
I have some item to delete from the database but I want to do it in Async manner. I also read about perform_later or perform_now method in delayed_job blogs. Here is my code which is working fine:
Controller class
def destroy
PostJob.perform_now(params[:id])
respond_to do |format|
format.xml { head :ok }
format.js { render 'posts.js.erb' }
end
end
Job class
class PostJob < ActiveJob::Base
queue_as :default
def perform(id)
#post = Post.find(id)
#post.destroy
end
end
According to official doc of delayed_jobs I can add handle_asynchronously in the end of method to run in async manner. How can I implement in this case?
My Question:
When I am looking at destroy method it is not deleting the element in Async way. However every steps written in destroy method is in Synchronous. Am I wrong?
If it's not then How can I implement destroy method to delete post in async manner?
Backgrounding task and cron job are same thing?
Edit -1
After giving suggestion by A Fader Darkly, I changed perform_now to perform_later which is working perfectly for Async process but it is not deleting the entry from table (code is fine because it works when i user perform_now).
Also when I am running job manually by following command, Everything works fine:
rake jobs:work
Is there any way to execute delay_job task as soon as the queue get some new data?
If you change your destroy method to call:
PostJob.perform_later(params[:id])
it should happen asynchronously. If not, you have some more set-up to do.
For your questions:
Yes you are right, but what you say is a tautology. Everything in that method is synchronous - the job queue isn't used because of the perform_now. Thus destroy isn't deleting in an async way.
See above.
Cron jobs work on the operating system level and are scheduled regularly for particular times. You could have a cron job working every minute, for example, or every day, or week (on a particular day at a particular time). They run from a schedule file called a crontab.
'Backgrounding' a task simply stops it from taking over the IO of your terminal session. So you can carry on using the terminal while the process runs in the background. Generally this is done on an ad-hoc basis, so you don't have to wait for a heavy operation to complete before going on to do different tasks.
EDIT
Based on edits to the question, it sounds like Delayed Job daemon needs to be started. From the instructions:
Note: For Rails 4 replace script/delayed_job with bin/delayed_job
When running a queue locally, omit the 'RAILS_ENV=production' part of commands.
Running Jobs
script/delayed_job can be used to manage a background process which will start working off jobs.
To do so, add gem "daemons" to your Gemfile and make sure you've run rails generate delayed_job.
You can then do the following:
RAILS_ENV=production script/delayed_job start
RAILS_ENV=production script/delayed_job stop
Runs two workers in separate processes.
RAILS_ENV=production script/delayed_job -n 2 start
RAILS_ENV=production script/delayed_job stop
Set the --queue or --queues option to work from a particular queue.
RAILS_ENV=production script/delayed_job --queue=tracking start
RAILS_ENV=production script/delayed_job --queues=mailers,tasks start
Use the --pool option to specify a worker pool. You can use this option multiple times to start different numbers of workers for different queues.
The following command will start 1 worker for the tracking queue,
2 workers for the mailers and tasks queues, and 2 workers for any jobs:
RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start
Runs all available jobs and then exits
RAILS_ENV=production script/delayed_job start --exit-on-complete
or to run in the foreground
RAILS_ENV=production script/delayed_job run --exit-on-complete
I've this worker that runs for ever.
class Worker
include Sidekiq::Worker
sidekiq_options queue: "infinity", retry: true
def perform(params)
# ...
self.class.perform_in(30.seconds, params)
end
end
The problem is that I load workers on start up, like this. config/initializers/load_workers.rb
Rails.application.config.after_initialize do
if ENV["SIDEKIQ"] == "1"
Worker.perform_async({})
end
end
Using this to start sidekiq SIDEKIQ=1 sidekiq --verbose --environment production -C config/sidekiq.yml.
This means that old workers as to stop, both those currently running but also the ones being rescheduled.
I tried running this on start up (just before loading new works), but that didn't work.
q = []
q += Sidekiq::RetrySet.new.select { |job| job.klass.match(/Worker/) }
q += Sidekiq::Queue.new("infinity").select { |job| job.klass.match(/Worker/) }
q += Sidekiq::ScheduledSet.new.select { |job| job.klass.match(/Worker/) }
q.each(&:delete)
After 5-ish deploys there are bunch of duplicate workers in the queue scheduled for later. So, is there a way to clear everyting in one queue and prevent already running jobs from rescheduling?
I'm using sidekiq 3.0.
Deletes all Jobs in a Queue, by removing the queue.
require 'sidekiq/api' # for the case of rails console
Sidekiq::Queue.new("infinity").clear
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
This will clear all queues, schedules and retries:
require 'sidekiq/api'
Sidekiq::Queue.all.each(&:clear)
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
Sidekiq::DeadSet.new.clear
Works for me for most sidekiq versions:
Sidekiq::RetrySet.new.clear
Sidekiq::ScheduledSet.new.clear
Clear statistics (Optional)
Sidekiq::Stats.new.reset
There is one more convenient way to clear all Sidekiq queues and sets for local env: Sidekiq.redis(&:flushdb)
Basically, it just flushes Redis that is configured for Sidekiq, so I would avoid using it in a non-local environment, as it can also remove some data you store in Redis. Anyways, it can be useful for the development or when you have a separate Redis instance for Sidekiq. And it's just one line instead of four.
You can clear your queue by running this code although there would be built-in methods.
queue = Sidekiq::Queue.new
queue.each do |job|
job.delete
end
I am using Sidekiq for my background jobs:
I have a worker app/workers/data_import_worker.rb
class DataImportWorker
include Sidekiq::Worker
sidekiq_options retry: false
def perform(job_id,file_name)
begin
#Some logic in it .....
end
end
Called from a file lib/parse_excel.rb
def parse_raw_data
#job_id and #filename are defined bfr
DataImportWorker.perform_async(job_id,filename)
end
As soon as i trigger it from my action the worker is not getting called.. Redis is running on localhost:6379
Any idea why this must be happening. The Environment is Linux.
I had a similar problem where Sidekiq was running but when I called perform_async it didn't do anything except return true.
The problem was rspec-sidekiq was added to my ":development, :test" group. I fixed the problem by moving rspec-sidekiq to the ":test" group only.
Start sidekiq from the root directory of your Rails app. For example,
bundle exec sidekiq -e staging -C config/sidekiq.yml
I encounter the same problem, it turns out that the argument I've passed in the function perform_async is not appropriate, it seems that one should not pass any query result in perform_async, you must do all the query in the function perform.
You need to specify the name of the queue that worker is for.
Example:
sidekiq_options retry: false, :queue => data_import_worker
data_import_worker can be any name you want to give it.
Then when you go to the web interface: yoursite.com/sidekiq, you'll be able to see the current workers for the queue "data_import_worker"
For me when doing a perform_later, it would enqueue but never remove from queue. I needed to add my queue name to the sidekiq.yml file
---
:concurrency: 25
:pidfile: ./tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:queues:
- default
- my_queue
Lost a good 15 min on this. To check if Sidekiq is correctly loading your config file (with the queues names), go to the web interface in the Busy tab and you'll find your Process ID and below it you'll find your queues.
In our case, we had misspelled mailer (the correct ActiveJob queue for Mailers is mailers, in plural).
My issue was simply having the worker file in the wrong path.
Needs to be in "project_root/app/worker/worker.rb", not "project_root/worker/worker.rb"
Check the file path!
is it realy run multiple workers on standalone sidekiq?
for example I have 2 workers:
ProccessWorker
CallbackWorker
when I am runnigs sidekiq:
bundle exec sidekiq -r ./workers/proccess_worker.rb -C ./config/sidekiq.yml
only one worker in same time.
I was calling perform_async(23) in a production console, however my sidekiq was started in staging mode.
After I started the Sidekiq in production mode, things have started working very well.