I have an active job that sends an email
class UpdateEmailJob < ActiveJob::Base
queue_as :default
def perform()
NotificationsMailer.create_client("mymail#gmail.com").deliver
end
end
Then on my code I call the perform later method
UpdateEmailJob.perform_later
I am getting a message that the job has been Enqueued
[ActiveJob] Enqueued UpdateEmailJob (Job ID: 1a57232b-f6a8-4dc6-b612-8aca1b1dba2c) to Sidekiq(default)
But the email is not been send, how can I debug that ? Is there a way to explore the queue ? is there a way to force the worker to run all the queue jobs ?
Do you have the sidekiq mailers queue running? If not, run this:
bundle exec sidekiq -q default -q mailers
You can interact and debug enqueued jobs via the Sidekiq API. Run rails c, then:
require 'sidekiq/api'
Sidekiq::Queue.new("mailers")
This provide all of the jobs currently enqueued in mailers. From there, look at the API docs for different ways to inspect them.
Related
I'm using delayed_job library as adapter for Active Jobs in Rails:
config.active_job.queue_adapter = :delayed_job
My delayed_job worker is configured to sleep for 60 seconds before checking the queue for new jobs:
Delayed::Worker.sleep_delay = 60
In my Rails code I add a new job to the queue like this:
MyJob.perform_later(data)
However, the job will not be picked up by the delayed_job worker immediately, even if there are no jobs in the queue, because of the sleep_delay. Is there a way to tell the delayed_job worker to wake up and start processing the job queue if it's sleeping?
There is a MyJob.perform_now method, but it blocks the thread, so it's not what I want because I want to execute a job asynchronously.
Looking at the delayed_job code and it appears that there's no way to directly control or communicate with the workers after they are daemonized.
I think the best you can do would be to start a separate worker with a small sleep_delay that only reads a specific queue, then use that queue for these jobs. A separate command is necessary because you can't start a worker pool where the workers have different sleep delays:
Start your main worker: bin/delayed_job start
Start your fast worker: bin/delayed_job start --sleep-delay=5 --queue=fast --identifier=999 (the identifier is necessary to differentiate the workers daemons)
Update your job to use that queue:
class MyJob < ApplicationJob
queue_as :fast
def perform...
end
Notes:
When you want to stop the workers you'll also have to do it separately: bin/delayed_job stop and bin/delayed_job stop --identifier=999.
This introduces some potential parallelism and extra load on the server when both workers are working at the same time.
The main worker will process jobs from the fast queue too, it's just a matter of which worker grabs the job first. If you don't want that you need to setup the main worker to only read from the other queue(s), by default that's only 'default', so: bin/delayed_job start --queue=default.
I'm trying to send email campaigns in a rails app with the Heya gem and GoodJob. The example in the Heya readme as well as the Heya example app uses Sidekiq as the Active Job backend.
I'm confused about how to actually send the Heya campaigns with GoodJob.
The docs for Heya show this example of starting Sidekick: bundle exec sidekiq -q default -q heya
I assume that there is a Job queue somewhere in the gem called "Heya", but I can't find this in the source code. Do I need to create one?
Do I need to create a job that runs the Heya scheduler? While the example app uses Sidekiq, I also don't see any custom jobs in that app.
I have the following setup for GoodJob and it appears to be running fine with good_job start which should run all of the jobs and queues, but I've also tried good_job start --queues=heya,default.
Here is the relevant code:
Profile.dev
web: bin/rails server -p 3000
css: bin/rails tailwindcss:watch
worker: bundle exec good_job start
config/initializers/heya.rb
Heya.configure do |config|
config.user_type = "User"
config.campaigns.priority = [
"WelcomeCampaign",
]
end
app/jobs/application_job.rb
class ApplicationJob < ActiveJob::Base
# Automatically retry jobs that encountered a deadlock
# retry_on ActiveRecord::Deadlocked
# Most jobs are safe to ignore if the underlying records are no longer available
# discard_on ActiveJob::DeserializationError
end
app/campaigns/application_campaign.rb
class ApplicationCampaign < Heya::Campaigns::Base
segment :email_subscriber?
default from: "#{I18n.t('settings.site_name')} <#{I18n.t('settings.newsletter_email')}>"
end
app/campaigns/welcome_campaign.rb
class WelcomeCampaign < ApplicationCampaign
default wait: 5.minutes,
layout: "newsletter"
step :intro, wait: 0.minutes,
subject: "Welcome to #{I18n.t('settings.site_name')}"
end
I also have a layout and views for the campaign similar to the Heya example app, and I'm using Mailcatcher to see if any email is being sent.
What am I missing to send these emails with Heya and GoodJob?
Note that I'm subscribing the users on signups like this:
class User < ApplicationRecord
after_create_commit :add_user_to_newsletters
private
def add_user_to_newsletters
WelcomeCampaign.add(self)
EvergreenCampaign.add(self)
self.update(email_subscriber: true)
end
end
And the default segment in campaigns/application_campaign.rb is segment :email_subscriber?
If I run User.last.email_subscriber? in the console to check this it returns true.
I feel like I'm missing something about how Heya connects to Active Job that is not obvious in the Heya docs.
Also, not sure if this is related, but I added this to config/puma.rb
# https://github.com/bensheldon/good_job#execute-jobs-async--in-process
before_fork do
GoodJob.shutdown
end
on_worker_boot do
GoodJob.restart
end
on_worker_shutdown do
GoodJob.shutdown
end
MAIN_PID = Process.pid
at_exit do
GoodJob.shutdown if Process.pid == MAIN_PID
end
preload_app!
Are you running the heya scheduler periodically? $ rails heya:scheduler
It looks like you could create your own background job to be run using GoodJob Cron, by executing Heya::Campaigns::Scheduler.new.run to run the scheduler and enqueue the emails.
Reading the "Running the Scheduler" part of the README explains what's happening:
To start queuing emails, run the scheduler task periodically:
rails heya:scheduler
Heya uses ActiveJob to send emails in the background. Make sure your
ActiveJob backend is configured to process the heya queue. By default, GoodJob runs from all queues "*".
You can change Heya's default queue using the queue option:
# app/campaigns/application_campaign.rb
class ApplicationCampaign < Heya::Campaigns::Base
default queue: "custom"
end
I built a simple test job for Sidekiq and added it to my schedule.yml file for Sidekiq Cron.
Here's my test job:
module Slack
class TestJob < ApplicationJob
queue_as :default
def perform(*args)
begin
SLACK_NOTIFIER.post(attachments: {"pretext": "test", "text": "hello"})
rescue Exception => error
puts error
end
end
end
end
The SLACK_NOTIFIER here is a simple API client for Slack that I initialize on startup.
And in my schedule.yml:
test_job:
cron: "* * * * *"
class: "Slack::TestJob"
queue: default
description: "Test"
So I wanted to have it run every minute, and it worked exactly as I expected.
However, I've now deleted the job file and removed the job from schedule.yml, and it still tries to run the job every minute. I've gone into my sidekiq dashboard, and I see a bunch of retries for that job. No matter how many times I kill them all, they just keep coming.
I've tried shutting down both the redis server and sidekiq several times. I've tried turning off my computer (after killing the servers, of course). It still keeps scheduling these jobs and it's interrupting my other jobs because it raises the following exception:
NameError: uninitialized constant Slack::TestJob
I've done a project-wide search for "TestJob", but get no results.
I only had the redis server open with this job for roughly 10 minutes...
Is there maybe something lingering in the redis database? I've looked into the redis-cli documentation, but I don't think any of it helps me.
Try calling $ FLUSHALL in the redis cli...
Other than that...
The sidekiq-cron documentation seems to expect that you check for the existence of schedule.yml explicitly...
#initializers/sidekiq.rb
schedule_file = "config/schedule.yml"
if File.exist?(schedule_file) && Sidekiq.server?
Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file)
end
The default for Sidekiq is to retry a failed job 25 times. Because the job doesn't exist, it fails... every time; Sidekiq only knows that the job fails (because attempting to perform a job that doesn't exist would raise an exception), so it flags it for retry... 25 times. Because you were scheduling that job every minute, you probably had a TON of these failed jobs queued up for retry. You can:
Wait ~3 weeks to hit the maximum number of retries
If you have a UI page set up for Sidekiq, you can see and clear these jobs in the Retries tab
Dig through the Redis CLI documentation for a way to identify these specific jobs and delete them (or flush the whole thing if you're not worried about losing other jobs)
While using ActiveJob with sidekiq,
# application.rb
config.active_job.queue_adapter = :sidekiq
I made a job using redis.
class SubmissionJob < ActiveJob::Base
queue_as :default
def perform(*args)
While checking through the log, after calling job as
SubmissionJob_perform_later args
But this doesn't work after giving message when doing rails s
Enqueued SubmissionJob (Job ID: 63320b9d-de2a-4791-99f6-e58f2adf73d7) to Sidekiq(development_default) with arguments: ["12345", "12346"]
when at last, I render the result calling with job_id from redis, the result supposed to be return the arguments, but it only returns empty array, and it seems SubmissionJob never runned.
However, when I call with perform_now , SubmissionJob.perform_now args the function works properly so it seems the method SubmissionJob itself is not the problem.
Where should I start to find problem and what is the possible solution to solve this.
Solved problem.
This was due to Work did not successfully dequeued job from suggested queue. When running the sidekiq, I gave -q option.
For example bundle exec sidekiq -e develop -q develop_submit_job_queue
I have a rails app using an engine where Sidekiq workers are defined. The worker's perform_async is invoked in a controller within the engine. The worker's perform does the work on arguments passed in through that controller. The worker specific queue is defined in the worker class too. However, when a request comes in to that controller, it gets pushed to the redis server from 'perform_async', to the right queue. A bundle exec sidekiq starts up Sidekiq. However, the worker's perform never gets executed. Checking the sidekiq UI, I can see that the job is in the right queue.
This is how my worker looks like
require 'pando'
class PandoWorker
include Sidekiq::Worker
sidekiq_options :queue => :pando, :backtrace=> true
def perform(*args)
puts "in here"
puts args
end
end
So in this case the sidekiq UI shows that the args are queued in 'pando'. The sidekiq process never processes from that queue or even the default.
You have to tell sidekiq process which queue to look at:
bundle exec sidekiq -q pando
Otherwise process is only watching 'default' queue.