Sidekiq worker enqueues but doesn't complete when paired with Sidetiq - ruby-on-rails

I have a Sidekiq worker that syncs the data of a Rails 3.2 application with a remote database. When I just execute it from the rails console, everything works as expected. I can verify that it connects to the remote DB and pulls in the data I am expecting.
The moment I try to set up the following:
recurrence {daily.hour_of_day(20).minute_of_hour(02)}
I see the following in Sidekiq but no signs of completion/execution:
2014-05-14T00:02:18Z 6747 TID-dvc2c INFO: [Sidetiq] Enqueue: BackendWorker (at: 1400112120.0) (last: -1.0)
I've got retries set to false (which I would like to change) currently, and I have attempted to clear out anything in redis before setting up another test run again.
Is there a way to squeeze some more information about what is going on? Is there a Sidekiq flag I can use to get some additional debug information?
Thanks in advance
Environment:
Rails 3.2
Ruby 2.1
Sidetiq 0.5.0
Sidekiq 2.17.7 (due to an issue with using 3.0)
UPDATE (5/14/2014):
I decided to test a simple example with Sidetiq and just have a worker that creates a DB entry with ActiveRecord:
class BackendTest
include Sidekiq::Worker
include Sidetiq::Schedulable
sidekiq_options :retry => false, :backtrace => true
recurrence {
daily.hour_of_day(11).minute_of_hour(05)
}
def perform
Book.create(title: "test", author: "testing", price: 9.99)
end
end
Another experiment I did was just trying to get Sidetiq and Sidekiq to just show something with logger.info:
class BackendTest
include Sidekiq::Worker
include Sidetiq::Schedulable
sidekiq_options :retry => false, :backtrace => true
recurrence {
daily.hour_of_day(16).minute_of_hour(00)
}
def perform
logger.info "test"
logger.info "test"
end
end
The jobs just still enters the default queue without being started or completed. Everything still works if I just fire it off from the rails console with perform_async, Sidekiq::Client.push, and Sidekiq::Client.enqueue still work from console.
I am a thoroughly confused newbie at this point.
Output:
2014-05-14T20:08:41Z 2149 TID-dfcfk BackendTest JID-6bc7ba08955f8340903f9063 INFO: start
2014-05-14T20:08:41Z 2149 TID-dfcfk BackendTest JID-6bc7ba08955f8340903f9063 INFO: test
2014-05-14T20:08:41Z 2149 TID-dfcfk BackendTest JID-6bc7ba08955f8340903f9063 INFO: test
2014-05-14T20:08:41Z 2149 TID-dfcfk BackendTest JID-6bc7ba08955f8340903f9063 INFO: done: 0.002 sec

After poking around other people's projects on github, I saw something interesting and tried it out and it seemed to solve my problem with Sidetiq.
I created /config/initializers/sidekiq.rb with the following content inside of it:
Sidekiq.configure_server do |config|
config.redis = { :url => 'redis://localhost:6379/3', :namespace => 'sidekiq' }
end
Sidekiq.configure_client do |config|
config.redis = { :url => 'redis://localhost:6379/3', :namespace => 'sidekiq' }
end
Job tasks enqueue and execute as expected.

Related

Sidekiq worker does not triggered as a AR callback on Heroku production

In my Rails 6 app I've got ActiveRecord callback (after_create) which should call SyncProductsWorker (Sidekiq worker) each time when record ProductsBatch is created:
ProductsBatch model with after_create:
module Imports
class ProductsBatch < ImportsRecord
attr_accessor :product_codes
after_create :enqueue
def enqueue
::Imports::SyncProductsWorker.perform_async(product_codes, self.id)
end
end
end
# base class for the above model
class ImportsRecord < ApplicationRecord
self.abstract_class = true
connects_to database: { writing: :imports }
end
SyncProductsWorker class:
module Imports
class SyncProductsWorker
include Sidekiq::Worker
sidekiq_options queue: 'imports_sync'
def perform(list, id)
# do some things
end
end
end
config/sidekiq.yml
:max_retries: 16
:queues:
- default
- imports_sync
- imports_fetch_all
:dynamic: true
config/initializers/sidekiq.rb
redis = { url: ENV['REDIS_URL'], ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
Sidekiq.configure_server do |config|
config.redis = redis
end
Sidekiq.configure_client do |config|
config.redis = redis
end
Everything works well locally but when I deploy code into Heroku worker doesn't seem to call. The strange thing is that, based on Heroku logs, it doesn't even work when I try to call it directly from Heroku rails console:
heroku run rails console --app test-app
› Warning: heroku update available from 7.47.7 to 7.60.2.
Running rails console on ⬢ test-app... up, run.8489 (Hobby)
Loading production environment (Rails 6.1.4.1)
irb(main):013:0> ::Imports::SyncProductsWorker.perform_async(['11'], 10)
=> "5edf93e27fa2f41245587d49"
But nothing happens inside Heroku logs:
2022-06-06T22:02:00.240650+00:00 app[worker.1]: [ActiveJob] [ProductAvailabilityAdjusterJob] [25c15f9d-e032-438e-bda8-16ffd557cc32] Performed ProductAvailabilityAdjusterJob (Job ID: 25c15f9d-e032-438e-bda8-16ffd557cc32) from Sidekiq(default) in 5.44ms
2022-06-06T22:02:00.240789+00:00 app[worker.1]: pid=4 tid=2xbk class=ProductAvailabilityAdjusterJob jid=91ad7e69e061df9f2f681ef3 elapsed=0.006 INFO: done
Is there anything special I should do to make this worker work on Heroku?

Using sidekiq/redistogo with heroku?

I followed the sideqik gem tutorial docs and heroku Redistogo addon docs
initializers/sidekiq.rb:
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://redistogo:xxx.redistogo.com:10076/' }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://redistogo:xxx.redistogo.com:10076/' }
end
app/workers/hard_woker.rb:
class HardWorker
include Sidekiq::Worker
def perform(shop_domain, webhook)
#performing stuffs
end
Webhook I am putting into background job (trying to at least):
class OrdersCreateJob < ActiveJob::Base
def perform(shop_domain:, webhook:)
shop = Shop.find_by(shopify_domain: shop_domain)
shop.with_shopify_session do
HardWorker.perform_async(shop_domain, webhook)
end
end
end
Procfile:
hardworker: bundle exec sidekiq -t 25
There are no errors in console.
Is something wrong here, did I miss something?
My queue:
irb(main):003:0> Sidekiq::Queue.all
=> [#<Sidekiq::Queue:0x000055b53a2d0920 #name="default", #rname="queue:default">]
I assume this means nothing is in the queue?
My goal is to take all my my CreateOrderWebhook code (which is almost 500 lines) into a background job to put less strain on the app and allow webhooks /prevent webhooks from being blocked

sneaker is not receiving messages on heroku - RabbitMQ Bigwig

I am trying to run message queues on heroku. For this I am using RabbitMQ Bigwig plugin.
I am publishing messages using bunny gem and trying to receive messages with sneakers gem. This whole setup works smoothly on local machine.
I take following steps to setup queue
I run this rake on server to setup queue:
namespace :rabbitmq do
desc 'Setup routing'
task :setup_test_commands_queue do
require 'bunny'
conn = Bunny.new(ENV['SYNC_AMQP'], read_timeout: 10, heartbeat: 10)
conn.start
ch = conn.create_channel
# get or create exchange
x = ch.direct('testsync.pcc', :durable => true)
# get or create queue (note the durable setting)
queue = ch.queue('test.commands', :durable => true, :ack => true, :routing_key => 'test_cmd')
# bind queue to exchange
queue.bind(x, :routing_key => 'test_cmd')
conn.close
end
end
I am able to see this queue in rabbitmq management plugin with mentioned binding.
class TestPublisher
def self.publish(test)
x = channel.direct("testsync.pcc", :durable => true)
puts "publishing this = #{Test}"
x.publish(Test, :persistent => true, :routing_key => 'pcc_cmd')
end
def self.channel
#channel ||= connection.create_channel
end
def self.connection
#conn = Bunny.new(ENV['RABBITMQ_BIGWIG_TX_URL'], read_timeout: 10, heartbeat: 10) # getting configuration from rabbitmq.yml
#conn.start
end
end
I am calling TestPublisher.publish() to publish message.
I have sneaker worker like this:
require 'test_sync'
class TestsWorker
include Sneakers::Worker
from_queue "test.commands", env: nil
def work(raw_event)
puts "^"*100
puts raw_event
# o = CaseNote.create!(content: raw_event, creator_id: 1)
# puts "#########{o}"
test = Oj.load raw_event
test.execute
# event_params = JSON.parse(raw_event)
# SomeWiseService.build.call(event_params)
ack!
end
end
My Procfile
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec rake jobs:work
sneaker: WORKERS=TestsWorker bundle exec rake sneakers:run
My Rakefile
require File.expand_path('../config/application', __FILE__)
require 'rake/dsl_definition'
require 'rake'
require 'sneakers/tasks'
Test::Application.load_tasks
My sneaker configuration
require 'sneakers'
Sneakers.configure amqp: ENV['RABBITMQ_BIGWIG_RX_URL'],
log: "log/sneakers.log",
threads: 1,
workers: 1
puts "configuring sneaker"
I am sure that message gets published. I am able to get message on rabbitmq management plugin. But sneaker does not work. There is nothing in sneakers.log that can help.
sneakers.log on heroku :
# Logfile created on 2016-04-05 14:40:59 +0530 by logger.rb/41212
Sorry for this late response. I was able to get this working on heroku. When I faced this error after hours of debugging I was not able to fix it. So I rewrote all above code and I did not check what was wrong with my previous code.
The only problem with this code and correct code is queue binding.
I had two queues on same exchange. pcc.commands with routing key pcc_cmd and test.commands with routing key test_cmd.
I was working with test_cmd but as per following line in TestPublisher
x.publish(Test, :persistent => true, :routing_key => 'pcc_cmd')
I was publishing to different queue(pcc.commands). Hence I was not able to recieve the message on test.commands queue.
In TestWorker
from_queue "test.commands", env: nil
This states that fetch messages only from test.commands queue.
Regarding sneakers.log file:
Above setup was not able to give me logs in sneakers.log file. Yes this setup works on your local development machine, but it was not working on heroku. Now days to debug such issue I ommit log attribute from configuration. like this:
require 'sneakers'
Sneakers.configure amqp: ENV['RABBITMQ_BIGWIG_RX_URL'],
# log: "log/sneakers.log",
threads: 1,
workers: 1
This way you will get sneaker logs (even heartbeat logs) in heroku logs which can be seen by running command heroku logs -a app_name --tail.

how setup active_job and rescue scheduler?

I'm trying to create background jobs for email notification and scraper.
I use resque-scheduler (4.0.0), resque (1.25.2) and rails 4.2.1.
My config.ru file:
# This file is used by Rack-based servers to start the application.
require ::File.expand_path('../config/environment', __FILE__)
run Rails.application
require 'resque/server'
run Rack::URLMap.new "/" => AppName::Application, "/resque" => Resque::Server.new
My /lib/tasks/resque.rake:
require 'resque/tasks'
require 'resque/scheduler/tasks'
namespace :resque do
task :setup do
require 'resque'
require 'resque-scheduler'
Resque.schedule = YAML.load_file("#{Rails.root}/config/resque_schedule.yml")
Dir["#{Rails.root}/app/jobs/*.rb"].each { |file| require file }
end
end
My /config/resque_scheduler.yml:
CheckFsUpdatesJob:
queue: fs_updates
every:
- '1h'
- :first_in: '10s'
class: CheckFsUpdatesJob
args:
description: scrape page
My /config/initializer/active_job.rb
ActiveJob::Base.queue_adapter = :resque
My /config/initializer/resque.rb:
#config/initializers/resque.rb
require 'resque-scheduler'
require 'resque/scheduler/server'
uri = URI.parse("redis://localhost:6379/")
Resque.redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
Resque.after_fork = Proc.new { ActiveRecord::Base.establish_connection }
Dir["#{Rails.root}/app/jobs/*.rb"].each { |file| require file }
Resque.schedule = YAML.load_file(Rails.root.join('config', 'resque_schedule.yml'))
Resque::Server.use(Rack::Auth::Basic) do |user, password|
user = 'admin'
password = 'admin'
end
My first job for emails notifications:
class EmailNotificationJob < ActiveJob::Base
queue_as :email_notifications
def perform(episode_id, email)
NotificationMailer.new_record_appears(record_id, email).deliver_now
end
end
My second job for scheduled runs:
class CheckFsUpdatesJob < ActiveJob::Base
queue_as :fs_updates
def perform()
FsStrategy.new.check_for_updates
end
end
So I have to jobs:
1. emails notifications - should sends email when new record in DB appears
2. scrape a page - should runs every hour
How I run it:
redis-server
rake environment resque:work QUEUE=fs_updates
rake environment resque:work QUEUE=email_notifications
rake environment resque:scheduler
rails s
After running these commands I see in Resque Dashboard two workers and two queues, as it is expected.
But!
After clicking on 'queue now' button at resque Schedule tab, I see that task was created and wroted to "fs_updates" queue. But it's not running and in a few second it dissapears.
When I run a job for emails sending from rails console - it does not work at all.
Please, help me to fix my configurations.
Thanks kindly!
As I understood: rails and active_job developers is not responsible for resque plugins. Maybe this problem will be fixed in new gem versions, but now it does not work (active_job does not work fine with resque-scheduler).
Currently I use gem 'active_scheduler' to fix current problem.
I had the same issue trying to configure Sucker Punch on rails 4.2.1 In the end I moved my custom initialiser logic into application.rb, not great but it got me up and running.
Not sure if there is an issue with the initialisers in this release. Try moving your code from active_job.rb and resque.rb into application.rb or the appropriate environment file. Obviously this isn't a long term solution but it will at least help you you identify whether it's an initialiser issue or Resque config problem.

sidekiq runs in bursts on production

In development, it runs as I would expect it, having 5 threads (limited at the moment due to redis connection limit) it averages at about 5-7 processes running, depending if the worker has to do anything or not (sometimes a worker would decide not to work, since the object it is working on was updated less than a few days ago)
on production, it behaves differently. It seems to run in bursts of around 400, and then immediatly reschedules the workers and waits a bit and then shoots a burst again
The workers work with facebook api (koala gem), which for this I use sidekiq-throttler (https://github.com/gevans/sidekiq-throttler)
with the options
sidekiq_options throttle: { threshold: 50, period: 60.seconds , key: ->(user_id){ "facebook:#{user_id}"} }
I am using heroku and redislabs (free plan at the moment) with the procfile
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -c 5
and sidekiq setup:
Sidekiq.configure_server do |config|
config.redis = { :url => "#{ENV['REDISCLOUD_URL']}", :namespace => 'sidekiq'}
config.server_middleware do |chain|
chain.add Sidekiq::Throttler, storage: :redis
end
end
Sidekiq.configure_client do |config|
config.redis = { :url => "#{ENV['REDISCLOUD_URL']}", :namespace => 'sidekiq' }
end
is this a known symptom for something?
Looks like it's being throttled, as expected.

Resources