I'm creating Elasticsearch callbacks using Sidekiq. I started Redis and Sidekiq, and created an object in rails console but it seems like perform_sync is not firing. If I add require 'sidekiq/testing';Sidekiq::Testing.inline! to sidekiq.rb, it logs and raises an error as I expect though. Am I missing something?
Ruby 2.3.0
Rails 5.0.2
Sidekiq 4.2.9
Redis 3.2.0
sidekiq.rb
REDIS_URL = 'localhost'
REDIS_PORT = '6379'
Sidekiq.configure_server do |config|
config.redis = { url: "redis://#{REDIS_URL}:#{REDIS_PORT}" }
end
Sidekiq.configure_client do |config|
config.redis = { url: "redis://#{REDIS_URL}:#{REDIS_PORT}" }
end
indexable.rb
module Indexable
extend ActiveSupport::Concern
included do
def index_elasticsearch
Rails.logger.debug 'call'
Rails.logger.debug Indexer::Logger
Rails.logger.debug Indexer::Client
Rails.logger.debug self.id.to_s
Indexer.perform_async(:index, self.id.to_s) # nothing happens here
Rails.logger.debug 'after'
self
end
end
class Indexer
include Sidekiq::Worker
sidekiq_options queue: :elasticsearch, retry: false, backtrace: true
Logger = Sidekiq.logger.level = Logger::DEBUG ? Sidekiq.logger : nil
raise 'No config/elasticsearch.yml' unless File.exists? "config/elasticsearch.yml"
erb = ERB.new( File.read('config/elasticsearch.yml') ).result
config = YAML.load(erb)[Rails.env].symbolize_keys
config.merge! logger: Logger
Client = Elasticsearch::Client.new(config)
def perform(operation, record_id)
Rails.logger.debug [ operation, "ID: #{record_id}"]
raise
end
end
end
some_mongoid_class.rb
class SomeMongoidClass
...
include ::Indexable
...
after_save :index_elasticsearch
end
console
bundle exec sidekiq -e development --queue elasticsearch --verbose
...Booting Sidekiq 4.2.9 with redis options {:url=>"redis://localhost:6379"}...
2017-11-10T ... DEBUG: {:queues=>["elasticsearch"], :labels=>[], :concurrency=>25, :require=>".", :environment=>"development", :timeout=>8, ...
rails console
SomeMongoidClass.create(...)
before
#<Logger:0x...
#<Elasticsearch::Transport::Client:0x...
BSON::ObjectId('...')
after
This problem was solved as soon as I posted the last comment.
It was just a silly mistake. I didn't put the worker file into the workers folder of the app!
Related
In my Rails 6 app I've got ActiveRecord callback (after_create) which should call SyncProductsWorker (Sidekiq worker) each time when record ProductsBatch is created:
ProductsBatch model with after_create:
module Imports
class ProductsBatch < ImportsRecord
attr_accessor :product_codes
after_create :enqueue
def enqueue
::Imports::SyncProductsWorker.perform_async(product_codes, self.id)
end
end
end
# base class for the above model
class ImportsRecord < ApplicationRecord
self.abstract_class = true
connects_to database: { writing: :imports }
end
SyncProductsWorker class:
module Imports
class SyncProductsWorker
include Sidekiq::Worker
sidekiq_options queue: 'imports_sync'
def perform(list, id)
# do some things
end
end
end
config/sidekiq.yml
:max_retries: 16
:queues:
- default
- imports_sync
- imports_fetch_all
:dynamic: true
config/initializers/sidekiq.rb
redis = { url: ENV['REDIS_URL'], ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE } }
Sidekiq.configure_server do |config|
config.redis = redis
end
Sidekiq.configure_client do |config|
config.redis = redis
end
Everything works well locally but when I deploy code into Heroku worker doesn't seem to call. The strange thing is that, based on Heroku logs, it doesn't even work when I try to call it directly from Heroku rails console:
heroku run rails console --app test-app
› Warning: heroku update available from 7.47.7 to 7.60.2.
Running rails console on ⬢ test-app... up, run.8489 (Hobby)
Loading production environment (Rails 6.1.4.1)
irb(main):013:0> ::Imports::SyncProductsWorker.perform_async(['11'], 10)
=> "5edf93e27fa2f41245587d49"
But nothing happens inside Heroku logs:
2022-06-06T22:02:00.240650+00:00 app[worker.1]: [ActiveJob] [ProductAvailabilityAdjusterJob] [25c15f9d-e032-438e-bda8-16ffd557cc32] Performed ProductAvailabilityAdjusterJob (Job ID: 25c15f9d-e032-438e-bda8-16ffd557cc32) from Sidekiq(default) in 5.44ms
2022-06-06T22:02:00.240789+00:00 app[worker.1]: pid=4 tid=2xbk class=ProductAvailabilityAdjusterJob jid=91ad7e69e061df9f2f681ef3 elapsed=0.006 INFO: done
Is there anything special I should do to make this worker work on Heroku?
I followed the sideqik gem tutorial docs and heroku Redistogo addon docs
initializers/sidekiq.rb:
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://redistogo:xxx.redistogo.com:10076/' }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://redistogo:xxx.redistogo.com:10076/' }
end
app/workers/hard_woker.rb:
class HardWorker
include Sidekiq::Worker
def perform(shop_domain, webhook)
#performing stuffs
end
Webhook I am putting into background job (trying to at least):
class OrdersCreateJob < ActiveJob::Base
def perform(shop_domain:, webhook:)
shop = Shop.find_by(shopify_domain: shop_domain)
shop.with_shopify_session do
HardWorker.perform_async(shop_domain, webhook)
end
end
end
Procfile:
hardworker: bundle exec sidekiq -t 25
There are no errors in console.
Is something wrong here, did I miss something?
My queue:
irb(main):003:0> Sidekiq::Queue.all
=> [#<Sidekiq::Queue:0x000055b53a2d0920 #name="default", #rname="queue:default">]
I assume this means nothing is in the queue?
My goal is to take all my my CreateOrderWebhook code (which is almost 500 lines) into a background job to put less strain on the app and allow webhooks /prevent webhooks from being blocked
TLDR;
Why Sneakers worker can't connect to the database or can't query it?
(General advices on "do's" and "dont's" are also welcome in comments)
Full question:
I am able to execute RPC call that returns a simple string, but I can't execute RPC call that is querying the database on the server side. I read the docs, tried many SO posts and blog tutorials, but I am still missing some piece.
I have two services. First service (Client) is using Bunny gem and is making an RPC call to second service (RPCServer) which is listening on workers using Sneakers gem. Both services are Rails apps.
RabbitMQ is serving in a docker container:
docker run -p 5672:5672 -p 15672:15672 rabbitmq:3-management
Postgres database is installed on a local machine.
Client service (mostly from Rabbitbunny docs ):
# app/services/client.rb
class Client
attr_accessor :call_id, :lock, :condition, :reply_queue, :exchange, :params, :response, :server_queue_name, :channel, :reply_queue_name
def initialize(rpc_route:, params:)
#channel = channel
#exchange = channel.fanout("Client.Server.exchange.#{params[:controller]}")
#server_queue_name = "Server.Client.queue.#{rpc_route}"
#reply_queue_name = "Client.Server.queue.#{params[:controller]}"
#params = params
setup_reply_queue
end
def setup_reply_queue
#lock = Mutex.new
#condition = ConditionVariable.new
that = self
#reply_queue = channel.queue(reply_queue_name, durable: true)
reply_queue.subscribe do |_delivery_info, properties, payload|
if properties[:correlation_id] == that.call_id
that.response = payload
that.lock.synchronize { that.condition.signal }
end
end
end
def call
#call_id = "NAIVE_RAND_#{rand}#{rand}#{rand}"
exchange.publish(params.to_json,
routing_key: server_queue_name,
correlation_id: call_id,
reply_to: reply_queue.name)
lock.synchronize { condition.wait(lock) }
connection.close
response
end
def channel
#channel ||= connection.create_channel
end
def connection
#connection ||= Bunny.new.tap { |c| c.start }
end
end
RPCServer service, using this gist (comments here are the "meat" of my question:
# app/workers/posts_worker.rb
require 'sneakers'
require 'sneakers/runner'
require 'byebug'
require 'oj'
class RpcServer
include Sneakers::Worker
from_queue 'Client.Server.queue.v1/filters/posts', durable: true, env: nil
def work_with_params(deserialized_msg, delivery_info, metadata)
post = {}
p "ActiveRecord::Base.connected?: #{ActiveRecord::Base.connected?}" # => true
##### This gets logged
Rails.logger.info "ActiveRecord::Base.connection_pool: #{ActiveRecord::Base.connection_pool}\n\n-------"
##### This never gets logged
Rails.logger.info "ActiveRecord::Base.connection_pool.with_connection: #{ActiveRecord::Base.connection_pool.with_connection}\n\n--------"
### interpreter never reaches this place when ActiveRecord methods like `with_connection`, `where`, `count` etc. are used
ActiveRecord::Base.connection_pool.with_connection do
post = Post.first.to_json
end
##### first commented `publish()` works fine and RPC works when no ActiveRecord is involved (this is, assuming above code using ActiveRecord is commented out)
##### second publish is not working
# publish("response from RPCServer", {
publish(post.to_json, {
to_queue: metadata[:reply_to],
correlation_id: metadata[:correlation_id],
content_type: metadata[:content_type]
})
ack!
end
end
Sneakers::Runner.new([RpcServer]).run
RPCServer sneakers configuration:
# config/initializers/sneakers.rb
Sneakers.configure({
amqp: "amqp://guest:guest#localhost:5672",
vhost: '/',
workers: 4,
log: 'log/sneakers.log',
pid_path: "tmp/pids/sneakers.pid",
timeout_job_after: 5,
prefetch: 10,
threads: 10,
durable: true,
ack: true,
heartbeat: 2,
exchange: "",
hooks: {
before_fork: -> {
Rails.logger.info('Worker: Disconnect from the database')
ActiveRecord::Base.connection_pool.disconnect!
Rails.logger.info("before_fork: ActiveRecord::Base.connected?: #{ActiveRecord::Base.connected?}") # => false
},
after_fork: -> {
ActiveRecord::Base.connection
Rails.logger.info("after_fork: ActiveRecord::Base.connected?: #{ActiveRecord::Base.connected?}") # => true
Rails.logger.info('Worker: Reconnect to the database')
},
timeout_job_after: 60
})
Sneakers.logger.level = Logger::INFO
RPCServer puma configuration:
# config/puma.rb
threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
threads threads_count, threads_count
port ENV.fetch("PORT") { 3000 }
environment ENV.fetch("RAILS_ENV") { "development" }
workers ENV.fetch("WEB_CONCURRENCY") { 2 }
preload_app!
### tried and did not work
# on_worker_boot do
# ActiveSupport.on_load(:active_record) do
# ActiveRecord::Base.establish_connection
# end
# end
before_fork do |server, worker|
# other settings
if defined?(ActiveRecord::Base)
ActiveRecord::Base.connection.disconnect!
end
end
after_worker_boot do |server, worker|
if defined?(ActiveRecord::Base)
ActiveRecord::Base.establish_connection
end
end
plugin :tmp_restart
for completeness, I also have an external Rakefile that is binding queues to exchanges (probably not important in this case)
namespace :rabbitmq do
desc "Setup routing"
task :setup do
conn = start_bunny
rpc_route service: :blog, from: 'v1/filters/posts_mappings', to: 'v1/filters/posts'
conn.close
end
def rpc_route(service:, from:, to:)
...
end
def start_bunny
...
end
end
I tried many sneakers configurations, and many orders of launching rabbitmq, resetting it, deleting queues, connections, etc. All of it is hard to list here and probably not the case.
Why I can't connect to the database or execute ActiveRecord methods? What Am I missing?
Ok I got it. The problem was last line of worker in RPCServer:
Sneakers::Runner.new([RpcServer]).run
It was running worker outside of Rails app. Commenting this out solved my problem of worker not being able to query database.
I have a Rails application that runs some background jobs via ActiveJob and Sidekiq. The sidekiq logs in both the terminal and the log file show the following:
2016-10-18T06:17:01.911Z 3252 TID-oukzs4q3k ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-97318b38b1391672d21feb93 INFO: start
Is there some way to show the class names of the jobs here similar to how logs work for a regular Sidekiq Worker?
Update:
Here is how a Sidekiq worker logs:
2016-10-18T11:05:39.690Z 13678 TID-or4o9w2o4 ClientJob JID-b3c71c9c63fe0c6d29fd2f21 INFO: start
Update 2:
My sidekiq version is 3.4.2
I'd like to replace ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper with Client Job
So I managed to do this by removing Sidekiq::Middleware::Server::Logging from the middleware configuration and adding a modified class that displays the arguments in the logs. The arguments themself contain the job and action names as well.
For latest version, currently 4.2.3, in sidekiq.rb
require 'sidekiq'
require 'sidekiq/middleware/server/logging'
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def log_context(worker, item)
klass = item['wrapped'.freeze] || worker.class.to_s
"#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}"
end
end
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.remove Sidekiq::Middleware::Server::Logging
chain.add ParamsLogging
end
end
For version 3.4.2, or similar, override the call method instead:
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def call(worker, item, queue)
klass = item['wrapped'.freeze] || worker.class.to_s
Sidekiq::Logging.with_context("#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}") do
begin
start = Time.now
logger.info { "start" }
yield
logger.info { "done: #{elapsed(start)} sec" }
rescue Exception
logger.info { "fail: #{elapsed(start)} sec" }
raise
end
end
end
end
You must be running some ancient version. Upgrade.
Sorry, looks like that's a Rails 5+ feature only. You'll need to upgrade Rails. https://github.com/rails/rails/commit/8d2b1406bc201d8705e931b6f043441930f2e8ac
I'm trying to make a small example with wisper-sidekiq gem. The queue is created, but it does not start. Why? I will describe the steps that you are doing, I think it will be easier. I hope for your help.
A. controller:
class BooksController < ApplicationController
def create
service = CreateBook.new
service.subscribe(ActivityListener, async: true)
service.on(:reserver_item_successfull) { |book| redirect_to book_path(book.id) }
service.on(:reserver_item_failed) { |book| #book = Book.new(book_params); respond_with(#book) }
service.execute(current_user, book_params)
end
B. service:
require 'wisper/sidekiq'
require 'sidekiq/api'
class CreateBook
include Wisper::Publisher
def execute(performer, attributes)
book = Book.new(attributes)
if book.valid?
book.save
broadcast(:reserver_item_successfull, performer, book)
else
broadcast(:book_failed, performer, book)
end
end
end
C. listener:
class ActivityListener
def self.reserver_item_successfull(performer, book)
puts performer.name.to_s + ", book: " + book.title.to_s
end
end
When I save the book, then of course creates a queue. But:
sidekiq silent (the logs are empty, but the queue was created)
redis silent too
Maybe I'm wrong start redis (redis-server) or sidekiq (bundle exec sidekiq)? Please help me.
P.S. Try bundle exec sidekiq -d -e production sidekiq -q default -C config/sidekiq.yml, but not result. The sidekiq.rb empty.
I think you can get this fixed by the following steps:
require 'sidekiq/web' and then, mount Sidekiq::Web => '/sidekiq'
go to /sidekiq to see if there're workers/tasks/queues
if there aint
something may be wrong with your redis
or your code, put a binding.pry there
if there're, must be something wrong with your code, put a binding.pry there
Hope that helps :-)
The problem is solved. I had to run sidekiq:
bundle exec sidekiq-r ./server.rb-L log/sidekiq.log
also in server.rb
require 'sidekiq'
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://localhost:6379/0' }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://localhost:6379/0' }
end