I am trying to deploy a RoR app that does some asynchronous task. I use workling for that and the message queue is RabbitMQ. This combination worked flawlessly with Starling but we decided to change the MQ for Rabbit.
I read somewhere that I should include the following code in my environment.rb
require 'mq'
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked|
if forked
if EM.reactor_running?
EM.stop_event_loop
EM.release_machine
EM.instance_variable_set( '#reactor_running', false )
end
Thread.current[:mq] = nil
AMQP.instance_variable_set('#conn', nil)
end
th = Thread.current
Thread.new{
AMQP.connect(:host => 'localhost'){
th.wakeup
}
}
Thread.stop
end
end
But that now Apache fails completely with message: The server encountered an internal error or misconfiguration and was unable to complete your request
EDIT: I've improved the code below somewhat since posting this. Available here: http://www.hiringthing.com/2011/11/04/eventmachine-with-rails.html
I just spent a milliioon years trying to get this to work, and finally did. Here is my code:
require 'amqp'
module HiringThingEM
def self.start
if defined?(PhusionPassenger)
PhusionPassenger.on_event(:starting_worker_process) do |forked|
if forked && EM.reactor_running?
EM.stop
end
Thread.new {
EM.run do
AMQP.channel ||= AMQP::Channel.new(AMQP.connect(:host=> Q_SERVER, :user=> Q_USER, :pass => Q_PASS, :vhost => Q_VHOST ))
end
}
die_gracefully_on_signal
end
end
end
def self.die_gracefully_on_signal
Signal.trap("INT") { EM.stop }
Signal.trap("TERM") { EM.stop }
end
end
HiringThingEM.start
Now I can use:
EM.next_tick { AMQP.channel.queue(Q_Q).publish("hi mom") }
Inside the controllers of my Rails app.
Hope this helps someone.
Not really an answer, but unless you're committed to AMQP, I would recommend using https://github.com/defunkt/resque - it does the asynchronous job + fork gig very nicely.
Related
I have a Rails application that runs some background jobs via ActiveJob and Sidekiq. The sidekiq logs in both the terminal and the log file show the following:
2016-10-18T06:17:01.911Z 3252 TID-oukzs4q3k ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-97318b38b1391672d21feb93 INFO: start
Is there some way to show the class names of the jobs here similar to how logs work for a regular Sidekiq Worker?
Update:
Here is how a Sidekiq worker logs:
2016-10-18T11:05:39.690Z 13678 TID-or4o9w2o4 ClientJob JID-b3c71c9c63fe0c6d29fd2f21 INFO: start
Update 2:
My sidekiq version is 3.4.2
I'd like to replace ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper with Client Job
So I managed to do this by removing Sidekiq::Middleware::Server::Logging from the middleware configuration and adding a modified class that displays the arguments in the logs. The arguments themself contain the job and action names as well.
For latest version, currently 4.2.3, in sidekiq.rb
require 'sidekiq'
require 'sidekiq/middleware/server/logging'
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def log_context(worker, item)
klass = item['wrapped'.freeze] || worker.class.to_s
"#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}"
end
end
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.remove Sidekiq::Middleware::Server::Logging
chain.add ParamsLogging
end
end
For version 3.4.2, or similar, override the call method instead:
class ParamsLogging < Sidekiq::Middleware::Server::Logging
def call(worker, item, queue)
klass = item['wrapped'.freeze] || worker.class.to_s
Sidekiq::Logging.with_context("#{klass} (#{item['args'].try(:join, ' ')}) JID-#{item['jid'.freeze]}") do
begin
start = Time.now
logger.info { "start" }
yield
logger.info { "done: #{elapsed(start)} sec" }
rescue Exception
logger.info { "fail: #{elapsed(start)} sec" }
raise
end
end
end
end
You must be running some ancient version. Upgrade.
Sorry, looks like that's a Rails 5+ feature only. You'll need to upgrade Rails. https://github.com/rails/rails/commit/8d2b1406bc201d8705e931b6f043441930f2e8ac
I'm trying to make a small example with wisper-sidekiq gem. The queue is created, but it does not start. Why? I will describe the steps that you are doing, I think it will be easier. I hope for your help.
A. controller:
class BooksController < ApplicationController
def create
service = CreateBook.new
service.subscribe(ActivityListener, async: true)
service.on(:reserver_item_successfull) { |book| redirect_to book_path(book.id) }
service.on(:reserver_item_failed) { |book| #book = Book.new(book_params); respond_with(#book) }
service.execute(current_user, book_params)
end
B. service:
require 'wisper/sidekiq'
require 'sidekiq/api'
class CreateBook
include Wisper::Publisher
def execute(performer, attributes)
book = Book.new(attributes)
if book.valid?
book.save
broadcast(:reserver_item_successfull, performer, book)
else
broadcast(:book_failed, performer, book)
end
end
end
C. listener:
class ActivityListener
def self.reserver_item_successfull(performer, book)
puts performer.name.to_s + ", book: " + book.title.to_s
end
end
When I save the book, then of course creates a queue. But:
sidekiq silent (the logs are empty, but the queue was created)
redis silent too
Maybe I'm wrong start redis (redis-server) or sidekiq (bundle exec sidekiq)? Please help me.
P.S. Try bundle exec sidekiq -d -e production sidekiq -q default -C config/sidekiq.yml, but not result. The sidekiq.rb empty.
I think you can get this fixed by the following steps:
require 'sidekiq/web' and then, mount Sidekiq::Web => '/sidekiq'
go to /sidekiq to see if there're workers/tasks/queues
if there aint
something may be wrong with your redis
or your code, put a binding.pry there
if there're, must be something wrong with your code, put a binding.pry there
Hope that helps :-)
The problem is solved. I had to run sidekiq:
bundle exec sidekiq-r ./server.rb-L log/sidekiq.log
also in server.rb
require 'sidekiq'
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://localhost:6379/0' }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://localhost:6379/0' }
end
Using the console, how can I tell if sidekiq is connected to a redis server? I want to be able to do something like this:
if (sidekiq is connected to redis) # psuedo code
MrWorker.perform_async('do_work', user.id)
else
MrWorker.new.perform('do_work', user.id)
end
You can use Redis info provided by Sidekiq:
redis_info = Sidekiq.redis { |conn| conn.info }
redis_info['connected_clients'] # => "16"
Took it from Sidekiq's Sinatra status app.
I make this method to Rails whit the obove answer, return true if connected and false if not.
def redis_connected?
!!Sidekiq.redis(&:info) rescue false
end
It sounds like you want to know if there is a Sidekiq process up and running to process jobs at a given point in time. With Sidekiq 3.0, you can do this:
require 'sidekiq/api'
ps = Sidekiq::ProcessSet.new
if ps.size > 0
MyWorker.perform_async(1,2,3)
else
MyWorker.new.perform(1,2,3)
end
Sidekiq::ProcessSet gives you almost real-time (updated every 5 sec) info about any running Sidekiq processes.
jumping off #overallduka's answer, for those using the okcomputer gem, this is the custom check i set up:
class SidekiqCheck < OkComputer::Check
def check
if sidekiq_accessible?
mark_message "ok"
else
mark_failure
end
end
private
def sidekiq_accessible?
begin
Sidekiq.redis { |conn| conn.info }
rescue Redis::CannotConnectError
end.present?
end
end
OkComputer::Registry.register "sidekiq", SidekiqCheck.new
begin
MrWorker.perform_async('do_work', user.id)
rescue Redis::CannotConnectError => e
MrWorker.new.perform('do_work', user.id)
end
I am working on rabbitmq and trying to push a method to a queue from my ruby on rails app and I am running a server side ruby script to read the queue and execute the method which is send in the payload. Here is my client side code.
module Rabbitesh
require 'amqp'
#debugger
def self.call_rabbits(payload,queue_name)
AMQP.start(:host => "localhost") do |connection|
channel = AMQP::Channel.new(connection)
queue = channel.queue(queue_name)
channel.default_exchange.publish(payload, :routing_key => queue.name)
#EM.add_timer(0.01) do
connection.close do
#end
end
end
end
end
This is now I call the Rabbitmq function
Rabbitesh::call_rabbits(obj,"welcome_mail")
where "welcome_mail" is the queue_name
This is the server side script
require 'rubygems'
require 'amqp'
require 'daemons'
options = { :backtrace => true, :dir => '.', :log_output => true}
Daemons.run_proc('raabbitmq_daemon',options) do
AMQP.start(:host => "localhost") do |connection|
channel = AMQP::Channel.new(connection)
queue = channel.queue("welcome_mail")
Signal.trap("INT") do
connection.close do
EM.stop { exit }
end
end
puts " [*] Waiting for messages. To exit press CTRL+C"
queue.subscribe do |body|
UserMailers.welcome_organic(body).deliver
end
end
end
The problem is when my rails app calls the rabbitmq function the console stops there saying "updating client properties" and though I will be running my server side ruby script, it will not read the queue and take execute the process. I am not able to understand whats wrong with the code, kindly help me out.
Hey, I just tested the TweetStream gem.
Example:
TweetStream::Client.new('myuser','mypass').track('ruby', 'rails') do |status|
puts "[#{status.user.screen_name}] #{status.text}"
end
This example works.
Questions:
I tried restarting my router (internet connection lost) and after that no new messages have arrived. Can someone explain this behavior to me?
I tested the daemon. What happens if no internet connection is available for a day or more? Will it reconnect automatically?
I like Rufus gem (for background processes). Can I somehow integrate this code with Rufus where I would check if the process is still active?
My reconnect solution (config/initializers/tweet_stream.rb):
client = nil
scheduler = Rufus::Scheduler.start_new
scheduler.every '30min', :first_in => '1s' do |job|
client.stop rescue nil
client = TweetStream::Client.new('user','pass').on_error do |message|
Rails.logger.info "[Rufus][#{Time.now}] TweetStream error: #{message}"
end.track('love') do |status|
Rails.logger.error "[TweetStream] Status: #{status.id}"
end
end
Thx!