I am working on rabbitmq and trying to push a method to a queue from my ruby on rails app and I am running a server side ruby script to read the queue and execute the method which is send in the payload. Here is my client side code.
module Rabbitesh
require 'amqp'
#debugger
def self.call_rabbits(payload,queue_name)
AMQP.start(:host => "localhost") do |connection|
channel = AMQP::Channel.new(connection)
queue = channel.queue(queue_name)
channel.default_exchange.publish(payload, :routing_key => queue.name)
#EM.add_timer(0.01) do
connection.close do
#end
end
end
end
end
This is now I call the Rabbitmq function
Rabbitesh::call_rabbits(obj,"welcome_mail")
where "welcome_mail" is the queue_name
This is the server side script
require 'rubygems'
require 'amqp'
require 'daemons'
options = { :backtrace => true, :dir => '.', :log_output => true}
Daemons.run_proc('raabbitmq_daemon',options) do
AMQP.start(:host => "localhost") do |connection|
channel = AMQP::Channel.new(connection)
queue = channel.queue("welcome_mail")
Signal.trap("INT") do
connection.close do
EM.stop { exit }
end
end
puts " [*] Waiting for messages. To exit press CTRL+C"
queue.subscribe do |body|
UserMailers.welcome_organic(body).deliver
end
end
end
The problem is when my rails app calls the rabbitmq function the console stops there saying "updating client properties" and though I will be running my server side ruby script, it will not read the queue and take execute the process. I am not able to understand whats wrong with the code, kindly help me out.
Related
We have a Rails 3.2 app and will listening on a RabbitMQ queue using Bunny. The code for the listener will be like:
require 'bunny'
class TestBunny
def self.do
connection = Bunny.new(host: RABBITMQ_HOST, user: RABBITMQ_USERNAME, pass: RABBITMQ_PASSWORD, automatically_recover: false)
connection.start
channel = connection.create_channel
queue = channel.queue('custom_reports_queue', durable: false)
channel.prefetch(1)
puts ' [*] Waiting for messages. To exit press CTRL+C'
begin
queue.subscribe(manual_ack: true, block: true) do |delivery_info, _properties, body|
puts " [x] Received '#{body}'"
# imitate some work
sleep body.count('.').to_i
puts ' [x] Done'
OfflineReportMailer.success.deliver
channel.ack(delivery_info.delivery_tag)
end
rescue Interrupt => _
connection.close
end
end
I have converted the above to a rake task:
RAILS_ENV=local rake report_queue:listen
It needs to be always running. How would I call this? Possibly as a rake background task? We are using nginx / unicorn as our webserver.
I'm using Faye websockets and Redis to attempt a master/client websocket setup for a slideshow presentation.
The clients' should all follow along with the master (i.e. the master moves forward a slide, a message is sent out to all of the clients, and they all move forward a slide).
The issue I'm having is that my list of client websockets is empty when I access inside the thread created in the initialize function. The separate thread is necessary as Redis' 'subscribe' is a blocking function.
This file is a middleware and runs on app boot up.
I know that the point of global variables in rails is that they're shared across threads, but in my case something seems to be preventing it.
Is there a way to store a list of websockets in a globally accessible place? Globally, as in, for all running instances of the app on the same server.
(can't use Redis for that cause it can't store objects).
require 'faye/websocket'
require 'redis'
class WsCommunication
KEEPALIVE_TIME = 15 #seconds
CHANNEL = 'vip'
def initialize(app)
#app = app
$clients = []
uri = URI.parse(ENV['REDISCLOUD_URL'])
Thread.new do
redis_sub = Redis.new(host: uri.host, port: uri.port, password: uri.password)
redis_sub.subscribe(CHANNEL) do |on|
on.message do |channel, msg|
puts 'client list on thread'
puts $clients
#### prints nothing
$clients.each { |ws| ws.send(msg) }
end
end
end
end
def call(env)
if Faye::WebSocket.websocket?(env)
ws = Faye::WebSocket.new(env, nil, {ping: KEEPALIVE_TIME})
ws.on :open do |event|
$clients << ws
end
ws.on :message do |event|
puts 'client list'
puts $clients
### prints the full list of clients
$redis.publish(CHANNEL, event.data)
end
ws.on :close do |event|
$clients.delete(ws)
ws = nil
end
# Return async Rack response
ws.rack_response
else
#app.call(env)
end
end
end
I have my main Rails server and a separate Faye server (localhost:9292) setup correctly to handle messaging. Everything is working as expected but I'm having problems with trying to test it.
I'm trying to setup my test as a worker that runs once a day, and if a message isn't received I want to have the app send me an email. Here's what I have so far based on the info in Faye's docs
require 'eventmachine'
require 'faye/websocket'
class PingFayeServerWorker < BaseWorker
def initialize
end
def process
EventMachine.run {
client = Faye::Client.new('http://localhost:9292/faye')
client.set_header('Authorization', 'OAuth abcd-1234')
client.subscribe('/foo') do |message|
puts message.inspect
end
publication = client.publish('/foo', 'text' => 'Hello world')
publication.callback do
puts 'Message received by server!'
end
publication.errback do |error|
puts 'There was a problem: ' + error.message
end
}
end
end
I am trying to run message queues on heroku. For this I am using RabbitMQ Bigwig plugin.
I am publishing messages using bunny gem and trying to receive messages with sneakers gem. This whole setup works smoothly on local machine.
I take following steps to setup queue
I run this rake on server to setup queue:
namespace :rabbitmq do
desc 'Setup routing'
task :setup_test_commands_queue do
require 'bunny'
conn = Bunny.new(ENV['SYNC_AMQP'], read_timeout: 10, heartbeat: 10)
conn.start
ch = conn.create_channel
# get or create exchange
x = ch.direct('testsync.pcc', :durable => true)
# get or create queue (note the durable setting)
queue = ch.queue('test.commands', :durable => true, :ack => true, :routing_key => 'test_cmd')
# bind queue to exchange
queue.bind(x, :routing_key => 'test_cmd')
conn.close
end
end
I am able to see this queue in rabbitmq management plugin with mentioned binding.
class TestPublisher
def self.publish(test)
x = channel.direct("testsync.pcc", :durable => true)
puts "publishing this = #{Test}"
x.publish(Test, :persistent => true, :routing_key => 'pcc_cmd')
end
def self.channel
#channel ||= connection.create_channel
end
def self.connection
#conn = Bunny.new(ENV['RABBITMQ_BIGWIG_TX_URL'], read_timeout: 10, heartbeat: 10) # getting configuration from rabbitmq.yml
#conn.start
end
end
I am calling TestPublisher.publish() to publish message.
I have sneaker worker like this:
require 'test_sync'
class TestsWorker
include Sneakers::Worker
from_queue "test.commands", env: nil
def work(raw_event)
puts "^"*100
puts raw_event
# o = CaseNote.create!(content: raw_event, creator_id: 1)
# puts "#########{o}"
test = Oj.load raw_event
test.execute
# event_params = JSON.parse(raw_event)
# SomeWiseService.build.call(event_params)
ack!
end
end
My Procfile
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec rake jobs:work
sneaker: WORKERS=TestsWorker bundle exec rake sneakers:run
My Rakefile
require File.expand_path('../config/application', __FILE__)
require 'rake/dsl_definition'
require 'rake'
require 'sneakers/tasks'
Test::Application.load_tasks
My sneaker configuration
require 'sneakers'
Sneakers.configure amqp: ENV['RABBITMQ_BIGWIG_RX_URL'],
log: "log/sneakers.log",
threads: 1,
workers: 1
puts "configuring sneaker"
I am sure that message gets published. I am able to get message on rabbitmq management plugin. But sneaker does not work. There is nothing in sneakers.log that can help.
sneakers.log on heroku :
# Logfile created on 2016-04-05 14:40:59 +0530 by logger.rb/41212
Sorry for this late response. I was able to get this working on heroku. When I faced this error after hours of debugging I was not able to fix it. So I rewrote all above code and I did not check what was wrong with my previous code.
The only problem with this code and correct code is queue binding.
I had two queues on same exchange. pcc.commands with routing key pcc_cmd and test.commands with routing key test_cmd.
I was working with test_cmd but as per following line in TestPublisher
x.publish(Test, :persistent => true, :routing_key => 'pcc_cmd')
I was publishing to different queue(pcc.commands). Hence I was not able to recieve the message on test.commands queue.
In TestWorker
from_queue "test.commands", env: nil
This states that fetch messages only from test.commands queue.
Regarding sneakers.log file:
Above setup was not able to give me logs in sneakers.log file. Yes this setup works on your local development machine, but it was not working on heroku. Now days to debug such issue I ommit log attribute from configuration. like this:
require 'sneakers'
Sneakers.configure amqp: ENV['RABBITMQ_BIGWIG_RX_URL'],
# log: "log/sneakers.log",
threads: 1,
workers: 1
This way you will get sneaker logs (even heartbeat logs) in heroku logs which can be seen by running command heroku logs -a app_name --tail.
I am using Resque and Resque Schedule to start a job that has to be run immediately on the application start. Other scheduled jobs are loaded every 30 seconds.
This is the code for my config/initializers/redis.rb
require 'rake'
require 'resque'
require 'resque/server'
require 'resque_scheduler/tasks'
# This will make the tabs show up.
require 'resque_scheduler'
require 'resque_scheduler/server'
uri = URI.parse(ENV["REDISTOGO_URL"])
REDIS = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
Resque.redis = REDIS
Dir["#{Rails.root}/app/workers/*.rb"].each { |file| require file }
Resque.enqueue(AllMessageRetriever)
Resque.schedule = YAML.load_file(Rails.root.join('config', 'schedule.yml'))
When the application is started up, the AllMessageRetriever gets run 2-3 times rather than only once. Do the initializers get called more than once? This happens both on Heroku and my local environment?
Is it possible to set a delayed job in Resque-Scheduler which will only get executed once and immediately on runtime?
The code for AllMessageRetriever. Basically it loops over a table and calls an external API to get data and then updates it to the table. This entire task happens 2-3 times if I add the enqueue method in initializer file
require 'socialcast'
module AllMessageRetriever
#queue = :message_queue
def self.perform()
Watchedgroup.all.each do |group|
puts "Running group #{group.name}"
continueLoading=true
page=1
per_page=500
while(continueLoading == true)
User.first.refresh_token_if_expired
token = User.first.token
puts "ContinueLoading: #{continueLoading}"
#test = Socialcast.get_all_messages(group.name,token,page,per_page)
messagesArray = ActiveSupport::JSON.decode(#test)["messages"]
puts "Message Count: #{messagesArray.count}"
if messagesArray.count == 0
puts 'count is zero now'
continueLoading = false
else
messagesArray.each do |message|
if not Message.exists?(message["id"])
Message.create_with_socialcast(message, group.id)
else
Message.update_with_socialcast(message)
end
end
end
page += 1
end
Resqueaudit.create({:watchedgroup_id => group.id,:timecompleted => DateTime.now})
end
# Do anything here, like access models, etc
puts "Doing my job"
end
end
Rake
Firstly, why are you trying to queue on init?
You'd be much better delegating to a rake task which is called from an initializer.
This will remove dependency on the initialize process, which should clear things up a lot. I wouldn't put this in an initializer itself, as it will be better handled elsewhere (modularity)
Problem
I think this line is causing the issue:
Resque.enqueue(AllMessageRetriever)
Without seeing the contents of AllMessageRetriever, I'd surmise that you're AllMessageRetriever (module / class?) will be returning the results 2/3 times, causing Resque to add the (2 / 3 times) data-set to the queue
Could be wrong, but it would make sense, and mean your issue is not with Resque / Initializers, but your AllMessageRetriever class
Would be a big help if you showed it!