In development, it runs as I would expect it, having 5 threads (limited at the moment due to redis connection limit) it averages at about 5-7 processes running, depending if the worker has to do anything or not (sometimes a worker would decide not to work, since the object it is working on was updated less than a few days ago)
on production, it behaves differently. It seems to run in bursts of around 400, and then immediatly reschedules the workers and waits a bit and then shoots a burst again
The workers work with facebook api (koala gem), which for this I use sidekiq-throttler (https://github.com/gevans/sidekiq-throttler)
with the options
sidekiq_options throttle: { threshold: 50, period: 60.seconds , key: ->(user_id){ "facebook:#{user_id}"} }
I am using heroku and redislabs (free plan at the moment) with the procfile
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -c 5
and sidekiq setup:
Sidekiq.configure_server do |config|
config.redis = { :url => "#{ENV['REDISCLOUD_URL']}", :namespace => 'sidekiq'}
config.server_middleware do |chain|
chain.add Sidekiq::Throttler, storage: :redis
end
end
Sidekiq.configure_client do |config|
config.redis = { :url => "#{ENV['REDISCLOUD_URL']}", :namespace => 'sidekiq' }
end
is this a known symptom for something?
Looks like it's being throttled, as expected.
Related
So I just migrated to Rails 5.1.4 and I'm trying to make Active Job work, but the jobs are just stuck in the queue and never processed.
rails: 5.1.4
ruby: 2.4.3
sidekiq: 5.0.5
redis: 4.0.1
sidekiq.yml
---
:verbose: true
:concurrency: 5
:timeout: 60
development:
:concurrency: 25
staging:
:concurrency: 50
production:
:concurrency: 5
:queues:
- default
- [high_priority, 2]
sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = {url: ENV['ACTIVE_JOB_URL'], network_timeout: 5}
end
Sidekiq.configure_client do |config|
config.redis = {url: ENV['ACTIVE_JOB_URL'], network_timeout: 5}
end
Here the is how I perform the task from the rails console:
TestJob.perform_later
TestJob.rb content:
class TestJob < ApplicationJob
queue_as :default
def perform(*args)
Rails.logger.debug "#{self.class.name}: I'm performing my job with arguments: #{args.inspect}"
end
end
The jobs are just stuck in the queue and never processed:
Have you started the worker, eg in development it might be:
bundle exec sidekiq
If in production Heroku should do this for you, if you have configured your Procfile, eg:
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq
You can also use your Procfile in development with foreman, eg:
foreman start -f Procfile
I'm trying to understand the Redis & Sidekiq configuration in a Passenger+Rails app and have run into some lack of understanding. I start the redis server independently of my rails app, while Sidekiq is a gem in my Rails app. I start it likewise: (no sidekiq.yml file needed for me)
bundle exec sidekiq
Following is my sidekiq.rb initializer:
require 'sidekiq'
require 'sidekiq-status'
Sidekiq.configure_client do |config|
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end
end
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
end
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end
end
I went through some library classes, but to no avail.
I want to understand where does Sidekiq configure it's Redis server details. It defaults to localhost:6379, but I am not quite sure how.
Also, if I wish to use Memcached in future, how can I change that?
From sidekiq docs:
By default, Sidekiq tries to connect to Redis at localhost:6379
https://github.com/mperham/sidekiq/wiki/Using-Redis
You can change the port in the initializer:
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://redis.example.com:7372/12' }
end
From the looks of it sidekiq works only with redis
Sidekiq uses Redis to store all of its job and operational data.
Here is what I did from the other answer:
# frozen_string_literal: true
Sidekiq.configure_server do |config|
config.redis = {
url: ENV.fetch("SIDEKIQ_REDIS_URL", "redis://localhost:6379/1")
}
end
My New Relic Insights is logging pageviews and User Agent info from local machine in dev environment. I have another dev in another city who is also having the development enviroment pageviews and other info being logged.
When I pull up samples, I see localhost:3000, which is my port.
However, the production info is also being logged.
I have New Relic running using Heroku's default set up. It automatically sets the license key as an environment variable. I do not have the license key anywhere in the app, it only set through an environment variable.
If I pull up my local development environment, navigate to port 3000, and refresh, then query New Relic Insights for events in the last minute, I see my city, my user agent info, my visited url and pageview. Our product is in beta, there is really no chance that an actual user in my location is hitting the same random page.
I have tried turning development mode off, monitor off. I cannot understand how this can be happening.
I do have some files hosted on AWS (images and some js), if that matters
Gemfile
group :production do
gem 'rails_12factor'
gem 'newrelic_rpm'
end
config/newrelic.yml
common: &default_settings
license_key: <%= ENV["NEW_RELIC_LICENSE_KEY"] %>
log_level: info
development:
<<: *default_settings
app_name: app-dev
developer_mode: false
monitor_mode: false
agent_enabled: false
test:
<<: *default_settings
monitor_mode: false
developer_mode: false
agent_enabled: false
production:
app_name: app-prod
monitor_mode: true
agent_enabled: false
<<: *default_settings
config/puma.rb
require 'puma_worker_killer'
ActiveRecord::Base.connection_pool.disconnect!
PumaWorkerKiller.config do |config|
config.ram = ENV['PUMA_WORKER_KILLER_RAM'] || 1024 # mb
config.frequency = 5 # seconds
config.percent_usage = 0.98
config.rolling_restart_frequency = 12 * 3600 # 12 hours in seconds
end
PumaWorkerKiller.start
end
workers Integer(ENV['WEB_CONCURRENCY'] || 5)
min_threads_count = Integer(ENV['MIN_THREADS'] || 1)
threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)
threads min_threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
# #sidekiq_pid ||= spawn('bundle exec sidekiq -c 2 -q default -q mailers')
ActiveSupport.on_load(:active_record) do
ActiveRecord::Base.establish_connection
end
end
config/initializers/sidekiq.rb
require 'sidekiq'
redis_url = ENV['REDISTOGO_URL']
redis_config = {
url: redis_url,
namespace: 'oct',
}
Sidekiq.configure_server do |config|
config.redis = {
url: ENV["REDISTOGO_URL"], namespace: 'app',
size: ENV["SIDEKIQ_SERVER_CONNECTIONS"].to_i || 6
}
config.error_handlers << Proc.new do |exception, context_hash|
SidekiqErrorService.new(exception, context_hash).notify
end
end
Sidekiq.configure_client do |config|
config.redis = {
url: ENV["REDISTOGO_URL"], namespace: 'app',
size: ENV["REDIS_CLIENT_CONNECTION_SIZE"].to_i || 2
}
end
So I believe it was the New Relic Browser JS that I included in the head of my pages. Once I set that to - if production_environment? (my helper method), then I only saw production environment traffic.
I believe that something in that JS was pinging my New Relic.
Fixed now.
I am trying to run message queues on heroku. For this I am using RabbitMQ Bigwig plugin.
I am publishing messages using bunny gem and trying to receive messages with sneakers gem. This whole setup works smoothly on local machine.
I take following steps to setup queue
I run this rake on server to setup queue:
namespace :rabbitmq do
desc 'Setup routing'
task :setup_test_commands_queue do
require 'bunny'
conn = Bunny.new(ENV['SYNC_AMQP'], read_timeout: 10, heartbeat: 10)
conn.start
ch = conn.create_channel
# get or create exchange
x = ch.direct('testsync.pcc', :durable => true)
# get or create queue (note the durable setting)
queue = ch.queue('test.commands', :durable => true, :ack => true, :routing_key => 'test_cmd')
# bind queue to exchange
queue.bind(x, :routing_key => 'test_cmd')
conn.close
end
end
I am able to see this queue in rabbitmq management plugin with mentioned binding.
class TestPublisher
def self.publish(test)
x = channel.direct("testsync.pcc", :durable => true)
puts "publishing this = #{Test}"
x.publish(Test, :persistent => true, :routing_key => 'pcc_cmd')
end
def self.channel
#channel ||= connection.create_channel
end
def self.connection
#conn = Bunny.new(ENV['RABBITMQ_BIGWIG_TX_URL'], read_timeout: 10, heartbeat: 10) # getting configuration from rabbitmq.yml
#conn.start
end
end
I am calling TestPublisher.publish() to publish message.
I have sneaker worker like this:
require 'test_sync'
class TestsWorker
include Sneakers::Worker
from_queue "test.commands", env: nil
def work(raw_event)
puts "^"*100
puts raw_event
# o = CaseNote.create!(content: raw_event, creator_id: 1)
# puts "#########{o}"
test = Oj.load raw_event
test.execute
# event_params = JSON.parse(raw_event)
# SomeWiseService.build.call(event_params)
ack!
end
end
My Procfile
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec rake jobs:work
sneaker: WORKERS=TestsWorker bundle exec rake sneakers:run
My Rakefile
require File.expand_path('../config/application', __FILE__)
require 'rake/dsl_definition'
require 'rake'
require 'sneakers/tasks'
Test::Application.load_tasks
My sneaker configuration
require 'sneakers'
Sneakers.configure amqp: ENV['RABBITMQ_BIGWIG_RX_URL'],
log: "log/sneakers.log",
threads: 1,
workers: 1
puts "configuring sneaker"
I am sure that message gets published. I am able to get message on rabbitmq management plugin. But sneaker does not work. There is nothing in sneakers.log that can help.
sneakers.log on heroku :
# Logfile created on 2016-04-05 14:40:59 +0530 by logger.rb/41212
Sorry for this late response. I was able to get this working on heroku. When I faced this error after hours of debugging I was not able to fix it. So I rewrote all above code and I did not check what was wrong with my previous code.
The only problem with this code and correct code is queue binding.
I had two queues on same exchange. pcc.commands with routing key pcc_cmd and test.commands with routing key test_cmd.
I was working with test_cmd but as per following line in TestPublisher
x.publish(Test, :persistent => true, :routing_key => 'pcc_cmd')
I was publishing to different queue(pcc.commands). Hence I was not able to recieve the message on test.commands queue.
In TestWorker
from_queue "test.commands", env: nil
This states that fetch messages only from test.commands queue.
Regarding sneakers.log file:
Above setup was not able to give me logs in sneakers.log file. Yes this setup works on your local development machine, but it was not working on heroku. Now days to debug such issue I ommit log attribute from configuration. like this:
require 'sneakers'
Sneakers.configure amqp: ENV['RABBITMQ_BIGWIG_RX_URL'],
# log: "log/sneakers.log",
threads: 1,
workers: 1
This way you will get sneaker logs (even heartbeat logs) in heroku logs which can be seen by running command heroku logs -a app_name --tail.
I have a Sidekiq worker that syncs the data of a Rails 3.2 application with a remote database. When I just execute it from the rails console, everything works as expected. I can verify that it connects to the remote DB and pulls in the data I am expecting.
The moment I try to set up the following:
recurrence {daily.hour_of_day(20).minute_of_hour(02)}
I see the following in Sidekiq but no signs of completion/execution:
2014-05-14T00:02:18Z 6747 TID-dvc2c INFO: [Sidetiq] Enqueue: BackendWorker (at: 1400112120.0) (last: -1.0)
I've got retries set to false (which I would like to change) currently, and I have attempted to clear out anything in redis before setting up another test run again.
Is there a way to squeeze some more information about what is going on? Is there a Sidekiq flag I can use to get some additional debug information?
Thanks in advance
Environment:
Rails 3.2
Ruby 2.1
Sidetiq 0.5.0
Sidekiq 2.17.7 (due to an issue with using 3.0)
UPDATE (5/14/2014):
I decided to test a simple example with Sidetiq and just have a worker that creates a DB entry with ActiveRecord:
class BackendTest
include Sidekiq::Worker
include Sidetiq::Schedulable
sidekiq_options :retry => false, :backtrace => true
recurrence {
daily.hour_of_day(11).minute_of_hour(05)
}
def perform
Book.create(title: "test", author: "testing", price: 9.99)
end
end
Another experiment I did was just trying to get Sidetiq and Sidekiq to just show something with logger.info:
class BackendTest
include Sidekiq::Worker
include Sidetiq::Schedulable
sidekiq_options :retry => false, :backtrace => true
recurrence {
daily.hour_of_day(16).minute_of_hour(00)
}
def perform
logger.info "test"
logger.info "test"
end
end
The jobs just still enters the default queue without being started or completed. Everything still works if I just fire it off from the rails console with perform_async, Sidekiq::Client.push, and Sidekiq::Client.enqueue still work from console.
I am a thoroughly confused newbie at this point.
Output:
2014-05-14T20:08:41Z 2149 TID-dfcfk BackendTest JID-6bc7ba08955f8340903f9063 INFO: start
2014-05-14T20:08:41Z 2149 TID-dfcfk BackendTest JID-6bc7ba08955f8340903f9063 INFO: test
2014-05-14T20:08:41Z 2149 TID-dfcfk BackendTest JID-6bc7ba08955f8340903f9063 INFO: test
2014-05-14T20:08:41Z 2149 TID-dfcfk BackendTest JID-6bc7ba08955f8340903f9063 INFO: done: 0.002 sec
After poking around other people's projects on github, I saw something interesting and tried it out and it seemed to solve my problem with Sidetiq.
I created /config/initializers/sidekiq.rb with the following content inside of it:
Sidekiq.configure_server do |config|
config.redis = { :url => 'redis://localhost:6379/3', :namespace => 'sidekiq' }
end
Sidekiq.configure_client do |config|
config.redis = { :url => 'redis://localhost:6379/3', :namespace => 'sidekiq' }
end
Job tasks enqueue and execute as expected.