I'm trying to understand the Redis & Sidekiq configuration in a Passenger+Rails app and have run into some lack of understanding. I start the redis server independently of my rails app, while Sidekiq is a gem in my Rails app. I start it likewise: (no sidekiq.yml file needed for me)
bundle exec sidekiq
Following is my sidekiq.rb initializer:
require 'sidekiq'
require 'sidekiq-status'
Sidekiq.configure_client do |config|
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end
end
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
end
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end
end
I went through some library classes, but to no avail.
I want to understand where does Sidekiq configure it's Redis server details. It defaults to localhost:6379, but I am not quite sure how.
Also, if I wish to use Memcached in future, how can I change that?
From sidekiq docs:
By default, Sidekiq tries to connect to Redis at localhost:6379
https://github.com/mperham/sidekiq/wiki/Using-Redis
You can change the port in the initializer:
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://redis.example.com:7372/12' }
end
From the looks of it sidekiq works only with redis
Sidekiq uses Redis to store all of its job and operational data.
Here is what I did from the other answer:
# frozen_string_literal: true
Sidekiq.configure_server do |config|
config.redis = {
url: ENV.fetch("SIDEKIQ_REDIS_URL", "redis://localhost:6379/1")
}
end
Related
I'm working on a site someone else made. There is a production version.
I'm trying to send emails to users whose properties haven't been updated in 30 days.
Everything works until I try to use deliver_later. The reason I'm using deliver_later is because deliver_now results in an issue of sending too many emails per second. I'm currently using Mailtrap for testing, but I assume I will run into that sort of issue on production.
So I opted to wait 1 second for each email:
#testproperties = Property.has_not_updated.includes(:user)
#testproperties.each do |property|
UserMailer.with(property: property, user: property.user).check_listing.deliver_later(wait:1.seconds)
end
This results in IO::EINPROGRESSWaitWritable Operation now in progress - connect(2) would block
And nothing sends.
I'm not sure how to solve this issue.
Edit:
I can see on the production site that I can visit the route /sidekiq. The routes file has this block:
authenticate :user, lambda { |u| u.admin? } do
mount Sidekiq::Web => '/sidekiq'
end
I can view the web interface and see all the jobs. It's all working there. But I need to access development version running on localhost:3000.
Trying to access this locally still results in:
Operation now in progress - connect(2) would block
# # Socket#connect
def connect_nonblock(addr, exception: true)
__connect_nonblock(addr, exception)
end
end
Sidekiq.rb:
require 'sidekiq'
unless Rails.env.test?
host = 'localhost'
port = '6379'
namespace = 'sitename'
Sidekiq.configure_server do |config|
config.redis = { url: "redis://#{host}:#{port}", namespace: namespace }
schedule_file = "config/schedule.yml"
if File.exists?(schedule_file)
Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file)
end
config.server_middleware do |chain|
chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
end
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
end
end
Sidekiq.configure_client do |config|
config.redis = { url: "redis://#{host}:#{port}", namespace: namespace }
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
end
end
end
for cable.yml:
development:
adapter: async
url: redis://localhost:6379/1
channel_prefix: sitename_dev
test:
adapter: async
production:
adapter: redis
url: redis://localhost:6379/1
channel_prefix: sitename_production
The production server is running Ubuntu and they already installed redis-server.
I had not installed that locally. (I'm using Ubuntu through Windows WSL)
sudo apt install redis-server
I can now access the web interface.
Make sure that Redis is started:
sudo service redis-server restart
in my nginx website config file (/etc/nginx/sites-available/my_website I have set:
passenger_env_var RAILS_ENV staging;
I'm using Sidekiq for sending emails, starting it like this:
bundle exec sidekiq -q mailers -d -L log/sidekiq.log
but my Sidekiq is having environment set to development. Why?
Do I have to set the environment explicitly when starting sidekiq? I thought it is set by the nginx config.
Thanks
Sidekiq's default environment is development. One way to set it is to run sidekiq like this
bundle exec sidekiq --environment production
It could be configured by creating a file under initializers/sidekiq.rb
If Rails.env.development?
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://localhost:6379/0', namespace: "name_{Rails.env}" }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://localhost:6379/0', namespace: "name_{Rails.env}" }
end
elsif Rails.env.staging?
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://redis-xx.xxxx.xxxx.xx.xxx.amazonaws.com:6379/12', namespace: "name_#{Rails.env}" }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://redis-xx.xxxx.xxxx.xx.xxx.amazonaws.com:6379/12', namespace: "name_#{Rails.env}" }
end
end
I have a contact form that send a email with the fields to a admin email. I'm using Sidekiq and Redis. When I send the form, the task stay lock up in sidekiq schedule task and is never sent.
Has anyone ever experienced this? I've already tried many things to try fix this, but without success. I configured something wrong?
# app/mailers/contact_mailer.rb
class ContactMailer < ActionMailer::Base
default from: "Facens Liga <no-reply#facens.br>"
def create(contact)
#contact = contact
mail(to: "felipe.marcon#atua.ag", subject: "Contato Através do Site")
end
end
# config/initializers/sidekiq.rb
require 'sidekiq'
require 'sidekiq-status'
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://127.0.0.1:6379/6', namespace: 'facenliga' }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://127.0.0.1:6379/6', namespace: 'facensliga' }
end
# config/sidekiq.yml
:pidfile: tmp/pids/sidekiq.pid
:logfile: ./log/sidekiq.log
:queues:
- default
- mailers
production:
:concurrency: 25
staging:
:concurrency: 15
development:
:concurrency: 25
I hope that someone can help me. Thank you.
The problem is with your namespace. Don't use namespaces, as I wrote in my blog last year.
The redis-namespace gem allows you to share a Redis database among several applications by prefixing every key with a namespace but it's a terrible hack that no one should use. Redis already has a native solution if you want to share a Redis instance: databases. The default database is 0. Here's how to point Sidekiq to use database 1 instead:
https://www.mikeperham.com/2017/04/10/migrating-from-redis-namespace/
Running an RSpec test with a Sidekiq::Queue instance is failing unless Redis is running separately.
Sidekiq::Queue.new('my-queue').select(&:item)
Raises error in test
Redis::CannotConnectError:
Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
I've added the usual to the spec helper:
require 'sidekiq/testing'
Sidekiq::Testing.inline!
And mock_redis to the gemfile.
# gemfile
gem 'mock_redis', '0.16.1'
Using sidekiq (3.4.2)
How can I update my configuration to allow this to work?
mock_redis only provides with a fake redis. It does not intercept/replace actual redis classes/connections. If you intend to use fake redis in tests, you should tell sidekiq so. In your config/initializers/sidekiq.rb (or whereever your sidekiq redis config is):
redis = if Rails.env.test?
require 'mock_redis'
MockRedis.new
else
{ url: 'redis://redis.example.com:7372/12' }
end
Sidekiq.configure_server do |config|
config.redis = redis
end
Sidekiq.configure_client do |config|
config.redis = redis
end
I solved this by mocking Redis for tagged RSpec tests in the spec_helper.rb file.
config.before(:each, redis: true) do
mock = MockRedis.new
allow(Redis).to receive(:new).and_return(mock)
end
Then in the scenario:
scenario "my scenario with redis", redis: true do
...
end
My New Relic Insights is logging pageviews and User Agent info from local machine in dev environment. I have another dev in another city who is also having the development enviroment pageviews and other info being logged.
When I pull up samples, I see localhost:3000, which is my port.
However, the production info is also being logged.
I have New Relic running using Heroku's default set up. It automatically sets the license key as an environment variable. I do not have the license key anywhere in the app, it only set through an environment variable.
If I pull up my local development environment, navigate to port 3000, and refresh, then query New Relic Insights for events in the last minute, I see my city, my user agent info, my visited url and pageview. Our product is in beta, there is really no chance that an actual user in my location is hitting the same random page.
I have tried turning development mode off, monitor off. I cannot understand how this can be happening.
I do have some files hosted on AWS (images and some js), if that matters
Gemfile
group :production do
gem 'rails_12factor'
gem 'newrelic_rpm'
end
config/newrelic.yml
common: &default_settings
license_key: <%= ENV["NEW_RELIC_LICENSE_KEY"] %>
log_level: info
development:
<<: *default_settings
app_name: app-dev
developer_mode: false
monitor_mode: false
agent_enabled: false
test:
<<: *default_settings
monitor_mode: false
developer_mode: false
agent_enabled: false
production:
app_name: app-prod
monitor_mode: true
agent_enabled: false
<<: *default_settings
config/puma.rb
require 'puma_worker_killer'
ActiveRecord::Base.connection_pool.disconnect!
PumaWorkerKiller.config do |config|
config.ram = ENV['PUMA_WORKER_KILLER_RAM'] || 1024 # mb
config.frequency = 5 # seconds
config.percent_usage = 0.98
config.rolling_restart_frequency = 12 * 3600 # 12 hours in seconds
end
PumaWorkerKiller.start
end
workers Integer(ENV['WEB_CONCURRENCY'] || 5)
min_threads_count = Integer(ENV['MIN_THREADS'] || 1)
threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)
threads min_threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
# #sidekiq_pid ||= spawn('bundle exec sidekiq -c 2 -q default -q mailers')
ActiveSupport.on_load(:active_record) do
ActiveRecord::Base.establish_connection
end
end
config/initializers/sidekiq.rb
require 'sidekiq'
redis_url = ENV['REDISTOGO_URL']
redis_config = {
url: redis_url,
namespace: 'oct',
}
Sidekiq.configure_server do |config|
config.redis = {
url: ENV["REDISTOGO_URL"], namespace: 'app',
size: ENV["SIDEKIQ_SERVER_CONNECTIONS"].to_i || 6
}
config.error_handlers << Proc.new do |exception, context_hash|
SidekiqErrorService.new(exception, context_hash).notify
end
end
Sidekiq.configure_client do |config|
config.redis = {
url: ENV["REDISTOGO_URL"], namespace: 'app',
size: ENV["REDIS_CLIENT_CONNECTION_SIZE"].to_i || 2
}
end
So I believe it was the New Relic Browser JS that I included in the head of my pages. Once I set that to - if production_environment? (my helper method), then I only saw production environment traffic.
I believe that something in that JS was pinging my New Relic.
Fixed now.