I'm working on a site someone else made. There is a production version.
I'm trying to send emails to users whose properties haven't been updated in 30 days.
Everything works until I try to use deliver_later. The reason I'm using deliver_later is because deliver_now results in an issue of sending too many emails per second. I'm currently using Mailtrap for testing, but I assume I will run into that sort of issue on production.
So I opted to wait 1 second for each email:
#testproperties = Property.has_not_updated.includes(:user)
#testproperties.each do |property|
UserMailer.with(property: property, user: property.user).check_listing.deliver_later(wait:1.seconds)
end
This results in IO::EINPROGRESSWaitWritable Operation now in progress - connect(2) would block
And nothing sends.
I'm not sure how to solve this issue.
Edit:
I can see on the production site that I can visit the route /sidekiq. The routes file has this block:
authenticate :user, lambda { |u| u.admin? } do
mount Sidekiq::Web => '/sidekiq'
end
I can view the web interface and see all the jobs. It's all working there. But I need to access development version running on localhost:3000.
Trying to access this locally still results in:
Operation now in progress - connect(2) would block
# # Socket#connect
def connect_nonblock(addr, exception: true)
__connect_nonblock(addr, exception)
end
end
Sidekiq.rb:
require 'sidekiq'
unless Rails.env.test?
host = 'localhost'
port = '6379'
namespace = 'sitename'
Sidekiq.configure_server do |config|
config.redis = { url: "redis://#{host}:#{port}", namespace: namespace }
schedule_file = "config/schedule.yml"
if File.exists?(schedule_file)
Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file)
end
config.server_middleware do |chain|
chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
end
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
end
end
Sidekiq.configure_client do |config|
config.redis = { url: "redis://#{host}:#{port}", namespace: namespace }
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware, expiration: 30.minutes
end
end
end
for cable.yml:
development:
adapter: async
url: redis://localhost:6379/1
channel_prefix: sitename_dev
test:
adapter: async
production:
adapter: redis
url: redis://localhost:6379/1
channel_prefix: sitename_production
The production server is running Ubuntu and they already installed redis-server.
I had not installed that locally. (I'm using Ubuntu through Windows WSL)
sudo apt install redis-server
I can now access the web interface.
Make sure that Redis is started:
sudo service redis-server restart
Related
I want to open sidekiq(bundle exec sidekiq) on my localhost, however I am getting ERR Client sent AUTH, but no password is set error even if I set a password on my redis.conf file.
I already set a password on my redis.conf file.
The password on redis.conf file is already on my secret.yml file like:
sidekiq_redis_url: redis://localhost:6379
sidekiq_redis_pwd: redispwd
Here is my sidekiq.rb
require 'sidekiq/web'
Sidekiq.configure_server do |config|
config.redis = { url: "#{Rails.application.secrets[:sidekiq_redis_url]}", password: "#{Rails.application.secrets[:sidekiq_redis_pwd]}" }
end
Sidekiq.configure_client do |config|
config.redis = { url: "#{Rails.application.secrets[:sidekiq_redis_url]}", password: "#{Rails.application.secrets[:sidekiq_redis_pwd]}" }
end
Sidekiq::Web.set :sessions, false
Redis server version is 5.0.5 and sidekiq version is 5.2.7.
Thanks for your help.
I'm trying to understand the Redis & Sidekiq configuration in a Passenger+Rails app and have run into some lack of understanding. I start the redis server independently of my rails app, while Sidekiq is a gem in my Rails app. I start it likewise: (no sidekiq.yml file needed for me)
bundle exec sidekiq
Following is my sidekiq.rb initializer:
require 'sidekiq'
require 'sidekiq-status'
Sidekiq.configure_client do |config|
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end
end
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add Sidekiq::Status::ServerMiddleware, expiration: 30.minutes
end
config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end
end
I went through some library classes, but to no avail.
I want to understand where does Sidekiq configure it's Redis server details. It defaults to localhost:6379, but I am not quite sure how.
Also, if I wish to use Memcached in future, how can I change that?
From sidekiq docs:
By default, Sidekiq tries to connect to Redis at localhost:6379
https://github.com/mperham/sidekiq/wiki/Using-Redis
You can change the port in the initializer:
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://redis.example.com:7372/12' }
end
From the looks of it sidekiq works only with redis
Sidekiq uses Redis to store all of its job and operational data.
Here is what I did from the other answer:
# frozen_string_literal: true
Sidekiq.configure_server do |config|
config.redis = {
url: ENV.fetch("SIDEKIQ_REDIS_URL", "redis://localhost:6379/1")
}
end
Running an RSpec test with a Sidekiq::Queue instance is failing unless Redis is running separately.
Sidekiq::Queue.new('my-queue').select(&:item)
Raises error in test
Redis::CannotConnectError:
Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)
I've added the usual to the spec helper:
require 'sidekiq/testing'
Sidekiq::Testing.inline!
And mock_redis to the gemfile.
# gemfile
gem 'mock_redis', '0.16.1'
Using sidekiq (3.4.2)
How can I update my configuration to allow this to work?
mock_redis only provides with a fake redis. It does not intercept/replace actual redis classes/connections. If you intend to use fake redis in tests, you should tell sidekiq so. In your config/initializers/sidekiq.rb (or whereever your sidekiq redis config is):
redis = if Rails.env.test?
require 'mock_redis'
MockRedis.new
else
{ url: 'redis://redis.example.com:7372/12' }
end
Sidekiq.configure_server do |config|
config.redis = redis
end
Sidekiq.configure_client do |config|
config.redis = redis
end
I solved this by mocking Redis for tagged RSpec tests in the spec_helper.rb file.
config.before(:each, redis: true) do
mock = MockRedis.new
allow(Redis).to receive(:new).and_return(mock)
end
Then in the scenario:
scenario "my scenario with redis", redis: true do
...
end
My New Relic Insights is logging pageviews and User Agent info from local machine in dev environment. I have another dev in another city who is also having the development enviroment pageviews and other info being logged.
When I pull up samples, I see localhost:3000, which is my port.
However, the production info is also being logged.
I have New Relic running using Heroku's default set up. It automatically sets the license key as an environment variable. I do not have the license key anywhere in the app, it only set through an environment variable.
If I pull up my local development environment, navigate to port 3000, and refresh, then query New Relic Insights for events in the last minute, I see my city, my user agent info, my visited url and pageview. Our product is in beta, there is really no chance that an actual user in my location is hitting the same random page.
I have tried turning development mode off, monitor off. I cannot understand how this can be happening.
I do have some files hosted on AWS (images and some js), if that matters
Gemfile
group :production do
gem 'rails_12factor'
gem 'newrelic_rpm'
end
config/newrelic.yml
common: &default_settings
license_key: <%= ENV["NEW_RELIC_LICENSE_KEY"] %>
log_level: info
development:
<<: *default_settings
app_name: app-dev
developer_mode: false
monitor_mode: false
agent_enabled: false
test:
<<: *default_settings
monitor_mode: false
developer_mode: false
agent_enabled: false
production:
app_name: app-prod
monitor_mode: true
agent_enabled: false
<<: *default_settings
config/puma.rb
require 'puma_worker_killer'
ActiveRecord::Base.connection_pool.disconnect!
PumaWorkerKiller.config do |config|
config.ram = ENV['PUMA_WORKER_KILLER_RAM'] || 1024 # mb
config.frequency = 5 # seconds
config.percent_usage = 0.98
config.rolling_restart_frequency = 12 * 3600 # 12 hours in seconds
end
PumaWorkerKiller.start
end
workers Integer(ENV['WEB_CONCURRENCY'] || 5)
min_threads_count = Integer(ENV['MIN_THREADS'] || 1)
threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)
threads min_threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 3000
environment ENV['RACK_ENV'] || 'development'
on_worker_boot do
# Worker specific setup for Rails 4.1+
# See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
# #sidekiq_pid ||= spawn('bundle exec sidekiq -c 2 -q default -q mailers')
ActiveSupport.on_load(:active_record) do
ActiveRecord::Base.establish_connection
end
end
config/initializers/sidekiq.rb
require 'sidekiq'
redis_url = ENV['REDISTOGO_URL']
redis_config = {
url: redis_url,
namespace: 'oct',
}
Sidekiq.configure_server do |config|
config.redis = {
url: ENV["REDISTOGO_URL"], namespace: 'app',
size: ENV["SIDEKIQ_SERVER_CONNECTIONS"].to_i || 6
}
config.error_handlers << Proc.new do |exception, context_hash|
SidekiqErrorService.new(exception, context_hash).notify
end
end
Sidekiq.configure_client do |config|
config.redis = {
url: ENV["REDISTOGO_URL"], namespace: 'app',
size: ENV["REDIS_CLIENT_CONNECTION_SIZE"].to_i || 2
}
end
So I believe it was the New Relic Browser JS that I included in the head of my pages. Once I set that to - if production_environment? (my helper method), then I only saw production environment traffic.
I believe that something in that JS was pinging my New Relic.
Fixed now.
Like Database cleaner, or the default clearing of the data store after a test run. I searched and couldn't find one. It could be either a separate test data store or just something simple that namespaces all Redis commands into a test namespace.
If anyone knows of any lemme know, otherwise I'll write one and OS it :)
When working with rails and redis I use a different redis db or namespace for the different environments. The setup is very simple and similar to ActiveRecords database config.
First, create a config: (namespace version commented out)
#config/redis.yml
default:
host: localhost
port: 6379
development:
db: 0
# namespace: appname_dev
test:
db: 1
# namespace: appname_test
production:
db: 2
host: 192.168.1.100
# namespace: appname_prod
Then load the config and connect to redis through an initializer:
#config/initializers/redis.rb
REDIS_CONFIG = YAML.load( File.open( Rails.root.join("config/redis.yml") ) ).symbolize_keys
dflt = REDIS_CONFIG[:default].symbolize_keys
cnfg = dflt.merge(REDIS_CONFIG[Rails.env.to_sym].symbolize_keys) if REDIS_CONFIG[Rails.env.to_sym]
$redis = Redis.new(cnfg)
#$redis_ns = Redis::Namespace.new(cnfg[:namespace], :redis => $redis) if cnfg[:namespace]
# To clear out the db before each test
$redis.flushdb if Rails.env == "test"
Remember to add 'redis-namespace' to your Gemfile if your using that version.
You can try fakeredis. It is an fake redis implementation written in pure ruby.
Oh, yes there is.
I use it in all of my projects where I need to test Redis logic.
it's very useful, and it's not on the same database as your local development so the data has no danger of being "mixed"
there it is: I put this code in my spec_helper.rb file, but you can put it in your test_helper.rb if you are using test unit.
# ==========================> Redis test configuration
REDIS_PID = "#{Rails.root}/tmp/pids/redis-test.pid"
REDIS_CACHE_PATH = "#{Rails.root}/tmp/cache/"
Dir.mkdir "#{Rails.root}/tmp/pids" unless Dir.exists? "#{Rails.root}/tmp/pids"
Dir.mkdir "#{Rails.root}/tmp/cache" unless Dir.exists? "#{Rails.root}/tmp/cache"
config.before(:suite) do
redis_options = {
"daemonize" => 'yes',
"pidfile" => REDIS_PID,
"port" => 9736,
"timeout" => 300,
"save 900" => 1,
"save 300" => 1,
"save 60" => 10000,
"dbfilename" => "dump.rdb",
"dir" => REDIS_CACHE_PATH,
"loglevel" => "debug",
"logfile" => "stdout",
"databases" => 16
}.map { |k, v| "#{k} #{v}" }.join('\n')
`echo '#{redis_options}' | redis-server -`
end
config.after(:suite) do
%x{
cat #{REDIS_PID} | xargs kill -QUIT
rm -f #{REDIS_CACHE_PATH}dump.rdb
}
end