I'm trying to run Sidekiq 5.2.2 on Rails 5.2 but I always get a connection lost error. Below are my server logs:
# bundle exec sidekiq -q v2_default,2 -q v2_low -e production
TID-2yed0 INFO: Booting Sidekiq 5.2.2 with redis options {:url=>"redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1", :network_timeout=>7, :id=>"Sidekiq-server-PID-3644"}
TID-2yed0 INFO: Running in ruby 2.3.5p376 (2017-09-14 revision 59905) [x86_64-linux]
TID-2yed0 INFO: See LICENSE and the LGPL-3.0 for licensing details.
TID-2yed0 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
TID-2yed0 INFO: Starting processing, hit Ctrl-C to stop
TID-19spa0 ERROR: heartbeat: Connection lost (ECONNRESET)
TID-xhh1g ERROR: Error fetching job: READONLY You can't write against a read only slave.
TID-xhh1g WARN: Redis::CommandError: READONLY You can't write against a read only slave.
TID-xhh1g WARN: /usr/local/rvm/gems/ruby-2.3.5/gems/redis-4.0.2/lib/redis/client.rb:119:in `call'
and this is my config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = { url: 'redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1', network_timeout: 7 }
end
Sidekiq.configure_client do |config|
config.redis = { url: 'redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1', network_timeout: 7 }
end
I verify that I can do ping PONG with the server but I don't know why sidekiq can't connect.
# rails c production
Running via Spring preloader in process 7372
Loading production environment (Rails 5.2.1)
2.3.5 :001 > require "redis"
=> false
2.3.5 :002 > redis = Redis.new(url: 'redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1')
=> #<Redis client v4.0.2 for redis://myserver.lmf979.0001.apse1.cache.amazonaws.com:6379/1>
2.3.5 :003 > redis.ping
=> "PONG"
2.3.5 :004 >
Any ideas?
I was just pointing to a wrong redis endpoint in AWS. Thanks to Pavel Mikhailyuk for pointing out.
Related
Rails 5.2
redis 4.0.1
sidekiq 5.1.3
RedisToGo on Heroku
Proc File:
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -e production -C config/sidekiq.yml
sidekiq.yml
development:
:concurrency: 5
production:
:concurrency: 6
:queues:
- default
config/initialize/sidekiq.rb
if Rails.env.production?
Sidekiq.configure_client do |config|
config.redis = { url: ENV['REDISTOGO_URL'] || ENV['REDIS_URL'], size: 1 }
end
Sidekiq.configure_server do |config|
pool_size = ENV['SIDEKIQ_DB_POOL'] || (Sidekiq.options[:concurrency] + 2)
config.redis = { url: ENV['REDISTOGO_URL'] || ENV['REDIS_URL'], size: pool_size }
Rails.application.config.after_initialize do
Rails.logger.info("DB Connection Pool size for Sidekiq Server before disconnect is: #{ActiveRecord::Base.connection.pool.instance_variable_get('#size')}")
ActiveRecord::Base.connection_pool.disconnect!
ActiveSupport.on_load(:active_record) do
db_config = Rails.application.config.database_configuration[Rails.env]
db_config['reaping_frequency'] = ENV['DATABASE_REAP_FREQ'] || 10 # seconds
db_config['pool'] = pool_size
ActiveRecord::Base.establish_connection(db_config)
Rails.logger.info("DB Connection Pool size for Sidekiq Server is now: #{ActiveRecord::Base.connection.pool.instance_variable_get('#size')}")
end
end
end
end
I have no idea why the job is stuck in the queue. The job works on my local in development.
Any assistance appreciated.
Update!
As per the comment here is the output of heroku ps:
=== web (Free): bundle exec puma -C config/puma.rb (1)
web.1: up 2018/06/15 18:49:56 +0700 (~ 5s ago)
=== worker (Free): bundle exec sidekiq -q default (1)
worker.1: crashed 2018/06/15 18:49:51 +0700 (~ 10s ago)
Seems a worker crashed for some reason.
Heroku logs:
2018-06-15T11:56:00.301716+00:00 app[web.1]: [4] Puma starting in cluster mode...
2018-06-15T11:56:00.301740+00:00 app[web.1]: [4] * Version 3.11.2 (ruby 2.4.1-p111), codename: Love Song
2018-06-15T11:56:00.301760+00:00 app[web.1]: [4] * Min threads: 5, max threads: 5
2018-06-15T11:56:00.301794+00:00 app[web.1]: [4] * Environment: production
2018-06-15T11:56:00.301801+00:00 app[web.1]: [4] * Process workers: 2
2018-06-15T11:56:00.301838+00:00 app[web.1]: [4] * Preloading application
2018-06-15T11:56:01.249511+00:00 app[worker.1]: DB Connection Pool size for Sidekiq Server before disconnect is: 5
2018-06-15T11:56:01.252654+00:00 app[worker.1]: could not connect to server: No such file or directory
2018-06-15T11:56:01.252657+00:00 app[worker.1]: Is the server running locally and accepting
2018-06-15T11:56:01.252659+00:00 app[worker.1]: connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
After talking with tech support at Heroku it turns out that I needed to change the way I had my db yaml file coded.
I changed it to the following:
production:
url: <%= ENV['DATABASE_URL'] %>
That seem to solve the issue.
Install Heroku Postgres as an addon and then add the environment variables to your project and everything should work fine. - https://elements.heroku.com/addons/heroku-postgresql
Im having trouble launching Sidekiq on my AWS ElasticBeanstalk environment.
I am running Sidekiq 4.1.0 with my Rails application.
I logged directly into the EC2 instance to start Sidekiq manually and when I run this:
bundle exec sidekiq -e staging -q default -q mailers
I get this error from the Insance:
2016-02-10T07:19:34.172Z 1655 TID-gn7dktok8 INFO: Booting Sidekiq 4.1.0 with redis options {:url=>"redis://xxxx.amazonaws.com:6379"}
'xxxx.xxxx.rds.amazonaws.com' database is not configured. Available: ["default", "development", "test", "production", "staging"]
I checked the RDS database URL its trying to connect to and its the right one for the environment I am trying to run this on.
My sidekiq config file looks like this:
redis_url = ENV['REDIS_URL']
Sidekiq.configure_server do |config|
config.redis = { url: redis_url, namespace: "swimlanes_#{Rails.env}" }
end
Sidekiq.configure_client do |config|
config.redis = { url: redis_url, namespace: "swimlanes_#{Rails.env}" }
end
I have tried everything. Do you know what I am missing?
Sounds like I helped you find the issue on GitHub: setting the Rails env incorrectly somewhere?
I need to run multiple instances of Sidekiq on a machine that is hosting multiple different Rails apps. In order to avoid conflicts, I had one of my apps use a different redis instance, pointed at 127.0.0.1/1 instead of the default 127.0.0.1/0, by using the following code in my config/initializers/sidekiq.rb
Sidekiq.configure_client do |config|
config.redis = { :url => 'redis://127.0.0.1:6379/1' }
end
When I boot my instance of Sidekiq using bundle exec sidekiq I get the following boot message:
2015-02-04T16:46:59.494Z 15104 TID-ovz94l9m0 INFO: Running in ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-darwin13.0]
2015-02-04T16:46:59.494Z 15104 TID-ovz94l9m0 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2015-02-04T16:46:59.494Z 15104 TID-ovz94l9m0 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro
2015-02-04T16:46:59.494Z 15104 TID-ovz94l9m0 INFO: Starting processing, hit Ctrl-C to stop
2015-02-04T16:46:59.541Z 15104 TID-ovz9c4dhk INFO: Booting Sidekiq 3.3.0 with redis options {}
The problem is that I want
2015-02-04T16:46:59.541Z 15104 TID-ovz9c4dhk INFO: Booting Sidekiq 3.3.0 with redis options {:url => 'redis://127.0.0.1:6379/1'}
instead. What command line options to boot Sidekiq or what config files should I edit to have Sidekiq boot with the redis options reflected in config/initializers/sidekiq.rb?
You must set the Redis config in both the configure_client and configure_server blocks. You are starting the server. More info here:
https://github.com/mperham/sidekiq/wiki/Using-Redis
A Rails 3.2.6 application running as a 'web' Heroku process is connecting to Redis using the ENV["REDISTOGO_URL"] environment variable:
irb(main):002:0> Redis.current
=> #<Redis client v2.2.2 connected to redis://xxx.redistogo.com:1234/0 (Redis v2.4.11)>
----- /initializers/redis.rb
if Rails.env.development?
Redis.current = Redis.new
elsif Rails.env.test?
Redis.current = Redis.new
elsif Rails.env.production?
uri = URI.parse(ENV["REDISTOGO_URL"])
Redis.current = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password)
end
A ruby daemon 'streaming' process runs as a secondary worker:
----- Procfile
web: bundle exec rails server thin -p $PORT -e $RACK_ENV
worker: bundle exec rake jobs:work
streaming: RAILS_ENV=production ruby bin/streaming.rb start
However, the streaming process is crashing when it calls methods in the main Rails application that connect with Redis - even though the streaming process supposed to be loading the same redis.rb initializer as the 'web' Rails application process.
----- /bin/streaming_ctl.rb
# encoding: UTF-8
require "rubygems"
require "bundler/setup"
require "daemons"
Daemons.run(File.expand_path("../streaming.rb", __FILE__))
----- /bin/streaming.rb
# encoding: UTF-8
# TODO: set rails env in init script
ENV["RAILS_ENV"] ||= "production"
# load rails environment
require File.expand_path('../../config/environment', __FILE__)
logger = ActiveSupport::BufferedLogger.new(File.expand_path("./../../log/streaming.log", __FILE__))
Streaming.start
logger.info("\nStarting streaming in #{Rails.env.to_s} mode.")
Why is the streaming process/worker using the default Redis host and port?
[streaming.1]: /app/vendor/bundle/ruby/1.9.1/gems/redis-2.2.2/lib/redis/client.rb:236:in `rescue in establish_connection': Connection refused - Unable to connect to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED)
Switching from "Redis.current" in /initializer/redis.rb and using a "$redis" variable convention instead (as illustrated here http://jimneath.org/2011/03/24/using-redis-with-ruby-on-rails.html) fixed the connection problem.
The app was using redis gem version 2.2.2 at the time. It appears as though the Redis.current in this gem version won't be consistent across 'web' and 'worker' processes because Heroku runs these processes on separate threads. The documentation on the gem repo suggests that updating to gem to >= version 3 will enable Redis.current to run in multi-threaded environments:
https://github.com/redis/redis-rb/blob/master/CHANGELOG.md#300
I am trying to execute the rake command on my brand new rails project with postgresql but I am getting this error:
rake aborted! could not connect to server: Connection refused Is the
server running on host "localhost" (127.0.0.1) and accepting TCP/IP
connections on port 5432?
Tasks: TOP => db:migrate (See full trace by running task with --trace)
real 0m3.245s user 0m1.668s sys 0m0.382s
This is my first rails app ever, whe I run rails server this is the output:
=> Booting WEBrick
=> Rails 3.2.3 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server [2012-06-19 01:17:30] INFO WEBrick 1.3.1 [2012-06-19 01:17:30] INFO ruby 1.9.3 (2012-04-20)
[x86_64-darwin11.3.0] [2012-06-19 01:17:30] INFO
WEBrick::HTTPServer#start: pid=4668 port=3000
Rails -v: Rails 3.2.3
gem env:
RubyGems Environment:
- RUBYGEMS VERSION: 1.8.24
- RUBY VERSION: 1.9.3 (2012-04-20 patchlevel 194) [x86_64-darwin11.3.0]
- INSTALLATION DIRECTORY: /Users/kinhow/.rvm/gems/ruby-1.9.3-p194
- RUBY EXECUTABLE: /Users/kinhow/.rvm/rubies/ruby-1.9.3-p194/bin/ruby
- EXECUTABLE DIRECTORY: /Users/kinhow/.rvm/gems/ruby-1.9.3-p194/bin
- RUBYGEMS PLATFORMS:
- ruby
- x86_64-darwin-11
- GEM PATHS:
- /Users/kinhow/.rvm/gems/ruby-1.9.3-p194
- /Users/kinhow/.rvm/gems/ruby-1.9.3-p194#global
- GEM CONFIGURATION:
- :update_sources => true
- :verbose => true
- :benchmark => false
- :backtrace => false
- :bulk_threshold => 1000
- REMOTE SOURCES:
- http://rubygems.org/
if it helps:
running psql
psql: could not connect to server: No such file or directory Is the
server running locally and accepting connections on Unix domain
socket "/tmp/.s.PGSQL.5432"?
First, you need to ensure that postgresql is running. Open a console and do a "ps auxw | grep post" and make sure that the server is running. You should have at least four processes running including "postgres: writer process". If you don't then you haven't started it. Even if you have started it, you must have it configured to listen on an appropriate port (127.0.0.1 and "localhost"). Did you edit the postgres.conf file and the pg_hba.conf files? If you did, did you restart the server? Are you using the latest pg gem? How do you start your server?
~Charles~