I am having issues converting from Resque to Sidekiq. I'm not getting any errors though a perform_async(ids) doesn't add anything to Redis. I can add keys directly to the Redis server via the Redis.current.append("test", "key")
Also my Sidekiq worker connects to the Redis server, though I get an empty array when I ask for Sidekiq::Client.registered_workers The web UI shows only the skeleton with no information other than the Redis info. I don't know if this matters but Sidekiq.redis { |conn| conn.info } returns information that is all correct with my local Redis server. Though Sidekiq.server? returns a nil value.
Update: When I perform a perform_async(args) it returns a string.
It sounds like your Sidekiq configuration isn't using the same redis config for the client and server components of Sidekiq.
The client executes within your Rails app server, while the server is a stand-alone separate process where work is performed. If they both aren't using the same redis queue via the redis configuration, work will not be processed.
Here is an example config/initializers/sidekiq.rb config block using the same redis for both components:
redis = { url: (ENV['REDIS_URL'] || 'redis://127.0.0.1'), namespace: 'sidekiq' }
Sidekiq.configure_server do |config|
config.redis = redis
end
Sidekiq.configure_client do |config|
config.redis = redis
end
My guess is that you have a different namespace or configuration between your client and server config.
I was hitting the same issue.
Turns out I had rspec-sidekiq configured in my Gemfile's development group and rspec-sidekiq apparently stubs the async calls for testing.
The solution was amending my Gemfile as such:
gem 'rspec-sidekiq', group: :test, require: false
Finally, I added require 'rspec-sidekiq' in my spec_helper.rb.
Related
I am using Sidekiq on an app hosted on Heroku.
I have been using the redis gem.
Now I am trying to upgrade to sidekiq 7 which uses redis-client instead of redis. And when doing so I get this error:
redis-client requires Redis 6+ with HELLO command available
The Redis instance I have on Heroku is on the version 6.2.6 (so Redis 6+) (picture attached)
Below my redis.rb file (which is in initializers). I replaced Redis with RedisClient but I still get the error in both cases (in the case where I use the redis gem with Sidekiq 7 and the case where I use redis-client).
It means that it is when the Sidekiq inner code is using redis-client that the error is happening.
Any thoughts or ideas would be highly appreciated 🙏🏼 Thanks in advance!
I was expecting this to work as my Redis instance is on a 6+ version
url = ENV["REDISCLOUD_URL"]
if url
Sidekiq.configure_server do |config|
config.redis = { url: url }
end
Sidekiq.configure_client do |config|
config.redis = { url: url }
end
end
Searching the Sidekiq issues for "HELLO" gives you this issue:
https://github.com/sidekiq/sidekiq/issues/5594#issuecomment-1302384905
I am running sidekiq on a production rails server, and some of its queues workers end up in my development sidekiq instance and, therefore, causes a bunch of errors. The same thing happens with my development sidekiq instance. When I'm in the development version of the application and I click something that kicks off a sidekiq worker, it sometimes tries to run it in the production sidekiq instance and the production database doesn't have the same information.
Is there a proper way to separate these queued jobs so that they only run in the instance which they are initiated?
Sidekiq uses Redis to store enqueued jobs. By default, all sidekiq instances will use the default Redis database. You can change this by creating an initializer in config/initializers/sidekiq.rb
if Rails.env == :production
redis_host = '127.0.0.1'
redis_port = 6379
redis_db = 1
else
redis_host = '127.0.0.1'
redis_port = 6379
redis_db = 2
end
Sidekiq.configure_server do |config|
config.redis = { url: "redis://#{redis_host}:#{redis_port}/#{redis_db}" }
end
Sidekiq.configure_client do |config|
config.redis = { url: "redis://#{redis_host}:#{redis_port}/#{redis_db}" }
end
And change redis_host, redis_port and redis_db with appropriate values. This way each environment will use a separate Redis database. Also don't forget to pass appropriate environment when starting a sidekiq instance with something like:
bundle exec sidekiq -e production
I'm trying to implement a background worker for payments using Sidekiq. I'm using nitrous.io for development and this is probably causing some problems when connecting to redis.
When I try to connect I get the following error:
Redis::CannotConnectError (Error connecting to Redis on 127.0.0.1:6379 (ECONNREFUSED)):
I assumed that this is the default (localhost).
I've tried to add the following initializer, sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = { :url => 'redis://test-box-123456.euw1-2.nitrousbox.com:6379/0'}
end
Sidekiq.configure_client do |config|
config.redis = { :url => 'redis://test-box-123456.euw1-2.nitrousbox.com:6379/0'}
end
This however, results in the following error:
Redis::ProtocolError ( Got '<' as initial reply byte. If you're in a forking environment, such as Unicorn, you need to connect to Redis after forking. ):
I haven't worked with background sidekiq/redis before, and I cannot figure out what exactly is going wrong here. How should I configure this connection in both development (nitrous) and production (heroku)?
I was getting the same error locally and figured out that this was because I did not start redis server- type in a new cmd window:
redis-server
Hope that helps at least when testing locally.
I have a RoR app with background jobs using whenever and sidekiq gems.
In development environment when I launch sidekiq with local redis instance (on localhost) the job keeps getting executed without problems. But when I switch to a remote redis instance (Heroku add-on) and restart sidekiq, it says it started processing, but nothing happens and workers aren't doing any jobs.
Here's my config/schedule.rb (for whenever gem)
every 2.minutes do
rake "crawler:crawl"
end
Here's my initializers/redis.rb:
Sidekiq.configure_server do |config|
config.redis = { :url => 'redis://user:pass#spinyfin.redistogo.com:9098/' }
end
Sidekiq.configure_client do |config|
config.redis = { :url => 'redis://user:pass#spinyfin.redistogo.com:9098/' }
end
If I comment out the content in redis.rb and run a local redis instance, the jobs are processed normally. But when I use this remote redis instance, this shows up and then nothing gets processed:
2013-11-29T15:09:26Z 95156 TID-ov6y7e14o INFO: Booting Sidekiq 2.13.0 using redis://redistogo:user#spinyfin.redistogo.com:9098/ with options {}
2013-11-29T15:09:26Z 95156 INFO: Running in ruby 1.9.3p327 (2012-11-10 revision 37606) [x86_64-darwin11.4.2]
2013-11-29T15:09:26Z 95156 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2013-11-29T15:09:26Z 95156 INFO: Starting processing, hit Ctrl-C to stop
Maybe you connecting to wrong redis database or not connected at all.
In my apps I use redis url without trailing slash. In your case:
This is for database "0"
redis://user:pass#spinyfin.redistogo.com:9098
And this for database "1"
redis://user:pass#spinyfin.redistogo.com:9098/1
I use the environment variable REDIS_URL to ensure that everything is using the same Redis.
Re: Heroku - I just read this here as I was searching for my own solution:
If you're running on Heroku, you can't rely on the config/database.yml as that platform relies on the DATABASE_URL environment variable to determine the database connection configuration. Heroku overwrites the database.yml during slug compilation so that it reads from DATABASE_URL.
I need to setup a connection to an external service in my Rails app. I do this in an initializer. The problem is that the service library uses threaded delivery (which I need, because I can't have it bogging down requests), but the Unicorn life cycle causes the thread to be killed and the workers never see it. One solution is to invoke a new connection on every request, but that is unnecessarily wasteful.
The optimal solution is to setup the up the connection in an after_fork block in the unicorn config. The problem there is that doesn't get invoked outside of unicorn, which means we can't test it in development/testing environments.
So the question is, what is the best way to determine whether a Rails app is running under Unicorn (either master or worker process)?
There is an environment variable that is accessible in Rails (I know it exists in 3.0 and 3.1), check the value of env['SERVER_SOFTWARE']. You could just put a regex or string compare against that value to determine what server you are running under.
I have a template in my admin that goes through the env variable and spits out its content.
Unicorn 4.0.1
env['SERVER_SOFTWARE'] => "Unicorn 4.0.1"
rails server (webrick)
env['SERVER_SOFTWARE'] => "WEBrick/1.3.1 (Ruby/1.9.3/2011-10-30)"
You can check for defined?(Unicorn) and in your Gemfile set: gem :unicorn, require: false
In fact you don't need Unicorn library loaded in you rails application.
Server is started by unicorn command from shell
Checking for Unicorn constant seems a good solution, BUT it depends very much on whether require: false is provided in the Gemfile. If it isn't (which is quite probable), the check might give a false positive.
I've solved it in a very straightforward manner:
# `config/unicorn.rb` (or alike):
ENV["UNICORN"] = 1
...
# `config/environments/development.rb` (or alike):
...
# Log to stdout if Web server is Unicorn.
if ENV["UNICORN"].to_i > 0
config.logger = Logger.new(STDOUT)
end
Cheers!
You could check to see if the Unicorn module has been defined with Object.constants.include?('Unicorn').
This is very specific to Unicorn, of course. A more general approach would be to have a method which sets up your connection and remembers it's already done so. If it gets called multiple times, it just returns doing nothing on subsequent calls. Then you call the method in after_fork and in a before_filter in your application controller. If it's been run in the after_fork it does nothing in the before_filter, if it hasn't been run yet it does its thing on the first request and nothing on subsequent requests.
Inside config/unicorn.rb
Define ENV variable as
ENV['RAILS_STDOUT_LOG']='1'
worker_processes 3
timeout 90
and then this variable ENV['RAILS_STDOUT_LOG'] will be accessible anywhere in your Rails app worker thread.
my issue:
I wanted to output all the logs(SQL queries) when on the Unicorn workers and not on any other workers on Heroku, so what I did is adding env variable in the unicorn configuration file
If you use unicorn_rails, below code will help
defined?(::Unicorn::Launcher)