I'm trying to get Sidekiq playing in Heroku. Without Luck.. My configs look like that:
Procfile
web: bundle exec passenger start -p $PORT --max-pool-size 5
worker: bundle exec sidekiq
Initializers/redis.rb
uri = URI.parse(ENV["REDISTOGO_URL"] || "redis://localhost:6379/")
REDIS = Redis.new(:url => ENV['REDISTOGO_URL'])
but when i do heroku ps only the web instance is shown. Not Sidekiq.
However i can manually run heroku run sidekiq and run my workers. What am i missing so that Heroku doesn't start that on it's own ?
The problem is that you didn't configure Sidekiq in the initializer so that Sidekiq knows exactly how to connect to redis. Create Initializers/sidekiq.rb and add the following code
Sidekiq.configure_server do |config|
config.redis = { :url => ENV["REDISTOGO_URL"] }
end
Also you can remove
uri = URI.parse(ENV["REDISTOGO_URL"] || "redis://localhost:6379/")
You don't need it as you are using the redis url from the environment.
Don't forget to restart your server.
Related
I am having a really strange problem on Heroku that I have been spinning my wheels to figure out for a while now.
My app has a few external API calls and mailers which I have set up ActiveJob to run in the background. On Heroku I have two workers set up as and I am using a Resque/Redis combo for the jobs as per the below snippets. I am using the Redis Cloud add-on on Heroku.
Config / setup
Procfile
web: bundle exec puma -C config/puma.rb
resque: env TERM_CHILD=1 QUEUE=* bundle exec rake resque:work
lib/tasks/resque.rake
require "resque/tasks"
require "resque/scheduler/tasks"
task "resque:setup": :environment do
Resque.before_fork = proc { ActiveRecord::Base.connection.disconnect! }
Resque.after_fork = proc { ActiveRecord::Base.establish_connection }
end
config/initializers/active_job.rb
Rails.application.config.active_job.queue_adapter = :resque
config/initializers/redis.rb
if ENV["REDISCLOUD_URL"]
$redis = Redis.new(url: ENV["REDISCLOUD_URL"])
end
config/initializers/resque.rb
if Rails.env.production?
uri = URI.parse ENV["REDISCLOUD_URL"]
Resque.redis = Redis.new(host: uri.host, port: uri.port,
password: uri.password)
else
Resque.redis = "localhost:6379"
end
The problem
The problem I am having is when a user is using the app in browser (i.e., interfacing with the web worker) and performs an action which triggers one of the ActiveJob jobs the job is run "inline" using the web worker and not the resque worker. When I run the specific model method that queues the job in my Heroku app console (opened by running heroku run rails console) it adds the job to Redis and runs it using the resque worker as expected.
Why would one way work properly and the other way not work? I have looked at almost every tutorial / SO question on the topic and have tried everything so any help getting the jobs to be run but the right worker would be amazing!
Thanks in advance!
I managed to solve the problem by playing with my config a little. It seems that actions were being tunnelled through ActiveJob's "Inline" default rather than via Resque. To get things working I just had to direct Resque.redis to be equal to the $redis variable set in config/initializers/redis.rb so everything was pointing to the same Redis instance and then move the config set in config/initializers/active_job.rb to application.rb.
For reference, the new & improved config that all works is:
Config / setup
Procfile
web: bundle exec puma -C config/puma.rb
resque: env TERM_CHILD=1 RESQUE_TERM_TIMEOUT=7 QUEUE=* bundle exec rake resque:work
lib/tasks/resque.rake
require "resque/tasks"
task "resque:setup" => :environment
config/application.rb
module App
class Application < Rails::Application
...
# Set Resque as ActiveJob queue adapter.
config.active_job.queue_adapter = :resque
end
end
config/initializers/redis.rb
if ENV["REDISCLOUD_URL"]
$redis = Redis.new(url: ENV["REDISCLOUD_URL"])
end
config/initializers/resque.rb
Resque.redis = Rails.env.production? ? $redis : "localhost:6379"
thanks a lot for providing the answer. It saved me a lot of time.
You have one typo inside your Procfile.
It should be resque instead of rescue.
resque: env TERM_CHILD=1 RESQUE_TERM_TIMEOUT=7 QUEUE=* bundle exec rake resque:work
Also, I had to type in one more command to get this all to work in production. Hopefully this helps someone.
heroku ps:scale resque=1 --app appname
This command scales the resque process to 1 dyno(free). You can also do this from the dashboard on heroku.
You can read more about it on the heroku docs https://devcenter.heroku.com/articles/scaling
Im having trouble launching Sidekiq on my AWS ElasticBeanstalk environment.
I am running Sidekiq 4.1.0 with my Rails application.
I logged directly into the EC2 instance to start Sidekiq manually and when I run this:
bundle exec sidekiq -e staging -q default -q mailers
I get this error from the Insance:
2016-02-10T07:19:34.172Z 1655 TID-gn7dktok8 INFO: Booting Sidekiq 4.1.0 with redis options {:url=>"redis://xxxx.amazonaws.com:6379"}
'xxxx.xxxx.rds.amazonaws.com' database is not configured. Available: ["default", "development", "test", "production", "staging"]
I checked the RDS database URL its trying to connect to and its the right one for the environment I am trying to run this on.
My sidekiq config file looks like this:
redis_url = ENV['REDIS_URL']
Sidekiq.configure_server do |config|
config.redis = { url: redis_url, namespace: "swimlanes_#{Rails.env}" }
end
Sidekiq.configure_client do |config|
config.redis = { url: redis_url, namespace: "swimlanes_#{Rails.env}" }
end
I have tried everything. Do you know what I am missing?
Sounds like I helped you find the issue on GitHub: setting the Rails env incorrectly somewhere?
There are quite a few questions like this, but none seem to resolve my issue. Here's the grub:
Rails 4, Ruby 2.0.0 app using Resque and Redis.
My redis initializer:
#config/initializers/redis.rb
if ENV["REDISCLOUD_URL"]
$redis = Redis.new(:url => ENV["REDISCLOUD_URL"])
end
Error is this, as reported by heroku logs:
Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (ECONNREFUSED)
My procfile:
web: bundle exec rackup config.ru -p $PORT
resque: env TERM_CHILD=1 QUEUES=* bundle exec rake resque:work
If it helps, my REDISCLOUD_URL is:
redis://rediscloud:somestuff#someotherstuff:17695
It isn't in http://. Is that an issue?
I've tried all the other stack overflow posts on this issues, and nothing works. Any tips? I've been follwing the RedisCloud Heroku page here: https://devcenter.heroku.com/articles/rediscloud
Nevermind, I found the answer. I did $redis = Resque.redis = Redis.new ... in my redis.rb initializer, and then I had to migrate my db. Silly me.
I've set up Sidekiq with my Rails project. It's running on Heroku with Unicorn. I've gone through all the configuration steps including setting the proper REDISTOGO_URL (as this question references), I've added the following to my after_fork in unicorn.rb:
after_fork do |server,worker|
if defined?(ActiveRecord::Base)
ActiveRecord::Base.establish_connection
Rails.logger.info('Connected to ActiveRecord')
end
Sidekiq.configure_client do |config|
config.redis = { :size => 1 }
end
end
My Procfile is as follows:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq
Right now I call my worker to perform_async and it adds the task to the queue. In fact in my Sidekiq web interface it says there are 7 items in the queue and it has all of the data there. Yet there are no workers processing the queue and for the life of me, I can't figure out why. If I run
heroku ps
I get the following output:
=== web: `bundle exec unicorn -p $PORT -c ./config/unicorn.rb`
web.1: up 2012/12/09 08:04:24 (~ 9m ago)
=== worker: `bundle exec sidekiq`
worker.1: up 2012/12/09 08:04:08 (~ 10m ago)
Anybody have any idea what's going on here?
Update
Here's the code for my worker class. Yes, I'm aware that the Oj gem has some issues potentially with sidekiq, but figured I'd give it a shot first. I'm not getting any error messages at this point (the workers don't even run).
require 'addressable/uri'
class DatasiftInteractionsWorker
include Sidekiq::Worker
sidekiq_options queue: "tweets"
def perform( stream_id , interactions )
interactions = Oj.load(interactions)
interactions.each{ |interaction|
if interaction['interaction']['type'] == 'twitter'
url = interaction['links']['normalized_url'] unless interaction['links']['normalized_url'][0].nil?
url = interaction['links']['url'] if interaction['links']['normalized_url'][0].nil?
begin
puts interaction['links'] if url[0].nil?
next if url[0].nil?
host = Addressable::URI.parse(url[0]).host
host = host.gsub(/^www\.(.*)$/,'\1')
date_str = Time.now.strftime('%Y%m%d')
REDIS.pipelined do
# Add domain to Redis domains Set
REDIS.sadd date_str , host
# INCR Redis host
REDIS.incr( host + date_str )
end
rescue
puts "ERROR: Could not store the following links: " + interaction['links'].to_s
end
end
}
end
end
My preference is to create a /config/sidekiq.yml file and then use worker: bundle exec sidekiq -C config/sidekiq.yml in your Procfile.
Here's an example sidekiq.yml file
Figured out that if you're using a custom queue, you need to make Sidekiq aware of that queue in the Procfile, as follows:
worker: bundle exec sidekiq -q tweets, 1 -q default
I'm not quite sure why this is the case since sidekiq is aware of all queues. I'll post this as an issue on the Sidekiq project.
I'm having a problem getting Sidekiq up and running on my Heroku deployed Rails app. I have my app working fine in development (and on Heroku without Sidekiq).
I created a Procfile with:
worker: bundle exec sidekiq
If I run heroku ps, the only process I see is web.1.
Should I see one for
Sidekiq?
I do get an error:
Redis::CannotConnectError (Error connecting to Redis on localhost:6379) in my Heroku logs.
UPDATE: Found I probably needed heroku addons:add redistogo. Still not working. I feel I'm missing some basic configuration.
Is there something I need to do to get Redis up and running for my Heroku app?
I've been using Redis/Sidekiq for about a day, so this is new to me.
Thanks!
Greg
No you do not need any config with Heroku for Sidekiq, just add the RedisToGo plugin and you're on.
Do not forget to attribute at least 1 worker to your app in your Heroku config.
Here is my default Procfile:
web: bundle exec thin start -p $PORT
worker: bundle exec sidekiq -c 5 -v
It's worth checking if the sidekiq process is really started with this command:
heroku ps
If there's no worker, then you might need to run this command:
heroku ps:scale worker+1
It turns out that there's a bug in the web UI in that some team members were not allowed to increase the number of workers from 0 to 1, even though the UI seemed to show that!
Starting with sidekiq version 3.0 there is an additional step, run heroku config:set REDIS_PROVIDER=REDISTOGO_URL in the console.
Here is the process I used for Rails 4:
In the console:
heroku addons:create redistogo
heroku config:set REDIS_PROVIDER=REDISTOGO_URL
In my Procfile I added:
worker: bundle exec sidekiq
In my gemfile.rb I added:
gem 'redis'
I added the following file, config/initializers/redis.rb:
uri = ENV["REDISTOGO_URL"] || "redis://localhost:6379/"
REDIS = Redis.new(:url => uri)
Here is the link to the sidekiq docs.
Complementing gdurelle answer:
You do need some config with Heroku for Sidekiq:
1) Have the Sidekiq and Redis gems installed (in gemfile and bundled), in my case:
Gemfile
gem 'redis', '~> 3.1'
gem 'sidekiq', '~> 2.7', '>= 2.7.1'
2) Add a worker, if you don't have any workers created locally I suggest you create at least one, just in case, use this:
rails g sidekiq:worker Hard # will create app/workers/hard_worker.rb
should create this:
app/workers/hard_worker.rb
class HardWorker
include Sidekiq::Worker
def perform(name, count)
# do something
end
end
3) Add the Redis add-on (in my case Heroku Redis):
heroku addons:create heroku-redis:hobby-dev
4) Add your redis.rb file, in my case:
config/initializers/redis.rb
$redis = Redis.new(url: ENV["REDIS_URL"])
5) Add Procfile or config/sidekiq.yml or both, here are mine:
Procfile
worker: bundle exec sidekiq -c 1 -q default -q mailers
which you can create easier by pasting this in your terminal
echo "worker: bundle exec sidekiq -c 1 -q default -q mailers" > Procfile
config/sidekiq.yml
:concurrency: 1
:queues:
- [mailers, 7]
- [default, 5]
6) Most important part go here:
and turn on the switch for your worker, click on the pencil and then turn on the missing switch. Things should be working fine now, have a great day!
What I found out is that you have to scale process manually like so:
heroku ps:scale worker+1
Makes no sense since my Procfile said:
web: bundle exec....
worker: bundle exec sidekiq
...and one would've expectd Heroku to start the worker automatically. In the end I didn't have to scale the web process...
Also, you might have problems with this line:
worker: bundle exec sidekiq
Add flags for concurency:
worker: bundle exec sidekiq -c 5 -v
Sounds like you're almost there. It may be that you just need to set REDISTOGO_URL on the heroku config?
heroku config
should show you the redistogo value?
If you copy that to REDISTOGO_URL then sidekiq should work?
heroku config:add REDISTOGO_URL=<redistogo value>
Edit:
Sidekiq will use any of these currently: https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/redis_connection.rb#L29-L33
Edit2:
Greg is correct in that you don't need to add the config if you're using RedisToGo. But if you're using OpenRedis, or other Redis providers, then you need to add REDISTOGO_URL for Sidekiq
You can of course run sidekiq alongside your current jobs queue.
Our Procfile currently looks like this:
web: bundle exec thin start -R config.ru -e $RAILS_ENV -p $PORT
worker: bundle exec rake jobs:work
sidekiq: bundle exec sidekiq -c 5 -v