I'm having a problem getting Sidekiq up and running on my Heroku deployed Rails app. I have my app working fine in development (and on Heroku without Sidekiq).
I created a Procfile with:
worker: bundle exec sidekiq
If I run heroku ps, the only process I see is web.1.
Should I see one for
Sidekiq?
I do get an error:
Redis::CannotConnectError (Error connecting to Redis on localhost:6379) in my Heroku logs.
UPDATE: Found I probably needed heroku addons:add redistogo. Still not working. I feel I'm missing some basic configuration.
Is there something I need to do to get Redis up and running for my Heroku app?
I've been using Redis/Sidekiq for about a day, so this is new to me.
Thanks!
Greg
No you do not need any config with Heroku for Sidekiq, just add the RedisToGo plugin and you're on.
Do not forget to attribute at least 1 worker to your app in your Heroku config.
Here is my default Procfile:
web: bundle exec thin start -p $PORT
worker: bundle exec sidekiq -c 5 -v
It's worth checking if the sidekiq process is really started with this command:
heroku ps
If there's no worker, then you might need to run this command:
heroku ps:scale worker+1
It turns out that there's a bug in the web UI in that some team members were not allowed to increase the number of workers from 0 to 1, even though the UI seemed to show that!
Starting with sidekiq version 3.0 there is an additional step, run heroku config:set REDIS_PROVIDER=REDISTOGO_URL in the console.
Here is the process I used for Rails 4:
In the console:
heroku addons:create redistogo
heroku config:set REDIS_PROVIDER=REDISTOGO_URL
In my Procfile I added:
worker: bundle exec sidekiq
In my gemfile.rb I added:
gem 'redis'
I added the following file, config/initializers/redis.rb:
uri = ENV["REDISTOGO_URL"] || "redis://localhost:6379/"
REDIS = Redis.new(:url => uri)
Here is the link to the sidekiq docs.
Complementing gdurelle answer:
You do need some config with Heroku for Sidekiq:
1) Have the Sidekiq and Redis gems installed (in gemfile and bundled), in my case:
Gemfile
gem 'redis', '~> 3.1'
gem 'sidekiq', '~> 2.7', '>= 2.7.1'
2) Add a worker, if you don't have any workers created locally I suggest you create at least one, just in case, use this:
rails g sidekiq:worker Hard # will create app/workers/hard_worker.rb
should create this:
app/workers/hard_worker.rb
class HardWorker
include Sidekiq::Worker
def perform(name, count)
# do something
end
end
3) Add the Redis add-on (in my case Heroku Redis):
heroku addons:create heroku-redis:hobby-dev
4) Add your redis.rb file, in my case:
config/initializers/redis.rb
$redis = Redis.new(url: ENV["REDIS_URL"])
5) Add Procfile or config/sidekiq.yml or both, here are mine:
Procfile
worker: bundle exec sidekiq -c 1 -q default -q mailers
which you can create easier by pasting this in your terminal
echo "worker: bundle exec sidekiq -c 1 -q default -q mailers" > Procfile
config/sidekiq.yml
:concurrency: 1
:queues:
- [mailers, 7]
- [default, 5]
6) Most important part go here:
and turn on the switch for your worker, click on the pencil and then turn on the missing switch. Things should be working fine now, have a great day!
What I found out is that you have to scale process manually like so:
heroku ps:scale worker+1
Makes no sense since my Procfile said:
web: bundle exec....
worker: bundle exec sidekiq
...and one would've expectd Heroku to start the worker automatically. In the end I didn't have to scale the web process...
Also, you might have problems with this line:
worker: bundle exec sidekiq
Add flags for concurency:
worker: bundle exec sidekiq -c 5 -v
Sounds like you're almost there. It may be that you just need to set REDISTOGO_URL on the heroku config?
heroku config
should show you the redistogo value?
If you copy that to REDISTOGO_URL then sidekiq should work?
heroku config:add REDISTOGO_URL=<redistogo value>
Edit:
Sidekiq will use any of these currently: https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/redis_connection.rb#L29-L33
Edit2:
Greg is correct in that you don't need to add the config if you're using RedisToGo. But if you're using OpenRedis, or other Redis providers, then you need to add REDISTOGO_URL for Sidekiq
You can of course run sidekiq alongside your current jobs queue.
Our Procfile currently looks like this:
web: bundle exec thin start -R config.ru -e $RAILS_ENV -p $PORT
worker: bundle exec rake jobs:work
sidekiq: bundle exec sidekiq -c 5 -v
Related
I recently added the dokku-shoreman plugin and a Procfile so that my app runs both a worker and web process when I deploy. It looks like this:
web: bundle exec rails server -p $PORT
worker: bundle exec rake jobs:work
However, now I when I use dokku run <app> <cmd> such as ...rake db:migrate a server and worker start running instead.
Currently the only way I know how to run the worker is with the Procfile. It's not a big issue to start it manually after I deploy - only I don't know how.
Though the ideal would still be to have the both the Procfile and dokku run working.
Update 2 Dokku now supports this http://dokku.viewdocs.io/dokku/deployment/process-management/
Update: checkout dokku-alt (no longer maintained) first - it's what I've switched to instead.
This is a known issue with dokku.
This was a temporary solution detailed in the issue discussion that worked for me:
rename actualProcfile to Procfile.real
create a new Procfile with the following content:
web: bundle exec foreman start -f Procfile.real
add gem 'foreman' as a dep in Gemfile
run bundle install
commit Procfile* and Gemfile*
push
Whenever I run heroku run bundle exec sidekiq, I see all my background jobs being done, however, I want them to be able to go without me needing to be there. When I exit out of that terminal tab, sidekiq stops working. How would I mitigate that?
Also, I've read something about procfiles and increasing workers. I don't know what procfiles are and I don't know how to increase workers either.
Basically, I'm a newbie trying to get sidekiq set up to run on Heroku for my Rails app. I want it to be running at all times.
Create a file named ./Procfile with the following in it:
web: bundle exec rails server -p $PORT
worker: bundle exec sidekiq
sidekiq on Heroku
more on Procfiles
foreman gem
The Heroku docs indicate that you can set environment variables for use in the procfile. I'm attempting to do set a SIDEKIQ_CONCURRENCY variable, so that my staging environment allots fewer workers for Sidekiq than my production environment.
I set the new variables like so:
heroku config:set SIDEKIQ_CONCURRENCY=3 --app mystaging
heroku config:set SIDEKIQ_CONCURRENCY=15 --app myproduction
Then I added the following line to my procfile:
worker: bundle exec sidekiq -c ENV[SIDEKIQ_CONCURRENCY]
I also tried this:
worker: bundle exec sidekiq -c $SIDEKIQ_CONCURRENCY
And when I deploy I get the following error in my logs:
invalid value for Integer(): "ENV[SIDEKIQ_CONCURRENCY]"
Just to close this out, worker: bundle exec sidekiq -c $SIDEKIQ_CONCURRENCY does work. I redeployed and the error resolved.
what is the "foreman way" for behaving differently in production vs
development? That is we want foreman start to start up a bunch of
stuff in dev, however in heroku production we don't need it to start
(for example) solr.
I follow the convention;
Procfile defines all processes
.foreman set specific foreman variables
Development:
.env sets environment variables for each developer
.env.example sets defaults for development
foreman start starts all processes
Production:
heroku config sets environment variables
heroku ps:scale turns on or off whichever processes are needed for production
Here's an example from a project.
Procfile:
web: bundle exec unicorn_rails -p $PORT -c ./config/unicorn.rb
worker: bundle exec rake jobs:work
search: bundle exec rake sunspot:solr:run
.env.example:
# default S3 bucket
S3_KEY=keykeykeykeykeykey
S3_SECRET=secretsecretsecret
S3_BUCKET=myapp-development
.env
# developer's private S3 bucket
S3_KEY=mememememememememe
S3_SECRET=mysecretmysecret
S3_BUCKET=myapp-development
.foreman:
# development port is 3000
port: 3000
Foreman takes arguments to use a different file (-d) and arguments to specify what to run. It also supports a .foreman file that allows those args to become default. See http://ddollar.github.com/foreman/ for more info
I've used environment-specific Procfiles before, which is pretty simple and works fine.
Basically you have Procfile.development, Procfile.production, etc. In each you can customize the procs you want to start, then run them via foreman like so:
foreman start -f Procfile.development
Another approach is to reference scripts in your Procfile, and within each script start up the appropriate process based on the environment. The creator of Foreman does this and has an example from his Anvil project your reference.
Our solution was to use a separate job type in our Procfile for dev vs. production. It is not the most DRY method, but it works...
sidekiq: bundle exec sidekiq
sidekiq-prod: bundle exec sidekiq -e production
Then we run foreman specifying the prod job on the production system:
foreman start -c sidekiq-prod=4
I have an app on (ruby/rails) heroku. It's running 1 web and 1 worker (for example)
I want to be able to tell what "type" of dyno the app is running under.
I suspect it's a simple thing to tell, but I can't see anything that tells me how to tell.
I don't know if there's a more elegant way to do this, but you can set an environment variable in your Procfile:
web: bundle exec ... PROC_TYPE=web
worker: bundle exec ... PROC_TYPE=worker
Then in your rails code, you can check ENV['PROC_TYPE']
EDIT: more detailed Procfile example, typical for a rails app:
web: bundle exec rails server -p $PORT PROC_TYPE=web
worker: bundle exec rake jobs:work PROC_TYPE=worker