I've been struggling for two days with getting sidekiq work on heroku production environment. I've read all the available documentation about similar issues, and still haven't been able to produce a working solution, I'd really like some help!
After deploying on heroku, my app crashes and I get the following error stack trace:
2014-09-23T23:38:40.905093+00:00 app[worker.1]: No such file or directory # rb_sysopen - /app/tmp/pids/sidekiq.pid
2014-09-23T23:38:40.905122+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.1.0/bin/sidekiq:23:in `<main>'
2014-09-23T23:38:40.905119+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.2.5/bin/sidekiq:7:in `<top (required)>'
2014-09-23T23:38:40.905117+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.2.5/lib/sidekiq/cli.rb:347:in `write_pid'
2014-09-23T23:38:40.905115+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.2.5/lib/sidekiq/cli.rb:347:in `open'
2014-09-23T23:38:40.905121+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.1.0/bin/sidekiq:23:in `load'
2014-09-23T23:38:40.905118+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.2.5/lib/sidekiq/cli.rb:41:in `parse'
2014-09-23T23:38:40.905114+00:00 app[worker.1]: /app/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.2.5/lib/sidekiq/cli.rb:347:in `initialize'
2014-09-23T23:38:39.588001+00:00 heroku[worker.1]: State changed from starting to up
First, sidekiq is working correctly on my local machine. I'm using heroku's REDISTOGO for redis, on local production, sidekiq have been pointed to the REDISTOGO correctly and running fine.
Second, according to the stack trace, especially this line No such file or directory # rb_sysopen - /app/tmp/pids/sidekiq.pid; it leads me to think that for some reason sidekiq.pid file is not generated correctly when running on heroku. On local environment, the sidekiq.pid file is generated everytime I start the app in the app/tmp/pids/ directory, and assigns a different pid number every time. I'm guessing that when running on heroku, sidekiq tried to read from this file yet could not find it.
Here is contents in my Procfile:
web: bundle exec rails server
worker: bundle exec sidekiq -C config/sidekiq.yml
Here is contents in my config/sidekiq.yml
---
:verbose: true
:pidfile: ./tmp/pids/sidekiq.pid
:concurrency: 25
# Set timeout to 8 on Heroku, longer if you manage your own systems.
:timeout: 8
:queues:
- carrierwave
Here is contents in my sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = { :url => ENV['REDISTOGO_URL'], :namespace => "mynamespece"}
end
Sidekiq.configure_client do |config|
config.redis = { :url => ENV['REDISTOGO_URL'], :namespace => "mynamespece"}
end
Update 1:
I'm using carrierwave and carrierwave-backgrounder in sync with sidekiq.
This issue has been resolved by the following actions:
1) Thanks to #MikePerham for pointing me to the right direction, first I removed this line in my sidekiq.yml file:
:pidfile: ./tmp/pids/sidekiq.pid
2) Then, in my Procfile, I had to use the following line to replace the origin:
web: bundle exec rails server -p $PORT
worker: bundle exec sidekiq -C config/sidekiq.yml
Now sidekiq is working correctly with redistogo on heroku for me.
I found it useful to add the directory to the repository and re-deploy the application.
mkdir -p tmp/pids
touch tmp/pids/.gitkeep
git add -f tmp/pids/.gitkeep
git commit -m 'Keep tmp/pids directory in repo'
Hope this helps.
It was fixed by making pids directory in tmp directory in root of project
Related
I need help with my sidekiq and redis set up in my rails (7) app.
The problem is that all my Sidekiq jobs (emails) are being enqueued in production and not sending, but they work fine in development.
Here are my relevant files
sidekiq.rb
require 'sidekiq/web'
Sidekiq.configure_server do |config|
config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') }
end
Sidekiq.configure_client do |config|
config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') }
end
sidekiq.yml
development:
:concurrency: 5
production:
verbose: false
:concurrency: 5
:timeout: 8
:max_retries: 1
:queues:
- default
- mailers
Procfile.dev
web: bundle exec rails server -p $PORT
worker: bundle exec sidekiq -c 5 -v
I have basically read all of the questions related to my problem and the lucid guidelines by Mike Perham. None of the answers work; to be fair, I am feeling my way in the dark. It's unclear what the issue is because I do not understand the processes thoroughly. I am new to Sidekiq. If you need more information, please let me know.
I try to move all the logging of my rails (3.2.17) app in production and staging to papertrail. Now I'm trying to set the logger for sidekiq in config/initializers/sidekiq.rb like this:
Sidekiq.configure_server do |config|
Sidekiq::Logging.logger = RemoteSyslogLogger.new('logs.papertrailapp.com', ENV.fetch('PAPERTRAIL_PORT'), program: "sidekiq-#{Rails.env}")
end
When I try to start sidekiq with:
bundle exec sidekiq --index 0 --pidfile <PATH_TO_PID> --environment staging --daemon
I get
You really should set a logfile if you're going to daemonize
...bundle/ruby/2.0.0/gems/sidekiq-2.17.4/lib/sidekiq/cli.rb:141:in `daemonize'
...bundle/ruby/2.0.0/gems/sidekiq-2.17.4/lib/sidekiq/cli.rb:39:in `parse'
...bundle/ruby/2.0.0/gems/sidekiq-2.17.4/bin/sidekiq:7:in `<top (required)>'
...bundle/ruby/2.0.0/bin/sidekiq:23:in `load'
...bundle/ruby/2.0.0/bin/sidekiq:23:in `<main>'
However if I give sidekiq a dummy for logging first:
bundle exec sidekiq --index 0 --pidfile <PATH_TO_PID> --environment staging --daemon --logfile /dev/null
it works like a charm (logs are sent to papertrail), since the initializer seems to override the previous option.
But I find this approach quite ugly. Anybody got a cleaner approach to this?
I discussed it with the owner of sidekiq and he made a good point: from the time the process is started until the time the initializer has run, a lot of things can go wrong. If the logger is only initialized in the initializer, everything that happended before, won't appear in any log.
I probably will go with the appoach recommended by the sidekiq owner:
bundle exec sidekiq | logger -t sidekiq
I've set up Faye on localhost, and works fine.
I'm now trying to deploy it on DigitalOcean VPS, with Cloud66 deployment service.
To run the command I use an "after_rails" hook.
rackup $RAILS_STACK_PATH/faye.ru -s thin -E production
This gives me the error:
Error during deployment: Error during after_rails hook: Server: not_specified. Failed to run command: /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in require': cannot load such file -- faye (LoadError) from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:inrequire' from /var/deploy/anabol/web_head/current/faye.ru:1:in block in <main>' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:55:ininstance_eval' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:55:in initialize' from /var/deploy/anabol/web_head/current/faye.ru:innew' from /var/deploy/anabol/web_head/current/faye.ru:in <main>' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:49:ineval' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:49:in new_from_string' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/builder.rb:40:inparse_file' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/server.rb:277:in build_app_and_options_from_config' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/server.rb:199:inapp' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/server.rb:314:in wrapped_app' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/server.rb:250:instart' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/lib/rack/server.rb:141:in start' from /usr/local/lib/ruby/gems/1.9.1/gems/rack-1.5.2/bin/rackup:4:in' from /usr/local/bin/rackup:23:in load' from /usr/local/bin/rackup:23:in'
github repo:
https://github.com/gwuix2/anabol
Any suggestions what could be the problem? Don't know where to start.
I've written up some documentation on how to solve this problem in case anyone else runs into it. You can find it here: http://community.cloud66.com/articles/faye-on-cloud-66
Disclaimer: I work for Cloud 66.
Cloud 66 Support solved my problem:
Hi there,
We run faye on our own stack. We actually run it behind thin - and monitor it as a process on the stack. The way we do this is with four files - see their details below:
1) RAILS_ROOT/.cloud66/deploy_hooks.yml
production:
before_rails:
- source: /.cloud66/files/add_thin_and_faye.sh
destination: ~/add_thin_and_faye.sh
target: rails
execute: true
sudo: true
apply_during: build_only
run_on: all_servers
2) RAILS_ROOT/.cloud66/files/add_thin_and_faye.sh
#!/bin/bash
sudo gem install thin --no-ri --no-rdoc
sudo gem install faye --no-ri --no-rdoc
3) RAILS_ROOT/Procfile
faye: thin -R $RAILS_STACK_PATH/faye/config.ru start
4) RAILS_ROOT/faye/config.ru
require 'faye'
faye_server = Faye::RackAdapter.new(:mount => '/your_faye_mount', :timeout => 45)
Faye::WebSocket.load_adapter('thin')
faye_server.listen(<<PUT-YOUR-PORT-HERE>>)
Note that for part 4) the settings will be different based on your requirements of course.
If you follow the setup above, you'll have faye running alongside your stack, and it will be available as a process on your stack.
I'm having a problem getting Sidekiq up and running on my Heroku deployed Rails app. I have my app working fine in development (and on Heroku without Sidekiq).
I created a Procfile with:
worker: bundle exec sidekiq
If I run heroku ps, the only process I see is web.1.
Should I see one for
Sidekiq?
I do get an error:
Redis::CannotConnectError (Error connecting to Redis on localhost:6379) in my Heroku logs.
UPDATE: Found I probably needed heroku addons:add redistogo. Still not working. I feel I'm missing some basic configuration.
Is there something I need to do to get Redis up and running for my Heroku app?
I've been using Redis/Sidekiq for about a day, so this is new to me.
Thanks!
Greg
No you do not need any config with Heroku for Sidekiq, just add the RedisToGo plugin and you're on.
Do not forget to attribute at least 1 worker to your app in your Heroku config.
Here is my default Procfile:
web: bundle exec thin start -p $PORT
worker: bundle exec sidekiq -c 5 -v
It's worth checking if the sidekiq process is really started with this command:
heroku ps
If there's no worker, then you might need to run this command:
heroku ps:scale worker+1
It turns out that there's a bug in the web UI in that some team members were not allowed to increase the number of workers from 0 to 1, even though the UI seemed to show that!
Starting with sidekiq version 3.0 there is an additional step, run heroku config:set REDIS_PROVIDER=REDISTOGO_URL in the console.
Here is the process I used for Rails 4:
In the console:
heroku addons:create redistogo
heroku config:set REDIS_PROVIDER=REDISTOGO_URL
In my Procfile I added:
worker: bundle exec sidekiq
In my gemfile.rb I added:
gem 'redis'
I added the following file, config/initializers/redis.rb:
uri = ENV["REDISTOGO_URL"] || "redis://localhost:6379/"
REDIS = Redis.new(:url => uri)
Here is the link to the sidekiq docs.
Complementing gdurelle answer:
You do need some config with Heroku for Sidekiq:
1) Have the Sidekiq and Redis gems installed (in gemfile and bundled), in my case:
Gemfile
gem 'redis', '~> 3.1'
gem 'sidekiq', '~> 2.7', '>= 2.7.1'
2) Add a worker, if you don't have any workers created locally I suggest you create at least one, just in case, use this:
rails g sidekiq:worker Hard # will create app/workers/hard_worker.rb
should create this:
app/workers/hard_worker.rb
class HardWorker
include Sidekiq::Worker
def perform(name, count)
# do something
end
end
3) Add the Redis add-on (in my case Heroku Redis):
heroku addons:create heroku-redis:hobby-dev
4) Add your redis.rb file, in my case:
config/initializers/redis.rb
$redis = Redis.new(url: ENV["REDIS_URL"])
5) Add Procfile or config/sidekiq.yml or both, here are mine:
Procfile
worker: bundle exec sidekiq -c 1 -q default -q mailers
which you can create easier by pasting this in your terminal
echo "worker: bundle exec sidekiq -c 1 -q default -q mailers" > Procfile
config/sidekiq.yml
:concurrency: 1
:queues:
- [mailers, 7]
- [default, 5]
6) Most important part go here:
and turn on the switch for your worker, click on the pencil and then turn on the missing switch. Things should be working fine now, have a great day!
What I found out is that you have to scale process manually like so:
heroku ps:scale worker+1
Makes no sense since my Procfile said:
web: bundle exec....
worker: bundle exec sidekiq
...and one would've expectd Heroku to start the worker automatically. In the end I didn't have to scale the web process...
Also, you might have problems with this line:
worker: bundle exec sidekiq
Add flags for concurency:
worker: bundle exec sidekiq -c 5 -v
Sounds like you're almost there. It may be that you just need to set REDISTOGO_URL on the heroku config?
heroku config
should show you the redistogo value?
If you copy that to REDISTOGO_URL then sidekiq should work?
heroku config:add REDISTOGO_URL=<redistogo value>
Edit:
Sidekiq will use any of these currently: https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/redis_connection.rb#L29-L33
Edit2:
Greg is correct in that you don't need to add the config if you're using RedisToGo. But if you're using OpenRedis, or other Redis providers, then you need to add REDISTOGO_URL for Sidekiq
You can of course run sidekiq alongside your current jobs queue.
Our Procfile currently looks like this:
web: bundle exec thin start -R config.ru -e $RAILS_ENV -p $PORT
worker: bundle exec rake jobs:work
sidekiq: bundle exec sidekiq -c 5 -v
I have a locally developed Rails application. It is very simple, with no database but only a controller that accesses Facebook data and renders them in view. By the way, gem RestGraph is used.
Before I push this application to heroku, I used foreman start to test it. Since I was using WEBrick before I need to install 'thin' gem and create a Procfile which reads:
web: bundle exec thin start -R config.ru -e $RACK_ENV -p $PORT
an '.env' file seems also required, which reads:
RACK_ENV=development
PORT=3000
The 'config.ru' file is generate by rails, which reads:
require ::File.expand_path('../config/environment', __FILE__)
run Myapp::Application
Now I entered 'foreman start', but all I get is one line:
15:33:18 web.1 | started with pid 27143
The server will not boot. And if I force terminate it, the error is:
/usr/local/foreman/lib/foreman/engine.rb:141: [BUG] rb_sys_fail() - errno == 0
Which is not very helpful.
It may be that instead of being "stuck," your log is just being buffered: foreman only shows line with “started wit pid #” and nothing else
I solved that issue by adding
$stdout.sync = true
to config.ru
Then you might get more helpful feedback from Foreman.
This seems a little over complex. From looking at a couple of apps I have I have the following in my procfile:
bundle exec rails server thin -p $PORT
I also don't have a .env file in these instances.
One thing worth noting is that Foreman will ignore the port you pass in most of the time, it likes to sit in a port range higher than 5000