I'm attempting to use sidekiq on Heroku Container, but no luck. I've been taking a week to fix this. Any help?
I describe what I did so far.
Setup sidekiq
Sidekiq.configure_server do |config|
config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') }
end
Sidekiq.configure_client do |config|
config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1') }
end
Set environment variables to Heroku app
Enable Redis add-on
redis screenshot1
redis screenshot2
setup 2 dynos for a worker and a web
Procfile
web: bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -c 3 -q default
commands
heroku container:push web
heroku container:release web
heroku container:push worker
heroku container:release worker
heroku ps:scale web=1 worker=1
Even when I triggered worker job, I can't see any logs about queue or worker.
web itself works fine.
What I feel weird is this description.
heroku screeen
web rails server -b 0.0.0.0
worker rails server -b 0.0.0.0
Is this correct?
Related
I'm new to Heroku and despite all the documentation, I'm a little unsure what the profile is for. I can set a port and the environment as follows, but Heroku always starts in production mode (which makes sense) and not with the specified port.
I suppose that the port cannot be set because it is determined by Heroku?
Is the Procfile only for the command "heroku local" to test?
Because when I run "heroku ps" I get info about the procfile, but the API runs without the procfile port in production mode.
Thank you for any explanation!
Procfile:
web: bundle exec puma -t 5:5 -p ${PORT:-3000} -e ${RACK_ENV:-development}
Output of heroku ps after deploying:
=== web (Hobby): bundle exec puma -t 5:5 -p ${PORT:-3000} -e ${RACK_ENV:-development} (1)
web.1: up 2020/07/27 13:50:23 +0200 (~ 1m ago)
Output of heroku logs at the same time:
Version 3.12.6 (ruby 2.5.8-p224), codename: Llamas in Pajamas
2020-07-27T11:49:32.219309+00:00 app[web.1]: * Min threads: 5, max threads: 5
2020-07-27T11:49:32.219309+00:00 app[web.1]: * Environment: production
2020-07-27T11:49:33.740321+00:00 app[web.1]: * Listening on tcp://0.0.0.0:10269
Heroku will set both $PORT and $RACK_ENV for Rails apps when they're deployed. You can confirm this by running heroku config --app <yourapp>. The construct ${PORT:-3000} means "use the PORT variable if it's present, otherwise use the value 3000.
In any case, you can't run a Heroku app on a port other than the one defined in $PORT, which is randomized for each dyno. Whatever that's set to will be forwarded to from ports 80 and 443 for HTTP/S.
If you want to override the RACK_ENV, you can run heroku config:set RACK_ENV=development.
My Heroku app is not starting any workers. I scale the worker first:
heroku ps:scale resque=1 -a test-eagle
Scaling dynos... done, now running resque at 1:Free
Then when I check the workers, I see:
heroku ps:workers -a test-eagle
<app> is running 0 workers
What could be worng here? This is how my Procfile looks:
web: bundle exec puma -C config/puma.rb
resque: env TERM_CHILD=1 bundle exec rake resque:work QUEUE=* COUNT=1
Or is it because it is a free app which can only handle 1 web worker and no other dynos?
Edit:
When I check with heroku ps -a <appname> I see that just after starting the worker is crashed: worker.1: crashed. This is without doing anything in the application itself.
UPDATE: Well, I have a "free" app running that happens to run Puma, too. So, I updated Procfile as follows:
web: bundle exec puma -C config/puma.rb
resque: env TERM_CHILD=1 bundle exec rake resque:work QUEUE=* COUNT=1
After that, I pushed the app to Heroku and ran heroku ps:scale, as you specified. It worked as follows:
D:\Bitnami\rubystack-2.2.5-3\projects\kctapp>heroku ps -a kctapp
=== web (Free): bundle exec puma -C config/puma.rb (1)
web.1: up 2016/06/06 19:38:24 -0400 (~ 1s ago)
D:\Bitnami\rubystack-2.2.5-3\projects\kctapp>heroku ps:scale resque=1 -a kctapp
Scaling dynos... done, now running resque at 1:Free
D:\Bitnami\rubystack-2.2.5-3\projects\kctapp>heroku ps -a kctapp
=== web (Free): bundle exec puma -C config/puma.rb (1)
web.1: up 2016/06/06 19:38:24 -0400 (~ 51s ago)
=== resque (Free): env TERM_CHILD=1 bundle exec rake resque:work QUEUE=* COUNT=1 (1)
resque.1: crashed 2016/06/06 19:39:18 -0400 (~ -3s ago)
Note that it did crash. But, I don't have any code running there, so that could be why? Also, note that I use the "heroku ps" command as "heroku ps:workers" for me throws an error saying it is deprecated.
This is my config/puma.rb, if that helps:
workers Integer(ENV['WEB_CONCURRENCY'] || 4)
threads_count = Integer(ENV['MAX_THREADS'] || 8)
threads threads_count, threads_count
preload_app!
rackup DefaultRackup
port ENV['PORT'] || 5000
environment ENV['RACK_ENV'] || 'development'
With edit: I missed the scale command...
See scaling in Heroku here. The options that I see are web, worker, rake or console, not resque. I tried your command and it didn't recognize "that formation". I'm curious about it.
Checking a free app, it does not give you the option to add a worker dyno. Checking a hobby app, you can add workers to it. With professional apps, you can mix and match the dyno type between web and worker using 1X, 2X, and Performance dynos.
I have a Rails app which uses Resque for background jobs. This works fine locally, but after deploying to Heroku I get a connection error:
Redis::CannotConnectError (Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)):
I see it tries to connect to localhost, which is not correct. I'm using the Heroku Redis :: Redis add-in and I have added the redis gem. This is how initializers.redis.rb looks like:
$redis = Redis.new(url: ENV["REDIS_URL"])
And this is my Procfile:
web: bundle exec puma -C config/puma.rb
resque: env TERM_CHILD=1 bundle exec rake resque:work QUEUE=* COUNT=1
In the config vars REDIS_URL is added. What is going wrong here?
It seems that when using Puma or Unicorn on Heroku, you need to add it also to the boot process. I'm using Puma, so I added this to config/puma.rb:
on_worker_boot do
# ...
if defined?(Resque)
Resque.redis = ENV["REDIS_URL"] || "redis://127.0.0.1:6379"
end
end
Here is a more detailed explanation for both Puma and Unicorn:
https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server
https://devcenter.heroku.com/articles/rails-unicorn#caveats
I am using Sidekiq with God in my Rails app. I am using Passenger and Nginx.
I see many processes (30-50) running by sidekiq which consume about 1000MB of RAM.
Processes like:
sidekiq 3.4.1 my_app_name [0 of 1 busy] - about 30 processes.
ruby /home/myuser/.rvm/ruby-2.1.5/bin/sidekiq --environment ... - about 20 processes.
How to tell sidekiq to not run so many threads.
my config for sidekiq (config/sidekiq.yml):
---
:concurrency: 1
:queues:
- default
- mailer
and config for sidekiq for god:
num_workers = 1
num_workers.times do |num|
God.watch do |w|
...
w.start = "bundle exec sidekiq --environment #{rails_env} --config #{rails_root}/config/sidekiq.yml --daemon --logfile #{w.log}"
The problem is with "--daemon" (or "-d") parameter which runs it as a daemon. No need to run it as daemon. Just remove this parameter.
I'm currently running my app on 2 Heroku dynos. From what I've looked up so far I need to add something similar to config/unicorn.rb:
worker_processes 3
timeout 30
#resque_pid = nil
before_fork do |server, worker|
#resque_pid ||= spawn("bundle exec rake " + \
"resque:work QUEUES=scrape,geocode,distance,mailer")
end
I have a few different background jobs to process, some need to be run single-threaded and some concurrently. The problem with this configuration is that on both Unicorn instances it will spawn exactly the same resque worker (same queues etc).
It would greatly simplify everything if I could change the type of queues each worker processes - or even have one instance running a resque worker and the other a sidekiq worker.
Is this possible?
Perhaps you are confusing unicorn worker_processes and Heroku workers?
you can use your Procfile to start Heroku workers for each queue and a unicorn process for handling web requests.
try this setup
/config/unicorn.rb
worker_processes 4 # amount of unicorn workers to spin up
timeout 30 # restarts workers that hang for 30 seconds
/Procfile
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq
worker: bundle exec rake resque:work QUEUES=scrape,geocode,distance,mailer