I am receiving the above error and trying to get more insight. My app has 3 types of background jobs and about 100 users so nothing too heavy.
My goal is to be able to process multiple background jobs at the same time (so if 10 users perform the same job, they don't need to wait for each other job to finish before starting).
I'm confused as to how many dynos I need, how many workers I need, how many redis connections I need. What's the difference between all these things?
My current setup has:
1 x professional web dyno
1 x professional scheduler dyno
3 x professional worker dyno
and my procfile:
web: bundle exec rails server -p $PORT
scheduler: bundle exec rake resque:scheduler
worker: env TERM_CHILD=1 QUEUE='*' COUNT='3' bundle exec rake resque:workers
And I am getting the error:
Redis::CommandError: ERR max number of clients reached
I am just surprised because it seems like what I'm trying to achieve is pretty simple.
Related
I want to run both the database and migration in foreman. However, I found that they usually run at the same time when I start foreman. Since at the time I run migration the database has not fully started yet, it causes the migration to fail.
Heroku using Procfile could facilitate a release phase. The phase would be run after all the commands are run. Can I do the same using foreman in my computer?
Heroku does not rely to Procfile to maintain the release process. The Heroku build stack does.
Since the foreman provides us the way to run multiple processes at the same time, not running processes in order, so your problem is not the responsibility of foreman
However, you have some other ways to do so.
Simple: since foreman can start your process with shell command, you can use basic shell command sleep (in seconds) for delaying your process
db_process: start_db_script.sh
migrarion_process: sleep 5; bundle exec rake db:migrate --trace
Full control: Instead of run default migration rake task, you can write another rake task which check the connection to database before execute the migration ( refer to this answer)
retried = 0
begin
# Establishes connection
ActiveRecord::Base.establish_connection
# Try to reconnect
# It will raise error if cannot reach your database
ActiveRecord::Base.connection.reconnect!
Rake::Task["db:migrate"].invoke if ActiveRecord::Base.connected?
rescue => e
retried += 1
if retried <= 5 # Retry only 5 times
sleep 1 # Wait 1 seconds before retry
retry
end
puts "#{e} Cannot connect to your database with 5 seconds"
end
I am working on a Rails app that runs regularly scheduled sidekiq jobs, and I understand how queues and background jobs works. I'm working with a 3rd party that requests that I batch jobs to them so that each worker handles one job at a time with 50 workers running in parallel.
I've been researching this for hours, but I'm unclear on how to do this and how to tell if it's actually working. Currently, my procfile looks like this:
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec sidekiq -C ./config/sidekiq.yml
Is it as simple as increasing the concurrency from the rake task to -c 50 in the worker line? Or do I need to use ConnectionPool inside the worker class? The Rails docs say that using find_each is "useful if you want multiple workers dealing with the same processing queue." If I run find_each inside the rake task and call the worker once for each item, will it run the jobs in parallel? I read one article that says that concurrency and parallelism are often confused, so I am, in turn, a little confused about which direction to take.
I'm birthing sidekiq queues via the Procfile:
worker: bundle exec sidekiq -q default -q events -q summaries -c 5 -v
And in development I mistakenly deleted the queues events and summaries from the sidekiq/queues page. I think they're still there and functioning but I can't SEE them. I thought they would once again be added the minute I called the bundle exec sidekiq again but they're not there....
Something I'm missing?
You have to push a job to them. Queues don't actually exist in Redis unless they contain jobs.
For production purposes I need three processes running. This is my procfile and I use Foreman to start them:
web: bundle exec rails s Puma -p $PORT
queuing: bundle exec clockwork clock.rb
workers: bundle exec rake resque:workers
For deployment I'm using Mina. What's the appropriate way to start Foreman at the end of deploy task? Currently I'm starting like this:
desc "Deploys the current version to the server."
task :deploy => :environment do
deploy do
invoke :'git:clone'
invoke :'deploy:link_shared_paths'
invoke :'bundle:install'
invoke :'rails:db_migrate'
invoke :'rails:assets_precompile'
to :launch do
queue "touch #{deploy_to}/tmp/restart.txt"
queue "bundle exec foreman start"
end
end
end
... but I don't think that's the proper way since the "mina deploy" command never successfully exits and the local console just starts outputting whatever these processes are doing.
Question number two: How do I initialize logging for each of these three processes separately in separate files?
And how do I prevent killing all of these three processes when one of them crashes? How do I make the process restart when it crashes?
Thanks!
OK, so that's 3 questions.
1) I think you want to detach foreman process from the terminal. That way the deployment process will finish and foreman process will be running even after you have disconnected from the server. nohup is great for that, e.g. this will launch your app and pipe all logs to server.log file:
nohup foreman start > server.log 2>&1 &
2) AFAIK, foreman doesn't let you do that. You should probably use another process management service (e.g. systemd, upstart). Thankfully, foreman lets you easily export your config to different process management formats (http://ddollar.github.io/foreman/#EXPORTING).
3) Again, you probably want to separate your processes and manage them separately via upstart, systemd, etc.
On a similar note, how can you know how many workers there are currently assigned?
If you are on your local machine, just run one of the following
# starts the worker
rake jobs:work
# kills it
Control + C on your keyboard
or
# starts the worker
script/delayed_job start
# kills it
script/delayed_job stop
Additionally, here are some commands to spawn multiple workers: https://github.com/collectiveidea/delayed_job/wiki/Delayed-job-command-details
If you want a list of currently running workers, you would do
script/delayed_job status
and this would return each process (which you'd then have to count to get the integer value)
If you are on Heroku, you can do heroku workers to get the number of current workers, and heroku workers 2 to start two workers or heroku workers 0 to kill all workers.
You can also use HireFireApp.com to manage all of your workers for you on Heroku.
Since you didn't specify what type of environment you are running DJ on, please let me know if these don't answer your question.
$ ps ax |grep delayed_job can show you any details directly
The "ps" suggestion works better after first looking int RAILS_ROOT/tmp/pids for delayed_job*.pid files.
Together this tells you what DJ thinks it's running and what is actually running. If one of those files contains a process ID that you can't find in the ps output, that's an indicator that DJ has a worker that has died that it hasn't realized.