The sidekiq workers get terminated in Heroku very frequently. It should be started every time.
How can we make these workers to run without getting terminated in Heroku?
Do we have any configurations for Sidekiq or Heroku / can we daemonize the Sidekiq and make it to run in Heroku without terminating.
Please help!
I have been using this sidekiq with rufus-scheduler for my Herokuapp.
Add your scheduler task in rails initializers(#app/config/initializers/task_scheduler.rb) It send daily status at 9am IST.
scheduler = Rufus::Scheduler.new
scheduler.cron '00 09 * * * Asia/Kolkata' do
puts "it's 9am! good morning!"
User.daily_status
end
scheduler.every '10s' do
Rails.logger.info "hello, it's #{Time.now}"
end
Related
I have a Rails app running on Nginx/Passenger. It has a rufus-scheduler cron job that runs in the background and sends out notifications via email.
When I start up the app in production on Nginx/Passenger, the emails don't get sent. In the production logs it doesn't show any logs for rufus-scheduler.
I'm stuck in this problem. Not able to debug the scheduler.
But after reading some issues & articles, I am sure that scheduler thread is kill on production by passenger.
Code snippet:
Example:
require 'rufus-scheduler'
scheduler = Rufus::Scheduler::singleton
scheduler.cron '0 12 * * *' do
puts "alert scheduler called at"
puts Time.now
system("rake trigger:email RAILS_ENV=production")
end
Kindly help me on the conf for the passenger/nginx that allow the scheduler thread to run & trigger the mails.
You need to set this in passanger config block
passenger_spawn_method direct;
I am using Rufus Scheduler to call a function once every 2 minutes.
In development mode when running rufus scheduler with WEBrick the system works as expected and my function is called every 2 minutes.
When I deploy to our production server which runs passenger and NGINX the scheduler does not call the function anymore.
I am assuming this is to do with threads, with the scheduler running on a separate thread that might be getting destroyed on our production server.
I have seen answers for similar issues when using Passenger and Apache together, but I am unable to come up with a solution for running rufus scheduler with passenger and NGINX in production mode.
Any help would be appreciated.
Code below setting up rufus
# :nocov:
require 'rufus-scheduler'
include TwitterParser
unless ENV['NODAEMON'] || Rails.env != 'production'
scheduler = Rufus::Scheduler.new
TwitterParser.set_up
scheduler.every '2m' do
begin
TwitterParser.parse_tweets
rescue Twitter::Error::TooManyRequests => error
puts 'Too Many Twitter requests, resets in: ' + error.rate_limit.reset_in.to_s
end
end
end
# :nocov:
Please read the rufus-scheduler FAQ.
https://github.com/jmettraux/rufus-scheduler#faq
It contains links to (old) articles about Passenger + rufus-scheduler
Amog them, this SO answer by one of the authors of Passenger might help:
rufus cron job not working in Apache/Passenger
Have a good time.
I've got Resque workers that typically shouldn't take longer than about 1-5 minutes to run, but frequently those workers will get "stuck" and go idle, clogging up workers and doing nothing.
So I'd like to regularly check for workers that have been running longer than X time and purge them. But I need to do this automatically, so I don't have to personally go in and manually clear them (Resque.workers.each {|w| w.unregister_worker}) every few hours.
This needs to work on Heroku.
Put this into a rake task:
allocated_time = 60 * 60 # 1 hour
Resque::WorkerRegistry.working.each do |worker|
if (worker.started <=> Time.now - allocated_time) < 1
worker.unregister
end
end
Use heroku scheduler, you can set it to minimum of 10 minutes if that suites.
For Resque v1,
# lib/tasks/clear_stale_workers.rake
namespace :clear do
desc 'Clearing stuck workers ...'
task :stale_workers => :environment do
Resque.workers.each do |w|
w.unregister_worker unless w.started > 1.hour.ago
end
end
end
From the command line, rake clear:stale_workers
On Heroku, set the set the scheduler to run this Rake task.
This worked for me to remove the specific workers running stale jobs. You could add it to a rake task.
Resque::Worker.working.each{|w| w.done_working }
Is it possible to run delayed jobs on Heroku for free?
I'm trying to use delayed_job_active_record on Heroku. However, it requires a worker dyno and it would cost money if I turned this dyno on for full time.
I thought using Unicorn and making its workers run delayed jobs instead of the Heroku worker, would cost nothing, while successfully running all the jobs. However, Unicorn workers do not seem to start "working" automatically.
I have the following in my Procfile.
web: bundle exec unicorn -p $PORT -c ./config/unicorn.rb
worker: bundle exec rake jobs:work
and the following in my unicorn.rb
worker_processes 3
timeout 30
preload_app true
before_fork do |server, worker|
# Replace with MongoDB or whatever
if defined?(ActiveRecord::Base)
ActiveRecord::Base.connection.disconnect!
Rails.logger.info('Disconnected from ActiveRecord')
end
# If you are using Redis but not Resque, change this
if defined?(Resque)
Resque.redis.quit
Rails.logger.info('Disconnected from Redis')
end
sleep 1
end
after_fork do |server, worker|
# Replace with MongoDB or whatever
if defined?(ActiveRecord::Base)
ActiveRecord::Base.establish_connection
Rails.logger.info('Connected to ActiveRecord')
end
# If you are using Redis but not Resque, change this
if defined?(Resque)
Resque.redis = ENV['REDIS_URI']
Rails.logger.info('Connected to Redis')
end
end
Delayed jobs only seem to work when I scale the Heroku worker from 0 to 1.
Again, is it not possible to use Unicorn workers instead of Heroku worker to do the delayed jobs?
Do I have to use a gem like workless to run delayed jobs on Heroku for free? (reference)
Splitting the process like that can incur problems - your best bet is it not try and get it 'free' but use something like http://hirefireapp.com/ which will start up a worker when there are jobs to perform reducing the cost significantly rather than running a worker 24x7.
Also note, Heroku will only ever autostart a 'web' process for you, starting other named processes is a manual task.
You can use Heroku Scheduler to run the jobs using the command
rake jobs:workoff
This way the jobs can run in your web dyno. According to Delayed_Job docs, this command will run all available jobs and exit.
You can configure the scheduler to run this command every 10 minutes for example, and it doesn't have sensible effect on the app's performance when no jobs are queued. Another good idea is to schedule it to run daily at a time with lower access rates.
Ideally there is no straight way to get this free, but you would find lots of workaround one can make to enjoy free background jobs. One of which is http://nofail.de/2011/07/heroku-cedar-background-jobs-for-free/
Also if you plan to use resque which is an excellent choice for background jobs you would need redis which comes free with nano version => https://addons.heroku.com/redistogo. https://devcenter.heroku.com/articles/queuing-ruby-resque
Simple solution is to buy a one dyno for the worker, whereas your web dyno would be free.
Let me if you need more help.
Thanks
Consider using the Workless gem: https://github.com/lostboy/workless
If you only have one web worker, Heroku will sleep it if it's inactive for an hour.
Also, Heroku will reboot all dynos at least once a day.
This makes it hard to do a within-ruby scheduler. It has to at least use persistent storage (e.g. database).
I've just switched to using Unicorn on Heroku. I'm also going to switch to resque from delayed_job and use the setup described at http://bugsplat.info/2011-11-27-concurrency-on-heroku-cedar.html
What I don't understand from this is how config/unicorn.rb:
worker_processes 3
timeout 30
#resque_pid = nil
before_fork do |server, worker|
#resque_pid ||= spawn("bundle exec rake " + \
"resque:work QUEUES=scrape,geocode,distance,mailer")
end
translates into:
"This will actually result in six processes in each web dyno: 1 unicorn master, 3 unicorn web workers, 1 resque worker, 1 resque child worker when it actually is processing a job"
How many workers will actually process background jobs? 1 or 2?
Lets say I wanted to increase the number of resque workers - what would I change?
I think if you run that block, you have your unicorn master already running, plus 3 web workers that you specify at the top of the file, and then the block below launches one Resque worker if it's not already started.
I'm guessing that Resque launches a child worker by itself when it actually performs work.
It would appear that if you wanted another Resque worker, you could just do
worker_processes 3
timeout 30
#resque_pid = nil
#resque_pid2 = nil
before_fork do |server, worker|
#resque_pid ||= spawn("bundle exec rake " + \
"resque:work QUEUES=scrape,geocode,distance,mailer")
#resque_pid2 ||= spawn("bundle exec rake " + \
"resque:work QUEUES=scrape,geocode,distance,mailer")
end
In my experience with Resque, it's as simple as launching another process as specified above. The only uncertainty I have is with Heroku and how it chooses to deal with giving you more workers.