Can Resque Cleaner be expanded to store >1000 failed jobs? - ruby-on-rails

I have Resque Cleaner set up with Redis and a Ruby on Rails framework. I noticed the cleaner fills with failed jobs up until the 1000 mark. I've been trying to find a way to increase the 1000 threshold with no luck yet, is there a way to do this that I'm missing?

Looks like this is covered in the readme here: https://github.com/ono/resque-cleaner
Add this to a resque-web config file:
# [app_dir]/config/resque-web.rb
require 'resque-cleaner'
module Resque::Plugins
ResqueCleaner::Limiter.default_maximum = 10_000
end
Then pass the file when you start resque-web
$ resque-web [app_dir]/config/resque-web.rb

Related

Ruby on Rails Cron Job Example

Hi um quite new to rails platform and um looking for a rails cron job scheduling tutorials . I ve gone through with tutorials use whenever and the other scheduling gems , but um looking for a core ruby implementation with the cron tab on rails . Thank you in advance
Railscasts has a decent tutorial on using Cron.
EDIT
If you would like an example of how it was implemented from scratch, you could have a look at how the Whenever Gem is implemented here
To set cron jobs, you can depend the help of a simple gem named WHENEVER
Its very easy to implement. Go for it.
Many people are against this approach (see this SO thread) but triggering your application's action with curl/wget from a cron job may be a quick-and-easy solution for periodic tasks.
You just have to keep a few things in mind:
Keep execution time of your action low (as it will block your application just like a regular web request would do)
Make sure that you do not allow anyone to trigger the action (by using IP restrictions, secret tokens or other security measures)
For more information on this topic, I have written an article about it.
For minimal setup of "cron-like" tasks in "core" rails/ruby, I created https://github.com/Ebbe/arask
No need to install anything (other than the gem) or setup anything outside of rails.
Add gem 'arask' to your Gemfile, run bundle install, rails generate arask:install and rails db:migrate.
Now you can setup your tasks in the file config/initializers/arask.rb:
arask.create task: 'send:logs', cron: '0 2 * * *' # At 02:00 every day
arask.create script: 'puts "IM ALIVE!"', interval: :daily
arask.create task: 'my:awesome_task', interval: :hourly
arask.create task: 'my:other_awesome_task', interval: 2.hours
The tasks will automatically run if the server is running.
I'm using rufus-scheduler, It uses threads to execute functions scheduled. Very simple to configure. just 3 steps:
1 - Add gem gem 'rufus-scheduler', '~> 3.6'
2 - Create file config/initializers/scheduler.rb
3 - Programming schedule in scheduler.rb:
require 'rufus-scheduler'
s = Rufus::Scheduler.singleton
s.every '5s' do
#do every 5 seconds exec this code
puts "WOWWWWWWWWWWWWWWWWWWWW"
end
s.in '2d' do
# every 2 days exec this
puts "Now it's me"
end
For more doubts : https://github.com/jmettraux/rufus-scheduler
You can also use and external free service to outsource cronjobs http://guardiano.getpeople.in
DISCLAIMER: I made it

How to log in a Ruby worker script of a Rails app that does not have the environment?

I'm using rufus-scheduler for handling cron jobs for a Rails 3.2.x app. The root worker.rb is being fired off by foreman (actually upstart on this particular server) and therefore when it starts off it does not have the Rails context in which to operate. Obviously when I attempt to call logger or Rails.logger it will fail each time.
I'm using log4r as a replacement for the default Rails Logger, which is quite nice, but I am wondering what the proper solution for this problem would be:
1) Do I need to give the script the Rails context at startup (it is simply running rake tasks right now so it ends up getting the Rails environment when the worker script hands off to the rake task and I fear giving the script the Rails env before running the timed task would be overkill since it takes so long to fire up the env)? or
2) Should I just set up log4r as one would do in a non-Rails Ruby script that simply reads in the same log4r.yml config that the Rails app is using and therefore participate in the same log structure?
3) Or some other option I'm not thinking of?
Additionally, I would appreciate either an example or the steps that I should consider with the recommended implementation.
For reference, I followed "How to configure Log4r with Rails 3.0.x?" and I think this could be helpful when integrated with the above: "Properly using Log4r in Ruby Application?"
I think this might be what you're looking for..
Use this within the worker itself, and create a custom named log file
require 'log4r'
logger = Log4r::Logger.new('test')
logger.outputters << Log4r::Outputter.stdout
logger.outputters << Log4r::FileOutputter.new('logtest', :filename => 'logtest.log')
logger.info('started script')
## You're actual worker methods go here
logger.info('finishing')

Can you use Redis alongside Resque?

I'm using Redis To Go on Heroku, and I'd like to use it to store some data alongside Resque jobs, much like I can store DelayedJobs in the same Postgres database as the rest of my model data.
Is this possible? Care to explain briefly how Resque stores jobs in Redis? I'm new to Redis, and though I understand on a higher level how it works, I'm unfamiliar with the implementation details.
Yes, you can. According to the Resque documentation:
Resque has a redis setter which can be given a string or a Redis object. This means if you're already using Redis in your app, Resque can re-use the existing connection.
Also, the documentation for Resque.redis= states that a namespace is accepted, to prevent clashes with your other apps running on the same instance:
A 'hostname:port/namespace' String (to set the Redis namespace)
Check out the "Configuration" section of the Resque readme for more information: https://github.com/resque/resque/blob/master/README.markdown
Redis is an advanced key value store which is used by Resque to store data about jobs. Exact details depend on your application however the two should be fine working together. As long as your code stays away from using the lists Resque uses then you will be golden.
yes, completely agree with other answers, the Redis can be used for other purposes, not only for background processing. By the way, Rescue provides easy interface to setup/config/use/browse Redis.
Setup/Run
$ git clone --depth=1 git://github.com/defunkt/resque.git
$ cd resque
$ rake redis:install dtach:install
$ vim config/resque.yml
"development: localhost:6379"
$ vim config/initializers/rescue.rb
"rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
rails_env = ENV['RAILS_ENV'] || 'development'
resque_config = YAML.load_file(rails_root + '/config/resque.yml')
Resque.redis = resque_config[rails_env]"
$ rake redis:start
Use
# model
def user_activity
a = Resque.redis.lrange "log_" + self.id.to_s, 0, -1
a.map{|i| JSON.parse(i)}
end
Browse
$ resque-web
It allows you to browse Redis content, not only background processing

How to run a cron job in Heroku, with a Sinatra app

I'm writing a tiny Sinatra app, and I want to host it on Heroku for simplicity sake. But, what I have is a task that scraps some sites and adds some data into my database every hour. Currently this is just written as a ruby script that needs to be executed. What Heroku has is a rake based cron job. Now if this was a rails app, I could easily do this, but I want to avoid the clutter for something as simple as this.
Is there a way to avoid this? Or do I have to install rake as well with my app?
Thank you.
Eric
You need a Rakefile like:
desc "This task is called by the Heroku cron add-on"
task :cron do
# Do something
end
Heroku periodically executes rake cron in your app depending on whether you have selected the "cron add-on" to be hourly or daily.
You need to check out Rufus. Rufus is your friend. Rufus will be your crontab while your app is loaded.
I did not try this stuff on Heroku but, give it a try and reply to us.
http://codex.heroku.com/past/2010/4/15/rufus_scheduler_for_my_tiny_sinatra_apps/
Why Rufus is cool? Well check this out, it's clean :)
$ sudo gem install rufus-scheduler
require 'rubygems'
require 'rufus/scheduler'
scheduler = Rufus::Scheduler.start_new
scheduler.cron '00 23 30 * *' do
# Run every 30 days at 23h00m
# ...your magic code goes here...
end
Looked again and looks like I jumped the gun on the question.
For applications that aren't Rails, one just has to create a Rakefile and put the task there.
Hope this helps other people.
Cheers!

How can I make sure the Sphinx daemon runs?

I'm working on setting up a production server using CentOS 5.3, Apache, and Phusion Passenger (mod_rails). I have an app that uses the Sphinx search engine and the Thinking Sphinx gem.
According to the Thinking Sphinx docs...
If you actually want to search against
the indexed data, then you’ll need
Sphinx’s searchd daemon to be running.
This can be controlled using the
following tasks:
rake thinking_sphinx:start
rake ts:start
rake thinking_sphinx:stop
rake ts:stop
What would be the best way to ensure that this takes place in production? I can deploy my app, then manually run rake thinking_sphinx:start, but I like to set things up so that if I have to bounce the server, everything will come back up.
Should I put a call to that Rake task in an initializer? Or something in rc.local?
rc.local is a good start, but its not enough. I would pair is with a monit rule to ensure it is running AND more importantly...
Sphinx requires a full-reindex to make all the latest and greatest available. There is some doco on the thinking sphinx site about delta indexing, but if your index is small, an hourly re-index will take care of things and you do not need the delta indexing stuff.
I run this hourly to take care of this:
0 * * * * cd /var/rails/my_site/current/ && RAILS_ENV=production /usr/bin/rake ts:rebuild
Note: for deployment, I will use the built in thinking sphinx capistrano tasks:
In your Capfile add
require 'thinking_sphinx/deploy/capistrano'
I used to chain the re-indexing in the cap task but stopped cause it is really slow, when I make schema changes I will remember to run it or wait for the hourly cron job to fix it up.
I haven't done this before with Spinix, so I hope someone can give you a better answer, but you should take a look at monit. Monit is designed for keeping daemons running, just like what you need to do.
A quick Google for spinix monit turned up this link: Capistrano recipes: sphinx:monit. That would be a good place to start.
For what it's worth, I'm running
thinking_sphinx:index
... in my cron job, instead of the "rebuild" task. This does not require the searchd process to be offline, but the indices are still rotated when it's done, so new changes are picked up. I think the "rebuild" task is only necessary when you actually change your index structure in your models, which happens very rarely for me.

Resources