In my application, I need to execute a task every 5 minutes. The task itself is quite lean, so that's not the problem: the point is that loading the Rails environment uses almost all the CPU of the server. Right now, I'm using whenever with a code like this in my schedule.rb:
every 5.minutes do
runner Task.perform
end
But the cost is too big. Is there anyway to execute this kind of tasks in a preloaded environment? Or any other better solution?
Consider using a background process (many gems, notably delayed job) and implementing a simple "cron" in your code with a polling loop something like
def perform
last = Time.now
while true
if last.sec % (5 * 60) != 0
do_task
last = Time.now
end
end
end
Warning: this example has more holes than a slice of swiss cheese. Write something better :-)
You could use a cron job or set it up to run in a bash script. These would have the advantage of not being tied to your application.
Related
I am creating an automatic raffling system. I have a draw button that will run a draw function to select a winner or winners and it sends an email to the admin.
I want this to be a completely automated system so that the admin only has to create the raffles and they receive an email with who won after the draw date has passed. My raffles have a draw date associated with them and once that passes, I need the function to be called.
How do I tell the application to check the time/date to see if any of the raffle draw times have passed? I have looked everywhere and cannot seem to find a way to do it.
You could use the whenever gem to define a job that runs hourly (or however often you want), checks the draw dates, and runs the draw for any that have passed.
I use Clockwork in my rails apps whenever I need to schedule things. Simply set it up to run a job when you want and do your logic within that job to see which raffles need to be processed. Example:
Clockwork config
every(1.day, 'Raffle::CheckJob', at: '01:00')
Job
Raffle.not_complete.find_each(batch_size: 10) do |raffle|
if raffle.has_ended?
// logic
end
end
You should write a rake task and add it's execution to your crontab on server. You can use whenever gem to simplify crontab scripting and auto update on each deploy (whenever-capistrano/whenever-mina). Example of your rake task:
namespace :raffle do
task :check do
Raffle.get_winners.each do |w|
Mailer.send_win_mail(w).deliver_later
end
end
end
deliver_later is background execution in queue by the queue driver you use (DelayedJob/Rescue/Backburner etc)
I have a rails 3 application and looked around in the internet for daemons but didnt found the right for me..
I want a daemon which fetches data permanently (exchange courses) from a web resource and saves it to the database..
like:
while true
Model.update_attribte(:course, http::get.new("asdasd").response)
end
I've only seen cron like jobs, but they only run after a specific time... I want it permanently, depending on how long it takes to end the query...
Do you understand what i mean?
The gem light-daemon I wrote should work very well in your case.
http://rubygems.org/gems/light-daemon
You can write your code in a class which has a perform method, use a queue system like this and at application startup enqueue the job with Resque.enqueue(Updater).
Obviously the job won't end until the application is stopped, personally I don't like that, but if this is the requirement.
For this reason if you need to execute other tasks you should configure more than one worker process and optionally more than one queue.
If you can edit your requirements and find a trigger for the update mechanism the same approach still works, you only have to remove the while true loop
Sample class needed:
Class Updater
#queue = :endless_queue
def self.perform
while true
Model.update_attribute(:course, http::get.new("asdasd").response)
end
end
end
Finaly i found a cool solution for my problem:
I use the god gem -> http://god.rubyforge.org/
with a bash script (link) for starting / stopping a simple rake task (with an infinite loop in it).
Now it works fine and i have even some monitoring with god running that ensures that the rake task runs ok.
I'm using thinking_sphinx and am delta indexing a model.
The delta index works but there is small bug. When I create a new product it is index. However, when I update that product it is not getting index right away. I have to update or create a new product before that old updated product is indexed.
Not quite sure where to start.
My recommendation would be to use delayed_delta indexing instead of straight delta indexing (which can be slow and if you have a few updates in a few seconds, can cause you all kinds of problems).
It takes two steps:
Change your define_index block to have a set_property :delta => :delayed
Create a short script to make sure the delayed indexing jobs get run. Here's the one I use:
#!/usr/bin/env ruby
## this script is for making sure and delayed_jobs get run
## it is used by thinking sphinx
require File.dirname(__FILE__) + '/../config/environment'
# you can also put the definition of this in config/environments/*.rb so it's different for test, production and development
JobRunnerPidFile = "#{RAILS_ROOT}/tmp/pids/job_runner.pid"
if File.exists?(JobRunnerPidFile)
old_pid = File.read(JobRunnerPidFile).to_i
begin
if Process.getpgid(old_pid) > 0
# still running, let's exit silently...
exit(0)
end
rescue
# looks like nothing is running, so let's carry on
end
end
File.open(JobRunnerPidFile, "w") {|f| f.write "#{$$}\n" }
Delayed::Worker.new.start
You can run that script from cron every 5 minutes (it'll only run one instance) or if you have a monitoring service (e.g., monit) you can have it make sure it's running.
Make sure to restart that script when ever you deploy a new version of your code.
When a new resource is created and it needs to do some lengthy processing before the resource is ready, how do I send that processing away into the background where it won't hold up the current request or other traffic to my web-app?
in my model:
class User < ActiveRecord::Base
after_save :background_check
protected
def background_check
# check through a list of 10000000000001 mil different
# databases that takes approx one hour :)
if( check_for_record_in_www( self.username ) )
# code that is run after the 1 hour process is finished.
user.update_attribute( :has_record )
end
end
end
You should definitely check out the following Railscasts:
http://railscasts.com/episodes/127-rake-in-background
http://railscasts.com/episodes/128-starling-and-workling
http://railscasts.com/episodes/129-custom-daemon
http://railscasts.com/episodes/366-sidekiq
They explain how to run background processes in Rails in every possible way (with or without a queue ...)
I've just been experimenting with the 'delayed_job' gem because it works with the Heroku hosting platform and it was ridiculously easy to setup!!
Add gem to Gemfile, bundle install, rails g delayed_job, rake db:migrate
Then start a queue handler with;
RAILS_ENV=production script/delayed_job start
Where you have a method call which is your lengthy process i.e
company.send_mail_to_all_users
you change it to;
company.delay.send_mail_to_all_users
Check the full docs on github: https://github.com/collectiveidea/delayed_job
Start a separate process, which is probably most easily done with system, prepending a 'nohup' and appending an '&' to the end of the command you pass it. (Make sure the command is just one string argument, not a list of arguments.)
There are several reasons you want to do it this way, rather than, say, trying to use threads:
Ruby's threads can be a bit tricky when it comes to doing I/O; you have to take care that some things you do don't cause the entire process to block.
If you run a program with a different name, it's easily identifiable in 'ps', so you don't accidently think it's a FastCGI back-end gone wild or something, and kill it.
Really, the process you start should be "deamonized," see the Daemonize class for help.
you ideally want to use an existing background job server, rather than writing your own. these will typically let you submit a job and give it a unique key; you can then use the key to periodically query the jobserver for the status of your job without blocking your webapp. here is a nice roundup of the various options out there.
I like to use backgroundrb, its nice it allows you to communicate to it during long processes. So you can have status updates in your rails app
I think spawn is a great way to fork your process, do some processing in background, and show user just some confirmation that this processing was started.
What about:
def background_check
exec("script/runner check_for_record_in_www.rb #{self.username}") if fork == nil
end
The program "check_for_record_in_www.rb" will then run in another process and will have access to ActiveRecord, being able to access the database.
>rails -v
Rails 1.2.6
>ruby -v
ruby 1.8.6 (2007-03-13 patchlevel 0) [i386-mswin32]
When I run a test fixture (that tests a rails model class) like this, it takes 20-30 secs to start executing these tests (show the "Loaded suite..."). What gives?
>ruby test\unit\category_test.rb
require File.dirname(__FILE__) + '/../test_helper'
class CategoryTest < Test::Unit::TestCase
def setup
Category.delete_all
end
def test_create
obCategoryEntry = Category.new({:name=>'Apparel'})
assert obCategoryEntry.save, obCategoryEntry.errors.full_messages.join(', ')
assert_equal 1, Category.count
assert_not_nil Category.find(:all, :conditions=>"name='Apparel'")
end
#.. 1 more test here
end
This one is Rails using a MySql DB with no fixtures. This time it clocked 30secs+ to startup.
Take a look at this Rails Test Server.
A quote from the author:
"Every time you run a test in a Rails
application, the whole environment is
loaded, including libraries that don’t
change between two consecutive runs.
That can take a considerable amount of
time. What if we could load the
environment once, and only reload the
changing parts before each run?
Introducing RailsTestServing.
With RailsTestServing, the run time of
a single test file has gone from 8
seconds down to .2 of a second on my
computer. That’s a x40 speed
improvement. Now, I don’t think twice
before hitting ⌘R in TextMate. It
feels liberating!"
(This was featured on the Rails Envy Podcast this past week which is where I found this.)
When starting any tests, Rails first loads any fixtures you have (in test/fixtures) and recreates the database with them.
20-30 seconds sounds very slow though. Do you have a lot of fixtures that need to be loaded before your tests run, or is your database running slow?
Ruby's gem tool follows a path discovery algorithm which, apparently, is not Windows (as I see from your ruby -v) friendly.
You can get a clear picture if you trace, for example, a Rails application loading with ProcMon. Every (I really mean every) require starts a scan over all directories in Ruby's path plus all gem directories. A typical require takes 20 ms on an average machine. Since Rails makes hundreds of requires, those 20 ms easily sum up to seconds every time you launch the Rails environment. Take in the time to initialize the fixtures in the database and you get a better idea of why it takes so much time to just begin running the test-cases.
Perhaps because of each file-system architecture and implementation (path caching etc.), this is less of a problem in Linux than in Windows. I don't know who you should blame, though. It looks like the NTFS file-system could be improved with a better path caching implementation, but clearly the gem tool could implement the caching itself and have its performance not so dependent on the platform.
It seems like Test::Unit is the simplest, but also one of the slowest ways to do unit testing with Ruby. One of alternatives is ZenTest.
Test unit startup isn't particularly slow, and nowhere near 20 seconds.
(11:39) ~/tmp $ cat test_unit.rb
require 'test/unit'
class MyTest < Test::Unit::TestCase
def test_test
assert_equal("this", "that")
end
end
(11:39) ~/tmp $ time ruby test_unit.rb
Loaded suite test_unit
Started
F
Finished in 0.007338 seconds.
1) Failure:
test_test(MyTest) [test_unit.rb:4]:
<"this"> expected but was
<"that">.
1 tests, 1 assertions, 1 failures, 0 errors
real 0m0.041s
user 0m0.027s
sys 0m0.012s
It's probably something you're doing in your tests. Are you doing anything complicated? Setting up a database? Retrieving something from the internet?
Complete shot in the dark, but the majority of the time I see long startup times on things, it is usually due to some sort of reverse DNS lookup happening with some TCP socket communication somewhere along the way.
Try adding:
require 'socket'
Socket.do_not_reverse_lookup = true
at the top of your test file after your other require line.
What does your test_helper.rb look like? Are you using instantiated fixtures?
self.use_instantiated_fixtures = true
[edit]
If this is set to true try setting it to false.