Saving and later importing xls, csv files in rails application - ruby-on-rails

Question

In my ruby on rails application, have implemented a facility seeing the railscast video http://railscasts.com/episodes/396-importing-csv-and-excel, that will directly import and update the db at same time, but it is creating problems when the file is too large.
So I want to write a facility to upload a csv or excel file in my application and save it in a directory. Then I want to add some kind of observer that will observe the contents of the directory and on an event like create or update of a file in that directory will trigger contents of those files to be uploaded in db. I am getting no idea about how to approach this.
Thanks in advance.

Answer 1

I think that the best approach is to use Resque to import and convert in a worker separately from the request.
Suppose you have a controller to add the Excel file, which I'm going to call Information model:
class InformationController < ApplicationController
def create
#information = Information.new(params[:information])
if #information.save
resque = Resque.enqueue(ImportDataJob, #information.id)
redirect_to #information, :notice => "Successfully created information for further processing."
else
render :new
end
end
end
You'll need make a job, in this case ImportDataJob:
class ImportDataJob
def self.perform(information_id)
information = Information.find(information_id)
# convert information.raw_csv or wherever attribute you saved the Excel or CSV into
# and save it into the database where you need to
end
end
You'll find a full tutorial in Resque RailsCast, where it shows how to add Resque into your existing Rails app.
Note: There's a conflict between README and the actual implementation for Resque. Apparently they want to change the way Resque is called (which is in the readme), but is not implemented yet. See this Issue in Github for more details.

Answer 2

As an alternative to Resque (and i do think doing it in a background job is the best approach), see also Spawnling, previously known as Spawn:
https://github.com/tra/spawnling
Super-low maintenance. In your controller action do something like
#file = <uploaded file here>
spawn do
#perform some long-running process using #file, in a new process
end
#current thread carries on straight away.
If you need to test if it's finished (or crashed for that matter) you can save the id of the new process like so:
#file = <uploaded file here>
spawner = spawn do
#perform some long-running process using #file, in a new process
end
#current thread carries on straight away.
spawner = spawn do
#make it in a temp file so serve_if_present doesn't serve a halfmade file
FileUtils.rm #filename if File.exists?(#filename)
temp_filename = "#{#filename}.temp"
ldb "temp_filename = #{temp_filename}"
current_user.music_service.build_all_schools_and_users_xls(:filename => temp_filename)
FileUtils.mv temp_filename, #filename
end
#spawn_id = spawner.handle