I am uploading a tab-delimited document and processing in the controller. Everything works fine, but can take some time on a large file. I want to move this to a delay_job, which I have working elsewhere in my app, but as this is in the controller, cannot be called in the same way.
The form calls on the process_file action, which in turn calls on the salesupload action. How should I turn this into a background job?
class SalesController < ApplicationController
def salesupload(file)
uploaded_io = file.read
numrows = "uploaded_io.size"
FasterCSV.parse(uploaded_io, {:headers => true, :col_sep =>"\t"}).each do |row_data|
full, upc, _discard, isrc = row_data[26].match(/^([^_]+)(_(.+))?/).to_a
new_record = AppleSale.new(
'provider' => row_data[0],
'provider_country' => row_data[1],
'vendor_identifier' => row_data[2]
)
new_record.save
end
end
def process_file
file = params[:apple_sale][:tsv_file]
salesupload(file)
end
end
I found when I had to do this that the method defined in the controller has to be a class method. I can't remember why this was, I think it had to do with having a more explicit receiver. So what I would do is make the salesupload method a class method, and then just call .delay on it.
def self.salesupload(files)
# code
end
def process_file
file = params[:apple_sale][:tsv_file]
SalesController.delay.salesupload(file)
head :no_content
end
And you should be good to go! I also made my original method (process_file in this case) called via AJAX, and then I appended the head :no_content so that it returned something without needing a redirect or anything.
I wrote a gem called delayed_action to do this.
In this case you'd write
delayed_action [:sales_upload]
and that's it.
I guess you should move this code to a model or a separated class (for instance inside the lib folder). But that is just a matter of organization and best practices.
About the running this code in background, you have many options.
If you want to use delayed_job I guess you should watch this screencast:
http://railscasts.com/episodes/171-delayed-job
but basically, after setting up delayed job, you just use something like send_later(:process_file) to tell to run that project in background.
It is pretty straightforward, your command and data structure will be saved in a database table and later a separated process can execute it. I imagine that after watching the railscast you will have your answers ;)
Related
I have a delayed job that runs perfect against a public schema in postgresql.
Most of my operations however are against other schemas (one for each client)
To handle different schemas I've followed the instructions and put code to switch search path, in my before_filter (in application controller).
I've noticed. That the code in the before_filter gets called perfectly during typical operations, but not at all during delayed job.
I trimmed and trimmed out everything but the simplest thing I could think of, to show entrance.
class ApplicationController < ActionController::Base
protect_from_forgery
def write_to_log(text)
File.open('c:\temp.txt', 'ab') do |f|
f.write text + "\r\n"
f.close
end
end
before_filter :on_before_filter
def on_before_filter
write_to_log('hey dave');
return if(use_token() == false);
set_active_schema if(goto_log_in? == false);
end
The code in the worker class
def run_job(id)
upload = Upload.find(id)
upload.run_job();
end
handle_asynchronously :run_job, :priority => 10, :queue => 'public'
Quite standard stuff? Though the code in the job runs, the before_filter code doesn't get called.
So my question is. Did I do something wrong? Or more importantly, how can I do something right?
I'm not recommending this approach; I'm just answering your question by providing this code. Since you essentially want your code to run before any attempted call to the database, you can monkey patch ActiveRecord. Add the following code to config/initializers/active_record_monkey_patch.rb
class ActiveRecord::ConnectionAdapters::ConnectionPool
# create an alias for the old 'connection' method
alias_method :old_connection, :connection
# redefine the 'connection' method
def connection
# output something just to make sure the monkey patch is working
puts "*** custom connection method called ***"
# your custom code is here
write_to_log('hey dave');
return if(use_token() == false);
set_active_schema if(goto_log_in? == false);
# call the old 'connection' method
old_connection
end
end
You'll see your custom connection method getting called frequently now, and it will work without a controller. You can test it by opening up a rails console and performing any database query, and you should see the "custom connection method called" message displayed several times.
If you want to manipulate the ActiveRecord search path for Postgres and schemas you can use a full-featured gem like apartment: https://github.com/bradrobertson/apartment
You can switch to a new schema:
Apartment::Database.switch('database_name')
Regardless if you call this in an application controller request or a background job.
I have a situation where i need to prevent users from explicitly calling say /town/addBuilding. Town is my controller and addBuilding is the action that is executed.
Now, the thing is that this action should only be executed in my program's code and not by a user requesting to execute it. Moreover, this action is executed like a callback. In my application_controller, when some condition is met, the controller action is triggered and there is a redirection. In php, a simple guard like defining a guard and checking against it would be enough. Is there an equivalent thing in rails and if so, what is the best way to implement it ?
Thanx for reading and i appreciate your help :)
EDIT: I'm pasting some code to make it clearer, note that /town/addBuilding was an example, the controller names and actions below are differently named.
Now, that is the actual application controller code, it is part of a browser game that i'm coding.
def checkQuest
if TavernQuest.hasQuest(current_user)
quest = TavernQuest.getQuest(current_user)
if quest.end_time < Time.now # get quest info and check if the quest has been completed
TavernQuest.deleteQuest(current_user)
redirect_to :controller => 'tavern', :action => 'monsterAttack'
end
end
end
The tavern controller action is just the plain code that i want to execute, but only if the redirection happens inside the application controller.
It seems that you are trying to put logic into a controller which actually should belong in a model or a library.
Why do i say this: aside from the current_user and the redirect, all the code is more related to your model (where the knowledge should be) and not your controller. Your model knows when a user's quest is expired.
Example implementation:
class TavernQuest
def self.user_quest_is_expired?(user)
quest = getQuest(current_user)
if quest && quest.end_time < Time.now
TavernQuest.deleteQuest(current_user)
true
else
false
end
end
end
and in your controller you just need to write
redirect_to :controller => 'tavern', :action => 'monsterAttack' if TavernQuest.user_quest_is_expired?(current_user)
Put the addBuilding method under a line that starts with protected, as follows
protected
def addBuilding
#your code
end
Enjoy!
EDIT: In addition to this you might also wanna use the before_filter in your controllers... I'll post the exact syntax soon.
before_filter :addBuilding, :only => :method_name
method_name is the method from which :addBuilding can be accessed, no other method can access this method after adding in this line..
EDIT: Ok, so based on the info you provided, protected wont work since if we put your secret action under protected only the tavern controller will have access to it.
EDIT: Please consider using Sessions to check if the users have a valid session when they try to to execute the monsterAttack action..
My rails app produces XML when I load /reports/generate_report.
On a separate page, I want to read this XML into a variable and save it to the database.
How can I do this? Can I somehow stream the response from the /reports/generate_report.xml URI into a variable? Or is there a better way to do it since the XML is produced by the same web app?
Here is my generate_report action:
class ReportsController < ApplicationController
def generate_report
respond_to do |format|
#products = Product.all
format.xml { render :layout => false }
end
end
end
Here is the action I am trying to write:
class AnotherController < ApplicationController
def archive_current
#output = # get XML output produced by /reports/generate_report
# save #output to the database
respond_to do |format|
format.html # inform the user of success or failure
end
end
end
Solved: My solution (thanks to Mladen Jablanović):
#output = render_to_string(:file => 'reports/generate_report.xml.builder')
I used the following code in a model class to accomplish the same task since render_to_string is (idiotically) a protected method of ActionController::Base:
av = ActionView::Base.new(Rails::Configuration.new.view_path)
#output = av.render(:file => "reports/generate_report.xml.builder")
Perhaps you could extract your XML rendering logic to a separate method within the same controller (probably a private one), which would render the XML to a string using render_to_string, and call it both from generate_report and archive_current actions.
What I typically do in this type of situation is to create a separate module/class/model to generate the report (it could even potentially be right in the Product model). This separate component could be in app/models or it could be in lib. In any case, once you have it extracted you can use it anywhere you need it. The controller can call it directly. You can generate it from the console. You can have a cron job generate it. This is not only more flexible, but it also can help smooth out your request response times if the report becomes slow to generate.
Since you are using a template it's understandable that the controller route is convenient, but even if you have to include some kind of ruby templating system in your auxiliary lib, it's still probably going to be less hassle and more flexible then trying to go through the controller.
#output = Product.all.to_xml
I'm sorry, is you question about Xml or about sessions? I mean is the fact that your action generates Xml material to the question? Or do you just want to save the output of the action for latter use?
You said on a "separate" page - you mean on another request? (like after user approved it?)
Why do you want to save the output? Because it should be saved exactly as rendered? (for example user can get frustrated if he clicked to save one report and you saved another)
Or is this thing expensive to generate?
Or may be, I got it wrong and it's about refactoring?
I'm working on a Rails app, where I'm using page caching to store static html output. The caching works fine. I'm having trouble expiring the caches, though.
I believe my problem is, in part, because I'm not expiring the cache from my controller. All of the actions necessary for this are being handled within the model. This seems like it should be doable, but all of the references to Model-based cache expiration that I'm finding seem to be out of date, or are otherwise not working.
In my environment.rb file, I'm calling
config.load_paths += %W( #{RAILS_ROOT}/app/sweepers )
And I have, in the /sweepers folder, a LinkSweeper file:
class LinkSweeper < ActionController::Caching::Sweeper
observe Link
def after_update(link)
clear_links_cache(link)
end
def clear_links_cache(link)
# expire_page :controller => 'links', :action => 'show', :md5 => link.md5
expire_page '/l/'+ link.md5 + '.html'
end
end
So ... why isn't it deleting the cached page when I update the model? (Process: using script/console, I'm selecting items from the database and saving them, but their corresponding pages aren't deleting from the cache), and I'm also calling the specific method in the Link model that would normally invoke the sweeper. Neither works.
If it matters, the cached file is an md5 hash off a key value in the Links table. The cached page is getting stored as something like /l/45ed4aade64d427...99919cba2bd90f.html.
Essentially, it seems as though the Sweeper isn't actually observing the Link. I also read (here) that it might be possible to simply add the sweeper to config.active_record.observers in environment.rb, but that didn't seem to do it (and I wasn't sure if the load_path of app/sweepers in environment.rb obviated that).
So I've tried a number of different approaches, to see what works, and what doesn't.
Again, to summarize the situation: My goal is to expire cached pages when an object updates, but to expire them without relying on a Controller action. Conventional sweepers use a line in the controller to notify the sweeper that it needs to function. In this case, I can't use a line in the controller, as the update is happening within the model. Normal sweeper tutorials aren't working, as they presume that your main interaction with the database object is through the controller.
If, in reading this, you see a way to tighten up my code, please comment and let me know.
First, let's look at the things that DO work, in case you're stuck on this, too, and need help.
Of all the things I tried, the only thing that really seemed to work was to declare an after_update command in the Observer for the model. In that command, I used the explicit command for the expire_page action, and included a path that had been declared in routes.rb.
So. This works:
In config/routes.rb:
map.link 'l/:md5.:format', :controller => 'links', :action => 'show'
In app/models/link_observer.rb:
def after_update(link)
ActionController::Base.expire_page(app.link_path(:md5 => link.md5))
end
Note that that "md5" is specific to my app. You might want to use :id or some other unique identifier.
I also found that declaring that ActionController::Base... line from the method in the model that's doing the updating worked. That is, within Link.rb, in the method that's actually updating the database, if I just stuck that whole line in, it worked. But since I might want to expire that page cache on other methods in the future, I'd rather have it extracted into the Observer.
Now, let's look at some things that DID NOT work, in case you're Googling around for this.
Calling "expire_page(...)" within the after_update(link) method within link_observer.rb did not work, as it returned an "undefined method `expire_page'" error
Creating a Sweeper file that observed the Model did not work. I couldn't find any error codes, but it just seemed to not even be aware that it had a job to do. This was after explicitly calling "config.load_paths += %W( #{RAILS_ROOT}/app/sweepers )" within environment.rb. Just in case I fat-fingered something in that code, here it is:
class LinkSweeper < ActionController::Caching::Sweeper
observe Link
def after_update(link)
clear_links_cache(link)
end
def clear_links_cache(link)
# DID NOT WORK expire_page :controller => 'links', :action => 'show', :md5 => link.md5
# DID NOT WORK expire_page '/l/'+ link.md5 + '.html'
# DID NOT WORK ActionController::Base.expire_page(app.link_path(:md5 => link.md5))
end
end
That above example had the link_sweeper.rb file in a directory, /app/sweepers. I also tried putting link_sweeper.rb within the app/models directory, and tried calling it with the config.active_record.observers command in environment.rb:
config.active_record.observers = :link_observer, :link_sweeper
But that didn't work, either.
So, yeah. It's quite possible that one of these methods would work, and that I messed up something in the code. But I think I did everything by the book.
Ultimately, to summarize: Rather than using a Sweeper to expire page caching, you want to set up an after_ callback in the model's Observer. You'll want to use the explicit path to the Base.expire_page method:
def after_update(<model>) # where <model> is the name of the model you're observing
ActionController::Base.expire_page(app.<model>_path(:id => <model>.id)) # where <model> is the name of the model you're observing
end
Hopefully this will help someone else down the road. Again, if you see anywhere in my not-working code where I should have done something differently, please let me know. If you see something in my working code that can be tighter, please let me know that, too.
Just a note: you can use cache_sweeper in ApplicationController.
class ApplicationController < ActionController::Base
cache_sweeper :my_sweeper
end
class MySweeper < ActionController::Caching::Sweeper
observe MyModel
def after_update(my_model)
expire_page(...)
end
end
I was experiencing the same problem when trying to do fragment caching (rails 3). Couldn't get the sweeper to observe, so I settled for the solution to make it an AR Observer as described above and calling ApplicationController.new.expire_fragment(...).
I did get this working. The only slight difference in my setup is that the sweeper is part of a Rails engine; which accounts for slight differences (loading the sweeper file with a require in the engine's init instead of adding it to the load path in environment.rb, etc).
So, the sweeper is loaded in the init.rb of the engine like this:
require File.join(File.dirname(__FILE__), 'app', 'sweepers', cached_category_count_sweeper')
I called it a sweeper because it "sweeps" the cache, but I guess its just an observer on the model:
class CachedCategoryCountSweeper < ActiveRecord::Observer
observe CategoryFeature
def before_save(cf)
expire_cache(cf.category_id_was) if cf.category_id_changed?
end
def after_save(cf)
expire_cache(cf.category_id)
end
def after_destroy(cf)
expire_cache(cf.category_id)
end
def expire_cache(c)
ApplicationController.expire_page("/categories/#{c}/counts.xml") if !c.nil?
end
end
Frankly, I don't like having to hard-code the path, but I tried adding:
include ActionController:UrlWriter
and then using the path method, but it only worked for me in development. It didn't work in production, because my production server uses a relative url root (instead of virtual hosts) and the internal method "page_cache_path" would consistently get the file path wrong so it couldn't expire.
Since this is an observer, I added to the environment.rb:
config.active_record.observers = :cached_category_count_sweeper
Finally the controller that uses the cache (doesn't expire it, that is done through the model observer):
class CachedCategoryCountsController < ApplicationController
caches_page :index
# GET /cached_category_counts.xml
def index
...
end
end
Anyhow, hope this helps.
Andres Montano
I've been able to get it to work, by way of adding
ActionController::Base.expire_page(app.link_path(:md5 => #link.md5))
to the method in the Model itself that's updating the database. This feels somewhat hacky, though, and I'd love to know if anyone can explain why it's not working with the normal sweeper setup, and if there's a more elegant way to handle this.
That snippet of code (apart from customizations I put in for my own app) came from this post on ruby-forum.com.
I wrote a bit about this topic here: Rails Cache Sweeper Confusion. Would love to hear your opinions.
Based on #moiristo and #ZoogieZork 's answers, I am guessing this would work (untested).
class LinkSweeper < ActiveRecord::Observer
include ActionController::Caching::Pages
# or if you want to expire fragments
#include ActionController::Caching::Fragments
observe Link
def after_update(link)
expire_page( ... )
#expire_fragment( ... )
end
end
I'd like to be able to dispatch from one controller action to another conditionally, based on a combination of query parameters and data in the database.
What I have right now is something like:
class OldController < ApplicationController
def old_controller_action
if should_use_new_controller
new_params = params.dup
new_params[:controller] = "new_controller_action"
redirect_to new_params
return
end
# rest of old and busted
end
end
class NewController < ApplicationController
def new_controller_action
# new hotness
end
end
This works just fine, but it issues an HTTP redirect, which is slow. I'd like to be able to do this same thing, but within the same HTTP request.
Is there a clean way to do this?
Edit: The bounty will go to someone who can show me a clean way to do this that leaves the controllers and their actions relatively untouched (other than the redirect code itself).
Instead of calling code across actions, extract the code to lib/ or something, and call that code from both controllers.
# lib/foo.rb
module Foo
def self.bar
# ...
end
end
# posts_controller
def index
Foo.bar
end
# things_controller
def index
Foo.bar
end
Create an instance of the controller class:
#my_other_controller = MyOtherController.new
Then call methods on it:
#my_other_controller.some_method(params[:id])
I prefer the module idea, but this should do the trick.
You can also pass parameters as a whole from another controller:
#my_other_controller.params = params
I suspect you want option 3, but lets go through the some alternatives first
Option 1 - Push the controller selection logic into a helper that inserts the right link into your view. Benifits - controllers remain clean, Cons - if decision logic depending on submitted values this approach won't work. If URL is being called by external websites then this won't work.
Option 2 - Push the logic back into your model. Pro's - keeps controller clean. Cons - doesn't work well if you've got lots of sesson, params or render / redirect_to interaction.
Option 3 - Stay within the same controller. I suspect you are trying to replace some existing functionality with some new functionality, but only in some cases. Pro's - Simple and have access to everything you need. Cons - only works if it makes sense to use the same controller i.e. you're working with the same entity such as user, place or company.
Lets look an an example for option 3. My links controller has totally diferent behavour for admins than other users ...
class LinksController < ApplicationController
#...
def new
#Check params and db values to make a choice here
admin? ? new_admin : new_user
end
#...
private
def new_admin
#All of the good stuff - can use params, flash, etc
render :action => 'new_admin'
end
def new_user
#All of the good stuff - can use params, flash, etc
render :action => 'new_user'
end
end
If two controllers are trying to do the same thing, there's a very good chance this should be in a model. Take a good look at your design and -- I'm sorry I don't know your experience level with MVC -- read up on thin controller techniques:
http://weblog.jamisbuck.org/2006/10/18/skinny-controller-fat-model
http://www.robbyonrails.com/articles/2007/06/19/put-your-controllers-on-a-diet-already
http://andrzejonsoftware.blogspot.com/2008/07/mvc-how-to-write-controllers.html
If the problem is that you need the other controller to do the render, then maybe the route should have pointed there to begin with, and still the skinny controller technique should save the day.
If extracting the common code between controllers into a module doesn't work for you, I would use Rack middleware. I haven't seen code that uses ActiveRecord within middleware but I don't know of any reason why it shouldn't be possible since people have used Redis and the like.
Otherwise I think your only option would be to restart processing of the request with something like (untested, pseudo example):
env['REQUEST_URI'] = new_controller_uri_with_your_params
call(env)
This is similar to how integration tests are implemented. But I don't know if everything from call until you hit a controller is idempotent and safe to rerun like this. You could trace through the source and see. But even if it's ok now, it might break in any future version of rails or rack.
Using middleware would avoid this by letting you intercept the request before it's been run. You should still be able to share code with your rails application by extracting it out into common modules included in both places.
Honestly I think just doing the simple thing of factoring the common controller code is likely cleaner, but it's hard to know without the details of your situation so I thought I'd go ahead and suggest this.
Do this:
class OldController < ApplicationController
def old_controller_action
if should_use_new_controller
new_controller_action
end
# rest of old and busted
end
end
and the new controller
class NewController < OldController
def new_controller_action
# new hotness
end
end