I've been struggling the last couple days trying to create an import process for users to upload CSV/Excel files. I have been trying to imitate the process done in this Railscast episode. However, my requirements are a bit more complex so I have been trying to modify what has been done there but I have hit a wall when it comes to enforcing model validation to ensure against any bad data.
I am trying to use Roo to allow my users the ability to upload CSV or Excel files. I also would like to save the OrderImport model with the uploaded spreadsheet and some other information for historical reference and what not.
Models
property.rb
has_many :owners
has_many :orders
owner.rb
belongs_to :property
order.rb
belongs_to :property
has_many :owners, through: :property
order_import.rb
#no assications
#uses paperclip to save the attached file
Order Import process in order_import.rb
def load_imported_orders(file, cid, uid) #loads file from controller along with client_id and user_id for use in order creation
spreadsheet = Roo::Spreadsheet.open(file.path)
header = spreadsheet.row(1)
order_imports = (2..spreadsheet.last_row).map do |i|
row = Hash[[header, spreadsheet.row(i)].transpose]
property = row["address"]
propparse = import_address_parse(property) #to parse submitted address
newprop = if Property.exists?(propparse)
Property.find_by(propparse)
else
Property.new(propparse)
end
owner = row["owner"]
newowner = Owner.find_or_initialize_by(name: owner)
newprop.owners << newowner #creates new property/owner association
neworder = Order.new(property_id: newprop.id,
client_id: cid,
task: "Property Report",
submitted_by: uid)
neworder
newprop
newowner
end
#imported_orders = order_imports
end
def import_address_parse(property)
address = StreetAddress::US.parse(property)
if address.blank?
new_blank_prop = { address1: nil,
city: nil,
state: nil,
zipcode: nil }
else
new_prop = { address1: address.to_s(:line1),
city: address.city,
state: address.state_name,
zipcode: address.postal_code}
end
end
def save
if #imported_orders.map(&:valid?).all?
#imported_orders.each(&:save!)
true
else
#imported_orders.each_with_index do |order, index|
if order.instance_of?(Property) #needed to reduce the amount of errors shown on the form
errors.add :base, "Row #{index+2}: Please re-check the Address of your Property."
else
order.errors.full_messages.each do |message| #shows any errors not related to property
errors.add :base, "Row #{index+2}: #{message} | #{order}"
end
end
end
false
end
end
order_import controller
def create
#order_import = OrderImport.new(params[:order_import_params])
#order_import.load_imported_orders(params[:order_import][:document], params[:order_import][:client_id], params[:order_import][:submitted_by])
if #order_import.save
redirect_to external_client_orders_path(current_user.client_id), notice: "Imported Orders successfully."
else
render :new
end
end
So as you can see, the load_imported_orders() method gets called from the controller and parses the spreadsheet. Then a property & owner are found (if they happen to exist already) or initialized and then an order is initialized from the items in that row. (I've tried using something like a :before_create filter in my OrderImport model but I have no idea how these different guides are opening files passed through OrderImport.new(params[:order_import_params]) without the model being saved first - and of course I don't want to save unless everything imports correctly).
The purpose for the import_address_parse method is because the address is submitted as one line (555 test rd, testington, tt, 55555) but is comprised of address1, city, state, zipcode in my database. The StreetAddress gem will return nil if it cannot parse the entire address string so I have put that catch in there to return a nil object in hopes to fail the property model validations when Property.new is initialized with all nil values in those fields.
The problems
For some reason the Property validation does not fail on its own, I am only alerted when owners go to be saved and associated to the new property in the save method. Property.exists?() with all nil attributes still loads a property for some reason instead of initializing Property.new and I can't for the life of me figure out why.
The order validation fails because a no property.id exists since the new property hasn't been saved yet, which I understand, but I'm not sure how create the Order association/creation from the save method (i.e. outside of the load_import_orders method which is parsing the import data).
I think my understanding of the entire validation side of importing bulk records is fundamentally wrong, especially since I cannot get the OrderImport record to save correctly even when adding create_or_update (Rails API docs) to my save method, which overwrites the default rails save method for the model.
This may be easier than I'm making it out to be but I would consider myself to still be an amateur Rails developer (having taught myself everything through tutorials, SO posts, guides/articles) and am diving into what I think is complex methodology so if someone could take the time to assist me with refining this process so that I can achieve what my goal is here and attain a deeper understanding of rails in general.
Added caveat, if I change all the find_or_initialize_by / new calls to create! and ONLY use data I know would pass all validations then this process works as intended but we all know thats not realistic in a real-world environment. I think since that part works its throwing me off from re-engineering this a bit (it's not the end of the world if using built in rails CSV.parse instead of Roo is the way to go but I'd like to provide the option of being able to import from excel but its not a necessary one).
Thank you to anyone who takes the time to break this down and help me out.
So after posting this question with no responses, I decided to brush up on my Ruby IRB skills to work my way through the process and analyze the data every step of the way.
After much trial and error, I've finally completed this task!
To provide a brief overview of my final process:
Following the methodology of the answer in this SO question, Ruby on Rails: Validate CSV file, I decided to
Break out the validations into its own Ruby class.
Using Roo's Parse method, I passed each row as a Hash to OrderImportValidator.new()
Pass the row[:property] pair through my import_address_parse method.
Instantiate both row[:property] && row[:owner] using .find_or_initialize_by method to their respective models.
Pair up each :property and :owner objects into a hash within a OrderImportValidator instance variable (#order_objects) and allow access to it from my OrderImport model via a method (very similar to how the #errors is returned in the linked SO CSV question).
Verify validation of all values (instantiated objects) in the returned hash.
If all were good, then I called save on each, create the correct associations, and appropriate order for each pair.
Added a custom validate :import_error_check, on: :create to my OrderImport model.
Run through each.with_index, check if errors exist, clear the errors on the object and add my custom message to errors[:base] of Order Import with their appropriate index and display them on the page.
Pretty pumped I figured it out and have furthered my knowledge of data manipulation, objects, and ruby syntax. Maybe the outline of my process will help someone else one day. Cheers.
Related
I have a nested resource in my routes.rb
# routes.rb
resources :users do
resource :preferences, :only => [:create,:show,:update,:destroy]
end
Now to update the preference of a user, I'm using
preferences = Preference.where(:user_id => user_id).update_all(preferences_request)
render json: preferences, status: 200
But I feel it's not a good practice to use update_all as each user has only one preference. And also I can't use render :json => preferences as preferences will have the value 1 instead of an actual hash object with all the table attributes.
What is the best way to update the preference?
Simply load the preference and then perform an update.
preference = Preference.find_by!(user_id: user_id)
preference.update(preferences_request)
render json: preference
You have to deal with the case the query returns nil (because the user doesn't exist, for example). preferences_request must be a Hash of attributes => values. Of course, you may want to validate it as well and/or use the strong parameters feature to filter our attributes you don't want to be updated.
Usually with this kind of request you'd make two queries. One to find the User and another to update it.
Your request should be to the URL users/1/preferences (replace 1 with the user id that you are trying to update)
Then the controller code can look like
def update
user = User.find(params[:user_id])
user.preferences.update!(preference_request)
render json: user.preferences
end
The benefit of doing it this way is it will appropriately throw a 404 error if the User does not exist, and it will throw a 500 with validation errors if the update fails for some reason.
Read about HTTP status codes and how they can help you here
It seems you have User 1-1 Preference with the following code:
class User < ActiveRecord::Base
has_one :preference
end
class Preference < ActiveRecord::Base
belongs_to :user
end
According to API method update_all has been designed for update related objects in batch straight through database layer bypass ActiveRecord:
Updates all records in the current relation with details given. This
method constructs a single SQL UPDATE statement and sends it straight
to the database. It does not instantiate the involved models and it
does not trigger Active Record callbacks or validations. Values passed
to update_all will not go through ActiveRecord's type-casting
behavior. It should receive only values that can be passed as-is to
the SQL database.
But probably you need to pass objects through validation and return objects back where update method may suit your needs better, see API:
Updates an object (or multiple objects) and saves it to the database,
if validations pass. The resulting object is returned whether the
object was saved successfully to the database or not.
Back to your question, you have 1-1 mapping, so there is no need to update multiple records. Correct me if I'm wrong:
class PreferencesController
def update
preference = User.find(params[:id]).preference
preference.update(params[:preference])
return json: preference.as_json
end
end
New to Rails and Ruby and trying to do things correctly.
Here are my models. Everything works fine, but I want to do things the "right" way so to speak.
I have an import process that takes a CSV and tries to either create a new record or update an existing one.
So the process is 1.) parse csv row 2.) find or create record 3.) save record
I have this working perfectly, but the code seems like it could be improved. If ParcelType wasn't involved it would be fine, since I'm creating/retrieving a parcel FROM the Manufacturer, that foreign key is pre-populated for me. But the ParcelType isn't. Anyway to have both Type and Manufacturer pre-populated since I'm using them both in the search?
CSV row can have multiple manufacturers per row (results in 2 almost identical rows, just with diff mfr_id) so that's what the .each is about
manufacturer_id.split(";").each do |mfr_string|
mfr = Manufacturer.find_by_name(mfr_string)
# If it's a mfr we don't care about, don't put it in the db
next if mfr.nil?
# Unique parcel is defined by it's manufacturer, it's type, it's model number, and it's reference_number
parcel = mfr.parcels.of_type('FR').find_or_initialize_by_model_number_and_reference_number(attributes[:model_number], attributes[:reference_number])
parcel.assign_attributes(attributes)
# this line in particular is a bummer. if it finds a parcel and I'm updating, this line is superfulous, only necessary when it's a new parcel
parcel.parcel_type = ParcelType.find_by_code('FR')
parcel.save!
end
class Parcel < ActiveRecord::Base
belongs_to :parcel_type
belongs_to :manufacturer
def self.of_type(type)
joins(:parcel_type).where(:parcel_types => {:code => type.upcase}).readonly(false) unless type.nil?
end
end
class Manufacturer < ActiveRecord::Base
has_many :parcels
end
class ParcelType < ActiveRecord::Base
has_many :parcels
end
It sounds like the new_record? method is what you're looking for.
new_record?() public
Returns true if this object hasn’t been saved yet — that is, a record
for the object doesn’t exist yet; otherwise, returns false.
The following will only execute if the parcel object is indeed a new record:
parcel.parcel_type = ParcelType.find_by_code('FR') if parcel.new_record?
What about 'find_or_create'?
I have wanted to use this from a long time, check these links.
Usage:
http://rubyquicktips.com/post/344181578/find-or-create-an-object-in-one-command
Several attributes:
Rails find_or_create by more than one attribute?
Extra:
How can I pass multiple attributes to find_or_create_by in Rails 3?
In my application I have the following relationship:
Document has_and_belongs_to_many Users
User has_and_belongs_to_many Documents
What I am trying to figure out is how to perform the following:
Let's say a document has 3 users which belong to it. If after an update they become for ex. 4, I would like to send an email
message (document_updated) to the first 3 and a different email message (document_assigned) to the 4th.
So I have to know the users belonging to my Document BEFORE and AFTER the Document update occurs.
My approach so far has been to create an Observer like this:
class DocumentObserver < ActiveRecord::Observer
def after_update(document)
# this works because of ActiveModel::Dirty
# #old_subject=document.subject_was #subject is a Document attribute (string)
# this is not working - I get an 'undefined method' error
#old_users=document.users_was
#new_users=document.users.all.dup
# perform calculations to find out who the new users are and send emails....
end
end
I knew that my chances of #old_users taking a valid value were slim because I guess it is populated dynamically by rails via the has_and_belongs_to_many relation.
So my question is:
How do I get all my related users before an update occurs?
(some other things I've tried so far:)
A. Obtaining document.users.all inside DocumentController::edit. This returns a valid array, however I do not know how to pass this array to
DocumentObserver.after_update in order to perform the calculations (just setting an instance variable inside DocumentController is of course not working)
B. Trying to save document.users inside DocumentObserver::before_update. This is not working either. I still get the new user values
Thanks in advance
George
Ruby 1.9.2p320
Rails 3.1.0
You could use a before_add callback
class Document
has_and_belongs_to_many :users, :before_add => :do_stuff
def do_stuff(user)
end
end
When you add a user to a document that callback will be called and at that point self.users will still it yet contain the user you are adding.
If you need something more complicated it might be simpler to have a set_users method on document
def set_users(new_user_set)
existing = users
new_users = users - new_user_set
# send your emails
self.users = new_user_set
end
Hey,
Not a Rails noob but this has stumped me.
With has many through associations in Rails. When I mass assign wines to a winebar through a winelist association (or through) table with something like this.
class WineBarController
def update
#winebar = WineBar.find(params[:id])
#winebar.wines = Wine.find(params[:wine_bar][:wine_ids].split(",")) // Mass assign wines.
render (#winebar.update_attributes(params[:wine_bar]) ? :update_success : :update_failure)
end
end
This will delete every winelist row associated with that winebar. Then it finds all of the wines in wine_ids, which we presume is a comma separated string of wine ids. Then it inserts back into the winelist a new association. This would be expensive, but fine if the destroyed association rows didn't have metadata such as the individual wine bar's price per glass and bottle.
Is there a way to have it not blow everything away, just do an enumerable comparison of the arrays and insert delete whatever changes. I feel like that's something rails does and I'm just missing something obvious.
Thanks.
Your problem looks like it's with your first statement in the update method - you're creating a new wine bar record, instead of loading an existing record and updating it. That's why when you examine the record, there's nothing showing of the relationship. Rails is smart enough not to drop/create every record on the list, so don't worry about that.
If you're using the standard rails setup for your forms:
<% form_for #wine_bar do |f| %>
Then you can call your update like this:
class WineBarController
def update
#winebar = WineBar.find(params[:id])
render (#winebar.update_attributes(params[:wine_bar]) ? :update_success : :update_failure)
end
end
You don't need to explicitly update your record with params[:wine_bar][:wine_ids], because when you updated it with params[:wine_bar], the wine_ids were included as part of that. I hope this helps!
UPDATE: You mentioned that this doesn't work because of how the forms are setup, but you can fix it easily. In your form, you'll want to rename the input field from wine_bar[wine_ids] to wine_bar[wine_ids_string]. Then you just need to create the accessors in your model, like so:
class WineBar < ActiveRecord::Base
def wine_ids_string
wines.map(&:id).join(',')
end
def wine_ids_string= id_string
self.wine_ids = id_string.split(/,/)
end
end
The first method above is the "getter" - it takes the list of associated wine ids and converts them to a string that the form can use. The next method is the "setter", and it accepts a comma-delimited string of ids, and breaks it up into the array that wine_ids= accepts.
You might also be interested in my article Dynamic Form Elements in Rails, which outlines how rails form inputs aren't limited to the attributes in the database record. Any pair of accessor methods can be used.
In one of my model objects I have an array of objects.
In the view I created a simple form to add additional objects to the array via a selection box.
In the controller I use the append method to add user selected objects to the array:
def add_adjacents
#site = Site.find(params[:id])
if request.post?
#site.adjacents << Site.find(params[:adjacents])
redirect_to :back
end
end
I added a validation to the model to validate_the uniqueness_of :neighbors but using the append method appears to be bypassing the validation.
Is there a way to force the validation? Or a more appropriate way to add an element to the array so that the validation occurs? Been googling all over for this and going over the books, but can't find anything on this.
Have you tried checking the validity afterwards by calling the ".valid?" method, as shown below?
def add_adjacents
#site = Site.find(params[:id])
#site.neighbors << Site.find(params[:neighbors])
unless #site.valid?
#it's not valid, do something to fix it!
end
end
A couple of comments:
Then only way to guarantee uniqueness is to add a unique constraint on your database. validates_uniqueness_of has it's gotchas when there are many users in the system:
Process 1 checks uniqueness, returns true.
Process 2 checks uniqueness, returns true.
Process 1 saves.
Process 2 saves.
You're in trouble.
Why do you have to test for request.post?? This should be handled by your routes, so in my view it's logic that is fattening your controller unnecessarily. I'd imagine something like the following in config/routes.rb: map.resources :sites, :member => { :add_adjacents => :post }
Need to know more about your associations to figure out how validates_uniqueness_of should play in with this setup...
I think you're looking for this:
#site.adjacents.build params[:adjacents]
the build method will accept an array of attribute hashes. These will be validated along with the parent model at save time.
Since you're validating_uniqueness_of, you might get some weirdness when you are saving multiple conflicting records at the same time, depending on the rails implementation for the save and validation phases of the association.
A hacky workaround would be to unique your params when they come in the door, like so:
#site.adjacents.build params[:adjacents].inject([]) do |okay_group, candidate|
if okay_group.all? { |item| item[:neighbor_id] != candidate[:neighbor_id] }
okay_group << candidate
end
okay_group
end
For extra credit you can factor this operation back into the model.