Having followed the RailsCast on importing CSV (http://railscasts.com/episodes/396-importing-csv-and-excel), I am trying to validate that the file being uploaded is a CSV file.
I have used the gem csv_validator to do so, as documented here https://github.com/mattfordham/csv_validator
And so my model looks like this:
class Contact < ActiveRecord::Base
belongs_to :user
attr_accessor :my_csv_file
validates :my_csv_file, :csv => true
def self.to_csv(options = {})
CSV.generate(options) do |csv|
csv << column_names
all.each do |contact|
csv << contact.attributes.values_at(*column_names)
end
end
end
def self.import(file, user)
allowed_attributes = ["firstname","surname","email","user_id","created_at","updated_at", "title"]
CSV.foreach(file.path, headers: true) do |row|
contact = find_by_email_and_user_id(row["email"], user) || new
contact.user_id = user
contact.attributes = row.to_hash.select { |k,v| allowed_attributes.include? k }
contact.save!
end
end
end
But my system still allows me to select to import non-CSV files (such as .xls), and I receive the resulting error: invalid byte sequence in UTF-8.
Can someone please tell me why and how to resolve this?
Please note that I am using Rails 4.2.6
You can create a new class, let's say ContactCsvRowValidator:
class ContactCsvRowValidator
def initialize(row)
#row = row.with_indifferent_access # allows you to use either row[:a] and row['a']
#errors = []
end
def validate_fields
if #row['firstname'].blank?
#errors << 'Firstname cannot be empty'
end
# etc.
end
def errors
#errors.join('. ')
end
end
And then use it like this:
# contact.rb
def self.import(file, user)
allowed_attributes = ["firstname","surname","email","user_id","created_at","updated_at", "title"]
if file.path.split('.').last.to_s.downcase != 'csv'
some_method_which_handle_the_fact_the_file_is_not_csv!
end
CSV.foreach(file.path, headers: true) do |row|
row_validator = ContactCsvRowValidator.new(row)
errors = row_validator.errors
if errors.present?
some_method_to_handle_invaid_row!(row)
return
end
# other logic
end
end
This pattern can easily be modified to fit your needs. For example, if you need to have import on several different models, you could create a base CsvRowValidator to provide basic methods such as validate_fields, initialize and errors. Then, you could create a class inheriting from this CsvRowValidator for each model you want, having its own validations implemented.
Related
Django admin shows you the dependent records that will be deleted when you delete a record as a confirmation.
Is there a way to do the same on Ruby on Rails?
I have been researching how to do it, but I am still looking for a way.
I couldn't find a gem, so I wrote this concern using association reflections:
module DependentDestroys
extend ActiveSupport::Concern
DEPENDENT_DESTROY_ACTIONS = %i[destroy delete destroy_async]
class_methods do
def dependent_destroy_reflections
#dependent_destroy_reflections ||= reflections.filter_map do |name, r|
r if DEPENDENT_DESTROY_ACTIONS.include?(r.options[:dependent])
end
end
end
def total_dependent_destroys
dependent_destroy_counts.sum { |r| r[1] }
end
def any_dependent_destroys?
dependent_destroy_counts.any?
end
# If you want all affected records...
def dependent_destroy_records
self.class.dependent_destroy_reflections.flat_map do |r|
relation = self.public_send(r.name)
if r.collection?
relation.find_each.to_a
else
relation
end
end
end
# If you only want the record type and ids...
def dependent_destroy_ids
self.class.dependent_destroy_reflections.flat_map do |r|
relation = self.public_send(r.name)
if r.collection?
relation.pluck(:id).map { |rid| [r.klass, rid] }
else
[[r.klass, relation.id]] if relation
end
end.compact
end
# If you only want counts...
def dependent_destroy_counts
self.class.dependent_destroy_reflections.filter_map do |r|
relation = self.public_send(r.name)
if r.collection?
c = relation.count
[r.klass, c] if c.positive?
else
[r.klass, 1] if relation
end
end
end
def dependent_destroy_total_message
"#{total_dependent_destroys} associated records will be destroyed"
end
def dependent_destroy_message
# Using #human means you can define model names in your translations.
"The following dependent records will be destroyed: #{dependent_destroy_ids.map { |r| "#{r[0].model_name.human}/#{r[1]}" }.join(', ')}"
end
def dependent_destroy_count_message
"The following dependent records will be destroyed: #{dependent_destroy_counts.map { |r| "#{r[0].model_name.human(count: r[1])} (#{r[1]})" }.join(', ')}"
end
end
Usage:
class User
include DependentDestroys
belongs_to :company
has_many :notes
has_one :profile
end
user = User.first
user.any_dependent_destroys?
# => true
user.total_dependent_destroys
# => 60
user.dependent_destroy_total_message
# => "60 associated records will be destroyed"
user.dependent_destroy_message
# => "The following dependent records will be destroyed: Note/1, Note/2, ..., Profile/1"
user.dependent_destroy_count_message
# => "The following dependent records will be destroyed: Notes (59), Profile (1)"
You can then use these methods in the controller to deal with the user flow.
With some improvements, options (like limiting it to the associations or modes (destroy, delete, destroy_async) you want) and tests, this could become a gem.
I am using roo-gem to import the following spreadsheet of authors designed as below:
I have tables in the database called authors, states, publishers and genres which the data in the respective excel columns map to. As you can see, the first row of headers is very detailed because the users uploading the data into the authors table are not very savvy. Because such long header names cannot map into the association columns in the table, I included a second row in the excel template and then added methods in the Author model as shown below:
class Author < ApplicationRecord
belongs_to :state
belongs_to :genre
belongs_to :publisher
#validations
validates :name, presence: true
validates :national_id, presence: true
def to_s
name
end
def home_state
state.try(:name)
end
def home_state=(name)
self.state = State.where(:name => name).first_or_create
end
def publisher_name
publisher.try(:name)
end
def publisher_name=(name)
self.publisher = Publisher.where(:name => name).first
end
def listed_genre
genre.try(:name)
end
def listed_genre=(name)
self.genre = Genre.where(:name => name).first
end
end
Here is authors_import.rb:
class AuthorsImport
include ActiveModel::Model
require 'roo'
attr_accessor :file
def initialize(attributes={})
attributes.each { |name, value| send("#{name}=", value) }
end
def persisted?
false
end
def open_spreadsheet
case File.extname(file.original_filename)
when ".csv" then Csv.new(file.path, nil, :ignore)
when ".xls" then Roo::Excel.new(file.path, nil, :ignore)
when ".xlsx" then Roo::Excelx.new(file.path)
else raise "Unknown file type: #{file.original_filename}"
end
end
def load_imported_authors
spreadsheet = open_spreadsheet
spreadsheet.default_sheet = 'Worksheet'
header = spreadsheet.row(2)
#header = header[0..25]
(3..spreadsheet.last_row).map do |i|
row = Hash[[header, spreadsheet.row(i)].transpose]
#author = Author.find_by_national_id(row["national_id"]) || Author.new
author = Author.find_by(national_id: row["national_id"], state_id: row["home_state"]) || Author.new
author.attributes = row.to_hash
author
end
end
def imported_authors
#imported_authors ||= load_imported_authors
end
def save
if imported_authors.map(&:valid?).all?
imported_authors.each(&:save!)
true
else
imported_authors.each_with_index do |author, index|
author.errors.full_messages.each do |msg|
errors.add :base, "Row #{index + 3}: #{msg}"
end
end
false
end
end
end
Here is my problem: when the users add any publisher or genre in the excel template that's not found in the respective publishers or genres table, that entry will be added into the tables, which I don't want. If list all the publishers or genres from the app, I find several entries which should not be there that have been created during import. The excel template has dropdowns of publishers and genres just as they appear in the corresponding tables in the database. One solution would be to lock the template, but that's not possible in the circumstances for now.
I thought first_or_create is the only method that will add a new record in case it is not found, but it seems even the first method is behaving the same here.
Is there some model validation or tweak in the method that will prevent unauthorized publishers and genres from being created?
Thanks
I need an example on how to import a CSV file with different attributes. So far when I run my code I get an error message saying unknown attribute 'email' for Student. The email attributes can coming from the Parent. Beside importing the attributes for students only and parent only. How can I import parent’s attributes within the student attributes?
My program work find when I exports my CSV with parents attributes in my student program.
Any help would be appreciated.
student model class:
class Student < ActiveRecord::Base
belongs_to :parent
delegate :email,:name,:phone_number,to: :parent, allow_nil: true
def self.to_csv
attributes = %w{parent_id email phone_number first_name last_name age workshop interest registration_date email }
CSV.generate(headers: true) do |csv|
csv << attributes
all.each do |script|
csv << attributes.map{ |attr| script.send(attr) }
end
end
end
def self.import(file)
spreadsheet = open_spreadsheet(file)
header=spreadsheet.row(1)
(2..spreadsheet.last_row).each do |i|
row = Hash[[header,spreadsheet.row(i)].transpose]
#CSV.foreach(file.path, headers: true) do |row|
student = find_by_id(row["id"])|| new
student.attributes = row.to_hash.slice(*row.to_hash.keys)
student.save!
end
end
def self.open_spreadsheet(file)
case File.extname(file.original_filename)
when ".csv" then Roo::CSV.new(file.path)
when ".xls" then Roo::CSV.new(file.path)
when ".xlsx" then Roo::CSV.new(file.path)
else raise "Unknown file type: #{file.original_filename}"
end
end
end
Student controller:
def import
Student.import(params[:file])
redirect_to root_url, notice: "student imported."
end
end
view folder:
<h2>Import Students</h2>
<%= form_tag import_students_path, multipart: true do %>
<%= file_field_tag :file %>
<%= submit_tag "Import" %>
<% end %>
First thing I want to advice you is 'keeping your model code thin'. The best practice is to keep your model to be responsible only for data otherwise it violets SRP.
class ImportStudentsService
SUPPORTED_TYPES = ['.csv', '.xls', '.xlsx'].freeze
STUDENT_ATTRIBUTES = %w(
parent_id first_name last_name
age workshop interest registration_date
).freeze
PARENT_ATTRIBUTES = %w(email phone_number email).freeze
def initialize(file)
#file = file
end
def call
import
end
private
def import
spreadsheet.each do |row|
student = Student.find_or_initialize_by(id: row['id'])
parent = student.parent || student.build_parent
student.attributes = row.slice(*STUDENT_ATTRIBUTES)
parent.attributes = row.slice(*PARENT_ATTRIBUTES)
student.save!
parent.save!
end
end
def spreadsheet
unless SUPPORTED_TYPES.include?(File.extname(#file.original_filename))
raise "Unknown file type: #{#file.original_filename}"
end
Roo::CSV.new(#file.path)
end
end
In controller
def import
ImportStudentsService.new(params[:file]).call
redirect_to root_url, notice: "student imported."
end
It's not 100% working code but the main idea is presented here.
I am trying to generate a CSV output with data from my database. I would like to provide these data to a third party, so I picture I would give to someone a URL (website.com/api_data/cars) and by accessing this URL the person would be able to work with it - I think I want to access the URL and then to see there (in the action) data displayed and separated by , or ;.
But how to do that?
So far, I am trying following approach:
csv_string = CSV.generate do |csv|
cols = ["column one", "column two", "column three"]
csv << cols
csv << ["A", "B", "C"]
#filename = "data-#{Time.now.to_date.to_s}.csv"
end
send_data(csv_string, :type => 'text/csv; charset=utf-8; header=present', :filename => #filename)
This is in the controller generate_data and action csv_cars.
When I run this action (webste.com/generate_data/csv_cars), it will automatically pop up a window to download the file.
But how to write the CSV content to the action? So when I open the URL, I'll see there written the content from the database?
I know this is an old thread but I came across it in my search so in case someone else does the same, here's my answer and what worked for me.
I think bhanu had a good way of going about it but I did change something. Instead of doing #cars within the respond_to, I just called send_data Cars.to_csv since, as Rob stated, it was made as a class method. It worked beautifully for me.
class Car < ActiveRecord::Base
def self.to_csv(make)
attributes = %w{id name price} #customize columns here
cars = Car.where(maker_name: make)
CSV.generate(headers: true) do |csv|
csv << attributes
cars.each do |car|
csv << attributes.map{ |attr| car.send(attr) }
end
end
end
end
And then in the controller
class CarsController < ApplicationController
def index
send_data Cars.to_csv('Chevy'), filename: "cars-#{Date.today}.csv"
end
end
I understand that this will be called when you go to cars/index but you can put that into any method, if statement or anything you want and just have it called whenever you would like from there. You can also have arguments, as I did above with make, and query for certain fields. It was definitely a lot easier than I thought it was going to be. Hope this helped someone.
You need to do something like this.
def csv_cars
headers = ['column one', 'column two', 'column three']
csv_data = CSV.generate(headers: true) do |csv|
csv << headers
csv << ["A", "B", "C"]
end
send_data csv_data, filename: "data-#{Date.today.to_s}.csv", disposition: :attachment
end
define a to_csv method in your model as shown below
class Car < ActiveRecord::Base
def self.to_csv
attributes = %w{id name price} #customize columns here
CSV.generate(headers: true) do |csv|
csv << attributes
all.each do |car|
csv << attributes.map{ |attr| car.send(attr) }
end
end
end
end
Later in your controller
class CarsController < ApplicationController
def index
#cars = Car.all
respond_to do |format|
format.html
format.csv { send_data #cars.to_csv, filename: "cars-#{Date.today}.csv" }
end
end
end
I'm creating my own gem and I want to enable user to save data to multiple NOSQL data stores. How can I make this happen? Where should I place the necessary files?
I've done the same thing in my gem. I think you have created an App folder in your gem/engine. Create another folder called "backend" and create classes for each datastore. For my case I created a seperate for Mongo and Redis
module Memberfier
class RedisStore
def initialize(redis)
#redis = redis
end
def keys
#redis.keys
end
def []=(key, value)
value = nil if value.blank?
#redis[key] = ActiveSupport::JSON.encode(value)
end
def [](key)
#redis[key]
end
def clear_database
#redis.keys.clone.each {|key| #redis.del key }
end
end
end
module Memberfier
class MongoStore
def initialize(collection)
#collection = collection
end
def keys
#collection.distinct :_id
end
def []=(key, value)
value = nil if value.blank?
collection.update({:_id => key},
{'$set' => {:value => ActiveSupport::JSON.encode(value)}},
{:upsert => true, :safe => true})
end
def [](key)
if document = collection.find_one(:_id => key)
document["value"]
else
nil
end
end
def destroy_entry(key)
#collection.remove({:_id => key})
end
def searchable?
true
end
def clear_database
collection.drop
end
private
def collection; #collection; end
end
end
You may have already seen one of Uncle Bob's presentations on application architecture. If not, it's here. I'd recommend having a single boundary object that select models inherit from. That boundary object could have multiple CRUD methods such as find, create, delete. That boundary object could inherit from whatever NOSQL adapter you configure. Example/source: http://hawkins.io/2014/01/pesistence_with_repository_and_query_patterns/