I have a page where a user can import data to the site. either in the form of copy and pasting into a text area from excel, or by uploading a .csv file.
The controller checks if a csv has been uploaded - if so it processes this, else it will process the pasted content. (working on the assumption the user will only choose one option for now).
The copy and paste part works perfectly, however, the problem arises when I try to process the uploaded csv file:
I get the error:
can't convert
ActionController::UploadedTempfile
into String
#events_controller
def invite_save
#event = Event.find(params[:id])
if params[:guest_list_csv]
lines = parse_csv_file(params[:guest_list_csv])
else
#csv file uploaded
lines = params[:guest_list_paste]
end
if lines.size > 0
lines.each do |line|
new_user(line.split)
end
flash[:notice] = "List processing was successful."
else
flash[:error] = "List data processing failed."
end
end
private
def parse_csv_file(path_to_csv)
lines = []
require 'fastercsv'
FasterCSV.foreach(path_to_csv) do |row|
lines << row
end
lines
end
def new_user(line)
#code to create new user would go here
end
I'm essentially trying to upload and process the csv in one smooth action, rather than have to get the user to press a "process" button.
On the line #6 above
if params[:guest_list_csv]
lines = parse_csv_file(params[:guest_list_csv])
else
#csv file uploaded
lines = params[:guest_list_paste]
end
The problem is params[:guest_list_csv] is not the actual string, neither is the path, since it's a file object. What you need is explicitly call #path on it.
# line 6
lines = parse_csv_file(params[:guest_list_csv].path)
Please try it and see if it fixes your problem.
Related
I have a doubt about showing a generated CSV file to the user (with a large amount of data). So here is the task I have to do.
App: I have a film that has many characters.
Task:
allow users to upload characters via CSV (ok, done)
if there are errors, show them for each row (ok, done)
in the results page, also show a link to a new CSV file only with the remaining characters - the ones that couldn’t be created (I’m stuck here)
Here is part of my code (upload method):
def upload
saved_characters = []
characters_with_errors = []
errors = {}
begin
CSV.parse(params[:csv].read, **csv_options) do |row|
row_hash = clear_input(row.to_h)
new_character = Character.new(row_hash)
if new_character.save
add_images_to(new_character, row)
saved_characters << new_character
else
characters_with_errors << new_character
errors[new_character.name] = new_character.errors.full_messages.join(', ')
end
end
rescue CSV::MalformedCSVError => e
errors = { 'General error': e.message }.merge(errors)
end
#upload = {
errors: errors,
characters: saved_characters,
characters_with_errors: characters_with_errors
}
end
The issue: large amount of data
In the end, the upload.html.erb almost everything works fine, it shows the results and errors per column BUT I’m not sure how create a link on this page to send the user to the new CSV file (only with characters with errors). If the link sends the user to another method / GET endpoint (for the view with CSV format), how can I send such a large amount of data (params won’t work because they will get too long)? What would be the best practice here?
You can use a session variable to store the data, and then redirect to a new action to download the file. In the new action, you can get the data from the session variable, and then generate the CSV file.
For example, In the upload action, you can do something like this:
session[:characters_with_errors] = characters_with_errors
redirect_to download_csv_path
In the download_csv action, you can do something like this:
characters_with_errors = session[:characters_with_errors]
session[:characters_with_errors] = nil
respond_to do |format|
format.csv { send_data generate_csv(characters_with_errors) }
end
In the generate_csv method, you can do something like this:
def generate_csv(characters_with_errors)
CSV.generate do |csv|
csv << ['name', 'age' ]
characters_with_errors.each do |character|
csv << [character.name, character.age]
end
end
end
Another option, you can use a temporary file to store the data and then send the user to the new CSV file. Here is an example:
def upload
saved_characters = []
characters_with_errors = []
errors = {}
begin
CSV.parse(params[:csv].read, **csv_options) do |row|
row_hash = clear_input(row.to_h)
new_character = Character.new(row_hash)
if new_character.save
add_images_to(new_character, row)
saved_characters << new_character
else
characters_with_errors << new_character
errors[new_character.name] = new_character.errors.full_messages.join(', ')
end
end
rescue CSV::MalformedCSVError => e
errors = { 'General error': e.message }.merge(errors)
end
#upload = {
errors: errors,
characters: saved_characters,
characters_with_errors: characters_with_errors
}
respond_to do |format|
format.html
format.csv do
# Create a temporary file
tmp = Tempfile.new('characters_with_errors')
# Write the CSV data to the temporary file
tmp.write(characters_with_errors.to_csv)
# Send the user to the new CSV file
send_file tmp.path, filename: 'characters_with_errors.csv'
# Close the temporary file
tmp.close
end
end
end
I'm trying to attach a zipped CSV file to an e-mail without any joy. I've tried following http://api.rubyonrails.org/classes/ActiveSupport/Gzip.html:
class UserExportProcessor
#queue = :user_export_queue
def self.perform(person_id, collection_ids)
person = Person.unscoped.find(person_id)
collection = Person.unscoped.where(id: [49522, 70789])
file = ActiveSupport::Gzip.compress(collection.to_csv)
PersonMailer.people_export(person, file).deliver
end
end
This sends an attachment - still as a CSV file - filled with symbols (no letters or numbers).
When I try and remove the compression:
class UserExportProcessor
#queue = :user_export_queue
def self.perform(person_id, collection_ids)
person = Person.unscoped.find(person_id)
collection = Person.unscoped.where(id: [49522, 70789])
PersonMailer.people_export(person, collection.to_csv).deliver
end
end
The system e-mails the CSV file as it should and the CSV is properly formed. What am I doing wrong? Do I need to make a new file out of the compressed data? I've tried various approaches with no joy..
Thanks in advance
EDIT: I'm trying
class UserExportProcessor
require 'zip'
#queue = :user_export_queue
def self.perform(person_id, collection_ids)
person = Person.unscoped.find(person_id)
collection = Person.unscoped.where(id: [49522, 70789])
file = Zip::ZipFile.open("files.zip", Zip::ZipFile::CREATE) { |zipfile|
puts zipfile.get_output_stream(collection.to_csv)
zipfile.mkdir("a_dir")
end
PersonMailer.people_export(person, file).deliver
end
end
However this fails with:
Errno::ENAMETOOLONG: File name too long - /Users/mark/projects/bla/Role,Title,First Name,Last Name,Address 1,Address 2,Address 3,City,Postcode,Country,Email,Telephone,Mobile,Job Title,Company,Area of work,Department,Regions,Account Manager,Sales Coordinator,Production Studios,Production Partners,Genres,Last login,Created date
Is there any way for me to set the file name with the above approach?
The mistake probably happens in the code of PersonMailer.people_export (which you didn't include). So this is my best guess: you probably add the attachment and do not properly define the mime-type.
Make sure you set the correct file extension when you add the attachment:
http://edgeguides.rubyonrails.org/action_mailer_basics.html#adding-attachments
something along this line should work:
attachments['archive.zip'] = ActiveSupport::Gzip.compress(collection.to_csv)
I have the below resque job that produces a csv file and sends it to a mailer. I want to validate that the csv file has data so I do not email blank files. For some reason, when I write a method outside of the perform method, it will not work. For example, the below code will print invalid when I know the csv file has data on the first line. If I uncomment the line below ensure it works properly, however I want to extract this checking of the file into a separate method. Is this correct?
class ReportJob
#queue = :report_job
def self.perform(application_id, current_user_id)
user = User.find(current_user_id)
client_application = Application.find(client_application_id)
transactions = application.transactions
file = Tempfile.open(["#{Rails.root}/tmp/", ".csv"]) do |csv|
begin
csv_file = CSV.new(csv)
csv_file << ["Application", "Price", "Tax"]
transactions.each do |transaction|
csv_file << [application.name, transaction.price, transaction.tax]
end
ensure
ReportJob.email_report(user.email, csv_file)
#ReportMailer.send_report(user.email, csv_file).deliver
csv_file.close(unlink=true)
end
end
end
def self.email_report(email, csv)
array = csv.to_a
if array[1].blank?
puts "invalid"
else
ReportMailer.send_report(email, csv).deliver
end
end
end
You should invoke your method as such:
ReportJob.email_report(email, csv)
Otherwise, get rid of the self in:
def self.email_report(email, csv)
# your implementation here.
end
and define your method as follows:
def email_report(email, csv)
# your implementation.
end
This is something that we call Class Methods and Instance Methods.
There is CSV-export of some objects (such as tasks, contacts, etc) in my application. It just renders CSV-file like this:
respond_to do |format|
format.html
format.csv { render text: Task.to_csv } # I have self.to_csv def in model
end
It generates a CSV file when I go to '/tasks.csv' without a problem.
Now I want to export all the objects and zip them. I'm using rubyzip gem to create zip-files. Now my code for creating zip-file with all the CSVs looks like that:
Zip::ZipFile.open("#{path_to_file}.zip", Zip::ZipFile::CREATE) do |zipfile|
zipfile.file.open("tasks.csv", "w") { |f| f << open("http://#{request.host}:#{request.port.to_s}/tasks.csv").read }
# the same lines for contacts and other objects
end
But it seems that there is something wrong with it because it's executing for a long time (I'm getting Timeout::Error even if there is just one line in CSV) and the resulting zip-archive contains something broken.
How can I save my "/tasks.csv", "/contacts.csv", etc as a file on server (inside of zip-archive in this case)?
I did it! The code is:
Zip::ZipFile.open("#{path_to_file}.zip", Zip::ZipFile::CREATE) do |zipfile|
zipfile.file.open("tasks.csv", "w") do |f|
CSV.open(f, "w") do |csv|
CSV.parse(Task.to_csv) { |row| csv << row }
end
end
end
I get a CSV:MalFormedCSVError when I try to import a file using the following code:
def import_csv(filename, model)
CSV.foreach(filename, :headers => true) do |row|
item = {}
row.to_hash.each_pair do |k,v|
item.merge!({k.downcase => v})
end
model.create!(item)
end
end
The csv files are HUGE, so is there a way I can just log the bad formatted lines and CONTINUE EXECUTION with the remainder of the csv file?
You could try handling the file reading yourself and let CSV work on one line at a time. Something like this:
File.foreach(filename) do |line|
begin
CSV.parse(line) do |row|
# Do something with row...
end
rescue CSV::MalformedCSVError => e
# complain about line
end
end
You'd have to do something with the header line yourself of course. Also, this won't work if you have embedded newlines in your CSV.
One problem with using File to manually go through each line in the file is that CSV files can contain fields with \n (newline character) in them. File will take that to indicate a newline and you will end up trying to parse a partial row.
Here is an another approach that might work for you:
#csv = CSV.new('path/to/file.csv')
loop do
begin
row = #csv.shift
break unless row
# do stuff
rescue CSV::MalformedCSVError => error
# handle the error
next
end
end
The main downside that I see with this approach is that you don't have access to the CSV row string when handling the error, just the CSV::MalformedCSVError itself.