I want to loop over a csv file using CSV.foreach, read the data, perform some operation with it, and write the result to the last column of that row, using the Row object.
So let's say I have a csv with data I need to save to a database using Rails ActiveRecord, I validate the record, if it is valid, I write true in the last column, if not I write the errors.
Example csv:
id,title
1,some title
2,another title
3,yet another title
CSV.foreach(path, "r+", headers: true) do |row|
archive = Archive.new(
title: row["title"]
)
archive.save!
row["valid"] = true
rescue ActiveRecord::RecordInvalid => e
row["valid"] = archive.errors.full_messages.join(";")
end
When I run the code it reads the data, but it does not write anything to the csv. Is this possible?
Is it possible to write in the same csv file?
Using:
Ruby 3.0.4
The row variable in your iterator exists only in memory. You need to write the information back to the file like this:
new_csv = ["id,title,valid\n"]
CSV.foreach(path, 'r+', headers: true) do |row| # error here, see edit note below
row["valid"] = 'foo'
new_csv << row.to_s
end
File.open(path, 'w+') do |f|
f.write new_csv
end
[EDIT] the 'r+' option to foreach is not valid, it should be 'r'
Maybe this is over-engineering things a bit. But I would do the following:
Read the original CSV file.
Create a temporary CSV file.
Insert the updated headers into the temporary CSV file.
Insert the updated records into the temporary CSV file.
Replace the original CSV file with the temporary CSV file.
csv_path = 'archives.csv'
input_csv = CSV.read(csv_path, headers: true)
input_headers = input_csv.headers
# using an UUID to prevent file conflicts
tmp_csv_path = "#{csv_path}.#{SecureRandom.uuid}.tmp"
output_headers = input_headers + %w[errors]
CSV.open(tmp_csv_path, 'w', write_headers: true, headers: output_headers) do |output_csv|
input_csv.each do |archive_data|
values = archive_data.values_at(*input_headers)
archive = Archive.new(archive_data.to_h)
archive.valid?
# error_messages is an empty string if there are no errors
error_messages = archive.errors.full_messages.join(';')
output_csv << values + [error_messages]
end
end
FileUtils.move(tmp_csv_path, csv_path)
Related
I am trying to read a form-uploaded .csv file. I am taking my answers in part from several answers: In Ruby, how to read data column wise from a CSV file?, how to read a User uploaded file, without saving it to the database, and Rails - Can't import data from CSV file. But so far nothing has worked.
Here is my code:
def upload_file
file = Tempfile.new(params[:search_file])
csv_text = File.read(file)
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
puts row
end
render json: {success: true}
end
I am sure that the file is not nil. It contains 4 columns and 2 rows of simple text. However, my file value above comes out as an empty array, and the csv_text value is an empty string. I am very sure the file contains values.
I have also tried params[:search_field].read and that throws an error every time, saying "undefined method 'read'".
How can I simply read these values from the user uploaded file? I am on rails 5.1.6 and ruby 2.3.
Edit:
I have tried some of the solutions below. However, the problem is that it doesn't write the contents of the file, when I call file.write--it simply writes the name of the file (like, myFileNameHere.csv) as a string to the temp file. The "ok testing now" never prints to terminal in the below code. Here is my code now:
file = Tempfile.new(['hello', '.csv'])
file.write(params[:search_file])
file.rewind
csv_text = file.read
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
puts "ok testing row"
puts row
end
file.close
file.unlink # deletes the temp file
You are reading from a empty tempfile. When you put params[:search_file], this value will become part of the new Tempfile filename (like this "/tmp/#{params[:search_file]}.24722.0").
So when you do File.read(file) it will try to read a tempfile that has params[:search_file] value in the it's filename but has no other value from the form inside it.
You should either skip the Tempfile part and load the params[:search_file] with File.read(params[:search_file]) or fill the new Tempfile object with params[:search_file] content. (I would recommend the first).
Tempfile.new('something') always returns an empty temporary file with 'something' in its basename.
First you create the tempfile (with the filename you want), then you can write the content from params[:search_file], rewind and read it.
Source : Class: Tempfile (Ruby 1.9.3)
I want to export some ActiveRecords in CSV format. After check some tutorials, I found this:
def export_as_csv(equipments)
attributes = %w[id title description category_id]
CSV.generate(headers: true) do |csv|
csv << attributes
equipments.each do |equipment|
csv << equipment.attributes.values_at(*attributes)
end
return csv
end
end
The problem is, I want to manipulate all in memory in my tests(i.e. I don't want to save the file in the disk). So, when I receive this csv object as return value, how I can iterate through rows and columns? I came from Python and so I tried:
csv = exporter.export_as_csv(equipments)
for row in csv:
foo(row)
But obviously didn't work. Also, the equipments are surely not nil.
CSV.generate returns string formatted according csv rules.
So the most obvious way is to parse it and iterate, like:
csv = exporter.expor_as_csv(equipments)
CSV.parse(csv).each do |line|
# line => ['a', 'b', 'c']
end
After some videos, I found that the return was the problem. Returning the CSV I was receiving a CSV object, and not the CSV itself.
I am new to RoR.
I want to dynamically add attributes from a csv file so that my code would be able to dynamically read any csv file and build the db (i.e. convert any CSV file into Ruby objects)
I was using the below code
csv_data = File.read('myData.csv')
csv = CSV.parse(csv_data, :headers => true, :header_converters => :symbol)
csv.each do |row|
MyModel.create!(row.to_hash)
end
However it will fail for the following example
myData.csv
Name,id
foo,1
bar,10
myData2.csv
Name,value
foo,1
bar,10
It will result an error for myData2 because the value is not a parameter in MyModel
unknown attribute 'value' for MyModel.
I have thought about using send(:attrAccessor, name) but I was not sure how can I integrate it when reading from csv, any ideas ?
You are doing it properly but you can also bulk upload the records
csv_data =
CSV.read("#{Rails.root}/myData.csv",
headers: true,
header_converters: :symbol
).map(&:to_hash)
MyModel.create(csv_data)
NOTE: If the data is going to be same you can use seeds.rb
I have already CSV file, the content like
a1 a2 a3
1 2 3
4 5 6
5 8 2
Now, What I want, when I read any row i want to add a flag in the csv file like
a1 a2 a3 flag
1 2 3 1
4 5 6 1
5 8 2
the above flag 1 that means this record is inserted in the table.
so How can I add flag in the csv file?
Thanks In Advance
I came up with two ways to append a column(s) to an existing CSV file.
Method 1 late merges the new column by reading the file into an array of hashes, then appending the columns to the end of each row. This method can exhibit anomalies if run multiple times.
require 'csv'
filename = 'test.csv'
# Load the original CSV file
rows = CSV.read(filename, headers: true).collect do |row|
row.to_hash
end
# Original CSV column headers
column_names = rows.first.keys
# Array of the new column headers
additional_column_names = ['flag']
# Append new column name(s)
column_names += additional_column_names
s = CSV.generate do |csv|
csv << column_names
rows.each do |row|
# Original CSV values
values = row.values
# Array of the new column(s) of data to be appended to row
additional_values_for_row = ['1']
values += additional_values_for_row
csv << values
end
end
# Overwrite csv file
File.open(filename, 'w') { |file| file.write(s) }
Method 2 early merges the new column(s) into the row hash. The nicety of this method is it is more compact and avoids duplicate column names if run more than once. This method can also be used to change any existing values in the CSV.
require 'csv'
filename = 'test.csv'
# Load the original CSV file
rows = CSV.read(filename, headers: true).collect do |row|
hash = row.to_hash
# Merge additional data as a hash.
hash.merge('flag' => '0')
# BONUS: Change any existing data here too!
hash.merge('a1' => hash['a1'].to_i + 1 )
end
# Extract column names from first row of data
column_names = rows.first.keys
txt = CSV.generate do |csv|
csv << column_names
rows.each do |row|
# Extract values for row of data
csv << row.values
end
end
# Overwrite csv file
File.open(filename, 'w') { |file| file.write(txt) }
You need to write new CSV file with additional column, and then replace original file with new one.
Not sure if you can append a new column in the same file, but you can append a new row into your csv:
CSV.open('your_csv.csv', 'w') do |csv|
customers.array.each do |row|
csv << row
end
end
Hope this helps.
I have a csv file with dump data of table and I would like to import it directly into my database using rails.
I am currently having this code:
csv_text = File.read("public/csv_fetch/#{model.table_name}.csv")
ActiveRecord::Base.connection.execute("TRUNCATE TABLE #{model.table_name}")
puts "\nUpdating table #{model.table_name}"
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
row = row.to_hash.with_indifferent_access
ActiveRecord::Base.record_timestamps = false
model.create!(row.to_hash.symbolize_keys)
end
with help from here..
Consider my Sample csv:
id,code,created_at,updated_at,hashcode
10,00001,2012-04-12 06:07:26,2012-04-12 06:07:26,
2,00002,0000-00-00 00:00:00,0000-00-00 00:00:00,temphashcode
13,00007,0000-00-00 00:00:00,0000-00-00 00:00:00,temphashcode
43,00011,0000-00-00 00:00:00,0000-00-00 00:00:00,temphashcode
5,00012,0000-00-00 00:00:00,0000-00-00 00:00:00,temphashcode
But problem with this code is :
It is generating `id' as autoincrement 1,2,3,.. instead of what in
csv file.
The timestamps for records where there is 0000-00-00 00:00:00 defaults to null automatically and throws error as the column created_at cannot be null...
Is there any way I can do it in generic way to import from csv to models?
or would i have to write custom code for each model to manipulate the attributes in each row manually??
for question1, I suggest you output the row.to_hash.symbolize_keys, e.g.
# ...
csv.each do |row|
#...
hash = row.to_hash.symbolize_keys
Rails.logger.info "hash: #{hash.inspect}"
model.create!(hash)
end
to see if the "id" is assigned.
for Question2, I don't think it's a good idea to store "0000-00-00" instead of nil for the date.
providing fields like 'id' and for timestamps fields too manually solved it...
model.id = row[:id]
and similar for created_at,updated_at if these exists in model..