I am new to RoR.
I want to dynamically add attributes from a csv file so that my code would be able to dynamically read any csv file and build the db (i.e. convert any CSV file into Ruby objects)
I was using the below code
csv_data = File.read('myData.csv')
csv = CSV.parse(csv_data, :headers => true, :header_converters => :symbol)
csv.each do |row|
MyModel.create!(row.to_hash)
end
However it will fail for the following example
myData.csv
Name,id
foo,1
bar,10
myData2.csv
Name,value
foo,1
bar,10
It will result an error for myData2 because the value is not a parameter in MyModel
unknown attribute 'value' for MyModel.
I have thought about using send(:attrAccessor, name) but I was not sure how can I integrate it when reading from csv, any ideas ?
You are doing it properly but you can also bulk upload the records
csv_data =
CSV.read("#{Rails.root}/myData.csv",
headers: true,
header_converters: :symbol
).map(&:to_hash)
MyModel.create(csv_data)
NOTE: If the data is going to be same you can use seeds.rb
Related
I want to loop over a csv file using CSV.foreach, read the data, perform some operation with it, and write the result to the last column of that row, using the Row object.
So let's say I have a csv with data I need to save to a database using Rails ActiveRecord, I validate the record, if it is valid, I write true in the last column, if not I write the errors.
Example csv:
id,title
1,some title
2,another title
3,yet another title
CSV.foreach(path, "r+", headers: true) do |row|
archive = Archive.new(
title: row["title"]
)
archive.save!
row["valid"] = true
rescue ActiveRecord::RecordInvalid => e
row["valid"] = archive.errors.full_messages.join(";")
end
When I run the code it reads the data, but it does not write anything to the csv. Is this possible?
Is it possible to write in the same csv file?
Using:
Ruby 3.0.4
The row variable in your iterator exists only in memory. You need to write the information back to the file like this:
new_csv = ["id,title,valid\n"]
CSV.foreach(path, 'r+', headers: true) do |row| # error here, see edit note below
row["valid"] = 'foo'
new_csv << row.to_s
end
File.open(path, 'w+') do |f|
f.write new_csv
end
[EDIT] the 'r+' option to foreach is not valid, it should be 'r'
Maybe this is over-engineering things a bit. But I would do the following:
Read the original CSV file.
Create a temporary CSV file.
Insert the updated headers into the temporary CSV file.
Insert the updated records into the temporary CSV file.
Replace the original CSV file with the temporary CSV file.
csv_path = 'archives.csv'
input_csv = CSV.read(csv_path, headers: true)
input_headers = input_csv.headers
# using an UUID to prevent file conflicts
tmp_csv_path = "#{csv_path}.#{SecureRandom.uuid}.tmp"
output_headers = input_headers + %w[errors]
CSV.open(tmp_csv_path, 'w', write_headers: true, headers: output_headers) do |output_csv|
input_csv.each do |archive_data|
values = archive_data.values_at(*input_headers)
archive = Archive.new(archive_data.to_h)
archive.valid?
# error_messages is an empty string if there are no errors
error_messages = archive.errors.full_messages.join(';')
output_csv << values + [error_messages]
end
end
FileUtils.move(tmp_csv_path, csv_path)
I am trying to read a form-uploaded .csv file. I am taking my answers in part from several answers: In Ruby, how to read data column wise from a CSV file?, how to read a User uploaded file, without saving it to the database, and Rails - Can't import data from CSV file. But so far nothing has worked.
Here is my code:
def upload_file
file = Tempfile.new(params[:search_file])
csv_text = File.read(file)
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
puts row
end
render json: {success: true}
end
I am sure that the file is not nil. It contains 4 columns and 2 rows of simple text. However, my file value above comes out as an empty array, and the csv_text value is an empty string. I am very sure the file contains values.
I have also tried params[:search_field].read and that throws an error every time, saying "undefined method 'read'".
How can I simply read these values from the user uploaded file? I am on rails 5.1.6 and ruby 2.3.
Edit:
I have tried some of the solutions below. However, the problem is that it doesn't write the contents of the file, when I call file.write--it simply writes the name of the file (like, myFileNameHere.csv) as a string to the temp file. The "ok testing now" never prints to terminal in the below code. Here is my code now:
file = Tempfile.new(['hello', '.csv'])
file.write(params[:search_file])
file.rewind
csv_text = file.read
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
puts "ok testing row"
puts row
end
file.close
file.unlink # deletes the temp file
You are reading from a empty tempfile. When you put params[:search_file], this value will become part of the new Tempfile filename (like this "/tmp/#{params[:search_file]}.24722.0").
So when you do File.read(file) it will try to read a tempfile that has params[:search_file] value in the it's filename but has no other value from the form inside it.
You should either skip the Tempfile part and load the params[:search_file] with File.read(params[:search_file]) or fill the new Tempfile object with params[:search_file] content. (I would recommend the first).
Tempfile.new('something') always returns an empty temporary file with 'something' in its basename.
First you create the tempfile (with the filename you want), then you can write the content from params[:search_file], rewind and read it.
Source : Class: Tempfile (Ruby 1.9.3)
I have the following two lines of a code that take an uploaded CSV file from params and return a hash of Contact objects. The code works fine when I input a CSV with UTF-8 encoding. If I try to upload a CSV with another type of encoding though, it breaks. How can I adjust the code to detect the encoding of the uploaded file and convert to UTF-8?
CSV::Converters[:blank_to_nil] = lambda { |field| field && field.empty? ? nil : field }
csv = CSV.new(params[:file].tempfile.open, headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
This question is not a duplicate! I've seen numerous other questions on here revolving around the same encoding issue, but the specifics of those are different than my case. Specifically, I need a way convert the encoding of a TempFile generated from my params hash. Other solutions I've seen involve encoding String and File objects, as well as passing an encoding option to CSV.parse or CSV.open. I've tried those solutions already without success.
I've tried passing in an encoding option to CSV.new, like so:
csv = CSV.new(params[:file].tempfile.open, encoding: 'iso-8859-1:utf-8', headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
I've also tried this:
csv = CSV.new(params[:file].tempfile.open, encoding: 'iso-8859-1:utf-8', headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
I've tried adjusting my converter as well, like so:
CSV::Converters[:blank_to_nil] = lambda { |field| field && field.empty? ? nil : field.encode('utf-8') }
I'm looking for a programatic solution here that does not require the user to convert their CSV to the proper encoding.
I've also had to deal with this problem and here is how I finally solved it.
CSV.open(new_csv_file, 'w') do |csv_object|
lines = File.open(uploaded_file).read
lines.each_line do |line|
csv_object << line.encode!("utf-8", "utf-8", invalid: :replace, undef: :replace, replace: '').parse_csv
end
end
CSV.new(File.read(new_csv_file))
Basically go through every line, sanitize it and shove it into a new CSV file.
Hope that leads you and other in the right direction.
You can use filemagic to detect the encoding of a file, although it's not 100% accurate. It bases on system's file command tool, so I'm not sure if it works on windows.
I need to validate headers in a CSV file before parsing data in it.
# convert the data into an array of hashes
CSV::Converters[:blank_to_nil] = lambda do |field|
field && field.empty? ? nil : field
end
csv = CSV.new(file, :headers => true, :header_converters => :symbol, :converters => [:all, :blank_to_nil])
csv_data = csv.to_a.map {|row| row.to_hash }
I know I can use headers method to get the headers
headers = csv.headers
But the problem with headers method is it "Returns nil if headers will not be used, true if they will but have not yet been read, or the actual headers after they have been read."
So if I put headers = csv.headers above csv_data = csv.to_a.map {|row| row.to_hash } line headers is true and if I put it after reading data, headers contain headers row in an array. It imposes an order of instructions on my method which is very hard to test and is bad programming.
Is there a way to read headers row without imposing order in this scenario? I'm using ruby 2.0.
CSV.open(file_path, &:readline)
I get the problem! I'm having the same one. Calling read seems to do what you want (populates the headers variable):
data = CSV.new(file, **flags)
data.headers # => true
data = CSV.new(file, **flags).read
data.headers # => ['field1', 'field2']
There might be other side effects I'm not aware of, but this works for me and doesn't smell too bad.
I don't quite get the problem. If you use one of the iterator methods, it's quite easy to do some validation on the headers:
CSV.foreach('tmp.txt', headers: true) do |csv|
return unless csv.headers[0] != 'xyz'
end
I have a csv file with dump data of table and I would like to import it directly into my database using rails.
I am currently having this code:
csv_text = File.read("public/csv_fetch/#{model.table_name}.csv")
ActiveRecord::Base.connection.execute("TRUNCATE TABLE #{model.table_name}")
puts "\nUpdating table #{model.table_name}"
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
row = row.to_hash.with_indifferent_access
ActiveRecord::Base.record_timestamps = false
model.create!(row.to_hash.symbolize_keys)
end
with help from here..
Consider my Sample csv:
id,code,created_at,updated_at,hashcode
10,00001,2012-04-12 06:07:26,2012-04-12 06:07:26,
2,00002,0000-00-00 00:00:00,0000-00-00 00:00:00,temphashcode
13,00007,0000-00-00 00:00:00,0000-00-00 00:00:00,temphashcode
43,00011,0000-00-00 00:00:00,0000-00-00 00:00:00,temphashcode
5,00012,0000-00-00 00:00:00,0000-00-00 00:00:00,temphashcode
But problem with this code is :
It is generating `id' as autoincrement 1,2,3,.. instead of what in
csv file.
The timestamps for records where there is 0000-00-00 00:00:00 defaults to null automatically and throws error as the column created_at cannot be null...
Is there any way I can do it in generic way to import from csv to models?
or would i have to write custom code for each model to manipulate the attributes in each row manually??
for question1, I suggest you output the row.to_hash.symbolize_keys, e.g.
# ...
csv.each do |row|
#...
hash = row.to_hash.symbolize_keys
Rails.logger.info "hash: #{hash.inspect}"
model.create!(hash)
end
to see if the "id" is assigned.
for Question2, I don't think it's a good idea to store "0000-00-00" instead of nil for the date.
providing fields like 'id' and for timestamps fields too manually solved it...
model.id = row[:id]
and similar for created_at,updated_at if these exists in model..