CSV Import issue - ruby-on-rails

I have this to parse a CSV file:
csv_file = CSV.parse(
file.read(),
headers: true,
header_converters: :symbol
)
It works, but I want to specify the encoding type so I add: {encoding: 'UTF-8'} into the read method as below:
csv_file = CSV.parse(
file.read({encoding: 'UTF-8'}),
headers: true,
header_converters: :symbol
)
But I get this error: can't convert Hash into Integer
I just can't see what is wrong. I have checked the docs but it says you can pass the encoding in like so but it does need the file as the first argument so it could be stalling there but surely as it already knows what file is is reading it should be ok.
How can I solve this?
Update:
I have updated to the following:
def import
if params[:import_coasters]
file = params[:import_coasters][:file]
Park.import_from_csv(file)
def self.import_from_csv(file)
Park.destroy_all
and
csv_file = CSV.parse(
File.read(file, {encoding: 'UTF-8'}),
headers: true,
header_converters: :symbol
)
But I get the following error:
can't convert ActionDispatch::Http::UploadedFile into String

You're close. Try this:
# Step 1: convert the uploaded file object to a file name
uploaded_file = params[:import_coasters][:file]
file_name = uploaded_file.path
# Step 2: To get the input text and see if it's what you expect
text = File.read(
file_name,
{encoding: 'UTF-8'}
)
# Parse the text
csv_file = CSV.parse(
text,
headers: true,
header_converters: :symbol
)
From the Ruby 1.9.3 docs on IO.read:
"If the last argument is a hash, it specifies option for internal open(). The key would be the following. open_args: is exclusive to others."
http://www.ruby-doc.org/core-1.9.3/IO.html#method-c-read
Also, look at the documentation for UploadedFile because it's not actually a typical Ruby File object:
http://api.rubyonrails.org/classes/ActionDispatch/Http/UploadedFile.html

It is because File.read() expect a filename as the first argument. You are passing ActionDispatch::Http::UploadedFile instead. If you look at the documentation, you'll see that UploadedFile already has a read method, so you can try:
csv_file = CSV.parse(
file.read({encoding: 'UTF-8'}),
headers: true,
header_converters: :symbol
)

Related

I can not get the exact number from a csv file if it starts with a zero in ruby

Descriptions
Read file csv in ruby.
I have a csv file with this content
longitude,latitude,phone
13,139.7113134,35.56712836,0311112222
I read file csv.
I get data is not expect at column phone number
Code
uploaded_io = params[:rooms][:file]
rooms_table = CSV.table(uploaded_io.tempfile, encoding: "UTF-8")
rooms_table.each_with_index do |row, i|
p row
end
puts row:
#<CSV::Row longitude:139.7113134 latitude:35.56712836 phone:52728978 >
I don't understand where is value phone number? I expect phone number is
0311112222 instead of 52728978
The reason this is happening is that, per the docs, CSV.table is:
A shortcut for:
CSV.read( path, { headers: true,
converters: :numeric,
header_converters: :symbol }.merge(options) )
Note converters: :numeric, which tells it to automatically (attempt to) convert numeric-looking fields to Numbers. Phone numbers, of course, aren't really numbers, but rather strings of digits.
If you don't want any converstions, you could pass converters: nil as an option to CSV.table.
Assuming you do want the :numeric converter to still operate on the other fields, though, you need to define your own converter. A converter is a Proc that takes two arguments: A field value and an (optional) FieldInfo object. Your converter might look like this:
NUMERIC_EXCEPT_PHONE_CONVERTER = lambda do |value, field_info|
if field_info.header == :phone
value
else
CSV::Converters[:float].call(
CSV::Converters[:integer].call(value))
end
end
Then you would use it by passing it to CSV.table as the converters: option, which will override the default converters: :numeric:
rooms_table = CSV.table("data.csv", encoding: "UTF-8", converters: NUMERIC_EXCEPT_PHONE_CONVERTER)
p rooms_table[0]
# => #<CSV::Row longitude:139.7113134 latitude:35.56712836 phone:"0311112222">
As you can see, the phone value is now a string with the leading 0.
You can see this code in action on repl.it: https://repl.it/#jrunning/WellmadeFarflungCron
Aside
Why, you might ask, is this bit so ugly?
CSV::Converters[:float].call(
CSV::Converters[:integer].call(value))
It's because the CSV module defines CSV::Converters thusly:
Converters = {
integer: lambda { |f|
Integer(f.encode(ConverterEncoding)) rescue f
},
float: lambda { |f|
Float(f.encode(ConverterEncoding)) rescue f
},
numeric: [:integer, :float],
# ...
}
Since the :numeric converter is not specified as a lambda, but rather an array that indicates that it's really just a "chain" of the :integer and :float converters, we can't just do CSV::Converters[:numeric].call(value); we have to call the two converters manually. (If anybody knows something I'm missing, please leave a comment.)
You can change:
rooms_table = CSV.table(uploaded_io.tempfile, encoding: "UTF-8")
to:
rooms_table = CSV.table(uploaded_io.tempfile, encoding: "UTF-8", converters: nil)
which will not convert/cast your fields (you will get strings). The default converter is :numeric which does this conversions that you don't want.
Possible converters that you can work with could be found here:
https://ruby-doc.org/stdlib-2.5.1/libdoc/csv/rdoc/CSV.html#Converters

CSV won't import by key in hash (Rails)

I'm having problems importing this CSV:
municipality,province,province abbrev,country,region
Vancouver,British Columbia,BC,Canada,Metro Vancouver - North
Specifically, Vancouver is not being returned when I look for its value by its key:
municipality_name = row["municipality"]
Here's the code:
def self.import_csv(file)
CSV.foreach(file, headers: true,
skip_blanks: true,
skip_lines: /^(?:,\s*)+$/,
col_sep: ",") do |row|
municipality_name = row["municipality"]
puts row.to_h
puts "municipality_name: #{municipality_name}"
puts "row[0]: #{row[0]}"
end
end
Here's the output:
irb(main):052:0> Importers::Municipalities.import_csv('tmp/municipalities.csv')
{"municipality"=>"Vancouver", "province"=>"British Columbia", "province abbrev"=>"BC", "country"=>"Canada", "region"=>"Metro Vancouver - North"}
municipality_name:
row['municipality']:
row[0]: Vancouver
Seems like I'm missing something obvious. I thought maybe there was a hidden character in the CSV but turned on hidden characters in Sublime and no dice.
Thanks in advance.
You need to call to_h on the row if you want to access it by its keys. Otherwise, it is an array-like object, accessible by indices.
def self.import_csv(file)
CSV.foreach(file, headers: true,
skip_blanks: true,
skip_lines: /^(?:,\s*)+$/,
col_sep: ",") do |row|
row = row.to_h
municipality_name = row["municipality"]
puts "municipality_name: #{municipality_name}"
end
end
Seems like it was a problem with the CSV and the code works fine. Created a new CSV, typed in the same content, and it worked. Maybe an invisible character that Sublime wasn't showing? Can't verify as I wiped the original CSV that was causing issues.

How to Dynamically add attributes from csv file

I am new to RoR.
I want to dynamically add attributes from a csv file so that my code would be able to dynamically read any csv file and build the db (i.e. convert any CSV file into Ruby objects)
I was using the below code
csv_data = File.read('myData.csv')
csv = CSV.parse(csv_data, :headers => true, :header_converters => :symbol)
csv.each do |row|
MyModel.create!(row.to_hash)
end
However it will fail for the following example
myData.csv
Name,id
foo,1
bar,10
myData2.csv
Name,value
foo,1
bar,10
It will result an error for myData2 because the value is not a parameter in MyModel
unknown attribute 'value' for MyModel.
I have thought about using send(:attrAccessor, name) but I was not sure how can I integrate it when reading from csv, any ideas ?
You are doing it properly but you can also bulk upload the records
csv_data =
CSV.read("#{Rails.root}/myData.csv",
headers: true,
header_converters: :symbol
).map(&:to_hash)
MyModel.create(csv_data)
NOTE: If the data is going to be same you can use seeds.rb

ArgumentError: invalid byte sequence in UTF-8 when creating CSV from TempFile

I have the following two lines of a code that take an uploaded CSV file from params and return a hash of Contact objects. The code works fine when I input a CSV with UTF-8 encoding. If I try to upload a CSV with another type of encoding though, it breaks. How can I adjust the code to detect the encoding of the uploaded file and convert to UTF-8?
CSV::Converters[:blank_to_nil] = lambda { |field| field && field.empty? ? nil : field }
csv = CSV.new(params[:file].tempfile.open, headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
This question is not a duplicate! I've seen numerous other questions on here revolving around the same encoding issue, but the specifics of those are different than my case. Specifically, I need a way convert the encoding of a TempFile generated from my params hash. Other solutions I've seen involve encoding String and File objects, as well as passing an encoding option to CSV.parse or CSV.open. I've tried those solutions already without success.
I've tried passing in an encoding option to CSV.new, like so:
csv = CSV.new(params[:file].tempfile.open, encoding: 'iso-8859-1:utf-8', headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
I've also tried this:
csv = CSV.new(params[:file].tempfile.open, encoding: 'iso-8859-1:utf-8', headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
I've tried adjusting my converter as well, like so:
CSV::Converters[:blank_to_nil] = lambda { |field| field && field.empty? ? nil : field.encode('utf-8') }
I'm looking for a programatic solution here that does not require the user to convert their CSV to the proper encoding.
I've also had to deal with this problem and here is how I finally solved it.
CSV.open(new_csv_file, 'w') do |csv_object|
lines = File.open(uploaded_file).read
lines.each_line do |line|
csv_object << line.encode!("utf-8", "utf-8", invalid: :replace, undef: :replace, replace: '').parse_csv
end
end
CSV.new(File.read(new_csv_file))
Basically go through every line, sanitize it and shove it into a new CSV file.
Hope that leads you and other in the right direction.
You can use filemagic to detect the encoding of a file, although it's not 100% accurate. It bases on system's file command tool, so I'm not sure if it works on windows.

Ruby CSV#fetch case sensitivness

I have simple csv file that has the following header: 'NYC'.
I use CSV fetch method:
http://ruby-doc.org/stdlib-2.0.0/libdoc/csv/rdoc/CSV/Row.html#method-i-fetch
the problem is that when I use fetch with 'NYC' it works properly but when I use fetch with 'nyc' it returns:
KeyError: key not found: nyc
How can I solve this problem?
There's an option :header_converters. You can set it to:
:downcase Calls downcase() on the header String.
:symbol The header String is downcased, spaces are replaced with underscores, non-word characters are dropped, and finally to_sym() is called.
Example:
require 'CSV'
CSV.parse("NYC\nfoo", headers: true, header_converters: :symbol) do |row|
row[:nyc] #=> "foo"
end

Resources