I'm having problems importing this CSV:
municipality,province,province abbrev,country,region
Vancouver,British Columbia,BC,Canada,Metro Vancouver - North
Specifically, Vancouver is not being returned when I look for its value by its key:
municipality_name = row["municipality"]
Here's the code:
def self.import_csv(file)
CSV.foreach(file, headers: true,
skip_blanks: true,
skip_lines: /^(?:,\s*)+$/,
col_sep: ",") do |row|
municipality_name = row["municipality"]
puts row.to_h
puts "municipality_name: #{municipality_name}"
puts "row[0]: #{row[0]}"
end
end
Here's the output:
irb(main):052:0> Importers::Municipalities.import_csv('tmp/municipalities.csv')
{"municipality"=>"Vancouver", "province"=>"British Columbia", "province abbrev"=>"BC", "country"=>"Canada", "region"=>"Metro Vancouver - North"}
municipality_name:
row['municipality']:
row[0]: Vancouver
Seems like I'm missing something obvious. I thought maybe there was a hidden character in the CSV but turned on hidden characters in Sublime and no dice.
Thanks in advance.
You need to call to_h on the row if you want to access it by its keys. Otherwise, it is an array-like object, accessible by indices.
def self.import_csv(file)
CSV.foreach(file, headers: true,
skip_blanks: true,
skip_lines: /^(?:,\s*)+$/,
col_sep: ",") do |row|
row = row.to_h
municipality_name = row["municipality"]
puts "municipality_name: #{municipality_name}"
end
end
Seems like it was a problem with the CSV and the code works fine. Created a new CSV, typed in the same content, and it worked. Maybe an invisible character that Sublime wasn't showing? Can't verify as I wiped the original CSV that was causing issues.
Related
I have a problem since I try to import my CSV created with Numbers on Mac,
Everything worked before on Ubuntu with LibreOffice,
When I try to import my CSV file I have error
unknown attribute 'adress user_id room_type etc...' for Bien.
I think it not detect separators and take the first hearder line rows as one string.
My import function:
def self.import(file)
CSV.foreach(file.path, headers: true) do |row|
#bien = Bien.create! row.to_hash
#bien.save
end
end
I would know how import the file and if I have things to change when I create my CSV on Numbers.
UPDATE
I think you're exactly right, it looks like the separators are not being respected so the header row is showing as one long string. To debug, you can try putting a pry in and running CSV.read(file.path) to see the whole output of the conversion to CSV. Once you do that, you should be able to see what Numbers uses for separators.
This post suggests Numbers uses semicolons as default separators, so if you define your col_sep: ';' as an option, that might do the trick. (Ref: CSV docs).
So, the code would be
def self.import(file)
CSV.foreach(file.path, col_sep: ';', headers: true) do |row|
#bien = Bien.create! row.to_hash
#bien.save
end
end
I have the following two lines of a code that take an uploaded CSV file from params and return a hash of Contact objects. The code works fine when I input a CSV with UTF-8 encoding. If I try to upload a CSV with another type of encoding though, it breaks. How can I adjust the code to detect the encoding of the uploaded file and convert to UTF-8?
CSV::Converters[:blank_to_nil] = lambda { |field| field && field.empty? ? nil : field }
csv = CSV.new(params[:file].tempfile.open, headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
This question is not a duplicate! I've seen numerous other questions on here revolving around the same encoding issue, but the specifics of those are different than my case. Specifically, I need a way convert the encoding of a TempFile generated from my params hash. Other solutions I've seen involve encoding String and File objects, as well as passing an encoding option to CSV.parse or CSV.open. I've tried those solutions already without success.
I've tried passing in an encoding option to CSV.new, like so:
csv = CSV.new(params[:file].tempfile.open, encoding: 'iso-8859-1:utf-8', headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
I've also tried this:
csv = CSV.new(params[:file].tempfile.open, encoding: 'iso-8859-1:utf-8', headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
I've tried adjusting my converter as well, like so:
CSV::Converters[:blank_to_nil] = lambda { |field| field && field.empty? ? nil : field.encode('utf-8') }
I'm looking for a programatic solution here that does not require the user to convert their CSV to the proper encoding.
I've also had to deal with this problem and here is how I finally solved it.
CSV.open(new_csv_file, 'w') do |csv_object|
lines = File.open(uploaded_file).read
lines.each_line do |line|
csv_object << line.encode!("utf-8", "utf-8", invalid: :replace, undef: :replace, replace: '').parse_csv
end
end
CSV.new(File.read(new_csv_file))
Basically go through every line, sanitize it and shove it into a new CSV file.
Hope that leads you and other in the right direction.
You can use filemagic to detect the encoding of a file, although it's not 100% accurate. It bases on system's file command tool, so I'm not sure if it works on windows.
I want to use comma OR semicolon as :col_sep when importing CSV data in rails:
CSV.foreach(file.path, :col_sep => (";"), headers: true) do |row|
user_hash = row.to_hash
User.create!(user_hash)
end
works.
But putting different col_seps inline won't work:
CSV.foreach(file.path, :col_sep => (";",","), headers: true) do |row|
Is it even possible? I haven't found anything in the docu nor here on stackoverflow.
That isn't possible. Col_sep can only accept one string. There are workarounds for this, mentioned here and here.
I am going to keep this question simple.
I am trying to import a CSV file into my application.
The file has long numbers in it such as :"9405510200830754182150"
but when the file is imported the data looks like this: "9.40551e+21"
does anyone know how to get around this?
Here is the code I am using
CSV.foreach(file.path, headers: true) do |row|
puts "row: #{row.inspect}"
end
UPDATE
Thank you for the comments, I am not sure why CSV is converting that number into a float i need to keep it as a string.
I should clarify that I am using Rails 3.2.18 for this project
If you want to reproduce my code:
1.create CSV with 9405510200830754182150 in it
2.run this code to terminal:
file = File.join(Rails.root, 'tracking.csv')
CSV.foreach(file, headers: true) do |row|
puts "row: #{row.inspect}"
end
I need to be able to keep "9405510200830754182150" is a string since this is a tracking number of an order and needs to be stored in the database
Are you sure that "9.40551e+21" is not a visual approximation? Try this:
CSV.foreach(file.path, headers: true) do |row|
puts row['my_numeric_header']
end
It's supposed to treat everything in a CSV file as a string by default. You could try the converters: numeric option
CSV.foreach(file.path, headers: true, converters: :numeric) do |row|
puts "row: #{row.inspect}"
end
Which will interpret numbers into the appropriate types. Otherwise you might have to debug the CSV module code to figure out what's going on.
The :float converter converted your number to a Float for you. Unfortunately Float cannot hold such a large number, see comments...
[14] pry(main)> val = CSV::Converters[:float].('9405510200830754182150')
=> 9.405510200830755e+21
[15] pry(main)> val.class
=> Float
[16] pry(main)> "%d" % val
=> "9405510200830754553856"
[17] pry(main)> "%f" % val
=> "9405510200830754553856.000000"
I'm using ruby 1.9.2. My csv file as follows..,
NAME, Id, No, Dept
Tom, 1, 12, CS
Hendry, 2, 35, EC
Bahamas, 3, 21, IT
Frank, 4, 61, EE
I want to print an specific row say ('Tom'). I tried out in many ways, but I didn't find the exact result. The most recommended options is "Fastercsv". But it is applicable for my version. Also, I noticed that csv print the field as column wise. How to print an entire row using csv in rails. My ruby code is as follows
require 'csv'
csv_text = File.read('sampler.csv')
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
puts "#{row[:NAME]},#{row[:Id]},#{row[:No]},#{row[:Dept]}"
end
Use .find
csv = CSV.read('sampler.csv', headers: true)
puts csv.find {|row| row['NAME'] == 'Tom'} #=> returns first `row` that satisfies the block.
Here's another approach that keeps the code within the CSV API.
csv_table is a CSV::Table
row is a CSV::Row
row_with_specified_name is a CSV::Row.
csv_table = CSV.table("./tables/example.csv", converters: :all)
row_with_specified_name = csv_table.find do |row|
row.field(:name) == 'Bahamas'
end
p row_with_specified_name.to_csv.chomp #=> "Bahamas,3,21,IT"
FYI, CSV.table is just a shortcut for:
CSV.read( path, { headers: true,
converters: :numeric,
header_converters: :symbol }.merge(options) )
As per the docs.
If you have a large CSV file and want to find an exact row it will be way faster and way less memory intense to read one line at a time.
require 'csv'
csv = CSV.open('sampler.csv', 'r', headers: true)
while row = csv.shift
if row['name'] == 'Bahamas'
break
end
end
pp row