I am going to keep this question simple.
I am trying to import a CSV file into my application.
The file has long numbers in it such as :"9405510200830754182150"
but when the file is imported the data looks like this: "9.40551e+21"
does anyone know how to get around this?
Here is the code I am using
CSV.foreach(file.path, headers: true) do |row|
puts "row: #{row.inspect}"
end
UPDATE
Thank you for the comments, I am not sure why CSV is converting that number into a float i need to keep it as a string.
I should clarify that I am using Rails 3.2.18 for this project
If you want to reproduce my code:
1.create CSV with 9405510200830754182150 in it
2.run this code to terminal:
file = File.join(Rails.root, 'tracking.csv')
CSV.foreach(file, headers: true) do |row|
puts "row: #{row.inspect}"
end
I need to be able to keep "9405510200830754182150" is a string since this is a tracking number of an order and needs to be stored in the database
Are you sure that "9.40551e+21" is not a visual approximation? Try this:
CSV.foreach(file.path, headers: true) do |row|
puts row['my_numeric_header']
end
It's supposed to treat everything in a CSV file as a string by default. You could try the converters: numeric option
CSV.foreach(file.path, headers: true, converters: :numeric) do |row|
puts "row: #{row.inspect}"
end
Which will interpret numbers into the appropriate types. Otherwise you might have to debug the CSV module code to figure out what's going on.
The :float converter converted your number to a Float for you. Unfortunately Float cannot hold such a large number, see comments...
[14] pry(main)> val = CSV::Converters[:float].('9405510200830754182150')
=> 9.405510200830755e+21
[15] pry(main)> val.class
=> Float
[16] pry(main)> "%d" % val
=> "9405510200830754553856"
[17] pry(main)> "%f" % val
=> "9405510200830754553856.000000"
Related
I have a problem since I try to import my CSV created with Numbers on Mac,
Everything worked before on Ubuntu with LibreOffice,
When I try to import my CSV file I have error
unknown attribute 'adress user_id room_type etc...' for Bien.
I think it not detect separators and take the first hearder line rows as one string.
My import function:
def self.import(file)
CSV.foreach(file.path, headers: true) do |row|
#bien = Bien.create! row.to_hash
#bien.save
end
end
I would know how import the file and if I have things to change when I create my CSV on Numbers.
UPDATE
I think you're exactly right, it looks like the separators are not being respected so the header row is showing as one long string. To debug, you can try putting a pry in and running CSV.read(file.path) to see the whole output of the conversion to CSV. Once you do that, you should be able to see what Numbers uses for separators.
This post suggests Numbers uses semicolons as default separators, so if you define your col_sep: ';' as an option, that might do the trick. (Ref: CSV docs).
So, the code would be
def self.import(file)
CSV.foreach(file.path, col_sep: ';', headers: true) do |row|
#bien = Bien.create! row.to_hash
#bien.save
end
end
I'm having problems importing this CSV:
municipality,province,province abbrev,country,region
Vancouver,British Columbia,BC,Canada,Metro Vancouver - North
Specifically, Vancouver is not being returned when I look for its value by its key:
municipality_name = row["municipality"]
Here's the code:
def self.import_csv(file)
CSV.foreach(file, headers: true,
skip_blanks: true,
skip_lines: /^(?:,\s*)+$/,
col_sep: ",") do |row|
municipality_name = row["municipality"]
puts row.to_h
puts "municipality_name: #{municipality_name}"
puts "row[0]: #{row[0]}"
end
end
Here's the output:
irb(main):052:0> Importers::Municipalities.import_csv('tmp/municipalities.csv')
{"municipality"=>"Vancouver", "province"=>"British Columbia", "province abbrev"=>"BC", "country"=>"Canada", "region"=>"Metro Vancouver - North"}
municipality_name:
row['municipality']:
row[0]: Vancouver
Seems like I'm missing something obvious. I thought maybe there was a hidden character in the CSV but turned on hidden characters in Sublime and no dice.
Thanks in advance.
You need to call to_h on the row if you want to access it by its keys. Otherwise, it is an array-like object, accessible by indices.
def self.import_csv(file)
CSV.foreach(file, headers: true,
skip_blanks: true,
skip_lines: /^(?:,\s*)+$/,
col_sep: ",") do |row|
row = row.to_h
municipality_name = row["municipality"]
puts "municipality_name: #{municipality_name}"
end
end
Seems like it was a problem with the CSV and the code works fine. Created a new CSV, typed in the same content, and it worked. Maybe an invisible character that Sublime wasn't showing? Can't verify as I wiped the original CSV that was causing issues.
I'm using ruby 1.9.2. My csv file as follows..,
NAME, Id, No, Dept
Tom, 1, 12, CS
Hendry, 2, 35, EC
Bahamas, 3, 21, IT
Frank, 4, 61, EE
I want to print an specific row say ('Tom'). I tried out in many ways, but I didn't find the exact result. The most recommended options is "Fastercsv". But it is applicable for my version. Also, I noticed that csv print the field as column wise. How to print an entire row using csv in rails. My ruby code is as follows
require 'csv'
csv_text = File.read('sampler.csv')
csv = CSV.parse(csv_text, :headers => true)
csv.each do |row|
puts "#{row[:NAME]},#{row[:Id]},#{row[:No]},#{row[:Dept]}"
end
Use .find
csv = CSV.read('sampler.csv', headers: true)
puts csv.find {|row| row['NAME'] == 'Tom'} #=> returns first `row` that satisfies the block.
Here's another approach that keeps the code within the CSV API.
csv_table is a CSV::Table
row is a CSV::Row
row_with_specified_name is a CSV::Row.
csv_table = CSV.table("./tables/example.csv", converters: :all)
row_with_specified_name = csv_table.find do |row|
row.field(:name) == 'Bahamas'
end
p row_with_specified_name.to_csv.chomp #=> "Bahamas,3,21,IT"
FYI, CSV.table is just a shortcut for:
CSV.read( path, { headers: true,
converters: :numeric,
header_converters: :symbol }.merge(options) )
As per the docs.
If you have a large CSV file and want to find an exact row it will be way faster and way less memory intense to read one line at a time.
require 'csv'
csv = CSV.open('sampler.csv', 'r', headers: true)
while row = csv.shift
if row['name'] == 'Bahamas'
break
end
end
pp row
I am using FasterCSV and i am looping with a foreach like this
FasterCSV.foreach("#{Rails.public_path}/uploads/transfer.csv", :encoding => 'u', :headers => :first_row) do |row|
but the problem is my csv has the first 3 lines as the headers...any way to make fasterCSV skip the first three rows rather then only the first??
Not sure about FasterCSV, but in Ruby 1.9 standard CSV library (which is made from FasterCSV), I can do something like:
c = CSV.open '/path/to/my.csv'
c.drop(3).each do |row|
# do whatever with row
end
I'm not a user of FasterCSV, but why not do the control yourself:
additional_rows_to_skip = 2
FasterCSV.foreach("...", :encoding => 'u', :headers => :first_row) do |row|
if additional_rows_to_skip > 0
additional_rows_to_skip -= 1
else
# do stuff...
end
end
Thanks to Mladen Jablanovic. I got my clue.. But I realized something interesting
In 1.9, reading seems to be from POS.
In this I mean if you do
c = CSV.open iFileName
logger.debug c.first
logger.debug c.first
logger.debug c.first
You'll get three different results in your log. One for each of the three header rows.
c.each do |row| #now seems to start on the 4th row.
It makes perfect sense that it would read the file this way. Then it would only have to have the current row in memory.
I still like Mladen Jablanovićs answer, but this is an interesting bit of logic too.
I am using ruby 1.8.7 , rails 2.3.8. I want to parse the data from TXT dump file separated by tab.
In this TXT dump contain some CSS property look like has some invalid data.
When run my code using FasterCSV gem
FasterCSV.foreach(txt_file, :quote_char => '"',:col_sep =>'\t', :row_sep =>:auto, :headers => :first_row) do |row|
col= row.to_s.split(/\t/)
puts col[15]
end
the error written in console as "Illegal quoting on line 38." Can any one suggest me how to skip the row which has invalid data and proceed data load process of remaining rows?
Here's one way to do it. We go to lower level, using shift to parse each row and then silent the MalformedCSVError exception, continuing with the next iteration. The problem with this is the loop doesn't look so nice. If anyone can improve this, you're welcome to edit the code.
FasterCSV.open(filename, :quote_char => '"', :col_sep => "\t", :headers => true) do |csv|
row = true
while row
begin
row = csv.shift
break unless row
# Do things with the row here...
rescue FasterCSV::MalformedCSVError
next
end
end
end
Just read the file as a regular one (not with FasterCSV), split it like you do know by \t and it should work
So the problem is that TSV files don't have a quote character. The specification simply specifies that you aren't allowed to have tabs in the data.
The CSV library doesn't really support this use case. I've worked around it by specifying a quote character that I know won't appear in my data. For example
CSV.parse(txt_file, :quote_char => '☎', :col_sep => "\t" do |row|
puts row[15]
end