I have a problem since I try to import my CSV created with Numbers on Mac,
Everything worked before on Ubuntu with LibreOffice,
When I try to import my CSV file I have error
unknown attribute 'adress user_id room_type etc...' for Bien.
I think it not detect separators and take the first hearder line rows as one string.
My import function:
def self.import(file)
CSV.foreach(file.path, headers: true) do |row|
#bien = Bien.create! row.to_hash
#bien.save
end
end
I would know how import the file and if I have things to change when I create my CSV on Numbers.
UPDATE
I think you're exactly right, it looks like the separators are not being respected so the header row is showing as one long string. To debug, you can try putting a pry in and running CSV.read(file.path) to see the whole output of the conversion to CSV. Once you do that, you should be able to see what Numbers uses for separators.
This post suggests Numbers uses semicolons as default separators, so if you define your col_sep: ';' as an option, that might do the trick. (Ref: CSV docs).
So, the code would be
def self.import(file)
CSV.foreach(file.path, col_sep: ';', headers: true) do |row|
#bien = Bien.create! row.to_hash
#bien.save
end
end
Related
I'm having problems importing this CSV:
municipality,province,province abbrev,country,region
Vancouver,British Columbia,BC,Canada,Metro Vancouver - North
Specifically, Vancouver is not being returned when I look for its value by its key:
municipality_name = row["municipality"]
Here's the code:
def self.import_csv(file)
CSV.foreach(file, headers: true,
skip_blanks: true,
skip_lines: /^(?:,\s*)+$/,
col_sep: ",") do |row|
municipality_name = row["municipality"]
puts row.to_h
puts "municipality_name: #{municipality_name}"
puts "row[0]: #{row[0]}"
end
end
Here's the output:
irb(main):052:0> Importers::Municipalities.import_csv('tmp/municipalities.csv')
{"municipality"=>"Vancouver", "province"=>"British Columbia", "province abbrev"=>"BC", "country"=>"Canada", "region"=>"Metro Vancouver - North"}
municipality_name:
row['municipality']:
row[0]: Vancouver
Seems like I'm missing something obvious. I thought maybe there was a hidden character in the CSV but turned on hidden characters in Sublime and no dice.
Thanks in advance.
You need to call to_h on the row if you want to access it by its keys. Otherwise, it is an array-like object, accessible by indices.
def self.import_csv(file)
CSV.foreach(file, headers: true,
skip_blanks: true,
skip_lines: /^(?:,\s*)+$/,
col_sep: ",") do |row|
row = row.to_h
municipality_name = row["municipality"]
puts "municipality_name: #{municipality_name}"
end
end
Seems like it was a problem with the CSV and the code works fine. Created a new CSV, typed in the same content, and it worked. Maybe an invisible character that Sublime wasn't showing? Can't verify as I wiped the original CSV that was causing issues.
I have the following two lines of a code that take an uploaded CSV file from params and return a hash of Contact objects. The code works fine when I input a CSV with UTF-8 encoding. If I try to upload a CSV with another type of encoding though, it breaks. How can I adjust the code to detect the encoding of the uploaded file and convert to UTF-8?
CSV::Converters[:blank_to_nil] = lambda { |field| field && field.empty? ? nil : field }
csv = CSV.new(params[:file].tempfile.open, headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
This question is not a duplicate! I've seen numerous other questions on here revolving around the same encoding issue, but the specifics of those are different than my case. Specifically, I need a way convert the encoding of a TempFile generated from my params hash. Other solutions I've seen involve encoding String and File objects, as well as passing an encoding option to CSV.parse or CSV.open. I've tried those solutions already without success.
I've tried passing in an encoding option to CSV.new, like so:
csv = CSV.new(params[:file].tempfile.open, encoding: 'iso-8859-1:utf-8', headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
I've also tried this:
csv = CSV.new(params[:file].tempfile.open, encoding: 'iso-8859-1:utf-8', headers: true, header_converters: :symbol, converters: [:all, :blank_to_nil]).to_a.map {|row| row.to_hash }
I've tried adjusting my converter as well, like so:
CSV::Converters[:blank_to_nil] = lambda { |field| field && field.empty? ? nil : field.encode('utf-8') }
I'm looking for a programatic solution here that does not require the user to convert their CSV to the proper encoding.
I've also had to deal with this problem and here is how I finally solved it.
CSV.open(new_csv_file, 'w') do |csv_object|
lines = File.open(uploaded_file).read
lines.each_line do |line|
csv_object << line.encode!("utf-8", "utf-8", invalid: :replace, undef: :replace, replace: '').parse_csv
end
end
CSV.new(File.read(new_csv_file))
Basically go through every line, sanitize it and shove it into a new CSV file.
Hope that leads you and other in the right direction.
You can use filemagic to detect the encoding of a file, although it's not 100% accurate. It bases on system's file command tool, so I'm not sure if it works on windows.
I want to use comma OR semicolon as :col_sep when importing CSV data in rails:
CSV.foreach(file.path, :col_sep => (";"), headers: true) do |row|
user_hash = row.to_hash
User.create!(user_hash)
end
works.
But putting different col_seps inline won't work:
CSV.foreach(file.path, :col_sep => (";",","), headers: true) do |row|
Is it even possible? I haven't found anything in the docu nor here on stackoverflow.
That isn't possible. Col_sep can only accept one string. There are workarounds for this, mentioned here and here.
I have the following CSV import action in my Miniatures model
def self.import(file)
CSV.foreach(file.path, headers: true) do |row|
Miniature.create! row.to_hash
end
end
That works fine but what I want to do is use further columns in the CSV to create associated objects.
This is one of my attempts:
def self.import(file)
CSV.foreach(file.path, headers: true) do |row|
Miniature.create! row.to_hash.slice(row[0..10])
#miniature.sizes.build(:scale_id => row[11])
end
end
My attempted at slicing the row have been very unsuccessful. If I don't slice off the first 10 rows then the Miniature.create tries to parse the 11th column which only applies to the associated sizes model. I want to slice off the first 10 and create a Miniature object with them and then build or create a line in my Sizes join table with the supplied scale_id.
Any help very much appreciated.
Another Update
This is my latest and cleanest attempt:
Miniature.create! row.to_hash.except!(:scale_id)
That throws "unknown attribute: scale_id" as an error. Possible I can't interact with the keys at all after the CSV.foreach(file.path, headers: true) do |row| ?
Updated again
I can see one reason why my above code won't work. I'm specifying a range for the fields in the row but hashes don't have an order. I've now tried specifying the keys that I want to deal with using indices but got undefined method 'indices'.
row.to_hash.indices(:name,:material,:release_date,:pcode,:notes,:set,:random,:quantity,:date_mask,:multipart)
I can't turn it into an array or I'll get stringily keys errors but I need to be able to specify which fields the create action should use so that I can use some for the Miniature create and some for the Size create.
My controller action by the way is as follows.
def import
Miniature.import(params[:file])
redirect_to miniatures_path, notice: "Miniatures imported."
end
Update
Here is the seed data I'm using to import
name,material,release_date,pcode,notes,set,random,quantity,date_mask,multipart,scale_id
A A A CSV test,Metal,03/01/2013,123123,Test notes,f,f,,6,f,1
With the above code, the error I get is
"Validation failed: Name can't be blank, Material can't be blank"
but through trying things out I've had a variety of errors which indicate my row.to_hash.slice is not being parsed in the way the simpler row.to_hash is.
The expected result is either a successfully created Miniature object and a Size object OR an error on creating the size object because it can't infer the miniature_id from my using #miniature.sizes.build and wants more params. Can't debug that until initial slicing stage is passed/parsed.
You're assuming the keys to the hash are symbols like :scale_id but in fact they're strings like 'scale_id' and this is where you're tripping up... you need to symbolize the keys if you want to use them as attributes to a create method.
symbolized_row = row.to_hash.inject({}){|memo,(k,v)| memo[k.to_sym] = v; memo}
Miniature.create! symbolized_row.except!(:scale_id)
EDIT
Actually if you use except instead of the mutating except! then you'll have access to the scale id in subsequent lines.
symbolized_row = row.to_hash.inject({}){|memo,(k,v)| memo[k.to_sym] = v; memo}
Miniature.create! symbolized_row.except(:scale_id)
#miniature.sizes.build(:scale_id => symbolized_row[:scale_id])
I am going to keep this question simple.
I am trying to import a CSV file into my application.
The file has long numbers in it such as :"9405510200830754182150"
but when the file is imported the data looks like this: "9.40551e+21"
does anyone know how to get around this?
Here is the code I am using
CSV.foreach(file.path, headers: true) do |row|
puts "row: #{row.inspect}"
end
UPDATE
Thank you for the comments, I am not sure why CSV is converting that number into a float i need to keep it as a string.
I should clarify that I am using Rails 3.2.18 for this project
If you want to reproduce my code:
1.create CSV with 9405510200830754182150 in it
2.run this code to terminal:
file = File.join(Rails.root, 'tracking.csv')
CSV.foreach(file, headers: true) do |row|
puts "row: #{row.inspect}"
end
I need to be able to keep "9405510200830754182150" is a string since this is a tracking number of an order and needs to be stored in the database
Are you sure that "9.40551e+21" is not a visual approximation? Try this:
CSV.foreach(file.path, headers: true) do |row|
puts row['my_numeric_header']
end
It's supposed to treat everything in a CSV file as a string by default. You could try the converters: numeric option
CSV.foreach(file.path, headers: true, converters: :numeric) do |row|
puts "row: #{row.inspect}"
end
Which will interpret numbers into the appropriate types. Otherwise you might have to debug the CSV module code to figure out what's going on.
The :float converter converted your number to a Float for you. Unfortunately Float cannot hold such a large number, see comments...
[14] pry(main)> val = CSV::Converters[:float].('9405510200830754182150')
=> 9.405510200830755e+21
[15] pry(main)> val.class
=> Float
[16] pry(main)> "%d" % val
=> "9405510200830754553856"
[17] pry(main)> "%f" % val
=> "9405510200830754553856.000000"