I have CSV's which I am trying to import into my oracle database but unfortunately I keep on getting the same error:
> CSV::MalformedCSVError: Unquoted fields do not allow \r or \n (line 1).
I know there are tons of similar questions which have been asked but none relate specifically to my issue other than this one, but unfortunately it didn't help.
To explain my scenario:
I have CSV's in which the rows don't always end with a value, but
rather, just a comma because it's a null value hence it stays blank.
I would like to import the CSV's regardless of whether the ending is with a comma or without a comma.
Here are the first 5 lines of my CSV with changed values due to privacy reasons,
id,customer_id,provider_id,name,username,password,salt,email,description,blocked,created_at,updated_at,deleted_at
1,,1,Default Administrator,admin,1,1," ",Initial default user.,f,2019-10-04 14:28:38.492000,2019-10-04 14:29:34.224000,
2,,2,Default Administrator,admin,2,1,,Initial default user.,,2019-10-04 14:28:38.633000,2019-10-04 14:28:38.633000,
3,1,,Default Administrator,admin,3,1," ",Initial default user.,f,2019-10-04 14:41:38.030000,2019-11-27 10:23:03.329000,
4,1,,admin,admin,4,1," ",,,2019-10-28 12:21:23.338000,2019-10-28 12:21:23.338000,
5,2,,Default Administrator,admin,5,1," ",Initial default user.,f,2019-11-12 09:00:49.430000,2020-02-04 08:20:06.601000,2020-02-04 08:20:06.601000
As you can see the ending is sometimes with or without a comma and this structure repeats quite often.
This is my code with which I have been playing around with:
def csv_replace_empty_string
Dir.foreach(Rails.root.join('db', 'csv_export')) do |filename|
next if filename == '.' or filename == '..' or filename == 'extract_db_into_csv.sh' or filename =='import_csv.rb'
read_file = File.read(Rails.root.join('db', 'csv_export', filename))
replace_empty_string = read_file.gsub(/(?<![^,])""(?![^,])/, '" "')
format_csv = replace_empty_string.gsub(/\r\r?\n?/, "\n")
# format_csv = remove_empty_lines.sub!(/(?:\r?\n)+\z/, "")
File.open(Rails.root.join('db', 'csv_export', filename), "w") {|file| file.puts format_csv }
end
end
I have tried using many different kinds of gsubs found online in similar forums, but it didn't help.
Here is my function for importing the CSV in the db:
def import_csv_into_db
Dir.foreach(Rails.root.join('db', 'csv_export')) do |filename|
next if filename == '.' or filename == '..' or filename == 'extract_db_into_csv.sh' or filename =='import_csv.rb'
filename_renamed = File.basename(filename, File.extname(filename)).classify
CSV.foreach(Rails.root.join('db', 'csv_export',filename), :headers => true, :skip_blanks => true) do |row|
class_name = filename_renamed.constantize
class_name.create!(row.to_hash)
puts "Insert on table #{filename_renamed} complete"
end
end
end
I have also tried the options provided by CSV such as :row_sep => :"\n" or :row_sep => "\r" but keep on getting the same error.
I am pretty sure I have some sort of thinking error, but I can't seem to figure it out.
I fixed the issue by using the following:
format_csv = replace_empty_string.gsub(/\r\r?\n?/, "\n")
This was originally #mgrims answer, but I had to adjust my code by further removing the :skip_blanks :row_sep options.
It is importing successfully now!
I have a file b.xls from excel I need to import it to my rails app
I have tried to open it
file = File.read(Rails.root.to_s+'/b.xls')
I have got this
file.encoding => #Encoding:UTF-8
I have few questions:
how to open without this symbols(normal language)?
how to convert this file to a hash?
File pretty large about 5k lines
You must have array of all rows then you can convert it to some hash if you like so.
I would recommend to use a batch_factory gem.
The gem is very simple and relies on the roo gem under the hood.
Here is the code example
require 'batch_factory'
factory = BatchFactory.from_file(
Rails.root.join('b.xlsx'),
keys: [:column1, :column2, ..., :what_ever_column_name]
)
Then you can do
factory.each do |row|
puts row[:column1]
end
You can also omit specifying keys. Then batch_factory will automatically fetch headers from the first row. But your keys would be in russian. Like
factory.each do |row|
puts row['Товар']
end
If you want to hash with product name as key you can do
factory.inject({}) do |hash, row|
hash.merge(row['Товар'] => row)
end
I am new to Ruby and I found this https://github.com/drodriguez/reversegeocoding and I think it sounds really cool. So I tried the sample app, installed everything, but when I'm calling
thor geocoder:database
I get this error:
/Users/xyz/.thor/f892c2a1d732c61bbf4ebff2abb70df6:194:in `initialize': wrong number of arguments(2 for 0) (ArgumentError)
Line 194 shows
csv = FasterCSV.new(io, CSV_OPTIONS.merge(COUNTRIES_CSV_OPTIONS))
and the whole method
def insert_countries(db, countries)
ids = Hash.new
country_insert = db.prepare("INSERT INTO countries (name) VALUES (:name)")
open(countries, 'rb') do |io|
io.rewind unless io.read(3) == "\xef\xbb\xbf" # Skip UTF-8 marker
io.readline while io.read(1) == '#' # Skip comments at the start of the file
io.seek(-1, IO::SEEK_CUR) # Unread the last character that wasn't '#'
csv = FasterCSV.new(io, CSV_OPTIONS.merge(COUNTRIES_CSV_OPTIONS))
csv.each do |row|
country_insert.execute :name => row['country']
ids[row['ISO']] = db.last_insert_row_id
end
end
country_insert.close
ids
end
I don't know how to fix this issue and I hope someone can help me with this.
Thanks.
I found the solution. The problem was my Ruby version. Since 1.9.x FasterCSV is no longer supported and now CSV is in the Ruby standard library... Look here for reference https://stackoverflow.com/a/6090840/749242
just started learning Rails and have managed to import a csv file into a database, but the price field in the csv has quotes and a comma like this: "560,000"
But if I make the price field as t.integer in the migration file, then add the data, the price gets imported as 560. So, how do I remove the quotes and the comma before importing it? thanks, Adam
edit: here's the rake file:
require 'csv'
task :csv_to_properties => [:environment] do
CSV.foreach("lib/assets/cbmb_sale.csv", :headers => true) do |row|
Property.create!(row.to_hash)
end
end
Try something like:
csvvalue = csvvalue.gsub!(/,/,'').to_i
Cheers!
Thanks for posting your code. I don't do a ton with converting csv's to hashes but something like this will probably work:
Property.create!(row.to_hash.each_pair{|k,v| row.store(k,v.gsub(/,/,'').to_i)})
Pretty ugly but probably pretty close to what you want.
In your code example, assuming the price field is in row element 4:
CSV.foreach("lib/assets/cbmb_sale.csv", :headers => true) do |row|
row[price=4].gsub!(/,/,'')
Property.create!(row.to_hash)
end
The price=4 is just a handy way to document the index value of the price element, it creates a variable called price assigns the value 4 to it, then immediately uses it as the array index.
Since Property.create! is already taking care of the string to integer conversion, we can perform an in-place substitution for the regular expression that contains a comma /,/ for an empty string ''.
Try:
"220,000".scan(/\d+/).join().to_i
I am using ruby 1.8.7 , rails 2.3.8. I want to parse the data from TXT dump file separated by tab.
In this TXT dump contain some CSS property look like has some invalid data.
When run my code using FasterCSV gem
FasterCSV.foreach(txt_file, :quote_char => '"',:col_sep =>'\t', :row_sep =>:auto, :headers => :first_row) do |row|
col= row.to_s.split(/\t/)
puts col[15]
end
the error written in console as "Illegal quoting on line 38." Can any one suggest me how to skip the row which has invalid data and proceed data load process of remaining rows?
Here's one way to do it. We go to lower level, using shift to parse each row and then silent the MalformedCSVError exception, continuing with the next iteration. The problem with this is the loop doesn't look so nice. If anyone can improve this, you're welcome to edit the code.
FasterCSV.open(filename, :quote_char => '"', :col_sep => "\t", :headers => true) do |csv|
row = true
while row
begin
row = csv.shift
break unless row
# Do things with the row here...
rescue FasterCSV::MalformedCSVError
next
end
end
end
Just read the file as a regular one (not with FasterCSV), split it like you do know by \t and it should work
So the problem is that TSV files don't have a quote character. The specification simply specifies that you aren't allowed to have tabs in the data.
The CSV library doesn't really support this use case. I've worked around it by specifying a quote character that I know won't appear in my data. For example
CSV.parse(txt_file, :quote_char => '☎', :col_sep => "\t" do |row|
puts row[15]
end