Iv recently written code that will help me export an SQL database into CSV using FasterCSV with rails. However some parts of my database contain Traditional Chinese Characters. When I export it i'm getting ?????? as the output in the CSV file. Iv already tried changing the $KCODE = 'u' so that FasterCSV uses UTF-8 to encode the CSV file, but no luck. Iconv to convert the encoding is giving me strange results as well. Here is the source code:
def csv
#lists = Project.find(:all, :order=> (params[:sort] + ' ' + params[:direction]), :conditions => ["name LIKE ?", "%#{params[:selection]}%"])
csv_string = FasterCSV.generate do |csv|
csv << [<bold> "Status","Name","Summary","Description","Creator","Comment","Contact Information","Created Date","Updated Date"]
#lists.each do |project|
csv << [project.status, project.name, project.summary, project.description, project.creator, project.statusreason, project.contactinfo, project.created_at, project.updated_at]
end
end
filename = Time.now.strftime("%Y%m%d") + ".csv"
send_data(csv_string,
:type => 'text/csv; charset=utf-8; header=present',
:filename => filename)
end
Thanks,
I'm not used to work with chinese characters, but you can try adding the :encoding option to 'u' (UTF-8) on the generate call:
...
csv_string = FasterCSV.generate(:encoding => 'u') do |csv|
...
As a side note, I'd recommend using named_scopes instead of writing this:
Project.find(:all, :order=> (params[:sort] + ' ' + params[:direction]), :conditions => ["name LIKE ?", "%#{params[:selection]}%"])
Good luck!
Related
This should have been such an easy thing... buy I can't for the life of me figure out how to parse a CSV file that doesn't seem to have a specific encoding.
File.open(Rails.root.join('data', 'mike/test-csv.csv'), 'rb') { |f| f.read }
=> "ID,\x00Q\x00u\x00a\x00n\x00t\x00i\x00t\x00y\n\x006\x00e\x005\x004\x009\x001\x00e\x007\x00-\x007\x00f\x001\x005\x00-\x004\x001\x007\x00d\x00-\x00a\x004\x000\x003\x00-345\x00,\x00\x005\x000\x00.\x000\x000\x000\x000\x000\x000\x000\x000\x00\n"
Here's a gist of it, can't figure out a way to post the specific CSV.
All I get from checking the encoding of the file is that it's in binary format, any thoughts on how I could get it into a normal csv?
Note: This is a downloaded CSV so converting it to another encoding via opening it in excel and exporting (or something like that) is not an option :)
Thanks!
Updating with attempted solution 1:
path = Rails.root.join('data', 'mike/test-csv.csv')
CSV.read(path, {:headers => true, :encoding => 'utf-8'}).each do |d|
puts d
end
Result: 6e5491e7-7f15-417d-a403-345,50.00000000
While this is correct, it ONLY works with puts, for example:
CSV.read(path, {:headers => true, :encoding => 'utf-8'}).map { |row| row }
=> [#<CSV::Row "ID":"\u00006\u0000e\u00005\u00004\u00009\u00001\u0000e\u00007\u0000-\u00007\u0000f\u00001\u00005\u0000-\u00004\u00001\u00007\u0000d\u0000-\u0000a\u00004\u00000\u00003\u0000-345\u0000" "\u0000Q\u0000u\u0000a\u0000n\u0000t\u0000i\u0000t\u0000y":"\u0000\u00005\u00000\u0000.\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u0000">]
CSV.read(path, {:headers => true, :encoding => 'utf-8'}).map(&:to_s)
=> ["\u00006\u0000e\u00005\u00004\u00009\u00001\u0000e\u00007\u0000-\u00007\u0000f\u00001\u00005\u0000-\u00004\u00001\u00007\u0000d\u0000-\u0000a\u00004\u00000\u00003\u0000-345\u0000,\u0000\u00005\u00000\u0000.\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u0000\n"]
It's unfortunately still not the correct string :(
Final Solution (via #ashmaroli below):
path = Rails.root.join('data', 'mike/test-csv.csv')
csv_text = ''
File.open(path, 'r') do |csv|
csv.each_line do |line|
csv_text << line.gsub(/\u0000/, '')
end
end
CSV.parse(csv_text, headers:true).map do |row| row end
Result:
[#<CSV::Row "ID":"6e5491e7-7f15-417d-a403-345" "Quantity":"50.00000000">]
Github Gist
Download Example CSV File
path = Rails.root.join('data', 'mike/test-csv.csv')
file = ""
File.open(path, 'r') do |csv|
csv.each_line do |line|
file << line.gsub(/\u0000/, '')
end
end
print file
print file.inspect # same as above just wraps the string in a
# single line with "\n" chars
I'm trying to create a CSV file for download in Rails, and cannot get it to send just the CSV without a tag around the data. In my controller, I have:
csv_string = CSV.generate do |csv|
headers = ['Header 1', 'Header 2']
csv << headers
#matches.each do |match|
csv << match
end
end
send_data(csv_string, :filename => filename, :layout => false)
The form to run this has:
=form_tag log_path, :id =>'log_search_form', :multipart => true, :remote=>true do
.search_fields
.panel.panel-default
.panel-heading
Search Log File:
=file_field_tag :search
=submit_tag "Find Matches", :class=>'btn btn-primary btn-xs'
When I press "Find Matches", I am prompted to download a csv file, but the first line has:
<textarea data-type="text/csv" data-status="200" data-statusText="OK">Header 1
and the file ends with
</textarea>
The (legacy) code uses remotipart - it seems I need to stop it from overriding render and adding the textarea. How can I do this to get a clean CSV download?
Thank you!
I've never seen that superfluous <textarea> issue before, but try this send_data(...) parameterization, which is taken from a Rails app I maintain:
# inside your format.csv handler...
# set the filename and csv_string variables...
send_data(csv_string, :type => 'text/csv',
:disposition => :attachment,
:filename => filename)
When exporting csv in Rails 4.2 app, there are ascii code in the csv output for Chinese characters (UTF8):
ä¸åˆåŒç†Šå·¥ç‰ç”¨é¤
We tried options in send_data without luck:
send_data #payment_requests.to_csv, :type => 'text/csv; charset=utf-8; header=present'
And:
send_data #payment_requests.to_csv.force_encoding("UTF-8")
In model, there is forced encoding utf8:
# encoding: utf-8
But it does not work. There are online posts talking about use gem iconv. However iconv depends on the platform's ruby version. Is there cleaner solution to fix the ascii in Rails 4.2 csv exporting?
If #payment_requests.to_csv includes ASCII text, then you should use encode method:
#payment_requests.to_csv.encode("UTF-8")
or
#payment_requests.to_csv.force_encoding("ASCII").encode("UTF-8")
depending on which internal encoding #payment_requests.to_csv has.
You can try:
#payment_requests.to_csv.force_encoding("ISO-8859-1")
for Chinese characters
CSV.generate(options) do |csv|
csv << column_names
all.each do |product|
csv << product.attributes.values_at(*column_names)
end
end.encode('gb2312', :invalid => :replace, :undef => :replace, :replace => "?")
This is what worked for me:
head = 'EF BB BF'.split(' ').map{|a|a.hex.chr}.join()
csv_str = CSV.generate(csv = head) do |csv|
csv << [ , , , ...]
#elements.each do |element|
csv << [ , , , ...]
end
end
I am trying to save multiple pngs in one pdf. I'm receiving the PNGs from an API Call to the Endicia Label Server, which is giving me a Base64 Encoded Image as response.
Based on this Question:
How to convert base64 string to PNG using Prawn without saving on server in Rails
def batch_order_labels
#orders = Spree::Order.ready_to_ship.limit(1)
dt = Date.current.strftime("%d %b %Y ")
title = "Labels - #{dt} - #{#orders.count} Orders"
Prawn::Document.generate("#{title}.pdf") do |pdf|
#orders.each do |order|
label = order.generate_label
if order.international?
#image = label.response_body.scan(/<Image PartNumber=\"1\">([^<>]*)<\/Image>/imu).flatten.last
else
#image = label.image
end
file = Tempfile.new('labelimg', :encoding => 'utf-8')
file.write Base64.decode64(#image)
file.close
pdf.image file
pdf.start_new_page
end
end
send_data("#{title}.pdf")
end
But I'm receiving following error:
"\x89" from ASCII-8BIT to UTF-8
Any Idea?
There's no need to write the image data to a tempfile, Prawn::Document#image can accept a StringIO.
Try replacing this:
file = Tempfile.new('labelimg', :encoding => 'utf-8')
file.write Base64.decode64(#image)
file.close
pdf.image file
With this:
require 'stringio'
.....
image_data = StringIO.new( Base64.decode64(#image) )
pdf.image(image_data)
The Problem is, that the Api is returning this thing in UTF-8 - So I dont have a great choice.
Anyhow, I found this solution to be working
file = Tempfile.new('labelimg', :encoding => 'utf-8')
File.open(file, 'wb') do |f|
f.write Base64.decode64(#image)
end
you can't convert the Base64 to UTF-8.
Leave it as plain ASCII:
file = Tempfile.new('labelimg', :encoding => 'ascii-8bit')
file.write Base64.decode64(#image)
file.close
or even better - leave it as binary:
file = Tempfile.new('labelimg')
file.write Base64.decode64(#image)
file.close
UTF-8 is multibite format and it's not usable for transferring binary data such as pics.
I tried to import data from csv in my rails app, but something went wrong:
CSV::MalformedCSVError in ArticlesController#index
Unclosed quoted field on line 1.
My csv looks like this:
"Код";"№ по каталогу (артикул)";"Наименование товара";"Ед. изм.";"Цена опт.";"Доп.";"Остатки";"Класс";"Группа";"Бренд";"Блок."
2223;15-562-44;15-562-44 (27-B07-F) VW Polo 95-R ;шт ;37,430;;;Амортизаторы ;Амортизаторы BOGE ;;
10327;24-052-1;24-052-1(46-A27-0) LAND ROVER 84- F ;шт ;68,750;;;Амортизаторы ;Амортизаторы BOGE ;;
10328;24-053-1;24-053-1(46-A28-0) LAND ROVER 84- R ;шт ;68,750;;;Амортизаторы ;Амортизаторы BOGE ;;
Maybe this is because of the first line (which has no ;;)
My code look like this:
def csv_import
require 'csv'
file = File.open("/#{Rails.public_path}/uploads/smallcsv.csv")
#csv = CSV.parse(file)
csv = CSV.open(file, "r:ISO-8859-15:UTF-8", {:col_sep => ";", :row_sep => ";;", :headers => :first_row})
file_path = "/#{Rails.public_path}/uploads/smallcsv.csv"
##parsed_file=CSV::Reader.parse(file_path)
csv.each do |row|
ename = row[2]
eprice = row[5]
eqnt = row[7]
esupp = row[10]
logger.warn(ename)
end
end
I'm running ruby 1.9+ with fastercsv gem
I figured this out myself using "CSV - Unquoted fields do not allow \r or \n (line 2)".
The problem was with the first line, so :auto helped me.