Generating a PDF With Images from Base64 with Prawn - ruby-on-rails

I am trying to save multiple pngs in one pdf. I'm receiving the PNGs from an API Call to the Endicia Label Server, which is giving me a Base64 Encoded Image as response.
Based on this Question:
How to convert base64 string to PNG using Prawn without saving on server in Rails
def batch_order_labels
#orders = Spree::Order.ready_to_ship.limit(1)
dt = Date.current.strftime("%d %b %Y ")
title = "Labels - #{dt} - #{#orders.count} Orders"
Prawn::Document.generate("#{title}.pdf") do |pdf|
#orders.each do |order|
label = order.generate_label
if order.international?
#image = label.response_body.scan(/<Image PartNumber=\"1\">([^<>]*)<\/Image>/imu).flatten.last
else
#image = label.image
end
file = Tempfile.new('labelimg', :encoding => 'utf-8')
file.write Base64.decode64(#image)
file.close
pdf.image file
pdf.start_new_page
end
end
send_data("#{title}.pdf")
end
But I'm receiving following error:
"\x89" from ASCII-8BIT to UTF-8
Any Idea?

There's no need to write the image data to a tempfile, Prawn::Document#image can accept a StringIO.
Try replacing this:
file = Tempfile.new('labelimg', :encoding => 'utf-8')
file.write Base64.decode64(#image)
file.close
pdf.image file
With this:
require 'stringio'
.....
image_data = StringIO.new( Base64.decode64(#image) )
pdf.image(image_data)

The Problem is, that the Api is returning this thing in UTF-8 - So I dont have a great choice.
Anyhow, I found this solution to be working
file = Tempfile.new('labelimg', :encoding => 'utf-8')
File.open(file, 'wb') do |f|
f.write Base64.decode64(#image)
end

you can't convert the Base64 to UTF-8.
Leave it as plain ASCII:
file = Tempfile.new('labelimg', :encoding => 'ascii-8bit')
file.write Base64.decode64(#image)
file.close
or even better - leave it as binary:
file = Tempfile.new('labelimg')
file.write Base64.decode64(#image)
file.close
UTF-8 is multibite format and it's not usable for transferring binary data such as pics.

Related

Parse binary CSV file in Ruby

This should have been such an easy thing... buy I can't for the life of me figure out how to parse a CSV file that doesn't seem to have a specific encoding.
File.open(Rails.root.join('data', 'mike/test-csv.csv'), 'rb') { |f| f.read }
=> "ID,\x00Q\x00u\x00a\x00n\x00t\x00i\x00t\x00y\n\x006\x00e\x005\x004\x009\x001\x00e\x007\x00-\x007\x00f\x001\x005\x00-\x004\x001\x007\x00d\x00-\x00a\x004\x000\x003\x00-345\x00,\x00\x005\x000\x00.\x000\x000\x000\x000\x000\x000\x000\x000\x00\n"
Here's a gist of it, can't figure out a way to post the specific CSV.
All I get from checking the encoding of the file is that it's in binary format, any thoughts on how I could get it into a normal csv?
Note: This is a downloaded CSV so converting it to another encoding via opening it in excel and exporting (or something like that) is not an option :)
Thanks!
Updating with attempted solution 1:
path = Rails.root.join('data', 'mike/test-csv.csv')
CSV.read(path, {:headers => true, :encoding => 'utf-8'}).each do |d|
puts d
end
Result: 6e5491e7-7f15-417d-a403-345,50.00000000
While this is correct, it ONLY works with puts, for example:
CSV.read(path, {:headers => true, :encoding => 'utf-8'}).map { |row| row }
=> [#<CSV::Row "ID":"\u00006\u0000e\u00005\u00004\u00009\u00001\u0000e\u00007\u0000-\u00007\u0000f\u00001\u00005\u0000-\u00004\u00001\u00007\u0000d\u0000-\u0000a\u00004\u00000\u00003\u0000-345\u0000" "\u0000Q\u0000u\u0000a\u0000n\u0000t\u0000i\u0000t\u0000y":"\u0000\u00005\u00000\u0000.\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u0000">]
CSV.read(path, {:headers => true, :encoding => 'utf-8'}).map(&:to_s)
=> ["\u00006\u0000e\u00005\u00004\u00009\u00001\u0000e\u00007\u0000-\u00007\u0000f\u00001\u00005\u0000-\u00004\u00001\u00007\u0000d\u0000-\u0000a\u00004\u00000\u00003\u0000-345\u0000,\u0000\u00005\u00000\u0000.\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u00000\u0000\n"]
It's unfortunately still not the correct string :(
Final Solution (via #ashmaroli below):
path = Rails.root.join('data', 'mike/test-csv.csv')
csv_text = ''
File.open(path, 'r') do |csv|
csv.each_line do |line|
csv_text << line.gsub(/\u0000/, '')
end
end
CSV.parse(csv_text, headers:true).map do |row| row end
Result:
[#<CSV::Row "ID":"6e5491e7-7f15-417d-a403-345" "Quantity":"50.00000000">]
Github Gist
Download Example CSV File
path = Rails.root.join('data', 'mike/test-csv.csv')
file = ""
File.open(path, 'r') do |csv|
csv.each_line do |line|
file << line.gsub(/\u0000/, '')
end
end
print file
print file.inspect # same as above just wraps the string in a
# single line with "\n" chars

Rails 4.2 - how to fix ascii code in CSV exporting without gem 'iconv'?

When exporting csv in Rails 4.2 app, there are ascii code in the csv output for Chinese characters (UTF8):
中åˆåŒç†Šå·¥ç­‰ç”¨é¤
We tried options in send_data without luck:
send_data #payment_requests.to_csv, :type => 'text/csv; charset=utf-8; header=present'
And:
send_data #payment_requests.to_csv.force_encoding("UTF-8")
In model, there is forced encoding utf8:
# encoding: utf-8
But it does not work. There are online posts talking about use gem iconv. However iconv depends on the platform's ruby version. Is there cleaner solution to fix the ascii in Rails 4.2 csv exporting?
If #payment_requests.to_csv includes ASCII text, then you should use encode method:
#payment_requests.to_csv.encode("UTF-8")
or
#payment_requests.to_csv.force_encoding("ASCII").encode("UTF-8")
depending on which internal encoding #payment_requests.to_csv has.
You can try:
#payment_requests.to_csv.force_encoding("ISO-8859-1")
for Chinese characters
CSV.generate(options) do |csv|
csv << column_names
all.each do |product|
csv << product.attributes.values_at(*column_names)
end
end.encode('gb2312', :invalid => :replace, :undef => :replace, :replace => "?")
This is what worked for me:
head = 'EF BB BF'.split(' ').map{|a|a.hex.chr}.join()
csv_str = CSV.generate(csv = head) do |csv|
csv << [ , , , ...]
#elements.each do |element|
csv << [ , , , ...]
end
end

winmail.dat attachment gets corrupted using ActionMailer in Rails app

I am using ActionMailer in a Ruby on Rails app to read emails (ruby 1.9.3, rails 3.2.13).
I have an email that has a winmail.dat file attached to it (ms-tnef) and I am using the tnef gem to extract its contents.
The problem is that when I read the attachment from the mail, it gets corrupted and tnef can not extract files from it.
$ tnef winmail.dat
ERROR: invalid checksum, input file may be corrupted
Extracting the winmail.dat attachment using any mail app, the extracted winmail.dat works fine with tnef and I got it's content.
Comparing the two files I noticed that:
- original file is bigger (76k against 72k)
- they differ on line breaks: Orginal file has the windows format (0D 0A) and the file saved by rails has the linux format (0A)
I wrote this test:
it 'should extract winmail.dat from email and extract its contents' do
file_path = "#{::Rails.root}/spec/files/winmail-dat-001.eml"
message = Mail::Message.new(File.read(file_path))
anexo = message.attachments[0]
files = []
Tnef.unpack(anexo) do |file|
files << File.basename(file)
end
puts files.inspect
files.size.should == 2
end
That fails with these messages:
WARNING: invalid checksum, input file may be corrupted
Invalid RTF CRC, input file may be corrupted
WARNING: invalid checksum, input file may be corrupted
Assertion failed: ((attr->lvl_type == LVL_MESSAGE) || (attr->lvl_type == LVL_ATTACHMENT)), function attr_read, file attr.c, line 240.
Errno::EPIPE: Broken pipe
anexo = message.attachments[0]
=> #<Mail::Part:2159872060, Multipart: false, Headers: <Content-Type: application/ms-tnef; name="winmail.dat">, <Content-Transfer-Encoding: quoted-printable>, <Content-Disposition: attachment; filename="winmail.dat">>
I tried to save it to disk as bynary, and read it again, but I got the same result
it 'should extract winmail.dat from email and extract its contents' do
file_path = "#{::Rails.root}/spec/files/winmail-dat-001.eml"
message = Mail::Message.new(File.read(file_path))
anexo = message.attachments[0]
tmpfile_name = "#{::Rails.root}/tmp/#{anexo.filename}"
File.open(tmpfile_name, 'w+b', 0644) { |f| f.write anexo.body.decoded }
anexo = File.open(tmpfile_name)
files = []
Tnef.unpack(anexo) do |file|
files << File.basename(file)
end
puts files.inspect
files.size.should == 2
end
How should I read the attachment?
The method anexo.body.decoded calls the decode method of the best suited encoding (Mail::Encodings) for the attachment, in your case quoted_printable.
Some of these encodings (7bit, 8bit and quoted_printable), perform a conversion, changing different types of line breaks to the platform specific line break.
the *quoted_printable" call .to_lf that corrupt the winmail.dat file
# Decode the string from Quoted-Printable. Cope with hard line breaks
# that were incorrectly encoded as hex instead of literal CRLF.
def self.decode(str)
str.gsub(/(?:=0D=0A|=0D|=0A)\r\n/, "\r\n").unpack("M*").first.to_lf
end
mail/core_extensions/string.rb:
def to_lf
to_str.gsub(/\n|\r\n|\r/) { "\n" }
end
To solve it you have perform the same encoding without the last .to_lf.
To do that you can create a new encoding that does not corrupt your file and use it to encode you attachment.
create the file:
lib/encodings/tnef_encoding.rb
require 'mail/encodings/7bit'
module Mail
module Encodings
# Encoding to handle Microsoft TNEF format
# It's pretty similar to quoted_printable, except for the 'to_lf' (decode) and 'to_crlf' (encode)
class TnefEncoding < SevenBit
NAME='tnef'
PRIORITY = 2
def self.can_encode?(str)
EightBit.can_encode? str
end
def self.decode(str)
# **difference here** removed '.to_lf'
str.gsub(/(?:=0D=0A|=0D|=0A)\r\n/, "\r\n").unpack("M*").first
end
def self.encode(str)
# **difference here** removed '.to_crlf'
[str.to_lf].pack("M")
end
def self.cost(str)
# These bytes probably do not need encoding
c = str.count("\x9\xA\xD\x20-\x3C\x3E-\x7E")
# Everything else turns into =XX where XX is a
# two digit hex number (taking 3 bytes)
total = (str.bytesize - c)*3 + c
total.to_f/str.bytesize
end
private
Encodings.register(NAME, self)
end
end
end
To use your custom encoding you must, first, register it:
Mail::Encodings.register('tnef', Mail::Encodings::TnefEncoding)
And then, set it as your preferred encoding for the attachment:
anexo.body.encoding('tnef')
Your test would, then, become:
it 'should extract winmail.dat from email and extract its contents' do
file_path = "#{::Rails.root}/spec/files/winmail-dat-001.eml"
message = Mail::Message.new(File.read(file_path))
anexo = message.attachments[0]
tmpfile_name = "#{::Rails.root}/tmp/#{anexo.filename}"
Mail::Encodings.register('tnef', Mail::Encodings::TnefEncoding)
anexo.body.encoding('tnef')
File.open(tmpfile_name, 'w+b', 0644) { |f| f.write anexo.body.decoded }
anexo = File.open(tmpfile_name)
files = []
Tnef.unpack(anexo) do |file|
files << File.basename(file)
end
puts files.inspect
files.size.should == 2
end
Hope it helps!

Why do I get a string encoding issue "\xE2" from ASCII-8BIT to UTF-8?

I'm trying to download a PDF from an email and write the contents to a file. For some reason, I'm getting this error:
An Encoding::UndefinedConversionError occurred in attachments#inbound: "\xE2" from ASCII-8BIT to UTF-8 app/controllers/api/attachments_controller.rb:70:in `write'
Here's my code:
def inbound
if Rails.env.production? or Rails.env.staging?
email = Postmark::Mitt.new(request.body.read)
else
email = Postmark::Mitt.new(File.binread "#{Rails.root}/app/temp_pdfs/email.json")
end
if email.attachments.count == 0
# notify aidin that we got an inbound email with no attachments
respond_to do |format|
format.json { head :no_content }
end
return
end
attachment = email.attachments.first
filename = "attachment" + (Time.now.strftime("%Y%m%d%H%M%S")+(rand * 1000000).round.to_s) + ".pdf"
base_path = "#{Rails.root}/temp_attachments/"
unless File.directory?(base_path)
Dir::mkdir(base_path)
end
file = File.new base_path + filename, 'w+'
file.write Base64.decode64(attachment.source['Content'].encode("UTF-16BE", :invalid=>:replace, :replace=>"?").encode("UTF-8"))
file.close
write_options = write_options()
write_options[:metadata] = {:filename => attachment.file_name, :content_type => attachment.content_type, :size => attachment.size }
obj = s3_object()
file = File.open file.path
obj.write(file.read, write_options)
file.close
FaxAttach.trigger obj.key.split('/').last
render :nothing => true, :status => 202 and return
end
I read around and it looked like the way to solve this was:
file.write Base64.decode64(attachment.source['Content'].encode("UTF-16BE", :invalid=>:replace, :replace=>"?").encode("UTF-8"))
but it doesn't seem to work.
The error message is actually being thrown on the file write, not by your encode/decode inside the params, because Ruby is trying to apply default character encoding on file.write. To prevent this, the quickest fix is to add the b flag when you open the file
file = File.new base_path + filename, 'wb+'
file.write Base64.decode64( attachment.source['Content'] )
That's assuming the incoming attachment is encoded in Base64, as your code implies (I have no way to verify this). The Base64 encoding stored inside attachment.source['Content'] should be the same bytes in ASCII-8BIT and UTF-8, so there is no point converting it inside the call to decode64.

Rails, i need to decode Base64 ziped pdf file

I need to decode base64 ziped pdf file and write decoded data to a tempfile . And need to save it on over local file system.in rails.
decoded_data=Base64.decode64(encoded_data)
file_name = "test.zip"
temp_file = Tempfile.new("filename-#{Time.now}")
File.open(temp_file, 'wb') {|f| f.write(decoded_data)}
send_file temp_file.path, :type => 'application/zip', :filename => file_name

Resources