I'm having trouble calling image.write with MiniMagick on a decoded base64 image in Rails. Every line seems to be working properly except for image.write. The code below is in my Rails API ImageController, which my React frontend is hitting through a POST request with the encoded image.
def create
uploaded_io = params["image_io"]["base64"] # base64 string + metadata
metadata = uploaded_io.split(',/')[0] + "," # "data:image/jpeg;base64,"
filetype = metadata.split("/")[1].split("base64")[0][0...-1] # "jpeg"
base64_string = uploaded_io[metadata.size..-1] # base64 string w/o metadata
blob = Base64.decode64(base64_string)
image = MiniMagick::Image.read(blob)
image.write `#{Time.new.to_i}.#{filetype}`
storage = Google::Cloud::Storage.new(
project_id: ENV['GOOGLE_CLOUD_PROJECT'],
credentials: JSON.parse(File.read('config/google_cloud_credentials.json'))
)
bucket = storage.bucket "auto-stock-189103.appspot.com"
bucket.create_file image,`test/#{Time.new.to_i}.jpg`
end
I added comments to the first few lines in the code describing their value. base64_string was too long to comment, so here is its value:
"/9j/4QAYRXhpZgAASUkqAAgAAAAAAAAAAAAAAP/sABFEdWNreQABAAQAAABkAAD/4QMtaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wLwA8P3hwYWNrZXQgYmVnaW49Iu+7vyIgaWQ9Ilc1TTBNcENlaGlIenJlU3pOVGN6a2M5ZCI/PiA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJBZG9iZSBYTVAgQ29yZSA1LjMtYzAxMSA2Ni4xNDU2NjEsIDIwMTIvMDIvMDYtMTQ6NTY6MjcgICAgICAgICI+IDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+IDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiIHhtbG5zOnhtcD0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wLyIgeG1sbnM6eG1wTU09Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9tbS8iIHhtbG5zOnN0UmVmPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvc1R5cGUvUmVzb3VyY2VSZWYjIiB4bXA6Q3JlYXRvclRvb2w9IkFkb2JlIFBob3Rvc2hvcCBDUzYgKE1hY2ludG9zaCkiIHhtcE1NOkluc3RhbmNlSUQ9InhtcC5paWQ6MTNDQjQyRjlEQTAxMTFFN0E4N0VBNzdGODEwREFGMTYiIHhtcE1NOkRvY3VtZW50SUQ9InhtcC5kaWQ6MTNDQjQyRkFEQTAxMTFFN0E4N0VBNzdGODEwREFGMTYiPiA8eG1wTU06RGVyaXZlZEZyb20gc3RSZWY6aW5zdGFuY2VJRD0ieG1wLmlpZDoxM0NCNDJGN0RBMDExMUU3QTg3RUE3N0Y4MTBEQUYxNiIgc3RSZWY6ZG9jdW1lbnRJRD0ieG1wLmRpZDoxM0NCNDJGOERBMDExMUU3QTg3RUE3N0Y4MTBEQUYxNiIvPiA8L3JkZjpEZXNjcmlwdGlvbj4gPC9yZGY6UkRGPiA8L3g6eG1wbWV0YT4gPD94cGFja2V0IGVuZD0iciI/Pv/uAA5BZG9iZQBkwAAAAAH/2wCEAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQECAgICAgICAgICAgMDAwMDAwMDAwMBAQEBAQEBAgEBAgICAQICAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDA//AABEIAAoACgMBEQACEQEDEQH/xABNAAEBAAAAAAAAAAAAAAAAAAAACQEBAQEAAAAAAAAAAAAAAAAAAAkKEAEAAAAAAAAAAAAAAAAAAAAAEQEAAAAAAAAAAAAAAAAAAAAA/9oADAMBAAIRAxEAPwCMKU7f4AAA/9k="
Testing it here renders the correct image (a red square), but when I run the image.write line it returns the following error:
bin/rails: No such file or directory - 1513397345.jpeg
*** NoMethodError Exception: undefined method `write' for nil:NilClass
nil
Here's the return value of image = MiniMagick::Image.read(blob) for reference:
#<MiniMagick::Image:0x00007f9ba76ba1f8 #path="/var/folders/pf/xhvv11092_j08hw47q6rt9z80000gn/T/mini_magick20171215-26353-l2lcyu", #tempfile=#<Tempfile:/var/folders/pf/xhvv11092_j08hw47q6rt9z80000gn/T/mini_magick20171215-26353-l2lcyu (closed)>, #info=#<MiniMagick::Image::Info:0x00007f9ba76ba1d0 #path="/var/folders/pf/xhvv11092_j08hw47q6rt9z80000gn/T/mini_magick20171215-26353-l2lcyu", #info={}>>
Ultimately, my goal is to upload the image to Google Cloud so please let me know if there's a better way to go about this. I'm following this answer from a similar question, which is why I have it structured this way.
I think your problem is that you're using backticks where you mean to use double quotes:
image.write `#{Time.new.to_i}.#{filetype}`
# ----------^----------------------------^
Backticks will attempt to execute their contents in the shell. You don't have an executable file named 1513397345.jpeg (which is what #{Time.new.to_i}.#{filetype} evaluates to) so you get an error.
You just want to use plain old double quotes to get the string interpolation you're expecting:
image.write "#{Time.new.to_i}.#{filetype}"
and again a few lines below that:
bucket.create_file image, "test/#{Time.new.to_i}.jpg"
Furthermore, you probably want to store that filename in a variable because Time.new.to_i isn't guaranteed to be the same in both invocation:
name = "#{Time.new.to_i}.#{filetype}
image.write name
#...
bucket.create_file image, name
Related
I am using Libreconv gem to convert word to doc but it's not working with S3
bucket = Aws::S3::Bucket.new('bucket-name')
object = bucket.object file.attachment.blob.key
path = object.presigned_url(:get)
Libreconv.convert(path, "public/test.pdf")
If I try to convert this path to PDF using Libreconv then it's give me filename too long error. I have wrriten this code under ActiveJobs. So kindly provide me solutions as per ActiveJobs.
Can someone please suggest me how can I convert word file to pdf.
Here path is https://domain.s3.amazonaws.com/Bf5qPUP3znZGCHCcTWHcR5Nn?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZ6RZ7J425ORVUYQ%2F20181206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181206T051240Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=b89c47a324b2aa423bf64dfb343e3b3c90dce9b54fa9fe1bc4efa9c248e912f9
and error I am getting is
Error: source file could not be loaded
*** Errno::ENAMETOOLONG Exception: File name too long # rb_sysopen - /tmp/Bf5qPUP3znZGCHCcTWHcR5Nn?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZ6RZ7J425ORVUYQ%2F20181206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181206T051240Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=b89c47a324b2aa423bf64dfb343e3b3c90dce9b54fa9fe1bc4efa9c248e912f9.pd
It seems that you PDF is created with all the params needed to fetch docx from S3.
I suppose it happens in this line:
target_tmp_file = "#{target_path}/#{File.basename(#source, ".*")}.#{File.basename(#convert_to, ":*")}"
#source is https://domain.s3.amazonaws.com/Bf5qPUP3znZGCHCcTWHcR5Nn?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZ6RZ7J425ORVUYQ%2F20181206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181206T051240Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=b89c47a324b2aa423bf64dfb343e3b3c90dce9b54fa9fe1bc4efa9c248e912f9 and
> File.basename(#source, ".*")
=> "Bf5qPUP3znZGCHCcTWHcR5Nn?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIZ6RZ7J425ORVUYQ%2F20181206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20181206T051240Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=b89c47a324b2aa423bf64dfb343e3b3c90dce9b54fa9fe1bc4efa9c248e912f9"
As a result Libreconv gem tries to create a tmp file with this long name and it's too long - that's why an error is raised.
Possible solution: split the process into separate steps of fetching file and converting it. Something like:
require "open-uri"
bucket = Aws::S3::Bucket.new('bucket-name')
object = bucket.object file.attachment.blob.key
path = object.presigned_url(:get)
doc_file = open(path)
begin
Libreconv.convert(doc_file.path, "public/test.pdf")
ensure
doc_file.delete
end
following is the answer using combine pdf gem
tape = Tape.new(file)
result = tape.preview
tempfile = Tempfile.new(['foo', '.pdf'])
File.open(tempfile, 'wb') do |f|
f.write result
end
path = tempfile.path
combine_pdf(path)
and for load file for S3 I have used
object = #bucket.object object_key
path = object.presigned_url(:get)
response = Net::HTTP.get_response(URI.parse(path)).body
I have working Paperclip gem in my app for documents (pdf, doc). I need to pass the document to some other third party application via post request.
I tried to convert the paperclip attachment via Base64 but it throws error:
no implicit conversion of Tempfile into String
Here is how I did it:
# get url from the paperclip file
url = document.doc.url # https://s3-ap-southeast-1.amazonaws.com/xx-eng/documents/xx/000/000/xx/original/doc.pdf
file_data = open(url)
# Encode the bytes to base64 - this line throw error
base_64_file = Base64.encode64(file_data)
Do you have any suggestion how to avoid the Tempfile error?
You need to read file first.
base_64_file = Base64.encode64(file_data.read)
Here is working example:
$ bundle exec rails c
=> file = open("tmp/file.pdf")
#> #<File:tmp/receipts.pdf>
=> base_64 = Base64.encode64(file)
#> TypeError: no implicit conversion of File into String
=> base_64 = Base64.encode64(file.read)
#> "JVBERi0xLjQKMSAwIG9iago8PAovVGl0b/BBQEPgQ ......J0ZgozMDM0OQolJUVPRgo=\n"
The answer from #3елёный didn't work to me - maybe because it's the S3 file.
However I managed to find a way with Paperclip method:
file_data = Paperclip.io_adapters.for(url).read
base_64_file = Base64.encode64(file_data)
The google vision API requires a bitmap sent as an argument. I am trying to convert a png from a URL to a bitmap to pass to the google api:
require "google/cloud/vision"
PROJECT_ID = Rails.application.secrets["project_id"]
KEY_FILE = "#{Rails.root}/#{Rails.application.secrets["key_file"]}"
google_vision = Google::Cloud::Vision.new project: PROJECT_ID, keyfile: KEY_FILE
img = open("https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png").read
image = google_vision.image img
ArgumentError: string contains null byte
This is the source code processing of the gem:
def self.from_source source, vision = nil
if source.respond_to?(:read) && source.respond_to?(:rewind)
return from_io(source, vision)
end
# Convert Storage::File objects to the URL
source = source.to_gs_url if source.respond_to? :to_gs_url
# Everything should be a string from now on
source = String source
# Create an Image from a HTTP/HTTPS URL or Google Storage URL.
return from_url(source, vision) if url? source
# Create an image from a file on the filesystem
if File.file? source
unless File.readable? source
fail ArgumentError, "Cannot read #{source}"
end
return from_io(File.open(source, "rb"), vision)
end
fail ArgumentError, "Unable to convert #{source} to an Image"
end
https://github.com/GoogleCloudPlatform/google-cloud-ruby
Why is it telling me string contains null byte? How can I get a bitmap in ruby?
According to the documentation (which, to be fair, is not exactly easy to find without digging into the source code), Google::Cloud::Vision#image doesn't want the raw image bytes, it wants a path or URL of some sort:
Use Vision::Project#image to create images for the Cloud Vision service.
You can provide a file path:
[...]
Or any publicly-accessible image HTTP/HTTPS URL:
[...]
Or, you can initialize the image with a Google Cloud Storage URI:
So you'd want to say something like:
image = google_vision.image "https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png"
instead of reading the image data yourself.
Instead of using write you want to use IO.copy_stream as it streams the download straight to the file system instead of reading the whole file into memory and then writing it:
require 'open-uri'
require 'tempfile'
uri = URI("https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png")
tmp_img = Tempfile.new(uri.path.split('/').last)
IO.copy_stream(open(uri), tmp_img)
Note that you don't need to set the 'r:BINARY' flag as the bytes are just streamed without actually reading the file.
You can then use the file by:
require "google/cloud/vision"
# Use fetch as it raises an error if the key is not present
PROJECT_ID = Rails.application.secrets.fetch("project_id")
# Rails.root is a Pathname object so use `.join` to construct paths
KEY_FILE = Rails.root.join(Rails.application.secrets.fetch("key_file"))
google_vision = Google::Cloud::Vision.new(
project: PROJECT_ID,
keyfile: KEY_FILE
)
image = google_vision.image(File.absolute_path(tmp_img))
When you are done you clean up by calling tmp_img.unlink.
Remember to read things in binary format:
open("https://www.google.com/..._272x92dp.png",'r:BINARY').read
If you forget this it might try and open it as UTF-8 textual data which would cause lots of problems.
I get image files sent from an Android app to my Rails API. I decode the images using this:
StringIO.new(Base64.decode64(image[1]))
The issue is that it takes too much time; on heroku it takes even longer.
Is there another way to do this that's faster and more efficient?
You can also use this for decode base64:
# this method for decode base64 code to file
def parse_image_data(image[1])
base64_file = image[1]
ext, string = base64_file.split(',')
ext = MIME::Types[base64_file].first.preferred_extension if ext.include?("base64")
tempfile = Tempfile.new(["#{DateTime.now.to_i}", ".#{ext}"])
tempfile.binmode
tempfile.write Base64.decode64(string)
tempfile.rewind
tempfile
end
The code below resizes my image. But I am not sure how to write it out to a temp file or blob so I can upload it to s3.
origImage = MiniMagick::Image.open(myPhoto.tempfile.path)
origImage.resize "200x200"
thumbKey = "tiny-#{key}"
obj = bucket.objects[thumbKey].write(:file => origImage.write("tiny.jpg"))
I can upload the original file just fine to s3 with the below command:
obj = bucket.objects[key].write('data')
obj.write(:file => myPhoto.tempfile)
I think I want to create a temp file, read the image file into it and upload that:
thumbFile = Tempfile.new('temp')
thumbFile.write(origImage.read)
obj = bucket.objects[thumbKey].write(:file => thumbFile)
but the origImage class doesn't have a read command.
UPDATE: I was reading the source code and found this out about the write command
# Writes the temporary file out to either a file location (by passing in a String) or by
# passing in a Stream that you can #write(chunk) to repeatedly
#
# #param output_to [IOStream, String] Some kind of stream object that needs to be read or a file path as a String
# #return [IOStream, Boolean] If you pass in a file location [String] then you get a success boolean. If its a stream, you get it back.
# Writes the temporary image that we are using for processing to the output path
And the s3 api docs say you can stream the content using a code block like:
obj.write do |buffer, bytes|
# writing fewer than the requested number of bytes to the buffer
# will cause write to stop yielding to the block
end
How do I change my code so
origImage.write(s3stream here)
http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html
UPDATE 2
This code successfully uploads the thumbnail file to s3. But I would still love to know how to stream it up. It would be much more efficient I think.
#resize image and upload a thumbnail
smallImage = MiniMagick::Image.open(myPhoto.tempfile.path)
smallImage.resize "200x200"
thumbKey = "tiny-#{key}"
newFile = Tempfile.new("tempimage")
smallImage.write(newFile.path)
obj = bucket.objects[thumbKey].write('data')
obj.write(:file => newFile)
smallImage.to_blob ?
below code copy from https://github.com/probablycorey/mini_magick/blob/master/lib/mini_magick.rb
# Gives you raw image data back
# #return [String] binary string
def to_blob
f = File.new #path
f.binmode
f.read
ensure
f.close if f
end
Have you looked into the paperclip gem? The gem offers direct compatibility to s3 and works great.