I'm struggling to attach a Vips:Image object to an ActiveStorage object.
I use Vips to compress a PNG image. I'm then looking to save this compressed PNG version into a second ActiveStorage attachment. The code is failing when attempting to attach. All I get is: Process finished with exit code -1073741819 (0xC0000005)
# compress the image and save the compressed version of the file as a PNG
# both img and img_to_compress are active storage attachments
def compress_charlie(img, img_to_compress)
# load the image as a Vips Image object
vips_img = Vips::Image.new_from_buffer (URI.open(img.url) { |f| f.read }), ''
# do compression, etc ... not bothering to show this code as it has no impact on the issue I have
# save the compressed png
img_to_compress.attach(
io: StringIO.new(vips_img .write_to_buffer('.png')),
filename: img.filename
)
end
Any help is appreciated, Charlie
Ruby 3.1
Rails 7.0.1
You're decompressing and recompressing twice. How about (untested):
def convert_to_png(img, img_to_compress)
# load the image as a Vips Image object
vips_img = Vips::Image.new_from_buffer (URI.open(img.url) { |f| f.read }), ''
# save as a PNG string
png = StringIO.new(vips_img.write_to_buffer('.png')),
img_to_compress.attach(io: png, filename: img.filename)
end
pngsave_buffer gives you a String with binary encoding, you don't need to save a second time. Try this test program:
require "vips"
x = Vips::Image.new_from_file ARGV[0]
y = x.pngsave_buffer
puts "y.class = #{y.class}"
I see:
$ ./pngsave.rb ~/pics/k2.jpg
y.class = String
If the image is already a PNG you'll be wasting a lot of time, of course. You could add something to detect the format and skip the conversion.
Related
I am using rails, active storage and minimagick to try to convert a multi-page PDF into individual PNG images and then store the images with active storage.
I have been able to successfully load a PDF and convert the pages into png images using MiniMagick. However, I have only been able to get Minimagic to store the png images in the app/assets/images directory (as opposed to storing the png images as active storage attachments).
Does anyone know how get MiniMagick to store images with active storage?
Here is the code I'm using:
# Doc model with active storage attachments
class Doc < ApplicationRecord
has_one_attached :pdf #PDF file to upload
has_many_attached :pdf_pages #PNG image for each page of the PDF
end
# PrepareDocs controller manages saving a PNG image of each PDF page.
# NOTE: at this point, the PDF has already been attached to the #doc instance.
# Now we are trying to create the PNG images of each page of the PDF
class PrepareDocsController < ApplicationController
def show
# Load #doc
#doc = Doc.find(params[:id])
# Get PDF from #doc
mini_magic_pdf = MiniMagick::Image.open(ActiveStorage::Blob.service.send(:path_for, #doc.pdf.key))
# Save first page of doc.pdf as a PNG (later I will update this to convert every page to a PNG, but I'm starting on page 1 for now)
MiniMagick::Tool::Convert.new do |convert|
# Prepare formatting for PNG image
convert.background "white"
convert.flatten
convert.density 150
convert.quality 100
# Save PNG Image - This successfully creates a PNG of the first page, but I don't want to store it in my assets.
convert << mini_magic_pdf.pages.first.path
convert << "app/assets/images/page_1.png" # This probably needs to be changed so that we are not storing in the app/assets/imdages directory
# [??? insert additional code to attach the png image to #doc.pdf_pages using active storage instead of to the images directory?]
# NOTE: if I try to access the newly created png in my views, I get "incorrect signature" OR "asset not available in asset pipeline" errors
end
end
end
Any help would be greatly appreciated!! Thanks!
UPDATE: I got it to work and here is the final code:
# Doc model with active storage attachments
class Doc < ApplicationRecord
has_one_attached :pdf #PDF file to upload
has_many_attached :pdf_pages #PNG image for each page of the PDF
end
# PrepareDocs controller manages saving a PNG image of each PDF page.
# NOTE: at this point, the PDF has already been attached to the #doc instance.
# Now we are trying to create the PNG images of each page of the PDF
class PrepareDocsController < ApplicationController
def show
# Load #doc
#doc = Doc.find(params[:id])
# path to current pdf (ie #doc.upoad)
pdf_path = ActiveStorage::Blob.service.path_for(#doc.pdf.key)
# set minimagick image wrapper for pdf stored in #doc.uplaod
magick = MiniMagick::Image.open(pdf_path)
# run repeat block to save each page as an individual image
magick.pages.each_with_index do |page, index|
# set file name to "page_N"
file_name = "page_#{(index+1).to_s}"
# set path for tempfile that you are about to create (using rails post ruby 2.5 approach. Pre 2.5 ruby uses make_tmpname; post 2.5 ruby uses create; I like rails post 2.5 version)
converted_file_path = File.join(Dir.tmpdir, "#{file_name}-#{Time.now.strftime("%Y%m%d")}-#{$$}-#{rand(0x100000000).to_s(36)}-.png")
# create png and save to tempfile path
MiniMagick::Tool::Convert.new do |convert|
# prep format
convert.background "white"
convert.flatten
convert.density 300
convert.quality 100
# add page to be converted
convert << page.path
# add path of page to be converted
convert << converted_file_path
end
# attach using active storage - NOTE: this needs to be done AFTER the convert block
#doc.pdf_pages.attach(io: File.open(converted_file_path), filename: file_name, content_type: "image/png")
# remove tempfile
FileUtils.rm(converted_file_path)
end
end
end
NOTE: I found 3 different methods for creating tempfiles (see here) depending on the ruby version and / or preference:
# pre ruby 2.5
converted_file_path = File.join(Dir.tmpdir, Dir::Tmpname.make_tempname(["page_1", ".png"], nil))
# ruby 2.5 and later
converted_file_path = File.join(Dir.tmpdir, Dir::Tmpname.create(["page_1", ".png"]) {} )
# rails version of pre ruby 2.5
converted_file_path = File.join(Dir.tmpdir, "#{file_name}-#{Time.now.strftime("%Y%m%d")}-#{$$}-#{rand(0x100000000).to_s(36)}-.png")
app/assets is a directory for your asset sources, images from there are copied to public/assets during asset compilation (usually done on deploy), so to be able to access created images - store them somewhere in public, like public/converted/page_1.png (it will have path /converted/page_1.png)
MiniMagick is just a wrapper around ImageMagick command line utilities, so you need to save the result in a temporary location before uploading to activestorage and then use something like:
converted_file_path = File.join(Dir.tmpdir, Dir::Tmpname.make_tmpname(["prefix-", ".png"], nil))
... # do convertation
#doc.pdf_pages.attach(io: File.open(converted_file_path), filename: 'page1.png')
FileUtils.rm(converted_file_path)
I have the following model:
class ScreenshotUploader < CarrierWave::Uploader::Base
include CarrierWave::MiniMagick
storage :file
convert :jpg
version :thumb do
process resize_to_fill: [50, 50]
end
def extension_whitelist
%w(jpg jpeg gif png)
end
version :print do
process border: ['black']
process quality: 80
end
end
The upload of the image happens via pasting an image from the clipboard via https://github.com/layerssss/paste.js and is saved as a base64 encoded string into a <textarea>, then uploaded using the https://github.com/y9v/carrierwave-base64 gem:
class Finding < ApplicationRecord
mount_base64_uploader :screenshot, ScreenshotUploader
end
In the HTML form, it looks like this:
After uploading, the result is the following files:
screenshot.png it's a PNG, not a JPG!
thumb_screenshot.jpg
print_screenshot.jpg
But I need the original file to be also converted to JPG, as I need to save disk space. How can I achieve this?
You can do it like it written on the carrier wave documentation
Just replace system("mogrify -resize '1200\>' #{file.file}") with system("mogrify -format jpg #{file.file}") and then remove original file.
Adding to Vasiliy's answer, I came up with the following:
after :store, :convert_original_to_jpg
def convert_original_to_jpg(new_file)
if version_name.nil?
system("mogrify -format jpg -quality 80 #{file.file}")
system("unlink #{file.file}") # Remove the old PNG file
model.update_column mounted_as, "#{mounted_as}.jpg" # The filename in the DB also needs to be manually set to .jpg!
end
end
While this works for creating the file, it does not when updating the file, as the new_file parameter then is nil, and thus all images are removed.
I think this is some quirk that has to do with the carrierwave-base64 gem, and I don't have any motivation to dig into this any further. So the proposed solution might not be too useful, but for the sake of documentation I wanted to post it here.
In my special case, I decided to let go of the idea of saving disk space by converting PNG to JPG. Instead, I simply set process quality: 80 to save at least some space on the versions.
For the original PNG (which is saved in lossless state by carrierwave-base64 gem), I simply use the following code to shrink its quality:
after :store, :optimise_images
def optimise_images(new_file)
return if Rails.env.test? # Optimising consumes quite some time, so let's disable it for tests
if version_name.nil?
image_optim = ImageOptim.new pngout: false,
svgo: false,
pngcrush: false,
optipng: false,
pngquant: {allow_lossy: true}, # Everything disabled except pngquant, to keep the performance at a good level
advpng: false
image_optim.optimize_images!(Dir["#{File.dirname(file.file)}/*.png"])
end
end
I am hitting an api to get an image that they have stored and use it as the profile pic for our application's users. I'm using Ruby on Rails and ActiveStorage with AWS to attach and store the image. What they send back is this:
{"status"=>"shared", "values"=>[{"$objectType"=>"BinaryData", "data"=>"/9j/4AAQSkZJRgABAQAASABIAAD/4QBMRXhpZgAAT.....KK5tT/9k=", "mime_type"=>"image/jpeg", "metadata"=>{"cropped"=>false}}]}
I tried a lot of different ways to attach it and manipulate the data such as just attaching it as it is, Base64.decode64, Base64.encode64. I also tried creating a new file and then attaching that. Here are some examples:
data = Base64.decode64(Base64.encode64(response[:selfie_image]["values"][0]["data"]))
user.profile_pic.attach!(
io: StringIO.new(open(data).read),
filename: "#{user.first_name}_#{user.last_name}_#{user.id}.jpg",
content_type: "image/jpeg"
)
data = Base64.decode64(Base64.encode64(response[:selfie_image]["values"][0]["data"]))
out_file = File.new("#{user.first_name}_#{user.last_name}_# . {user.id}.jpg", "w")
out_file.puts(data)
out_file.close
user.profile_pic.attach(
io: StringIO.new(open(out_file).read),
filename: "#{user.first_name}_#{user.last_name}_#{user.id}.jpg",
content_type: "image/jpeg"
)
I also tried:
user.profile_pic.attach(out_file)
It keeps either saying attachment is nil or depending on how I manipulate the data it will say not a jpeg file content header type wrong and throw that as an image magick error.
How can I manipulate this data to be able to attach it as an image for our users with ActiveStorage?
To get this to work I had to add gem "mini_magick" to my gemfile and then use it to decode the image data I was receiving from the api call and then turn it into a blob so that ActiveStorage could handle it.
data = response[:selfie_image]["values"][0]["data"]
decoded_data = Base64.decode64(data)
image = MiniMagick::Image.read(decoded_data)
image.format("png")
user.profile_pic.attach(
io: StringIO.new(image.to_blob),
filename: "#{user.id}_#{user.first_name}_#{user.last_name}.png",
content_type: "image/jpeg"
)
In command line ImageMagick you can use the inline: feature to decode base64 data into a gif or jpg. The base64 image here has transparency, it is most proper to save to gif or png.
convert 'inline:data:image/gif;base64,
R0lGODlhIAAgAPIEAAAAAB6Q/76+vvXes////wAAAAAAAAAAACH5BAEAAAUALAAA
AAAgACAAAAOBWLrc/jDKCYG1NBcwegeaxHkeGD4j+Z1OWl4Yu6mAYAu1ebpwL/OE
YCDA0YWAQuJqRwsSeEyaRTUwTlxUqjUymmZpmeI3u62Mv+XWmUzBrpeit7YtB1/r
pTAefv942UcXVX9+MjNVfheGCl18i4ddjwwpPjEslFKDUWeRGj2fnw0JADs=
' b64_noseguy.gif
But you can still save it to jpg. The transparency will be lost and the background will be black.
convert 'inline:data:image/jpeg;base64,
R0lGODlhIAAgAPIEAAAAAB6Q/76+vvXes////wAAAAAAAAAAACH5BAEAAAUALAAA
AAAgACAAAAOBWLrc/jDKCYG1NBcwegeaxHkeGD4j+Z1OWl4Yu6mAYAu1ebpwL/OE
YCDA0YWAQuJqRwsSeEyaRTUwTlxUqjUymmZpmeI3u62Mv+XWmUzBrpeit7YtB1/r
pTAefv942UcXVX9+MjNVfheGCl18i4ddjwwpPjEslFKDUWeRGj2fnw0JADs=
' b64_noseguy.jpg
See https://imagemagick.org/Usage/files/#inline
Sorry, I do not know the equivalent in RMagick
The code below resizes my image. But I am not sure how to write it out to a temp file or blob so I can upload it to s3.
origImage = MiniMagick::Image.open(myPhoto.tempfile.path)
origImage.resize "200x200"
thumbKey = "tiny-#{key}"
obj = bucket.objects[thumbKey].write(:file => origImage.write("tiny.jpg"))
I can upload the original file just fine to s3 with the below command:
obj = bucket.objects[key].write('data')
obj.write(:file => myPhoto.tempfile)
I think I want to create a temp file, read the image file into it and upload that:
thumbFile = Tempfile.new('temp')
thumbFile.write(origImage.read)
obj = bucket.objects[thumbKey].write(:file => thumbFile)
but the origImage class doesn't have a read command.
UPDATE: I was reading the source code and found this out about the write command
# Writes the temporary file out to either a file location (by passing in a String) or by
# passing in a Stream that you can #write(chunk) to repeatedly
#
# #param output_to [IOStream, String] Some kind of stream object that needs to be read or a file path as a String
# #return [IOStream, Boolean] If you pass in a file location [String] then you get a success boolean. If its a stream, you get it back.
# Writes the temporary image that we are using for processing to the output path
And the s3 api docs say you can stream the content using a code block like:
obj.write do |buffer, bytes|
# writing fewer than the requested number of bytes to the buffer
# will cause write to stop yielding to the block
end
How do I change my code so
origImage.write(s3stream here)
http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html
UPDATE 2
This code successfully uploads the thumbnail file to s3. But I would still love to know how to stream it up. It would be much more efficient I think.
#resize image and upload a thumbnail
smallImage = MiniMagick::Image.open(myPhoto.tempfile.path)
smallImage.resize "200x200"
thumbKey = "tiny-#{key}"
newFile = Tempfile.new("tempimage")
smallImage.write(newFile.path)
obj = bucket.objects[thumbKey].write('data')
obj.write(:file => newFile)
smallImage.to_blob ?
below code copy from https://github.com/probablycorey/mini_magick/blob/master/lib/mini_magick.rb
# Gives you raw image data back
# #return [String] binary string
def to_blob
f = File.new #path
f.binmode
f.read
ensure
f.close if f
end
Have you looked into the paperclip gem? The gem offers direct compatibility to s3 and works great.
I have a Dragonfly processor which should take a given PDF and return a PNG of the first page of the document.
When I run this processor via the console, I get back the PNG as expected, however, when in the context of Rails, I'm getting it as a PDF.
My code is roughly similar to this:
def to_pdf_thumbnail(temp_object)
tempfile = new_tempfile('png')
args = "'#{temp_object.path}[0]' '#{tempfile.path}'"
full_command = "convert #{args}"
result = `#{full_command}`
tempfile
end
def new_tempfile(ext=nil)
tempfile = ext ? Tempfile.new(['dragonfly', ".#{ext}"]) : Tempfile.new('dragonfly')
tempfile.binmode
tempfile.close
tempfile
end
Now, tempfile is definitely creating a .png file, but the convert is generating a PDF (when run from within Rails 3).
Any ideas as to what the issue might be here? Is something getting confused about the content type?
I should add that both this and a standard conversion (asset.png.url) both yield a PDF with the PDF content as a small block in the middle of the (A4) image.
An approach I’m using for this is to generate the thumbnail PNG on the fly via the thumb method from Dragonfly’s ImageMagick plugin:
<%= image_tag rails_model.file.thumb('100x100#', format: 'png', frame: 0).url %>
So long as Ghostscript is installed, ImageMagick/Dragonfly will honour the format / frame (i.e. page of the PDF) settings. If file is an image rather than a PDF, it will be converted to a PNG, and the frame number ignored (unless it’s a GIF).
Try this
def to_pdf_thumbnail(temp_object)
ret = ''
tempfile = new_tempfile('png')
system("convert",tmp_object.path[0],tmpfile.path)
tempfile.open {|f| ret = f.read }
ret
end
The problem is you are likely handing convert ONE argument not two
Doesn't convert rely on the extension to determine the type? Are you sure the tempfiles have the proper extensions?