Custom Paperclip::Processor with AWS S3 - ruby-on-rails

This code works fine if the uploads are stored locally, but sometimes they are on S3 so it's not possible to just source: "#{File.expand_path(src.path)}[0]". How do I make the Paperclip's run method load images from S3 and replace them afterwards?
module Paperclip
class KskCrop < Processor
def initialize(file, options = {}, attachment = nil)
super
#crop = options
#format = File.extname(#file.path)
#basename = File.basename(#file.path, #format)
end
def make
src = #file
dst = Tempfile.new([#basename, #format])
dst.binmode
parameters = []
parameters << ":source"
parameters << "-crop '#{#crop[2]}x#{#crop[3]}+#{#crop[0]}+#{#crop[1]}'"
parameters << ":dest"
parameters = parameters.flatten.compact.join(' ').strip.squeeze(' ')
success = Paperclip.run('convert', parameters, source: "#{File.expand_path(src.path)}[0]", dest: File.expand_path(dst.path))
dst
end
end
end

So, as per the conversation. It turned out that path has to be a complete url instead of a relative path. Hence, this line will look something like this:
success = Paperclip.run('convert', parameters, source: src.url, dest: File.expand_path(dst.path)
As, OP has already answered his question with an if.. else. I think that's a better approach to check if the attachment is on a local file or on a CDN.
P.S.: I also learned that Paperclip.run(..) method can actually locate and download a file for processing if it's a url without making an IO operations at developer's end.

well i didnt got your question well but for Amazon S3 READ/WRITE, this is what is use...
this is my s3.yml
development:
bucket: app_development
access_key_id: xxxxxxxxxxxxxx
secret_access_key: xxxxxxxxxxxxx+xxxxxxxxxxx
production:
bucket: app_production
access_key_id: xxxxxxxxxxxxxxxxx
secret_access_key: xxxxxxxxxxxx+xxxxxxxxxxxxx
my config/initializers/paperclip.rb
Paperclip::Attachment.default_options[:url] = ':s3_domain_url'
Paperclip::Attachment.default_options[:path] = '/:class/:id/:style/:filename'
##will store in the foll way shown below in the bucket u specify in s3.yml
##http://app_development.s3.amazonaws.com/videos/1/original/small.mp4
##if Rails.env == "production"
#S3_CREDENTIALS = { :access_key_id => ENV['S3_KEY'], :secret_access_key => ENV['S3_SECRET'], :bucket => "ourbucket"}
##else
S3_CREDENTIALS = Rails.root.join("config/s3.yml")
##end
so after this configuration,any paperclip model with has_attached_file:avatar will get uploaded on s3
##upload to s3..code snippet (can also use in _form.html.erb as f.file_field :avatar)
#video=Video.create!({ :avatar => File.new("#{Rails.root}/app/assets/images/dashboard/default_avatar.jpg")
})
##this is how u can display/access any uploaded object/model using url(which shows original unless you specify some styles -thumbnail,small)
#video.avatar.url
#video.avatar.url(:thumbnail)
#video.avatar.url(:small)

Looks like this is the solution!
module Paperclip
class KskCrop < Processor
def initialize(file, options = {}, attachment = nil)
super
#crop = options
#format = File.extname(#file.path)
#basename = File.basename(#file.path, #format)
end
def make
src = #file
dst = Tempfile.new([#basename, #format])
dst.binmode
parameters = []
parameters << ":source"
parameters << "-crop '#{#crop[2]}x#{#crop[3]}+#{#crop[0]}+#{#crop[1]}'"
parameters << ":dest"
parameters = parameters.flatten.compact.join(' ').strip.squeeze(' ')
path = if #file.options && #file.options[:storage] == :s3
src.url
else
File.expand_path(src.path)
end
success = Paperclip.run('convert', parameters, source: path, dest: File.expand_path(dst.path))
dst
end
end
end

Related

how to read a file from blob and save it to public folder in rails

i am reading a pdf file from blob and saving it to local folder
#doc = Document.find(params[:id])
blob = #doc.docs.first.blob
ActiveStorage::Downloader.new(blob.service).open(blob.key, checksum: blob.checksum) do |tempFile|
temppdf = Tempfile.new ["mypdf", ".pdf"]
pathpdf = "app/assets/images/" + current_account.email+'.pdf'
tempFile.save(pathpdf)
end
You can use ActiveStorage::Blob#open to steam the file directly to disk circumventing the tempfile:
#doc = Document.find(params[:id])
blob = #doc.docs.first.blob
# app/assets certainly isn't a public path...
fn = Rails.root.join('public', 'uploads', 'mypdf.pdf')
File.open(fn, "wb+") do |file|
blob.download { |chunk| file.write(chunk) }
end

Access ActiveStorageBlob or ActiveStorageAttachment like it would be a native model

Would it be possible to access the ActiveStorageBlob or ActiveStorageAttachment like it would be a native model ?
E.g.
I want to do ActiveStorageBlob.first to access the first record of this model/table.
or. ActiveStorageAttachment.all.as_json to generate json formated print.
The background idea is to find a way how to dump the content of these ActiveStorage related tables as json formated files. Then change simething on these files, and load it back.
----Extending this text after got correct answer-----
Thank you very much Sarah Marie.
And I hope you know how to load the JSON data back into these tables ?
I have tried this :
dump_file_path = File.join(Rails.root, "backup", active_storage_blobs_file)
load_json = JSON.parse(File.read(dump_file_path))
load_json.each do |j|
ActiveStorage::Blob.create(j)
end
But thats not working.
ActiveModel::UnknownAttributeError (unknown attribute
'attachable_sgid' for ActiveStorage::Blob.)
ActiveStorage::Blob.first
ActiveStorage::Attachment.all.as_json
---- For second extended question ----
ActiveStorage::Blob.create_before_direct_upload!(
filename: j[:filename],
content_type: j[:content_type],
byte_size: j[:byte_size],
checksum: j[:checksum]
)
# or
ActiveStorage::Blob.create_before_direct_upload!(**j.symbolize_keys)
Reference: https://github.com/rails/rails/blob/5f3ff60084ab5d5921ca3499814e4697f8350ee7/activestorage/app/controllers/active_storage/direct_uploads_controller.rb#L8-L9
https://github.com/rails/rails/blob/098fd7f9b3d5c6f540911bc0c17207d6b48d5bb3/activestorage/app/models/active_storage/blob.rb#L113-L120
Now I have a complete solution, how to dump and load the ActiveStorage tables as JSON files.
...dump it
active_storage_blobs_file = "active_storage_blob.json"
active_storage_attachments_file = "active_storage_attachment.json"
puts("...dump active_storage_blob")
dump_file_path = File.join(Rails.root, "backup",active_storage_blobs_file)
dump_file = File.open(dump_file_path, "w")
dump_file.write(JSON.pretty_generate(ActiveStorage::Blob.all.as_json))
dump_file.close()
puts("...dump active_storage_attachment")
dump_file_path = File.join(Rails.root, "backup",
active_storage_attachments_file)
dump_file = File.open(dump_file_path, "w")
dump_file.write(JSON.pretty_generate(ActiveStorage::Attachment.all.as_json))
dump_file.close()
...load it back
puts("...load active_storage_blob")
dump_file_path = File.join(Rails.root, "backup", active_storage_blobs_file)
abort("File does not exist (" + dump_file_path + ") > abort <") unless File.exist?(dump_file_path)
load_json = JSON.parse(File.read(dump_file_path))
load_json.each do |j|
j = j.except("attachable_sgid")
result = ActiveStorage::Blob.create(j)
if (not result.errors.empty?)
puts(result.errors.full_messages.to_s)
puts(j.inspect)
exit(1)
end
end
puts("...load active_storage_attachment")
dump_file_path = File.join(Rails.root, "backup", active_storage_attachments_file)
abort("File does not exist (" + dump_file_path + ") > abort <") unless File.exist?(dump_file_path)
load_json = JSON.parse(File.read(dump_file_path))
load_json.each do |j|
result = ActiveStorage::Attachment.create(j)
if (not result.errors.empty?)
puts(result.errors.full_messages.to_s)
puts(j.inspect)
exit(1)
end
end

How to upload base64 image to s3 from rails controller?

I am making an ajax reject to my controller with some data and base64 image and now I want to upload this image to s3 and replace base64 with the image url. I am following this https://sebastiandobrincu.com/blog/how-to-upload-images-to-rails-api-using-s3
def split_base64(uri_str)
if uri_str.match(%r{^data:(.*?);(.*?),(.*)$})
uri = Hash.new
uri[:type] = $1 # "image/gif"
uri[:encoder] = $2 # "base64"
uri[:data] = $3 # data string
uri[:extension] = $1.split('/')[1] # "gif"
return uri
else
return nil
end
end
def convert_data_uri_to_upload(image_url)
image_data = split_base64(image_url)
image_data_string = image_data[:data]
image_data_binary = Base64.decode64(image_data_string)
temp_img_file = Tempfile.new("")
temp_img_file.binmode
temp_img_file << image_data_binary
temp_img_file.rewind
img_params = {:filename => "image.#{image_data[:extension]}", :type => image_data[:type], :tempfile => temp_img_file}
uploaded_file = ActionDispatch::Http::UploadedFile.new(img_params)
end
return uploaded_file
end
Now from my controller I am passing
convert_data_uri_to_upload(base64_image)
Now I don't know where to write the AWS credentials. According to the url I have to write the credentials in Fog.rb file but I don't have any such file. I have created one ImageUploader inside uploader folder which extends CarrierWave and wrote the configurations but it is also not working.
You can use dotenv ruby gem follow this : http://www.rubydoc.info/gems/dotenv-rails/2.1.1

AWS::S3::Errors::NoSuchKey: No Such Key error

I'm trying to create a method that deletes files on an S3 instance, but I am getting a AWS::S3::Errors::NoSuchKey: No Such Key error when I try to call .head or .read on an object.
app/models/file_item.rb
def thumbnail
{
exists: thumbnailable?,
small: "http://#{bucket}.s3.amazonaws.com/images/#{id}/small_thumb.png",
large: "http://#{bucket}.s3.amazonaws.com/images/#{id}/large_thumb.png"
}
end
lib/adapters/amazons3/accessor.rb
module Adapters
module AmazonS3
class Accessor
S3_BUCKET = AWS::S3.new.buckets[ENV['AMAZON_BUCKET']]
...
def self.delete_file(thumbnail)
prefix_pattern = %r{http://[MY-S3-HOST]-[a-z]+.s3.amazonaws.com/}
small_path = thumbnail[:small].sub(prefix_pattern, '')
large_path = thumbnail[:large].sub(prefix_pattern, '')
small = S3_BUCKET.objects[small_path]
large = S3_BUCKET.objects[large_path]
binding.pry
S3_BUCKET.objects.delete([small, large])
end
end
end
end
example url1
"http://projectname-staging.s3.amazonaws.com/images/994/small_thumb.png"
example url2
"http://projectname-production.s3.amazonaws.com/images/994/large_thumb.png"
assuming awssdk v1 for ruby.
small = S3_BUCKET.objects[small_path]
does not actually get any objects.
from: https://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/Bucket.html
bucket.objects['key'] #=> makes no request, returns an S3Object
bucket.objects.each do |obj|
puts obj.key
end
so you would need to alter your code to something like:
to_delete = []
S3_BUCKET.objects[small_path].each do |obj|
to_delete << obj.key
end
S3_BUCKET.objects[large_path].each do |obj|
to_delete << obj.key
end
S3_BUCKET.objects.delete(to_delete)
just banged out the code, so the idea is there, you might need to correct/polish it a bit
I was able to come of with a kind of different solution thanks to your answer of #Mircea above.
def self.delete_file(thumbnail)
folder = thumbnail[:small].match(/(\d+)(?!.*\d)/)
to_delete = []
S3_BUCKET.objects.with_prefix("images/#{folder}").each do |thumb|
to_delete << thumb.key
end
# binding.pry
S3_BUCKET.objects.delete(to_delete)
end

Is there an easy way to automatic vertical/horizontal align images with paperclip?

I always want square images of 100x100 but i dont want to scale.
If the it is an image of 100X80 i want it vertical aligned with 10 white pixels on top and 10 white pixels on bottm.
Is there an easy way to handle this?
Using Rails 2.1 with paperclip 2.3.0
Might not be appropriate, however you can use CSS to easily acchieve this:
html:
<div class="paperclip-holder">
<%= your_paperclip_image_tag %>
</div>
css:
.paperclip-holder{
width: 100px;
height: 100px;
background: #ffffff;
vertical-align: center;
text-align: center;
}
Ok i finally managed it.
i used this script to crop the image the way i want.
http://www.fmwconcepts.com/imagemagick/squareup/index.php
I made a new Paperclip processor and named CropFile. I copy pasted the Thumbnail processor from paper clip and added some lines to run the squareup script.
in my model:
has_attached_file :widget_image, :styles => { :square_60 => ["60x60", :png], :square_100 => ["100x100", :png] }, :processors => [:crop_file]
I added paperclip_postprocess.rb in initializers
#paperclip_postprocess.rb
module Paperclip
# Handles thumbnailing images that are uploaded.
class CropFile < Processor
attr_accessor :current_geometry, :target_geometry, :format, :whiny, :convert_options, :source_file_options
# Creates a Thumbnail object set to work on the +file+ given. It
# will attempt to transform the image into one defined by +target_geometry+
# which is a "WxH"-style string. +format+ will be inferred from the +file+
# unless specified. Thumbnail creation will raise no errors unless
# +whiny+ is true (which it is, by default. If +convert_options+ is
# set, the options will be appended to the convert command upon image conversion
def initialize file, options = {}, attachment = nil
super
geometry = options[:geometry]
#file = file
#crop = geometry[-1,1] == '#'
#target_geometry = Geometry.parse geometry
#current_geometry = Geometry.from_file #file
#source_file_options = options[:source_file_options]
#convert_options = options[:convert_options]
#whiny = options[:whiny].nil? ? true : options[:whiny]
#format = options[:format]
#path = options[:path]
#source_file_options = #source_file_options.split(/\s+/) if #source_file_options.respond_to?(:split)
#convert_options = #convert_options.split(/\s+/) if #convert_options.respond_to?(:split)
#current_format = File.extname(#file.path)
#basename = File.basename(#file.path, #current_format)
# reg exp to get the with and the height
geometry.match(/(\d+)(#|>|<)?x/)
#width = $1
#height = $3
end
# Returns true if the +target_geometry+ is meant to crop.
def crop?
#crop
end
# Returns true if the image is meant to make use of additional convert options.
def convert_options?
!#convert_options.nil? && !#convert_options.empty?
end
# Performs the conversion of the +file+ into a thumbnail. Returns the Tempfile
# that contains the new image.
def make
src = #file
dst = Tempfile.new(['bla', ".png"])
dst.binmode
begin
parameters = []
parameters << source_file_options
parameters << ":source"
parameters << transformation_command
parameters << convert_options
parameters << ":dest"
parameters = parameters.flatten.compact.join(" ").strip.squeeze(" ")
Paperclip.run("convert", parameters, :source => "#{File.expand_path(src.path)}[0]", :dest => File.expand_path(dst.path))
# Cropping it with vertical and horizontal aligning
`#{Rails.root}/public/scripts/squareup.sh -c trans -s #{#width} #{File.expand_path(src.path)} #{File.expand_path(dst.path)}`
rescue PaperclipCommandLineError => e
raise PaperclipError, "There was an error processing the thumbnail for #{#basename}" if #whiny
end
dst
end
# Returns the command ImageMagick's +convert+ needs to transform the image
# into the thumbnail.
def transformation_command
scale, crop = #current_geometry.transformation_to(#target_geometry, crop?)
trans = []
trans << "-resize" << %["#{scale}"] unless scale.nil? || scale.empty?
trans << "-crop" << %["#{crop}"] << "+repage" if crop
trans
end
end
end
I added the following line to crop the images after paperclip has converted them
# Cropping it with vertical and horizontal aligning
`#{Rails.root}/public/scripts/squareup.sh -c trans -s #{#width} #{File.expand_path(src.path)} #{File.expand_path(dst.path)}`
If somebody gets a better idea, please give it to me :)
Thanks

Resources