S3 save old url, change paperclip config, set new url as old - ruby-on-rails

So here is the thing: currently our files, when user downloads them, have names like 897123uiojdkashdu182uiej.pdf. I need to change that to file-name.pdf.
And logically I go and change paperclip.rb config from this:
Paperclip::Attachment.default_options.update({
path: '/:hash.:extension',
hash_secret: Rails.application.secrets.secret_key_base
})
to this:
Paperclip::Attachment.default_options.update({
path: "/attachment/#{SecureRandom.urlsafe_base64(64)}/:filename",
hash_secret: Rails.application.secrets.secret_key_base
})
which works just fine, filenames are great. However, old files are now unaccessable due to the change in the path. So I came up with the following decision
First I made a rake task which will store the old paths in the database:
namespace :paperclip do
desc "Set old urls for attachments"
task :update_old_urls => :environment do
Asset.find_each do |asset|
if asset.attachment
attachment_url = asset.attachment.try!(:url)
file_url = "https:#{attachment_url}"
puts "Set old url asset attachment #{asset.id} - #{file_url}"
asset.update(old_url: file_url)
else
puts "No attachment found in asset #{asset.id}"
end
end
end
end
Now the asset.old_url stores the current url of the file. Then I go and change the config, making the file unaccessable.
Now it's time for the new rake task:
require 'uri'
require 'open-uri'
namespace :paperclip do
desc "Recreate attachments and save them to new destination"
task :move_attachments => :environment do
Asset.find_each do |asset|
unless asset.old_url.blank?
url = asset.old_url
filename = File.basename(asset.attachment.path)
file = File.new("#{Rails.root}/tmp/#{filename}", "wb")
file.write(open(url).read)
if File.exists? file
puts "Re-saving asset attachment #{asset.id} - #{filename}"
asset.attachment = file
asset.save
# if there are multiple styles, you want to recreate them :
asset.attachment.reprocess!
file.close
else
puts "Missing file attachment #{asset.id} - #{filename}"
end
File.delete(file)
end
end
end
end
But my plan didn't work at all, I didn't get access to the files, and the asset.url still isn't equal to asset.old_url.
Would appreciate help very much!

With S3, you can set the "filename upon saving" as a header. Specifically, the user will get to an url https://foo.bar.com/mangled/path/some/weird/hash/whatever?options and when the browser will offer to save, you can control the filename (not the url).
The trick to that relies on the browser reading the Content-Disposition header from the response, if it reads Content-Disposition: attachment; filename="filename.jpg" it will save (or ask the user to save as) filename.jpg, independently on the original URL.
You can force S3 to add this header by adding one more parameter to the URL or by setting a metadata on the file.
The former can be done by passing it to the url method:
has_attached_file :attachment,
s3_url_options: ->(instance) {
{response_content_disposition: "attachment; filename=\"#{instance.filename}\""}
}
Check https://github.com/thoughtbot/paperclip/blob/v6.1.0/lib/paperclip/storage/s3.rb#L221-L225 for the relevant source code.
The latter can be done in bulk via paperclip (and you should also configure it to do it on new uploads). It will also take a long time!!
Asset.find_each do |asset|
next unless asset.attachment
s3_object = asset.attachment.s3_object
s3_object.copy_to(
s3_object,
metadata_directive: 'REPLACE',
content_disposition: "attachment; filename=\"#{asset.filename}\")"
)
end
# for new uploads
has_attached_file :attachment,
s3_headers: ->(att) {
{content_disposition: "attachment; filename=\"#{att.model.filename}\""}
}

Related

Rails download file direct from S3 with content-disposition = attachment?

This is my controller
Cotroller
def download
data = open(#attachment.file.url).read
#attachment.clicks = #attachment.clicks.to_i + 1
#attachment.save
send_data data, :type => #attachment.content_type, :filename => #attachment.name
end
example:
#attachment.file.url = "http://my_bucket.cloudfront.net/uploads/attachment/file/50/huge_file.pptx"
I did this, but if #attachement is a huge file (eg. 300MB), my server crash.
I want to allow users to download the file in the browser directly from my AWS server?
2) tip: Do you suggest to download file from S3 (where they are stored) or with CloudFront?
If you using carrierwave gem, you can try this to track number of clicks
def download
#attachment.clicks.to_i += 1
#attachment.save
redirect_to #attachment.file.url(query: {"response-content-disposition" => "attachment;"})
end
references:
Rails carrierwave S3 get url with Content-Disposition header

Invalid encoding with rqrcode

I'm having an invalid encoding error that doesn't let me save the image to a carrierwave uploader.
require 'rqrcode_png'
img = RQRCode::QRCode.new( 'test', :size => 4, :level => :h ).to_img.to_s
img.valid_encoding?
=> false
I'm not sure if this is what you're looking for, in my case I needed to associate the generated QR code with a Rails model using carrierwave, what I ended up doing was saving the image to a temp file, associating that file with the model and afterwards deleting the temp file, here's my code:
def generate_qr_code!
tmp_path = Rails.root.join('tmp', "some-filename.png")
tmp_file = RQRCode::QRCode.new(self.hash_value).to_img.resize(200,200).save(tmp_path)
# Stream is handed closed, we need to reopen it
File.open(tmp_file.path) do |file|
self.qr_code = file
end
File.delete(tmp_file.path)
self.save!
end

Ruby on Rails - How to Save Remote File over HTTPS and Basic Authentication

I am using a supplier's api and the response they send to our server includes a url to a file, upon trying to save this file locally I fail miserably.
def self.create_file_new(filename, ext, url)
require 'open-uri'
file = Tempfile.new(filename + ext)
file.binmode
# data = open(url).read
# data = open(url, :http_basic_authentication => [username, password])
file << open(url, :http_basic_authentication => [username, password]).read
# file.write CGI::unescape(data)
file.close
file = File.open(file.path)
return file
end
I was originally getting a OpenURI::HTTPError (401 Unauthorised): but I have since created a file named bypass_ssl_verification_for_open_uri in app/initializers containing the following:
# Make open-uri work with https
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
which I found whilst Googling on how to fix it.
I then started to get this error message: NoMethodError (undefined method 'tr' for #<StringIO:0xb5b728c4>):, I tried creating another file (cgi_escape_fix.rb in app/initializers) containing this:
require 'cgi'
class << CGI
alias_method :orig_escape, :escape
def escape(str)
orig_escape(str.to_str)
end
end
Which I also found on my Google travels but that doesn't seem to have solved anything, so I commented out the file.write CGI::unescape(data) to try a different way but still no joy.
Now in the log I am just getting a plain 500 Internal Server Error with no useful information.
The file I'm attempting to save will always be a pdf.
Ruby 1.8.7
Rails 2.3.14
Got it to work with the following (two new initializer scripts removed):
file = Tempfile.new(filename + ext)
file.binmode
file << open(url, :http_basic_authentication => [username, password]).read
file.close
file = File.open(file.path)
return file
Should also mention that this is being passed to the attachment_fu plugin incase anyone else has problems with it.

403 Error when calling AWS::S3.new

I'm working on a rake task for a Rails 4.0 application in order to copy some old image URLs from outside my website or S3 into S3. I can verify that my credentials are correct and loaded with printenv. For some reason though, I get a 403 error every time I run the task. My system time is synced with a reliable time server and the avatars directory I'm attempting to save to in S3 is set to public. Any ideas on what could be causing this? Full rake task code below:
namespace :arthackday do
desc "Grabs old profile pics from participants and saves them to S3."
task :update_participant_photos => :environment do
require 'open-uri'
require 'aws/s3'
s3_base_url = "https://s3-us-west-1.amazonaws.com/arthackday/participants/avatars/"
Participant.all.each do |participant|
if participant.photo_url?
file_name = "#{participant.name.gsub(/ /,"-")}.jpg"
File.open("#{Rails.root}/public/img/temp/#{file_name}", 'wb') do |fo|
fo.write open(participant.photo_url).read
end
s3 = AWS::S3.new(:access_key_id => ENV['AWS_ACCESS_KEY_ID'], :secret_access_key => ENV['AWS_SECRET_ACCESS_KEY'])
s3.buckets[ENV['S3_BUCKET_NAME']].objects["participants/avatars/#{file_name}"].write(:file => "#{Rails.root}/public/img/temp/#{file_name}")
participant.photo_url = s3_base_url + file_name
participant.save!
puts "New URL for #{participant.name}'s photo: #{participant.photo_url}"
else
puts "#{participant.name} does not have a photo_url"
end
end
end
end

Migrating paperclip S3 images to new url/path format

Is there a recommended technique for migrating a large set of paperclip S3 images to a new :url and :path format?
The reason for this is because after upgrading to rails 3.1, new versions of thumbs are not being shown after cropping (previously cached version is shown). This is because the filename no longer changes (since asset_timestamp was removed in rails 3.1). I'm using :fingerprint in the url/path format, but this is generated from the original, which doesn't change when cropping.
I was intending to insert :updated_at in the url/path format, and update attachment.updated_at during cropping, but after implementing that change all existing images would need to be moved to their new location. That's around half a million images to rename over S3.
At this point I'm considering copying them to their new location first, then deploying the code change, then moving any images which were missed (ie uploaded after the copy), but I'm hoping there's an easier way... any suggestions?
I had to change my paperclip path in order to support image cropping, I ended up creating a rake task to help out.
namespace :paperclip_migration do
desc 'Migrate data'
task :migrate_s3 => :environment do
# Make sure that all of the models have been loaded so any attachments are registered
puts 'Loading models...'
Dir[Rails.root.join('app', 'models', '**/*')].each { |file| File.basename(file, '.rb').camelize.constantize }
# Iterate through all of the registered attachments
puts 'Migrating attachments...'
attachment_registry.each_definition do |klass, name, options|
puts "Migrating #{klass}: #{name}"
klass.find_each(batch_size: 100) do |instance|
attachment = instance.send(name)
unless attachment.blank?
attachment.styles.each do |style_name, style|
old_path = interpolator.interpolate(old_path_option, attachment, style_name)
new_path = interpolator.interpolate(new_path_option, attachment, style_name)
# puts "#{style_name}:\n\told: #{old_path}\n\tnew: #{new_path}"
s3_copy(s3_bucket, old_path, new_path)
end
end
end
end
puts 'Completed migration.'
end
#############################################################################
private
# Paperclip Configuration
def attachment_registry
Paperclip::AttachmentRegistry
end
def s3_bucket
ENV['S3_BUCKET']
end
def old_path_option
':class/:id_partition/:attachment/:hash.:extension'
end
def new_path_option
':class/:attachment/:id_partition/:style/:filename'
end
def interpolator
Paperclip::Interpolations
end
# S3
def s3
AWS::S3.new(access_key_id: ENV['S3_KEY'], secret_access_key: ENV['S3_SECRET'])
end
def s3_copy(bucket, source, destination)
source_object = s3.buckets[bucket].objects[source]
destination_object = source_object.copy_to(destination, {metadata: source_object.metadata.to_h})
destination_object.acl = source_object.acl
puts "Copied #{source}"
rescue Exception => e
puts "*Unable to copy #{source} - #{e.message}"
end
end
Didn't find a feasible method for migrating to a new url format. I ended up overriding Paperclip::Attachment#generate_fingerprint so it appends :updated_at.

Resources