I'm getting an error testing the ActiveStorage attachment. The code is something like this:
class AssemblyTest < ActiveSupport::TestCase
test 'Updating svg attachment should upload the updated file' do
#assembly = Assembly.create(name: assemblies(:head_gasket).name,
image:
fixture_file_upload('files/track-bar.svg', 'image/svg+xml'))
assert #assembly.image.attached?
assert_not_empty #assembly.image.download
end
end
I'm getting the following error
Minitest::UnexpectedError: ActiveStorage::FileNotFoundError: ActiveStorage::FileNotFoundError when #assembly.image.download is called. The attached? assertion is passing, but I can't figure out why the download of the file is failing. Also, nothing shows up in the tmp/storage directory, where the ActiveStorage is configured to store files.
While digging in the ActiveStorage code I found this snipped which relies on actual database commits to execute the document upload (or save to disc):
after_commit(on: %i[ create update ]) { attachment_changes.delete(name.to_s).try(:upload) }
In case you use database transactions in the test environment this will then not store the document.
To solve this you can trigger the commit callback manually:
run_callbacks(:commit)
So in your case this might work:
class AssemblyTest < ActiveSupport::TestCase
test 'Updating svg attachment should upload the updated file' do
#assembly = Assembly.create(name: assemblies(:head_gasket).name,
image:
fixture_file_upload('files/track-bar.svg', 'image/svg+xml'))
#assembly.run_callbacks(:commit) # Run commit callback to store on disk
assert #assembly.image.attached?
assert_not_empty #assembly.image.download
end
end
Try this
#assembly = Assembly.create(name: assemblies(:head_gasket).name)
#assembly.image.attach(io: File.open('/path/to/file'), filename: 'file.name', content_type: 'mime/type')
You can create the blob directly (which is how the direct upload process works) and then attach it so the blob is guaranteed to already be uploaded.
blob = ActiveStorage::Blob.create_and_upload!(
io: File.open(Rails.root.join("test/fixtures/files/test.csv")),
filename: "test.csv",
content_type: "text/csv",
identify: false
)
#model.file.attach(blob)
Related
So here is the thing: currently our files, when user downloads them, have names like 897123uiojdkashdu182uiej.pdf. I need to change that to file-name.pdf.
And logically I go and change paperclip.rb config from this:
Paperclip::Attachment.default_options.update({
path: '/:hash.:extension',
hash_secret: Rails.application.secrets.secret_key_base
})
to this:
Paperclip::Attachment.default_options.update({
path: "/attachment/#{SecureRandom.urlsafe_base64(64)}/:filename",
hash_secret: Rails.application.secrets.secret_key_base
})
which works just fine, filenames are great. However, old files are now unaccessable due to the change in the path. So I came up with the following decision
First I made a rake task which will store the old paths in the database:
namespace :paperclip do
desc "Set old urls for attachments"
task :update_old_urls => :environment do
Asset.find_each do |asset|
if asset.attachment
attachment_url = asset.attachment.try!(:url)
file_url = "https:#{attachment_url}"
puts "Set old url asset attachment #{asset.id} - #{file_url}"
asset.update(old_url: file_url)
else
puts "No attachment found in asset #{asset.id}"
end
end
end
end
Now the asset.old_url stores the current url of the file. Then I go and change the config, making the file unaccessable.
Now it's time for the new rake task:
require 'uri'
require 'open-uri'
namespace :paperclip do
desc "Recreate attachments and save them to new destination"
task :move_attachments => :environment do
Asset.find_each do |asset|
unless asset.old_url.blank?
url = asset.old_url
filename = File.basename(asset.attachment.path)
file = File.new("#{Rails.root}/tmp/#{filename}", "wb")
file.write(open(url).read)
if File.exists? file
puts "Re-saving asset attachment #{asset.id} - #{filename}"
asset.attachment = file
asset.save
# if there are multiple styles, you want to recreate them :
asset.attachment.reprocess!
file.close
else
puts "Missing file attachment #{asset.id} - #{filename}"
end
File.delete(file)
end
end
end
end
But my plan didn't work at all, I didn't get access to the files, and the asset.url still isn't equal to asset.old_url.
Would appreciate help very much!
With S3, you can set the "filename upon saving" as a header. Specifically, the user will get to an url https://foo.bar.com/mangled/path/some/weird/hash/whatever?options and when the browser will offer to save, you can control the filename (not the url).
The trick to that relies on the browser reading the Content-Disposition header from the response, if it reads Content-Disposition: attachment; filename="filename.jpg" it will save (or ask the user to save as) filename.jpg, independently on the original URL.
You can force S3 to add this header by adding one more parameter to the URL or by setting a metadata on the file.
The former can be done by passing it to the url method:
has_attached_file :attachment,
s3_url_options: ->(instance) {
{response_content_disposition: "attachment; filename=\"#{instance.filename}\""}
}
Check https://github.com/thoughtbot/paperclip/blob/v6.1.0/lib/paperclip/storage/s3.rb#L221-L225 for the relevant source code.
The latter can be done in bulk via paperclip (and you should also configure it to do it on new uploads). It will also take a long time!!
Asset.find_each do |asset|
next unless asset.attachment
s3_object = asset.attachment.s3_object
s3_object.copy_to(
s3_object,
metadata_directive: 'REPLACE',
content_disposition: "attachment; filename=\"#{asset.filename}\")"
)
end
# for new uploads
has_attached_file :attachment,
s3_headers: ->(att) {
{content_disposition: "attachment; filename=\"#{att.model.filename}\""}
}
I'm trying to seed my development database. One of the models Project has images associated with it.
I have put a placeholder image in ./db/seed_files/. My seed file looks like this:
# Add projects
1000.times do
project = Project.new(
name: Faker::Marketing.buzzwords.capitalize,
description: Faker::Lorem.sentence(rand(1..30))
)
image_file = File.open("./db/seed_files/placeholder_image.png")
project.images.attach(io: image_file, filename: "placeholder_image.png", content_type: "image/png")
project.save
end
This runs fine. It attaches one image to each project.
However, I want to seed each project with multiple images. I thought I could attach the same image multiple times.
I have tried:
# Add projects
1000.times do
project = Project.new(
name: Faker::Marketing.buzzwords.capitalize,
description: Faker::Lorem.sentence(rand(1..30))
)
image_file = File.open("./db/seed_files/placeholder_image.png")
rand(1..3).times do
project.images.attach(io: image_file, filename: "placeholder_image.png", content_type: "image/png")
end
project.save
end
But this results in an error: ActiveStorage::FileNotFoundError.
/Users/greidods/.rvm/gems/ruby-2.6.1/bundler/gems/rails-b366be3b5b28/activestorage/lib/active_storage/service/disk_service.rb:136:in `rescue in stream'
/Users/greidods/.rvm/gems/ruby-2.6.1/bundler/gems/rails-b366be3b5b28/activestorage/lib/active_storage/service/disk_service.rb:129:in `stream'
/Users/greidods/.rvm/gems/ruby-2.6.1/bundler/gems/rails-b366be3b5b28/activestorage/lib/active_storage/service/disk_service.rb:28:in `block in download'
/Users/greidods/.rvm/gems/ruby-2.6.1/bundler/gems/rails-b366be3b5b28/activesupport/lib/active_support/notifications.rb:180:in `block in instrument'
/Users/greidods/.rvm/gems/ruby-2.6.1/bundler/gems/rails-b366be3b5b28/activesupport/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
...
I have a feeling that there's a approach to seeding a row with multiple attachments.
What is causing this error? Why can I attach the image once but not multiple times?
In my case, what I noticed is that attaching attachments with ActiveStorage is not working properly when doing it inside a transaction. This applies to migrations, seeds or callbacks.
What I finally did to avoid my logic being wrapped inside a transaction is to run the logic inside a thread:
Thread.new do
# your logic here
end.join
I can't exactly reproduce your problem (I keep getting ActiveStorage::IntegrityError exceptions rather than ActiveStorage::FileNotFoundError) but I think I know what's going on. After the first time you attach the image:
project.images.attach(io: image_file, filename: "placeholder_image.png", content_type: "image/png")
the image_file's current position will be at the end of the file. Now when Active Storage tries to read the file again, it won't get any data so either the checksum fails (my IntegrityError) or Active Storage figures that there's no file there (your FileNotFoundError).
The solution is to reset the file position back to the beginning by calling #rewind:
rand(1..3).times do
project.images.attach(io: image_file, filename: "placeholder_image.png", content_type: "image/png")
image_file.rewind
end
You can image_file.rewind before or after the project.images.attach call, rewinding a freshly opened file doesn't do anything interesting. #rewind won't always be supported (or desired) by the io you pass to #attach so Active Storage can't really do this itself.
Alternatively, you could open the file on each iteration:
rand(1..3).times do
image_file = File.open("./db/seed_files/placeholder_image.png")
project.images.attach(io: image_file, filename: "placeholder_image.png", content_type: "image/png")
end
I'm assuming that the missing do for the times block in your question is just a typo BTW.
I got it working by simply using an array as parameter:
image_file = File.open("./db/seed_files/placeholder_image.png")
files = []
rand(1..3).times do
files << {io: image_file,
filename: "placeholder_image.png",
content_type: "image/png"
}
project.images.attach(files)
end
I'm trying to upload a file that is protected under http_basic_authentication with CarrierWave. Here is the tested code:
hs = House.new
hs.remote_house_url = "http://username:password#127.0.0.1:3000/houses/export.csv"
hs.save!
I'm expecting the file to be uploaded, but I get the following:
(13.2ms) BEGIN
(0.8ms) ROLLBACK
ActiveRecord::RecordInvalid: The validation failed : House could not download file: userinfo not supported. [RFC3986]
from /Users/htaidirt/.rvm/gems/ruby-2.1.1/gems/activerecord-4.0.0/lib/active_record/validations.rb:57:in `save!'
I know it's a problem with giving http_basic_authentication credentials (username & password) thanks to the message http_basic_authentication. But what is the right way to do it? Thanks.
I just encountered a similar problem. OpenURI does not allow you to supply basic auth credentials as part of the url, instead it should be like
open("http://www.your-website.net",
http_basic_authentication: ["user", "password"])
(which I found here: http://blog.andreamostosi.name/2013/04/open-uri-and-basic-authentication/)
Carrierwave does not seem to support this by default. For now, I have monkey patched the CarrierWave::Uploader::Download::RemoteFile class to add the required basic auth. I will try and submit a better version of this as a pull request so hopefully it can be added to the gem, but for now I created config/initializers/overrides.rb with the contents:
#add basic auth to carrierwave
module CarrierWave
module Uploader
module Download
class RemoteFile
private
def file
if #file.blank?
#file = Kernel.open(#uri.to_s, http_basic_authentication: ["USERNAME", "PASSWORD])
#file = #file.is_a?(String) ? StringIO.new(#file) : #file
end
#file
rescue Exception => e
raise CarrierWave::DownloadError, "could not download file: #{e.message}"
end
end
end
end
end
I'm having an invalid encoding error that doesn't let me save the image to a carrierwave uploader.
require 'rqrcode_png'
img = RQRCode::QRCode.new( 'test', :size => 4, :level => :h ).to_img.to_s
img.valid_encoding?
=> false
I'm not sure if this is what you're looking for, in my case I needed to associate the generated QR code with a Rails model using carrierwave, what I ended up doing was saving the image to a temp file, associating that file with the model and afterwards deleting the temp file, here's my code:
def generate_qr_code!
tmp_path = Rails.root.join('tmp', "some-filename.png")
tmp_file = RQRCode::QRCode.new(self.hash_value).to_img.resize(200,200).save(tmp_path)
# Stream is handed closed, we need to reopen it
File.open(tmp_file.path) do |file|
self.qr_code = file
end
File.delete(tmp_file.path)
self.save!
end
Is there a recommended technique for migrating a large set of paperclip S3 images to a new :url and :path format?
The reason for this is because after upgrading to rails 3.1, new versions of thumbs are not being shown after cropping (previously cached version is shown). This is because the filename no longer changes (since asset_timestamp was removed in rails 3.1). I'm using :fingerprint in the url/path format, but this is generated from the original, which doesn't change when cropping.
I was intending to insert :updated_at in the url/path format, and update attachment.updated_at during cropping, but after implementing that change all existing images would need to be moved to their new location. That's around half a million images to rename over S3.
At this point I'm considering copying them to their new location first, then deploying the code change, then moving any images which were missed (ie uploaded after the copy), but I'm hoping there's an easier way... any suggestions?
I had to change my paperclip path in order to support image cropping, I ended up creating a rake task to help out.
namespace :paperclip_migration do
desc 'Migrate data'
task :migrate_s3 => :environment do
# Make sure that all of the models have been loaded so any attachments are registered
puts 'Loading models...'
Dir[Rails.root.join('app', 'models', '**/*')].each { |file| File.basename(file, '.rb').camelize.constantize }
# Iterate through all of the registered attachments
puts 'Migrating attachments...'
attachment_registry.each_definition do |klass, name, options|
puts "Migrating #{klass}: #{name}"
klass.find_each(batch_size: 100) do |instance|
attachment = instance.send(name)
unless attachment.blank?
attachment.styles.each do |style_name, style|
old_path = interpolator.interpolate(old_path_option, attachment, style_name)
new_path = interpolator.interpolate(new_path_option, attachment, style_name)
# puts "#{style_name}:\n\told: #{old_path}\n\tnew: #{new_path}"
s3_copy(s3_bucket, old_path, new_path)
end
end
end
end
puts 'Completed migration.'
end
#############################################################################
private
# Paperclip Configuration
def attachment_registry
Paperclip::AttachmentRegistry
end
def s3_bucket
ENV['S3_BUCKET']
end
def old_path_option
':class/:id_partition/:attachment/:hash.:extension'
end
def new_path_option
':class/:attachment/:id_partition/:style/:filename'
end
def interpolator
Paperclip::Interpolations
end
# S3
def s3
AWS::S3.new(access_key_id: ENV['S3_KEY'], secret_access_key: ENV['S3_SECRET'])
end
def s3_copy(bucket, source, destination)
source_object = s3.buckets[bucket].objects[source]
destination_object = source_object.copy_to(destination, {metadata: source_object.metadata.to_h})
destination_object.acl = source_object.acl
puts "Copied #{source}"
rescue Exception => e
puts "*Unable to copy #{source} - #{e.message}"
end
end
Didn't find a feasible method for migrating to a new url format. I ended up overriding Paperclip::Attachment#generate_fingerprint so it appends :updated_at.