I am using Paperclip/RSpec and StackOverflow has helped me successfully stub file uploads to S3 using this code:
spec/rails_helper.rb
config.before(:each) do
allow_any_instance_of(Paperclip::Attachment).to receive(:save).and_return(true)
end
This is working great.
On my model I have two Paperclip fields:
class MyModel < ActiveRecord::Base
has_attached_file :pdf
has_attached_file :resource
end
My code uses the #copy_to_local_file method (Docs) to retrieve a file from S3.
#copy_to_local_file takes two params: the style (:original, :thumbnail, etc) and the local file path to copy to.
Example:
MyModel.resource.copy_to_local_file(:original, local_file.path)
When the system under test tries to access MyModel#pdf#copy_to_local_file or MyModel#resource#copy_to_local_file, I originally got errors like the following:
No Such Key - cannot copy /email_receipts/pdfs/000/000/001/original/email_receipt.eml.pdf to local file /var/folders/4p/1mm86g0n58x7d9rvpy88_s9h0000gn/T/receipt20150917-4906-13evk95.pdf
No Such Key - cannot copy /email_receipts/resources/000/000/001/original/email_receipt.eml to local file /var/folders/4p/1mm86g0n58x7d9rvpy88_s9h0000gn/T/resource20150917-4906-1ysbwr3.eml
I realize these errors were happening because uploads to S3 are stubbed, so when it encounters MyModel#pdf#copy_to_local_file or MyModel#resource#copy_to_local_file it tries to grab a file in S3 that isn't there.
Current Solution:
I've managed to quash the errors above, but I feel it's not a complete solution and gives my tests a false sense of security. My half-solution is to stub this method in the following way:
spec/rails_helper.rb
before(:each) do
allow_any_instance_of(Paperclip::Storage::S3).to receive(:copy_to_local_file)
end
While this does stub out the #copy_to_local_file method and removes the errors, it doesn't actually write any content to the local file that is provided as the second argument to #copy_to_local_file, so it doesn't quite simulate the file being downloaded from S3.
Question:
Is there a way to stub #copy_to_local_file AND have it write the contents of a canned file in my spec/factories/files directory to the local file (its second argument)?
Or am I overthinking this? Is this something I shouldn't be worrying about?
You don't need to worry about whether the 'downloaded' files actually exist in your tests. You've decided to stub out Paperclip, so do it completely, by stubbing out both #save and #copy_to_file. You may also need to stub out reads of downloaded files from the filesystem.
All this stubbing raises the possibility of integration errors, so you should probably write a feature spec (using a captive browser like poltergeist) that actually uploads and downloads something and reads it from the filesystem.
That said, you can do anything you want in an RSpec stub by passing it a block:
allow_any_instance_of(Paperclip::Storage::S3).to receive(:copy_to_local_file) do |style, local_dest_path|
# write a file here, or do anything you like
end
Related
We have an Attachment model with a copy_for_edit! method which helps an Attachment copy itself. The attachment data is stored in AWS S3 bucket(s). We make use of the Bucket copy_to technique to perform the copy remotely on the AWS S3 server, without transferring the data back to us.
https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#copy_to-instance_method
I'm writing a unit test (in minitest) for this method, and getting this error caused by the Aws::S3::Bucket#copy_to instance method:
Aws::S3::Errors::NoSuchKey: Aws::S3::Errors::NoSuchKey: The specified key does not exist.
I've seen countless solutions for how to stub an AWS S3 Client, but not a Bucket. I'm sure I'm missing something simple here. The code itself works in staging and production, but testing in my development environment I'd obviously prefer not to hit the AWS S3 server. But even if I configure the test environment to use our staging credentials for the bucket, that doesn't work either (same error).
I'd like to know how to stub (or similar) the Aws::S3::Bucket#copy_to instance method in minitest.
I know I've left out some details. I will be watching this closely and editing to add context if needed.
Edit 1: A simplified version of the test looks like this:
test '#copy_for_edit! should copy the attachment, excluding some attributes' do
source = attachments(:attachment_simple) #From an existing fixture.
result = nil
assert_difference(-> { Attachment.count }, 1) do
result = source.copy_for_edit!
end
assert_nil(result.owner)
assert_nil(result.draft_id)
end
Narrowing it down to an instance method (not a class method or attribute) helped me narrow down my options.
I finally got the syntax correct, and believe I have a working test now.
This was basically my solution: https://stackoverflow.com/a/29042835/14837782
I can't say I've stubbed the AWS S3 Bucket#copy_to method. I've actually just stubbed our own method (copy_attached_file_to) that eventually calls it, since I'm not actually testing that method. When it comes time to test that method, I might be in similar trouble. Although maybe this solution will work to stub Bucket similarly.
And here is the test now, seemingly working properly:
test '#copy_for_edit! should copy the attachment, excluding some attributes' do
source = attachments(:attachment_simple) # From an existing fixture.
source.stub(:copy_attached_file_to, true) do
result = nil
assert_difference(-> { Attachment.count }, 1) do
result = source.copy_for_edit!
end
assert_nil(result.owner)
assert_nil(result.draft_id)
end
end
I'm using AmazonS3 to store Paperclip attachments on all non-test environments.
For test specifically I use a local path/url setup to avoid interacting with S3 remotely
Paperclip::Attachment.default_options[:path] =
":rails_root/public/system/:rails_env/:class/:attachment/:id_partition/:filename"
Paperclip::Attachment.default_options[:url] =
"/system/:rails_env/:class/:attachment/:id_partition/:filename"
I define my attachment as follows in the model
has_attached_file :source_file, use_timestamp: false
In my Production code I need to access the file using Model.source_file.url because .url returns the remote fully qualified Amazon S3 path to the file. This generally works fine for non-test environments.
However on my test environment I can't use .url because Paperclip creates and stores the file under the path defined by :path above. So I need to use .path. If I use .url I get the error -
Errno::ENOENT:
No such file or directory # rb_sysopen - /system/test/imports/source_files/000/000/030/sample.txt
which makes sense because paperclip didn't store the file there...
How do I get paperclip on my test environment to store/create my file under the :url path so I can use .url correctly?
Edit: If it helps, in test I create the attachment from a locally stored fixture file
factory :import do
source_file { File.new(Rails.root + "spec/fixtures/files/sample.tsv") }
end
Edit2: Setting :path and :url to be the same path in the initializer might seem like a quick fix, but I'm working on a larger app with several contributors, so I don't the have the luxury to do that or break any one else's specs. Plus it looks like Thoughtbot themselves recommend this setup, so there should be a "proper" way to get it working as is.
Thanks!
Have you tried using s3proxy in your test environment to simulate S3 instead of directly have paperclip write to local files?
I am building an online translation platform. When a job is done being translated and saved, I want the user to be able to download the translated version as a text file. It is currently saved in a string in the model called "target_text".
I know in ruby I can use this method:
File.open("translation.txt", 'w') {|f| f.write("my translated string") }
I am assuming I could tack the location for the file to be saved in front of the "translation.txt", but I am not sure what folder within my app I should specify?
Furthermore I want this file to be attached to the "job" object in the same way that paperclip can attach files, the difference being it's initiated server side. How should I go about this?
I have googled all over looking for an answer to this, and I want to make sure I do it in the cleanest way possible. I would really appreciate even directions to a good place to look to understand this concept.
I don't quite understand the question, but I hope this could help...
Instead of using
File.open("translation.txt", 'w') {|f| f.write("my translated string") }
try using the following
Tempfile.open(['translation', '.txt'], Rails.root.join('tmp')) do |file|
# this will create a temp file in RAILS_ROOT/tmp/ folder
# you can replace the 'translation' text part to any auto generated text for example
# Tempfile.open([#user.id.to_s, '_translation.txt'] will create
# RAILS_ROOT/tmp/1_translation.1fe2ed.txt
# the 1fe2ed is generated by Tempfile to avoid conflicting
begin
file << "my translated string"
# this creates the file
# add all the processing you need here... cause the next ensure block
# will close and delete this temp file... so that the tmp dir doesn't get big.
# you can for example add the file to paperclip attachment
#user.translation = file
# assuming that user has paperclip attachment called translation
ensure
# close and delete file
file.close
file.unlink
end
end
also check the Tempfile docs... this is the practice i've been using... not sure if it's the best or not.. but it didn't create any issues so far
(even with paperclip s3 storage)
I want to stub carrierwave to prevent it from fetch images on the web during my tests. How would I stub things to achieve this?
My crawler parses a remote web page, and saves one image url into the model. Carrierwave will fetch that image automatically during the save operation. It works well.
However I have a test about the parsing of pages, and every-time it will download the file, which slows down the testing.
UPDATE:
I mount the uploader as the following (in the pre-existing paperclip column)
mount_uploader :image, TopicImageUploader, :mount_on => :image_file_name
I tried to stub the following, but neither worked:
Topic.any_instance.stub(:store_image!)
Topic.any_instance.stub(:store_image_file_name!)
Topic.any_instance.stub(:store_image_remote_url!)
TopicImageUploader.any_instance.stub(:download!)
This is what I'm using in my spec_helper:
class CarrierWave::Mount::Mounter
def store!
end
end
This completely blocks all real file uploads (note that I'm using this with carrier wave 0.5.8, which is newest version at the time of writing, if you're using much older version, it might differ). If you want to control tests which stub uploads, you could use:
CarrierWave::Mount::Mounter.any_instance.stub(:store!)
I reduced my test-suite time from 25 seconds to just 2 seconds with a simple config in the CarrierWave initializer:
# config/initializers/carrier_wave.rb
CarrierWave.configure do |config|
config.enable_processing = false if Rails.env.test?
end
This config skips the image manipulation (resizing, cropping, ...) of ImageMagick, MiniMagick ect.
allow_any_instance_of(CarrierWave::Uploader::Base).to receive(:store!).and_return nil
I'm am adding tests to a Rails app that remotely stores files. I'm using the default Rails functional tests. How can I add file uploads to them? I have:
test "create valid person" do
post(:create, :person => { :avatar => fixture_file_upload('avatar.jpeg') })
end
This for some reason uploads a Tempfile and causes the AWS/S3 gem to fail with:
NoMethodError: undefined method `bytesize' for Tempfile
Is their any way that I can get the test to use an ActionDispatch::Http::UploadedFile and perform more like it does when testing with the web browser? Is fixture_file_upload the way to test uploading files to a controller? If so why doesn't it work like the browser?
As a note, I really don't want to switch testing frameworks. Thanks!
I use the s3 gem instead of the aws/s3 gem. The main reasons for this are no support for european buckets and development of aws/s3 seems to be stopped.
If you want to test file upload than using the fixtures_file_upload method is correct, it maps directly to Rack::Test::UploadedFile.new (you can use this if the test file isn't in the fixtures folder).
But I've also noticed that the behavior of the Rack::Test::Uploaded file objects isn't exactly the same as the ActionDispatch::Http::UploadedFile object (that's the class of uploaded files). The basic methods (original_filename, read, size, ...) all work but there are some differences when working with the file method. So limit your controller to these methods and all will be fine.
An other possible solution is by creating an ActionDispatch::Http::Uploaded file object and using that so:
upload = ActionDispatch::Http::UploadedFile.new({
:filename => 'avatar.jpeg',
:type => 'image/jpeg',
:tempfile => File.new("#{Rails.root}/test/fixtures/avatar.jpeg")
})
post :create, :person => { :avatar => upload }
I'd recommend using mocks.
A quick google search reveals:
http://www.ibm.com/developerworks/web/library/wa-mockrails/index.html
You should be able to create an object that will respond to the behaviors you want it to. Mostly used in a Unit test environment, so you can test your stuff in isolation, as integration tests are supposed to fully exercise the entire stack. However, I can see in this case it'd be useful to mock out the S3 service because it costs money.
I'm not familiar with the AWS/S3 gem, but it seems that you probably aren't using the :avatar param properly. bytesize is defined on String in ruby1.9. What happens if you call read on the uploaded file where you pass it into AWS/S3?