I want to stub carrierwave to prevent it from fetch images on the web during my tests. How would I stub things to achieve this?
My crawler parses a remote web page, and saves one image url into the model. Carrierwave will fetch that image automatically during the save operation. It works well.
However I have a test about the parsing of pages, and every-time it will download the file, which slows down the testing.
UPDATE:
I mount the uploader as the following (in the pre-existing paperclip column)
mount_uploader :image, TopicImageUploader, :mount_on => :image_file_name
I tried to stub the following, but neither worked:
Topic.any_instance.stub(:store_image!)
Topic.any_instance.stub(:store_image_file_name!)
Topic.any_instance.stub(:store_image_remote_url!)
TopicImageUploader.any_instance.stub(:download!)
This is what I'm using in my spec_helper:
class CarrierWave::Mount::Mounter
def store!
end
end
This completely blocks all real file uploads (note that I'm using this with carrier wave 0.5.8, which is newest version at the time of writing, if you're using much older version, it might differ). If you want to control tests which stub uploads, you could use:
CarrierWave::Mount::Mounter.any_instance.stub(:store!)
I reduced my test-suite time from 25 seconds to just 2 seconds with a simple config in the CarrierWave initializer:
# config/initializers/carrier_wave.rb
CarrierWave.configure do |config|
config.enable_processing = false if Rails.env.test?
end
This config skips the image manipulation (resizing, cropping, ...) of ImageMagick, MiniMagick ect.
allow_any_instance_of(CarrierWave::Uploader::Base).to receive(:store!).and_return nil
Related
I am using Paperclip/RSpec and StackOverflow has helped me successfully stub file uploads to S3 using this code:
spec/rails_helper.rb
config.before(:each) do
allow_any_instance_of(Paperclip::Attachment).to receive(:save).and_return(true)
end
This is working great.
On my model I have two Paperclip fields:
class MyModel < ActiveRecord::Base
has_attached_file :pdf
has_attached_file :resource
end
My code uses the #copy_to_local_file method (Docs) to retrieve a file from S3.
#copy_to_local_file takes two params: the style (:original, :thumbnail, etc) and the local file path to copy to.
Example:
MyModel.resource.copy_to_local_file(:original, local_file.path)
When the system under test tries to access MyModel#pdf#copy_to_local_file or MyModel#resource#copy_to_local_file, I originally got errors like the following:
No Such Key - cannot copy /email_receipts/pdfs/000/000/001/original/email_receipt.eml.pdf to local file /var/folders/4p/1mm86g0n58x7d9rvpy88_s9h0000gn/T/receipt20150917-4906-13evk95.pdf
No Such Key - cannot copy /email_receipts/resources/000/000/001/original/email_receipt.eml to local file /var/folders/4p/1mm86g0n58x7d9rvpy88_s9h0000gn/T/resource20150917-4906-1ysbwr3.eml
I realize these errors were happening because uploads to S3 are stubbed, so when it encounters MyModel#pdf#copy_to_local_file or MyModel#resource#copy_to_local_file it tries to grab a file in S3 that isn't there.
Current Solution:
I've managed to quash the errors above, but I feel it's not a complete solution and gives my tests a false sense of security. My half-solution is to stub this method in the following way:
spec/rails_helper.rb
before(:each) do
allow_any_instance_of(Paperclip::Storage::S3).to receive(:copy_to_local_file)
end
While this does stub out the #copy_to_local_file method and removes the errors, it doesn't actually write any content to the local file that is provided as the second argument to #copy_to_local_file, so it doesn't quite simulate the file being downloaded from S3.
Question:
Is there a way to stub #copy_to_local_file AND have it write the contents of a canned file in my spec/factories/files directory to the local file (its second argument)?
Or am I overthinking this? Is this something I shouldn't be worrying about?
You don't need to worry about whether the 'downloaded' files actually exist in your tests. You've decided to stub out Paperclip, so do it completely, by stubbing out both #save and #copy_to_file. You may also need to stub out reads of downloaded files from the filesystem.
All this stubbing raises the possibility of integration errors, so you should probably write a feature spec (using a captive browser like poltergeist) that actually uploads and downloads something and reads it from the filesystem.
That said, you can do anything you want in an RSpec stub by passing it a block:
allow_any_instance_of(Paperclip::Storage::S3).to receive(:copy_to_local_file) do |style, local_dest_path|
# write a file here, or do anything you like
end
I'm writing tests with rspec and am a bit struggling with Paperclip 4. At the moment I'm using webmock to stub requests but image processing is slowing tests down.
Everything I read suggest to use stubs like so:
Profile.any_instance.stub(:save_attached_files).and_return(true)
It doesn't work since :save_attached_files disappeared with Paperclip 4 as far as I can tell.
What's the proper way to do it now ?
Thanks
Add this to your rails_helper.rb in your config block:
RSpec.configure do |config|
# Stub saving of files to S3
config.before(:each) do
allow_any_instance_of(Paperclip::Attachment).to receive(:save).and_return(true)
end
end
It doesn't exactly answer my question, but I found a way to run faster specs thanks to this dev blog, so I'm posting it if it can help someone else.
Just add this piece of code at the beginning of your spec file or in a helper. My tests are running 3x faster now.
# We stub some Paperclip methods - so it won't call shell slow commands
# This allows us to speedup paperclip tests 3-5x times.
module Paperclip
def self.run cmd, params = "", expected_outcodes = 0
cmd == 'convert' ? nil : super
end
end
class Paperclip::Attachment
def post_process
end
end
Paperclip > 3.5.2
For newer versions of Paperclip, use the following:
module Paperclip
def self.run cmd, arguments = "", interpolation_values = {}, local_options = {}
cmd == 'convert' ? nil : super
end
end
class Paperclip::Attachment
def post_process
end
end
I had a heckuva time dealing with this. I didn't want to test Paperclip per se -- rather, I needed a file to exist on my model so I can test a very sensitive custom class.
None of the other answers worked for me (Rails 5, Rspec 3.5) so I gave up on stubbing AWS or using the Paperclip::Shoulda::Matchers. Nothing worked and it ate up an entire day! Finally I gave up on Paperclip altogether and went this route:
list = build(:list)
allow(list).to receive_message_chain("file.url") { "#{Rails.root}/spec/fixtures/guess_fullname.csv" }
importer = Importer.new(list)
expect(importer.set_key_mapping).to eq({nil=>:email_address, :company=>:company, :fullname=>:full_name})
Where my models/list.rb has has_attached_file :file and Importer is a class I wrote that takes the file, which was already uploaded to S3 by paperclip before the class is initialized, and processes it. The file.url would otherwise be the S3 URL, so I give it the path in my repo to a testing csv file. set_key_mapping is a method in my class that I can use to verify the processing part worked.
Hope this saves somebody a few hours...
What about adding AWS.stub! in your spec/spec_helper.rb? Give that a try.
I was using the Paperclip gem: https://github.com/thoughtbot/paperclip
I'm now using Carrierwave: https://github.com/carrierwaveuploader/carrierwave
My production website is currently using Paperclip. I'm going to be updating the production website to use Carrierwave.
The folder structure for uploads in Paperclip differs from Carrierwave.
I'm also using Amazon S3 to store uploads.
I'm wondering if there's a way to convert my production files uploaded with Paperclip to Carrierwave.
For example, with Paperclip in production I currently have something like the following for resumes:
bucket_name/model_name/resume/000/000/model_id/original/test.pdf
With Carrierwave it should be:
bucket_name/uploads/model_name/resume/model_id/original/test.pdf
Right now it seems I have to make this conversion manually. I was wondering if there's a better approach.
Please advise.
The CarrierWave::Compatibility::Paperclip module already provides this functionality. Just do the following in your uploader:
class MyUploader < CarrierWave::Uploader::Base
include CarrierWave::Compatibility::Paperclip
# The :id_partition symbol will trigger a proc in the Paperclip compatibility module that will build out the properly partition directory structure
def store_dir
"#{model.class.to_s.underscore}/#{mounted_as.to_s}/:id_partition"
end
end
Have you tried changing the store_dir options define in carrierwave uploader to look exactly like that of paperclip
def store_dir
"#{model.class.to_s.underscore}/resume/#{id_partitioning}/original/"
end
def id_partitioning
("%09d" % model.id).scan(/.{3}/).join("/")
end
Note : I just done remember how the paperclip does the id_partitioning (how much '0' it pad to the left based on object id )
but based upon your format 000/000/model_id look to me like 9 character Please confirm
Hope this help
I'm new to rails, and I'm writing a RESTful website using the CRUD technique. So far I have created three pages, all of which allow the user to create, edit, and delete a row from the database. However, my fourth page will need to include an upload file form, but a) I don't know how the filesystem works with Rails thus I don't know where files should be stored. The file would be around 100kb and couldn't be stored in temporary storage because it will be constantly downloaded. And b) I don't know how to write to a file.
It would be great if you could tell me how to do what I mentioned above - create an upload input on an input form, and to then write the file to a filepath in a separate directory.
Update 2018
While everything written below still holds true, Rails 5.2 now includes active_storage, which allows stuff like uploading directly to S3 (or other cloud storage services), image transformations, etc. You should check out the rails guide and decide for yourself what fits your needs.
While there are plenty of gems that solve file uploading pretty nicely (see https://www.ruby-toolbox.com/categories/rails_file_uploads for a list), rails has built-in helpers which make it easy to roll your own solution.
Use the file_field-form helper in your form, and rails handles the uploading for you:
<%= form_for #person do |f| %>
<%= f.file_field :picture %>
<% end %>
You will have access in the controller to the uploaded file as follows:
uploaded_io = params[:person][:picture]
File.open(Rails.root.join('public', 'uploads', uploaded_io.original_filename), 'wb') do |file|
file.write(uploaded_io.read)
end
It depends on the complexity of what you want to achieve, but this is totally sufficient for easy file uploading/downloading tasks. This example is taken from the rails guides, you can go there for further information: http://guides.rubyonrails.org/form_helpers.html#uploading-files
Sept 2018
For anyone checking this question recently, Rails 5.2+ now has ActiveStorage by default & I highly recommend checking it out.
Since it is part of the core Rails 5.2+ now, it is very well integrated & has excellent capabilities out of the box (still all other well-known gems like Carrierwave, Shrine, paperclip,... are great but this one offers very good features that we can consider for any new Rails project)
Paperclip team deprecated the gem in favor of the Rails ActiveStorage.
Here is the github page for the ActiveStorage & plenty of resources are available everywhere
Also I found this video to be very helpful to understand the features of Activestorage
There is a nice gem especially for uploading files : carrierwave. If the wiki does not help , there is a nice RailsCast about the best way to use it . Summarizing , there is a field type file in Rails forms , which invokes the file upload dialog. You can use it , but the 'magic' is done by carrierwave gem .
I don't know what do you mean with "how to write to a file" , but I hope this is a nice start.
Okay. If you do not want to store the file in database and store in the application, like assets (custom folder), you can define non-db instance variable defined by attr_accessor: document and use form_for - f.file_field to get the file,
In controller,
#person = Person.new(person_params)
Here person_params return whitelisted params[:person] (define yourself)
Save file as,
dir = "#{Rails.root}/app/assets/custom_path"
FileUtils.mkdir(dir) unless File.directory? dir
document = #person.document.document_file_name # check document uploaded params
File.copy_stream(#font.document, "#{dir}/#{document}")
Note, Add this path in .gitignore & if you want to use this file again add this path asset_pathan of application by application.rb
Whenever form read file field, it get store in tmp folder, later you can store at your place, I gave example to store at assets
note: Storing files like this will increase the size of the application, better to store in the database using paperclip.
In your intiallizer/carrierwave.rb
if Rails.env.development? || Rails.env.test?
config.storage = :file
config.root = "#{Rails.root}/public"
if Rails.env.test?
CarrierWave.configure do |config|
config.storage = :file
config.enable_processing = false
end
end
end
use this to store in a file while running on local
I am trying to create a custom Paperclip::Processor that integrates with an external web service (the processor will call the web service whenever a new file is uploaded). The external service needs the file to be present in S3 and will handle uploading the processed versions to S3 automatically.
Can this be done using a custom Paperclip::Processor or should it be done with an ActiveRecord callback? If a Paperclip::Processor will work, what is the best way to trigger the upload? Ideally I'd like to do a processor, but the requirement is that the original file MUST be uploaded to S3 first. I have taken a look at using after_create calls, but it sometimes seems to conflict with the after_create used in paperclip. Thanks.
You can do this to create a local copy of the file. If it's on S3 it will be downloaded.
tmp_file = #model.attached_file.to_file => TempFile<...>
You can then do your operations on this TempFile. When you're don:
#model.attached_file = tmp_file
#model.save
Edit: misread your question. You can use the before_post_process and after_post_process hooks to perform tasks before or after the file was processed.
class Model < AR::Base
has_attached_file :avatar
after_post_process :ping_webservice
private
def ping_webservice
# Do your magic here.
end
end
I dealt with a similar issue recently, and it was with the after_save callback. I managed to fix my problem by defining paperclip (has_attached_file ...) after I defined my after_save. This way, paperclip's callback will fire after mine.