My users store an external image URL (http://their-site.com/photo.jpg) in - for example - #user.external_image. I'm trying to write a method for the User class that takes that URL and saves it to S3 using Carrierwave.
So on the above #user, I'd like to run #user.save_to_s3 and have it "upload" the image to S3. I've tried to do this by mounting an uploader on :s3_image to the User class and writing the following method:
def save_to_s3
self.remote_s3_image_url = self.external_image
save
end
But I get the following error when I call that method on a #user record:
"ArgumentError: Missing required arguments: aws_access_key_id, aws_secret_access_key"
So it's getting close, but it's not retrieving my S3 credentials - even though they're set. I'd appreciate any thoughts or suggestions.
The problem turned out to be unrelated to Carrierwave or Fog. It was an oversight on my part that the ENV variables which I'd set (in my app's .env file) were not being loaded into the bootstrapped Rails environment (e.g. rails console). Once I added http://github.com/bkeepers/dotenv (which solves precisely that issue) to my bundle, the save_to_s3 method worked.
Related
We have an Attachment model with a copy_for_edit! method which helps an Attachment copy itself. The attachment data is stored in AWS S3 bucket(s). We make use of the Bucket copy_to technique to perform the copy remotely on the AWS S3 server, without transferring the data back to us.
https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Object.html#copy_to-instance_method
I'm writing a unit test (in minitest) for this method, and getting this error caused by the Aws::S3::Bucket#copy_to instance method:
Aws::S3::Errors::NoSuchKey: Aws::S3::Errors::NoSuchKey: The specified key does not exist.
I've seen countless solutions for how to stub an AWS S3 Client, but not a Bucket. I'm sure I'm missing something simple here. The code itself works in staging and production, but testing in my development environment I'd obviously prefer not to hit the AWS S3 server. But even if I configure the test environment to use our staging credentials for the bucket, that doesn't work either (same error).
I'd like to know how to stub (or similar) the Aws::S3::Bucket#copy_to instance method in minitest.
I know I've left out some details. I will be watching this closely and editing to add context if needed.
Edit 1: A simplified version of the test looks like this:
test '#copy_for_edit! should copy the attachment, excluding some attributes' do
source = attachments(:attachment_simple) #From an existing fixture.
result = nil
assert_difference(-> { Attachment.count }, 1) do
result = source.copy_for_edit!
end
assert_nil(result.owner)
assert_nil(result.draft_id)
end
Narrowing it down to an instance method (not a class method or attribute) helped me narrow down my options.
I finally got the syntax correct, and believe I have a working test now.
This was basically my solution: https://stackoverflow.com/a/29042835/14837782
I can't say I've stubbed the AWS S3 Bucket#copy_to method. I've actually just stubbed our own method (copy_attached_file_to) that eventually calls it, since I'm not actually testing that method. When it comes time to test that method, I might be in similar trouble. Although maybe this solution will work to stub Bucket similarly.
And here is the test now, seemingly working properly:
test '#copy_for_edit! should copy the attachment, excluding some attributes' do
source = attachments(:attachment_simple) # From an existing fixture.
source.stub(:copy_attached_file_to, true) do
result = nil
assert_difference(-> { Attachment.count }, 1) do
result = source.copy_for_edit!
end
assert_nil(result.owner)
assert_nil(result.draft_id)
end
end
I'm using AmazonS3 to store Paperclip attachments on all non-test environments.
For test specifically I use a local path/url setup to avoid interacting with S3 remotely
Paperclip::Attachment.default_options[:path] =
":rails_root/public/system/:rails_env/:class/:attachment/:id_partition/:filename"
Paperclip::Attachment.default_options[:url] =
"/system/:rails_env/:class/:attachment/:id_partition/:filename"
I define my attachment as follows in the model
has_attached_file :source_file, use_timestamp: false
In my Production code I need to access the file using Model.source_file.url because .url returns the remote fully qualified Amazon S3 path to the file. This generally works fine for non-test environments.
However on my test environment I can't use .url because Paperclip creates and stores the file under the path defined by :path above. So I need to use .path. If I use .url I get the error -
Errno::ENOENT:
No such file or directory # rb_sysopen - /system/test/imports/source_files/000/000/030/sample.txt
which makes sense because paperclip didn't store the file there...
How do I get paperclip on my test environment to store/create my file under the :url path so I can use .url correctly?
Edit: If it helps, in test I create the attachment from a locally stored fixture file
factory :import do
source_file { File.new(Rails.root + "spec/fixtures/files/sample.tsv") }
end
Edit2: Setting :path and :url to be the same path in the initializer might seem like a quick fix, but I'm working on a larger app with several contributors, so I don't the have the luxury to do that or break any one else's specs. Plus it looks like Thoughtbot themselves recommend this setup, so there should be a "proper" way to get it working as is.
Thanks!
Have you tried using s3proxy in your test environment to simulate S3 instead of directly have paperclip write to local files?
I am using Paperclip/RSpec and StackOverflow has helped me successfully stub file uploads to S3 using this code:
spec/rails_helper.rb
config.before(:each) do
allow_any_instance_of(Paperclip::Attachment).to receive(:save).and_return(true)
end
This is working great.
On my model I have two Paperclip fields:
class MyModel < ActiveRecord::Base
has_attached_file :pdf
has_attached_file :resource
end
My code uses the #copy_to_local_file method (Docs) to retrieve a file from S3.
#copy_to_local_file takes two params: the style (:original, :thumbnail, etc) and the local file path to copy to.
Example:
MyModel.resource.copy_to_local_file(:original, local_file.path)
When the system under test tries to access MyModel#pdf#copy_to_local_file or MyModel#resource#copy_to_local_file, I originally got errors like the following:
No Such Key - cannot copy /email_receipts/pdfs/000/000/001/original/email_receipt.eml.pdf to local file /var/folders/4p/1mm86g0n58x7d9rvpy88_s9h0000gn/T/receipt20150917-4906-13evk95.pdf
No Such Key - cannot copy /email_receipts/resources/000/000/001/original/email_receipt.eml to local file /var/folders/4p/1mm86g0n58x7d9rvpy88_s9h0000gn/T/resource20150917-4906-1ysbwr3.eml
I realize these errors were happening because uploads to S3 are stubbed, so when it encounters MyModel#pdf#copy_to_local_file or MyModel#resource#copy_to_local_file it tries to grab a file in S3 that isn't there.
Current Solution:
I've managed to quash the errors above, but I feel it's not a complete solution and gives my tests a false sense of security. My half-solution is to stub this method in the following way:
spec/rails_helper.rb
before(:each) do
allow_any_instance_of(Paperclip::Storage::S3).to receive(:copy_to_local_file)
end
While this does stub out the #copy_to_local_file method and removes the errors, it doesn't actually write any content to the local file that is provided as the second argument to #copy_to_local_file, so it doesn't quite simulate the file being downloaded from S3.
Question:
Is there a way to stub #copy_to_local_file AND have it write the contents of a canned file in my spec/factories/files directory to the local file (its second argument)?
Or am I overthinking this? Is this something I shouldn't be worrying about?
You don't need to worry about whether the 'downloaded' files actually exist in your tests. You've decided to stub out Paperclip, so do it completely, by stubbing out both #save and #copy_to_file. You may also need to stub out reads of downloaded files from the filesystem.
All this stubbing raises the possibility of integration errors, so you should probably write a feature spec (using a captive browser like poltergeist) that actually uploads and downloads something and reads it from the filesystem.
That said, you can do anything you want in an RSpec stub by passing it a block:
allow_any_instance_of(Paperclip::Storage::S3).to receive(:copy_to_local_file) do |style, local_dest_path|
# write a file here, or do anything you like
end
In my rails app, I'm using Carrierwave to upload images on Amazon S3. I'd like to point to existing Amazon S3 images without having to re-upload the image. For example, if I have an existing Amazon S3 image at http://test.s3.amazonaws.com/image/path/0001/image.jpg, can I update an image's path to point to this image? I don't want to use the remote upload option because I really just want to use the same exact image that's already there (but save it in my record's "path" attribute).
In the console, I've tried:
image.update_attributes(:path=> "http://test.s3.amazonaws.com/image/path/0001/image.jpg")
but this fails to override the image's path.
Chiming in, better late than never! Caveat: This is for rails 4, and I am testing on rails 4.1 only at the moment.
This is harder than it should be, methinks! The reason this was absolutely crucial to me was that I am attaching 100MB+ MP3 files, which I cannot receive on my host, due to CloudFlare SSL limitations (and common sense). Fortunately, AWS supports preauthorized uploads, and I got carrierwave to do the right thing for me:
Step 1: get carrierwave to tell me where it would store a file if it could:
m.raw_write_attribute('file','file.mp3');
url = m.file.url
signed = aws_presigned_url(url)
raw_write_attribute does not save anything, it just bypasses carrierwave when setting the value. This makes the object act as if it read 'file.mp3' out of the database. Then you can ask Carrierwave "where the file lives". I then upload the file directly from the client to S3. When that's done, I make another API call to Rails, which performs the following code:
m.raw_write_attribute('file','file.mp3');
m.update_attribute('file','file.mp3');
These two paired get around Carrierwave. The first makes carrierwave think that the 'file' column is set to 'file.mp3', the second explicitly tells rails to persist 'file.mp3' to the DB. Because of the raw_write_attribute call, Carrierwave allows the second through un-changed.
In my case update_column and update_columns worked great:
model.update_columns file_1: 'filename.txt'
Update column is with comma:
model.update_column :file_1, 'filename.txt'
This will not run any callback and set column to filename.txt.
When I do model.file_1.url I get the right S3 URL.
I am a bit late to the party, but you can use Active Record's raw_write_attribute method something like:
#image.raw_write_attribute(:path, "http://test.s3.amazonaws.com/image/path/0001/image.jpg")
I found that you can actually do this, for example if your mount_uploader is :path, then:
image.remote_path_url = "http://test.s3.amazonaws.com/image/path/0001/image.jpg"
image.save
I have uploaded a file on s3 using paperclip.. the file upload process works fine..
Now i wanted to download it. In my model i have set my :s3_host_alias.. now as the file is private.. so if i am trying to fetch the file using paperclip url method... it's giving me access denied error...
and if i am using S3Object.url_for method then the url return is s3.amazonaws.com/mybucket/path_of_file.
I don't want tht s3.amazonaws.com to be shown in the url so used :s3_host_alias in my model
and created a CNAME inmy DNS server... now if i am directly using #object.url then its giving the correct url but throws access denied error. because i guess the access_key and signature is not passed..
Is there a way to fetch private file from s3 using paperclip by using canonical url..
I don't use paperclip, but yes, you can sign a S3 request using a virtual hostname.
I had this problem using Paperclip and the AWS::S3 gem. Paperclip set up everything fine for non-authenticated requests. But falling back to AWS::S3 to generate an authenticated URL didn't use the S3 host alias.
You can pass AWS::S3 a server option on connect, but I didn't need or want a connection just to get the URL. I also couldn't see a way to set it via configuration (so it would apply outside of a connection). Even glancing at the source, it looks like it's non-configurable.
So, I created a monkey patch. My Ruby-fu (and maybe my OO-fu) aren't super high, so there may be a better way to do this, but it works for what I need. Basically, I pass url_for an :s3_host_alias param on the option hash, and then the monkey patch uses that if it's passed. If it's passed, it also has to remove the bucket from the path that's generated.
So....
You can create this 1-line file, RAILS_ROOT/initializers/load_patches.rb, to load all patches in RAILS_ROOT/lib:
Dir[File.join(Rails.root, 'lib', 'patches', '**', '*.rb')].sort.each { |patch| require(patch) }
Then create the file RAILS_ROOT/lib/patches/aws.rb with this code:
http://pastie.org/1622881
And you can call for an authenticated url with something along these lines (Configuration is a custom class for storing, natch, configuration values) :
AWS::S3::S3Object.url_for(media.path(style || media.default_style), media.bucket_name, :expires_in => expires_in, :use_ssl => false, :s3_host_alias => Configuration.s3_host_alias)