I have been able to have third party clients upload files directly to AWS s3 and then process those files with paperclip with the following line in the model:
my_object.file_attachment = URI.parse(URI.escape(my_bucket.s3.amazonaws.com/whatever.ext))
That line downloads the file, processes it and then saves it appropriately. The problem is, in order for that line to work, I have to provide anonymous read privileges for the upload location. So my question is: How do avoid that? My thought is to use the aws-sdk to download the file - so I have been trying stuff like:
file = Tempfile.new('temp', :encoding => 'ascii-8bit')
bucket.objects[aws_key].read do |chunk|
file.write chunk
end
my_object.file_attachment = file
and variations on that theme, but nothing is working so far. Any insights would be most helpful.
Solution I am not very happy with
You can generate a temporary privileged URL using the AWS SDK:
s3 = AWS::S3.new
bucket = s3.buckets['bucket_name']
my_object.file_attachment = bucket.objects['relative/path/of/uploaded/file.ext'].url_for(:read)
As #laertiades says in his amended question, one solution is to create a temporary, pre-signed URL using the AWS SDK.
AWS SDK version 1
In AWS SDK version 1, that looks like this:
s3 = AWS::S3.new
bucket = s3.buckets['bucket_name']
my_object.file_attachment = bucket.objects['relative/path/of/uploaded/file.ext'].url_for(:read)
AWS documentation: http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html#url_for-instance_method
AWS SDK version 2
In AWS SDK version 2, it looks like this with the optional expires_in parameter (credit to this answer on another question):
presigner = Aws::S3::Presigner.new
my_object.file_attachment = presigner.presigned_url(:get_object, # get_object method means read-only
bucket: 'bucket-name',
key: "relative/path/of/uploaded/file.ext",
expires_in: 10.minutes.to_i # time should be in seconds
).to_s
AWS documentation: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Presigner.html
Related
I am using aws-skd-s3 gem in my Rails project.
Create S3 resoure
s3 = Aws::S3::Resource.new(access_key_id: #####,
secret_access_key: #####,
region: 'us-east-1')
Create an S3 object
path = 'sample'
key = test.csv
obj = s3.bucket(#{bucket_name}).object("#{path}" + key)
Store CSV in S3
obj.put(body: csv_response, content_type: 'text/csv')
How to verify that put method stored the csv in S3 without any issues?
Is there any status code available for put method in S3 to verify?
Two ways to go about it:
Store the result. It should be a PutObjectOutput type object. You can check out the official method documentation of the put request method.
The second way to go about it is to make a exists? call right after your put request is completed. Something like this:
s3 = Aws::S3::Resource.new(region: 'ap-southeast-1') # change to the region you use
obj = s3.bucket('bucket-name').object("path/to/object/in/bucket")
if obj.exists?
# Object was uploaded successfully!
else
# No it wasn't!
end
Hope that helps!
One way I've seen or read other people doing it is calculating a md5 hash of the original file before upload and then match that with the etag value from the response of obj.put
In Rails, what is the best (ie. efficient, elegant) way to download a file from a public HTTP url, upload to Amazon S3 and delete the file in the server.
I am using Heroku so I have the additional restriction of a ephemeral file system.
If reading into memory isn't an option.
You can use the /tmp directory. As long as this is in the same process using that shouldn't be a problem.
The answer is to read your image into memory instead of storing it on the filesystem. Here's an example.
s3 = Aws::S3::Resource.new
obj = s3.bucket(your_bucket_name).object(your_object_key)
s3_put_url = URI.parse(obj.presigned_url(:put))
image_url = 'http://www.google.com/google.jpg'
image_file = open(image_url).read
Net::HTTP.start(s3_put_url.host) { |http| http.send_request('PUT', s3_put_url.request_uri, image_file); }
# Let's get the URL
s3.bucket(your_bucket_name).object(your_object_key).presigned_url(:get)
I'm using the s3_direct_upload gem to store images and videos on Amazon s3. When the image or video is changed or deleted, I want to nuke the old image or video on s3 and save everyone money and space.
This solution uses the V1 Aws SDK and is no longer valid:
http://blog.littleblimp.com/post/53942611764/direct-uploads-to-s3-with-rails-paperclip-and
This solution deletes files that were initially uploaded in a batch, but does nothing for the final files post-processing:
github - waynehoover/s3_direct_upload
Here is the Aws v2 SDK doc, which seems clear enough:
http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html#delete_object-instance_method
Yet this solution:
class Image < ActiveRecord::Base
validates :name, :s3_image_file_url, :s3_image_file_path, :s3_image_file_key, presence: true
before_destroy :delete_s3_files
private
def delete_s3_files
s3 = Aws::S3::Client.new()
response = s3.delete_object(
bucket: Rails.configuration.aws[:bucket],
key: self.s3_image_file_key
)
end
end
...returns only:
=> #<struct delete_marker=nil, version_id=nil>
(And the file is still available on s3 at the original url.)
Thoughts? Hasn't everyone had to do this?
I've written code using the v2 SDK to delete objects from S3. Here is a sample from my codebase:
def delete_avatar_from_s3
Aws::S3::Client.new.delete_object(
bucket: "user-avatars",
key: "#{#id}"
)
return true
rescue => e
Rails.logger.error "Error deleting avatar of user #{#id}. Failure with S3 call. Details: #{e}; #{e.backtrace}"
return false
end
It looks similar to yours, so I don't think that this code is the issue. Have you confirmed your bucket & key names and ensured that your method is actually being called?
AWS SDK for Ruby V2
delete one or more file, just pass objects array containing keys.
def delete_from_s3
S3_BUCKET.delete_objects({
delete: {
objects: [
{key: key}
]
}
})
end
In my Rails app I save customer RMA shipping labels to an S3 bucket on creation. I just updated to V2 of the aws-sdk gem, and now my code for setting the ACL doesn't work.
Code that worked in V1.X:
# Saves label to S3 bucket
s3 = AWS::S3.new
obj = s3.buckets[ENV['S3_BUCKET_NAME']].objects["#{shippinglabel_filename}"]
obj.write(open(label.label('pdf').postage_label.label_pdf_url, 'rb'), :acl => :public_read)
.write seems to have been deprecated, so I'm using .put now. Everything is working, except when I try to set the ACL.
New code for V2.0:
# Saves label to S3 bucket
s3 = Aws::S3::Resource.new
obj = s3.bucket(ENV['S3_BUCKET_NAME']).object("#{shippinglabel_filename}")
obj.put(Base64.decode64(label_base64), { :acl => :public_read })
I get an Aws::S3::Errors::InvalidArgument error, pointed at the ACL.
This code works for me:
photo_obj = bucket.object object_name
photo_obj.upload_file path, {acl: 'public-read'}
so you need to use the string 'public-read' for the acl. I found this by seeing an example in object.rb
I have the following problem,
In my rails 4 app I am hosting images / videos on s3. Currently I made all the files public and for example an image I can access by storing the public link in the database.
However, I want some of the images videos to be private.
I looked at the presigned url options using the following
s3 = Aws::S3::Client.new(
region: AWS_REGION,
access_key_id: S3_CONFIG['access_key_id'],
secret_access_key: S3_CONFIG['secret_access_key']
)
resource = Aws::S3::Resource.new(client: s3)
bucket = resource.bucket(BUCKET_NAME)
utilities = bucket.objects(prefix: '/folder').each do |obj|
obj.presigned_url(:get, expires_in: 3600).to_s
end
This works fine, but how would I use the presigned url since I can obviously not store them in the db like the public links.
I am using aws-sdk version 2
I am also wondering if this in general is a good solution?
Thanks for any hints,
Jean
Here is the Presigner Doc
Example:
signer = Aws::S3::Presigner.new
url = signer.presigned_url(:put_object, bucket: "bucket", key: "path")