rails version 5.2
I have a scenario where I need to access the public URL of Rails Active Storage with Amazon S3 storage to make a zip file with Sidekiq background job.
I am having difficulty getting the actual file URL. I have tried rails_blob_url but it gives me following
http://localhost:3000/rails/active_storage/blobs/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBZUk9IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--9598613be650942d1ee4382a44dad679a80d2d3b/sample.pdf
How do I access the real file URL through Sidekiq?
storage.yml
test:
service: Disk
root: <%= Rails.root.join("tmp/storage") %>
local:
service: Disk
root: <%= Rails.root.join("storage") %>
development:
service: S3
access_key_id: 'xxxxx'
secret_access_key: 'xxxxx'
region: 'xxxxx'
bucket: 'xxxxx'
development.rb
config.active_storage.service = :development
I can access fine these on web interface but not within Sidekiq
Use ActiveStorage::Blob#service_url. For example, assuming a Post model with a single attached header_image:
#post.header_image.service_url
Update: Rails 6.1
Since Rails 6.1 ActiveStorage::Blob#service_url is deprecated in favor of ActiveStorage::Blob#url.
So, now
#post.header_image.url
is the way to go.
Sources:
Link to the corresponding PR.
Link to source.
If you need all your files public then you must make public your uploads:
In file config/storage.yml
amazon:
service: S3
access_key_id: zzz
secret_access_key: zzz
region: zzz
bucket: zzz
upload:
acl: "public-read"
In the code
attachment = ActiveStorage::Attachment.find(90)
attachment.blob.service_url # returns large URI
attachment.blob.service_url.sub(/\?.*/, '') # remove query params
It will return something like:
"https://foo.s3.amazonaws.com/bar/buz/2yoQMbt4NvY3gXb5x1YcHpRa"
It is public readable because of the config above.
My use case was to upload images to S3 which would have public access for ALL images in the bucket so a job could pick them up later, regardless of request origin or URL expiry. This is how I did it. (Rails 5.2.2)
First, the default for new S3 bucked is to keep everything private, so to defeat that there are 2 steps.
Add a wildcard bucket policy. In AWS S3 >> your bucket >> Permissions >> Bucket Policy
{
"Version": "2008-10-17",
"Statement": [
{
"Sid": "AllowPublicRead",
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::your-bucket-name/*"
}
]
}
In your bucket >> Permissions >> Public Access Settings, be sure Block public and cross-account access if bucket has public policies is set to false
Now you can access anything in your S3 bucket with just the blob.key in the url. No more need for tokens with expiry.
Second, to generate that URL you can either use the solution by #Christian_Butzke: #post.header_image.service.send(:object_for, #post.header_image.key).public_url
However, know that object_for is a private method on service, and if called with public_send would give you an error. So, another alternative is to use the service_url per #George_Claghorn and just remove any params with a url&.split("?")&.first. As noted, this may fail in localhost with a host missing error.
Here is my solution or an uploadable "logo" stored on S3 and made public by default:
#/models/company.rb
has_one_attached :logo
def public_logo_url
if self.logo&.attachment
if Rails.env.development?
self.logo_url = Rails.application.routes.url_helpers.rails_blob_url(self.logo, only_path: true)
else
self.logo_url = self.logo&.service_url&.split("?")&.first
end
end
#set a default lazily
self.logo_url ||= ActionController::Base.helpers.asset_path("default_company_icon.png")
end
Enjoy ^_^
I had a few problems getting this working. Thought I'd document them for posterity.
In rails 6.0 use #post.header_image.service_url
In rails >= 6.1 use #post.header_image.url as #GeorgeClaghorn recommends.
I got this error:
error: uninitialized constant Analyzable
It's a weird bug in rails 6.0, which is fixed by placing this in config/application.rb
config.autoloader = :classic
I then see this error:
URI::InvalidURIError (bad URI(is not URI?): nil) Active Storage service_url
Fix it by simply adding this to your application_controller.rb
include ActiveStorage::SetCurrent
Now something like #post.image.blob.service_url will work as you expect =)
Using the service_url method combined with striping the params to get a public URL was good idea, thanks #genkilabs and #Aivils_Štoss!
There is however a potential scaling issue involved if you are using this method on large number of files, eg. if you are showing a list of records that have files attached. For each call to service_url you will in your logs see something like:
DEBUG -- : [8df9220c-e8c9-45b7-a1ee-b746e623ca1b] S3 Storage (1.4ms) Generated URL for file at key: ...
You can't eager load these calls either, so you can potentially have a large number of calls to S3 Storage to generate those URLs for each record you are showing.
I worked around it by creating a Presenter like this:
class FilePresenter < SimpleDelegator
def initialize(obj)
super
end
def public_url
return dev_url if Rails.env.development? || Rails.env.test? || assest_host.nil?
"#{assest_host}/#{key}"
end
private
def dev_url
Rails.application.routes.url_helpers.rails_blob_url(self, only_path: true)
end
def assest_host
#assest_host ||= ENV['ASSET_HOST']
end
end
Then I set an ENV variable ASSET_HOST with this:
https://<your_app_bucket>.s3.<your_region>.amazonaws.com
Then when I display the image or just the file link, I do this:
<%= link_to(image_tag(company.display_logo),
FilePresenter.new(company.logo).public_url, target: "_blank", rel:"noopener") %>
<a href=<%= FilePresenter.new(my_record.file).public_url %>
target="_blank" rel="noopener"><%= my_record.file.filename %></a>
Note, you still need to use display_logo for images so that it will access the variant if you are using them.
Also, this is all based on setting my AWS bucket public as per #genkilabs step #2 above, and adding the upload: acl: "public-read" setting to my 'config/storage.yml' as per #Aivils_Štoss!'s suggestion.
If anyone sees any issues or pitfalls with this approach, please let me know! This seemed to work great for me in allowing me to display a public URL but not needing to hit the S3 Storage for each record to generate that URL.
Also see public access in rails active storage. This was introduced in Rails 6.1.
Specify public: true in your app's config/storage.yml. Public services will always return a permanent URL.
Just write this if You are using minio or aws S3 to get attachment url on server.
#post.header_image&.service_url&.split("?")&.first
A bit late, but you can get the public URL also like this (assuming a Post model with a single attached header_image as in the example above):
#post.header_image.service.send(:object_for, #post.header_image.key).public_url
Update 2020-04-06
You need to make sure, that the document is saved with public ACLs (e.g. setting the default to public)
rails_blob_url is also usable. Requests will be served by rails, however, those requests will be probably quite slow, since a private URL needs to be generated on each request.
(FYI: outside the controller you can generate that URL also like this: Rails.application.routes.url_helpers.rails_blob_url(#post, only_path: true))
I am trying to generate a pre-signed url on my Rails server to send to the browser, so that the browser can upload to S3.
It seems like aws-sdk-s3 is the gem to use going forward. But unfortunately, I haven't come across documentation for the gem that would provide clarity. There seem to be a few different ways of doing so, and would appreciate any guidance on the difference in the following methods -
Using Aws::S3::Presigner.new (https://github.com/aws/aws-sdk-ruby/blob/master/aws-sdk-core/lib/aws-sdk-core/s3/presigner.rb) but it doesn't seem to take in an object parameter or auth credentials.
Using Aws::S3::Resource.new, but it seems like aws-sdk-resources is not going to be maintained. (https://aws.amazon.com/blogs/developer/upgrading-from-version-2-to-version-3-of-the-aws-sdk-for-ruby-2/)
Using Aws::S3::Object.new and then calling the put method on that object.
Using AWS::SigV4 directly.
I am wondering how they differ, and the implications of choosing one over the other? Any recommendations are much appreciated, especially with aws-sdk-s3.
Thank you!
So, thanks to the tips by #strognjz above, here is what worked for me using `aws-sdk-s3'.
require 'aws-sdk-s3'
#credentials below for the IAM user I am using
s3 = Aws::S3::Client.new(
region: 'us-west-2', #or any other region
access_key_id: AWS_ACCESS_KEY_ID,
secret_access_key: AWS_SECRET_ACCESS_KEY
)
signer = Aws::S3::Presigner.new(client: s3)
url = signer.presigned_url(
:put_object,
bucket: S3_BUCKET_NAME,
key: "${filename}-#{SecureRandom.uuid}"
)
This will work using the aws-sdk-s3 gem
aws_client = Aws::S3::Client.new(
region: 'us-west-2', #or any other region
access_key_id: AWS_ACCESS_KEY_ID,
secret_access_key: AWS_SECRET_ACCESS_KEY
)
s3 = Aws::S3::Resource.new(client: aws_client)
bucket = s3.bucket('bucket-name')
obj = bucket.object("${filename}-#{SecureRandom.uuid}")
url = obj.presigned_url(:put)
additional http verbs:
obj.presigned_url(:put)
obj.presigned_url(:head)
obj.presigned_url(:delete)
In my Rails app I save customer RMA shipping labels to an S3 bucket on creation. I just updated to V2 of the aws-sdk gem, and now my code for setting the ACL doesn't work.
Code that worked in V1.X:
# Saves label to S3 bucket
s3 = AWS::S3.new
obj = s3.buckets[ENV['S3_BUCKET_NAME']].objects["#{shippinglabel_filename}"]
obj.write(open(label.label('pdf').postage_label.label_pdf_url, 'rb'), :acl => :public_read)
.write seems to have been deprecated, so I'm using .put now. Everything is working, except when I try to set the ACL.
New code for V2.0:
# Saves label to S3 bucket
s3 = Aws::S3::Resource.new
obj = s3.bucket(ENV['S3_BUCKET_NAME']).object("#{shippinglabel_filename}")
obj.put(Base64.decode64(label_base64), { :acl => :public_read })
I get an Aws::S3::Errors::InvalidArgument error, pointed at the ACL.
This code works for me:
photo_obj = bucket.object object_name
photo_obj.upload_file path, {acl: 'public-read'}
so you need to use the string 'public-read' for the acl. I found this by seeing an example in object.rb
I have been able to have third party clients upload files directly to AWS s3 and then process those files with paperclip with the following line in the model:
my_object.file_attachment = URI.parse(URI.escape(my_bucket.s3.amazonaws.com/whatever.ext))
That line downloads the file, processes it and then saves it appropriately. The problem is, in order for that line to work, I have to provide anonymous read privileges for the upload location. So my question is: How do avoid that? My thought is to use the aws-sdk to download the file - so I have been trying stuff like:
file = Tempfile.new('temp', :encoding => 'ascii-8bit')
bucket.objects[aws_key].read do |chunk|
file.write chunk
end
my_object.file_attachment = file
and variations on that theme, but nothing is working so far. Any insights would be most helpful.
Solution I am not very happy with
You can generate a temporary privileged URL using the AWS SDK:
s3 = AWS::S3.new
bucket = s3.buckets['bucket_name']
my_object.file_attachment = bucket.objects['relative/path/of/uploaded/file.ext'].url_for(:read)
As #laertiades says in his amended question, one solution is to create a temporary, pre-signed URL using the AWS SDK.
AWS SDK version 1
In AWS SDK version 1, that looks like this:
s3 = AWS::S3.new
bucket = s3.buckets['bucket_name']
my_object.file_attachment = bucket.objects['relative/path/of/uploaded/file.ext'].url_for(:read)
AWS documentation: http://docs.aws.amazon.com/AWSRubySDK/latest/AWS/S3/S3Object.html#url_for-instance_method
AWS SDK version 2
In AWS SDK version 2, it looks like this with the optional expires_in parameter (credit to this answer on another question):
presigner = Aws::S3::Presigner.new
my_object.file_attachment = presigner.presigned_url(:get_object, # get_object method means read-only
bucket: 'bucket-name',
key: "relative/path/of/uploaded/file.ext",
expires_in: 10.minutes.to_i # time should be in seconds
).to_s
AWS documentation: http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Presigner.html
What's the more straightforward way to download a file from given URL and uploading it immediately to Amazon S3 (+ save into database some information about the file, like name, size etc)?
Right now, I am not using Paperclip neither Carrierwave.
Thank you
Straightforward:
require 'open-uri'
require 's3'
amazon = S3::Service.new(access_key_id: 'KEY', secret_access_key: 'KEY')
bucket = amazon.buckets.find('image_storage')
url = 'http://www.example.com/url'
download = open(url)
file = bucket.objects.build('image.png')
file.content = (File.read download)
if file.save
# Make a new ActiveRecord::Base class for this
LogFile.create(size: download.size, type: download.type, name: url)
end
https://github.com/qoobaa/s3