I have uploaded a file on s3 using paperclip.. the file upload process works fine..
Now i wanted to download it. In my model i have set my :s3_host_alias.. now as the file is private.. so if i am trying to fetch the file using paperclip url method... it's giving me access denied error...
and if i am using S3Object.url_for method then the url return is s3.amazonaws.com/mybucket/path_of_file.
I don't want tht s3.amazonaws.com to be shown in the url so used :s3_host_alias in my model
and created a CNAME inmy DNS server... now if i am directly using #object.url then its giving the correct url but throws access denied error. because i guess the access_key and signature is not passed..
Is there a way to fetch private file from s3 using paperclip by using canonical url..
I don't use paperclip, but yes, you can sign a S3 request using a virtual hostname.
I had this problem using Paperclip and the AWS::S3 gem. Paperclip set up everything fine for non-authenticated requests. But falling back to AWS::S3 to generate an authenticated URL didn't use the S3 host alias.
You can pass AWS::S3 a server option on connect, but I didn't need or want a connection just to get the URL. I also couldn't see a way to set it via configuration (so it would apply outside of a connection). Even glancing at the source, it looks like it's non-configurable.
So, I created a monkey patch. My Ruby-fu (and maybe my OO-fu) aren't super high, so there may be a better way to do this, but it works for what I need. Basically, I pass url_for an :s3_host_alias param on the option hash, and then the monkey patch uses that if it's passed. If it's passed, it also has to remove the bucket from the path that's generated.
So....
You can create this 1-line file, RAILS_ROOT/initializers/load_patches.rb, to load all patches in RAILS_ROOT/lib:
Dir[File.join(Rails.root, 'lib', 'patches', '**', '*.rb')].sort.each { |patch| require(patch) }
Then create the file RAILS_ROOT/lib/patches/aws.rb with this code:
http://pastie.org/1622881
And you can call for an authenticated url with something along these lines (Configuration is a custom class for storing, natch, configuration values) :
AWS::S3::S3Object.url_for(media.path(style || media.default_style), media.bucket_name, :expires_in => expires_in, :use_ssl => false, :s3_host_alias => Configuration.s3_host_alias)
Related
In a Rails 6 api-only project running in a K8s cluster I'm using S3 to store file uploaded by users. When retrieving files I'm using
polymorphic_url(pdf_document, only_path: true) (because I need to manipulate host path. I tried rails_blob_path, url_for but the final result is always the same)
and it returns the right path:
/rails/active_storage/blobs/redirect/SIGNED_ID/document.pdf
When accessing this path, I get a redirect to S3 private object uri, like
https://BUCKETNAME.s3.us-east-1.amazonaws.com/ID?response-content-disposition=inline%3B%20filename%3D%22document.pdf%22%3B%20filename%2A%3DUTF-8%27%27document.pdf&response-content-type=application%2Fpdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AWS_ACCESS_KEY_ID%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20221209T112219Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=SIGNATURE
that exposes private informations like AWS_ACCESS_KEY_ID and SIGNATURE.
How do I return the public S3 object uri, or a safe uri without sensitive data?
I have a paperclip instance that I am migrating my files to a different area. Originally the files were stored on my server and just given a filename based on the id of the record created and the original id. Now I'm moving them to s3 and want to update the filenames to work appropriately. I setup my paperclip config like so:
:path => ":class/:attachment/:hash-:style.:extension",
:url => ":s3_domain_url",
:hash_secret => SECRET,
:hash_data => ":class/:attachment/:id/:updated_at"
I updated the original records filenames for my files to be unique and moved them over to my s3 instance. Unfortunately now I am unable to pull down the files from s3 and I think it is because paperclip is using the wrong path for the filenames. One that is based off the path default that is now set using my config file. I want to be able to update my files file_name field so that the path is correct for the new files and I am able to download them appropriately. Is there a way to call paperclips hashing function based on my secret and hash_data directly so I can update those file_name fields and be able to pull those records now? Everything that has been uploaded since the move from my original servers seems to work appropriately.
Say you have a model User with an attachment named profile_pic;
Go into the rails console eg. rails c and then get an object for the model you have the attachment on, eg. u = User.find(100).
Now type u.profile_pic.url to get the url or u.profile_pic_file_name to get the filename.
To see the effect of other options (for example your old options) you can do;
p = u.profile_pic # gets the paperclip attachment for profile_pic
puts p.url # gets the current url
p.options.merge!(url: '/blah/:class/:attachment/:id_partition/:style/:filename')
puts p.url # now shows url with the new options
Similarly p.path will show the local file path with whatever options you pick.
Long story short, something like;
User.where('created_at < some_date').map do |x|
"#{x.id} #{x.profile_pic_file_name} #{x.profile_pic.path}"
end
should give you what you want :)
In my rails app, I'm using Carrierwave to upload images on Amazon S3. I'd like to point to existing Amazon S3 images without having to re-upload the image. For example, if I have an existing Amazon S3 image at http://test.s3.amazonaws.com/image/path/0001/image.jpg, can I update an image's path to point to this image? I don't want to use the remote upload option because I really just want to use the same exact image that's already there (but save it in my record's "path" attribute).
In the console, I've tried:
image.update_attributes(:path=> "http://test.s3.amazonaws.com/image/path/0001/image.jpg")
but this fails to override the image's path.
Chiming in, better late than never! Caveat: This is for rails 4, and I am testing on rails 4.1 only at the moment.
This is harder than it should be, methinks! The reason this was absolutely crucial to me was that I am attaching 100MB+ MP3 files, which I cannot receive on my host, due to CloudFlare SSL limitations (and common sense). Fortunately, AWS supports preauthorized uploads, and I got carrierwave to do the right thing for me:
Step 1: get carrierwave to tell me where it would store a file if it could:
m.raw_write_attribute('file','file.mp3');
url = m.file.url
signed = aws_presigned_url(url)
raw_write_attribute does not save anything, it just bypasses carrierwave when setting the value. This makes the object act as if it read 'file.mp3' out of the database. Then you can ask Carrierwave "where the file lives". I then upload the file directly from the client to S3. When that's done, I make another API call to Rails, which performs the following code:
m.raw_write_attribute('file','file.mp3');
m.update_attribute('file','file.mp3');
These two paired get around Carrierwave. The first makes carrierwave think that the 'file' column is set to 'file.mp3', the second explicitly tells rails to persist 'file.mp3' to the DB. Because of the raw_write_attribute call, Carrierwave allows the second through un-changed.
In my case update_column and update_columns worked great:
model.update_columns file_1: 'filename.txt'
Update column is with comma:
model.update_column :file_1, 'filename.txt'
This will not run any callback and set column to filename.txt.
When I do model.file_1.url I get the right S3 URL.
I am a bit late to the party, but you can use Active Record's raw_write_attribute method something like:
#image.raw_write_attribute(:path, "http://test.s3.amazonaws.com/image/path/0001/image.jpg")
I found that you can actually do this, for example if your mount_uploader is :path, then:
image.remote_path_url = "http://test.s3.amazonaws.com/image/path/0001/image.jpg"
image.save
My users store an external image URL (http://their-site.com/photo.jpg) in - for example - #user.external_image. I'm trying to write a method for the User class that takes that URL and saves it to S3 using Carrierwave.
So on the above #user, I'd like to run #user.save_to_s3 and have it "upload" the image to S3. I've tried to do this by mounting an uploader on :s3_image to the User class and writing the following method:
def save_to_s3
self.remote_s3_image_url = self.external_image
save
end
But I get the following error when I call that method on a #user record:
"ArgumentError: Missing required arguments: aws_access_key_id, aws_secret_access_key"
So it's getting close, but it's not retrieving my S3 credentials - even though they're set. I'd appreciate any thoughts or suggestions.
The problem turned out to be unrelated to Carrierwave or Fog. It was an oversight on my part that the ENV variables which I'd set (in my app's .env file) were not being loaded into the bootstrapped Rails environment (e.g. rails console). Once I added http://github.com/bkeepers/dotenv (which solves precisely that issue) to my bundle, the save_to_s3 method worked.
I have a Rails application hosted on Heroku. The app generates and stores PDF files on Amazon S3. Users can download these files for viewing in their browser or to save on their computer.
The problem I am having is that although downloading of these files is possible via the S3 URL (like "https://s3.amazonaws.com/my-bucket/F4D8CESSDF.pdf"), it is obviously NOT a good way to do it. It is not desirable to expose to the user so much information about the backend, not to mention the security issues that rise.
Is it possible to have my app somehow retrieve the file data from S3 in a controller, then create a download stream for the user, so that the Amazon URL is not exposed?
You can create your s3 objects as private and generate temporary public urls for them with url_for method (aws-s3 gem). This way you don't stream files through your app servers, which is more scalable. It also allows putting session based authorization (e.g. devise in your app), tracking of download events, etc.
In order to do this, change direct links to s3 hosted files into links to controller/action which creates temporary url and redirects to it. Like this:
class HostedFilesController < ApplicationController
def show
s3_name = params[:id] # sanitize name here, restrict access to only some paths, etc
AWS::S3::Base.establish_connection!( ... )
url = AWS::S3::S3Object.url_for(s3_name, YOUR_BUCKET, :expires_in => 2.minutes)
redirect_to url
end
end
Hiding of amazon domain in download urls is usually done with DNS aliasing. You need to create CNAME record aliasing your subdomain, e.g. downloads.mydomain, to s3.amazonaws.com. Then you can specify :server option in AWS::S3::Base.establish_connection!(:server => "downloads.mydomain", ...) and S3 gem will use it for generating links.
Yes, this is possible - just fetch the remote file with Rails and either store it temporarily on your server or send it directly from the buffer. The problem with this is of course the fact that you need to fetch the file first before you can serve it to the user. See this thread for a discussion, their solution is something like this:
#environment.rb
require 'open-uri'
#controller
def index
data = open(params[:file])
send_data data, :filename => params[:name], ...
end
This issue is also somewhat related.
First you need create a CNAME in your domain, like explain here.
Second you need create a bucket with the same name that you put in CNAME.
And to finish you need add this configurations in your config/initializers/carrierwave.rb:
CarrierWave.configure do |config|
...
config.asset_host = 'http://bucket_name.your_domain.com'
config.fog_directory = 'bucket_name.your_domain.com'
...
end