Rails Active Storage upload to a public S3 remote url - ruby-on-rails

I need to upload a video to a 3rd party via API.
Using the API I have requested this "Upload Location" which is valid for 15-minute. Now I need to upload my Active Storage video directly to this remote upload location. This remote location is not managed by me.
I have read the official documentation but it's not clear where I can change the default upload location url with this one.
Doc: https://edgeguides.rubyonrails.org/active_storage_overview.html#direct-uploads
Upload location:
{"uploadLocation"=>"https://storage-3rd-party.s3.eu-west-1.amazonaws.com/staging/folderr/12345/xxxx?X-Amz-
Security-Token=xxx...."}

If I'm understanding correctly, you're going to want to implement a custom ActiveStorage::Service for this 3rd party API. Behind the scenes, rails invokes url_for_direct_upload to get the URL that you're wanting to customize.
You should be able to something close to working if you implemented a new service like so:
class ThirdPartyStorageService < ActiveStorage::Service
def url_for_direct_upload(key, expires_in:, content_type:, content_length:, checksum:)
ThirdPartyAPI.get_upload_location(...)
end
# Implement other abstract methods...
end
You then need to add your service in config/storage.yml:
third_party:
service: ThirdPartyStorageService
# username: ...
# password: ...
# other config...
And then you can set it up to be used in a specific model, or globally.
# app/models/my_model.rb
class MyModel
has_one_attached :file, service: :third_party
end
# or config/application.rb
config.active_storage.service = :third_party
It's a bit of work, but I think this should set you up for success! Make sure to read the docs on ActiveStorage::Service and you can look at the implementations for Azure, AWS and Google storage services for inspiration if you aren't sure how to implement a certain method.

Related

Rails Active Storage set folder to store files

I'm using Active Storage to store files in a Rails 5.2 project. I've got files saving to S3, but they save with random string filenames and directly to the root of the bucket. I don't mind the random filenames (I actually prefer it for my use case) but would like to keep different attachments organized into folders in the bucket.
My model uses has_one_attached :file. I would like to specify to store all these files within a /downloads folder within S3 for example. I can't find any documentation regarding how to set these paths.
Something like has_one_attached :file, folder: '/downloads' would be great if that's possible...
The ultimate solution is to add an initializer. You can add a prefix based on an environment variable or your Rails.env :
# config/initializer/active_storage.rb
Rails.configuration.to_prepare do
ActiveStorage::Blob.class_eval do
before_create :generate_key_with_prefix
def generate_key_with_prefix
self.key = if prefix
File.join prefix, self.class.generate_unique_secure_token
else
self.class.generate_unique_secure_token
end
end
def prefix
ENV["SPACES_ROOT_FOLDER"]
end
end
end
It works perfectly with this. Other people suggest using Shrine.
Credit to for this great workaround : https://dev.to/drnic/how-to-isolate-your-rails-blobs-in-subfolders-1n0c
As of now ActiveStorage doesn't support that kind of functionality. Refer to this link. has_one_attached just accepts name and dependent.
Also in one of the GitHub issues, the maintainer clearly mentioned that they have clearly no idea of implementing something like this.
The workaround that I can imagine is, uploading the file from the front-end and then write a service that updates key field in active_storage_blob_statement
There is no official way to change the path which is determined by ActiveStorage::Blob#key and the source code is:
def key
self[:key] ||= self.class.generate_unique_secure_token
end
And ActieStorage::Blog.generate_unique_secure_token is
def generate_unique_secure_token
SecureRandom.base36(28)
end
So a workaround is to override the key method like the following:
# config/initializers/active_storage.rb
ActiveSupport.on_load(:active_storage_blob) do
def key
self[:key] ||= "my_folder/#{self.class.generate_unique_secure_token}"
end
end
Don't worry, this will not affect existing files. But you must be careful ActiveStorage is very new stuff, its source code is variant. When upgrading Rails version, remind yourself to take look whether this patch causes something wrong.
You can read ActiveStorage source code from here: https://github.com/rails/rails/tree/master/activestorage
Solution using Cloudinary service
If you're using Cloudinary you can set the folder on storage.yml:
cloudinary:
service: Cloudinary
folder: <%= Rails.env %>
With that, Cloudinary will automatically create folders based on your Rails env:
This is a long due issue with Active Storage that seems to have been worked around by the Cloudinary team. Thanks for the amazing work ❤️
# config/initializers/active_storage.rb
ActiveSupport.on_load(:active_storage_blob) do
def key
sql_find_order_id = "select * from active_storage_attachments where blob_id = #{self.id}"
active_storage_attachment = ActiveRecord::Base.connection.select_one(sql_find_order_id)
# this variable record_id contains the id of object association in has_one_attached
record_id = active_storage_attachment['record_id']
self[:key] = "my_folder/#{self.class.generate_unique_secure_token}"
self.save
self[:key]
end
end
Active Storage by default doesn't contain a path/folder feature but you can override the function by
model.file.attach(key: "downloads/filename", io: File.open(file), content_type: file.content_type, filename: "#{file.original_filename}")
Doing this will store the key with the path where you want to store the file in the s3 subdirectory and upload it at the exact place where you want.

Active Storage, specify a Google bucket directory ?

I've been implementing an Active Storage Google strategy on Rails 5.2, at the moment I am able to upload files using the rails console without problems, the only thing I am missing is if there is a way to specify a directory inside a bucket. Right now I am uploading as follows
bk.file.attach(io: File.open(bk.source_dir.to_s), filename: "file.tar.gz", content_type: "application/x-tar")
The configuration on my storage.yml
google:
service: GCS
project: my-project
credentials: <%= Rails.root.join("config/myfile.json") %>
bucket: bucketname
But in my bucket there are different directories such as bucketname/department1 and such. I been through the documentation and have not found a way to specify further directories and all my uploads end up in bucket name.
Sorry, I’m afraid Active Storage doesn’t support that. You’re intended to configure Active Storage with a bucket it can use exclusively.
Maybe you can try metaprogramming, something like this:
Create config/initializers/active_storage_service.rb to add set_bucket method to ActiveStorage::Service
module Methods
def set_bucket(bucket_name)
# update config bucket
config[:bucket] = bucket_name
# update current bucket
#bucket = client.bucket(bucket_name, skip_lookup: true)
end
end
ActiveStorage::Service.class_eval { include Methods }
Update your bucket before uploading or downloading files
ActiveStorage::Blob.service.set_bucket "my_bucket_name"
bk.file.attach(io: File.open(bk.source_dir.to_s), filename: "file.tar.gz", content_type: "application/x-tar")

How to list directory folders in rails

Hi I am newbie in rails and wanted to know any way to list folders and files for a url say public . So example user types example.com/public he gets all folders and files listed as we do in php if we start a wamp server . I need this so I can create files and have their url shared to public i.e. simply send url link online like example.com/public/test.pdf . Currently I am getting page with no routes defined .enter image description here
Thanks.
Create a controller to serve the file using a route parameter, ex.
get '/public/:filename', to: 'file#serve'
Then use the send_file method in your controller.
class FileController < ApplicationController
def serve
# For authorization:
#authorize!
# For authentication: (a way or another)
#redirect_to :access_denied unless user_signed_in?
send_file "/www/yourapp/var/files/#{params[:filename]}"
end
end
This way the file can be anywhere within your app, or even on a Cloud storage. It also gives you the ability to use authentication or authorization to check if the user has access.
Please note that this code is very simple, but there are much more options like Fog gem for Cloud storage and everything else.
See https://apidock.com/rails/ActionController/Streaming/send_file for more information.

Connecting to Google Calendar's API through Rails

I'm creating a really simple Rails application with one specific purpose: add a Calendar event to a Google Calendar account.
I'm following the Ruby Guide from the Google Calendar API.
I was able to run the provided testing code (Ruby only, no framework) but I'm having a hard time accessing the credentials from a Rails project and I'm not sure the proper ("idiomatic"?) way to do it and organize the project.
Part of the process is using OAuth 2.0 since this goal requires access to Google User's private data (both read and write), I'm following the Using OAuth 2.0 for Web Services instructions.
Right now I have several different questions regarding best practices and/or the proper way to organize code:
Google provides a client_secret.json that have the credentials to access the application. Where should I keep it? Should I keep it in a .env file in the Development environment and (in my case) in Heroku's ENV VARS in the Production Environment?
I tried keeping the client_secret.json file in the project's root folder (same path as the Gemfile), added it to the .gitignore but I wasn't able to require "#{Rails.root}/client_secret.json":
/Users/jsoifer/.rbenv/versions/2.3.1/lib/ruby/gems/2.3.0/gems/activesupport-5.0.1/lib/active_support/dependencies.rb:293:in `require': No such file to load -- /Users/jsoifer/Developer/Tiny-Things/tiny-booking/client_secret.json (LoadError)
I created a services/ folder to put the Google Calendar related code inside, I wasn't sure if I should put is in a controller though. How should I organize this?
Important consideration:
I'm not using any other method of Authentication/Authorization such as Devise or others and I'm not planning to do so right now. I just want to get Google's Authorization token and create a Calendar event.
Github Project Link
I was able to figure this out and will post the answer to each of the questions below:
One of the possible locations for the client_secret.json file is config/client_secret.json.
When shipping to Production in Heroku, use ENV Vars.
Require is not the appropriate way to import the credentials in the json file.
Use Google::APIClient::ClientSecrets.load( File.join( Rails.root, 'config', 'client_secret.json' ) ) (assuming the file is indeed in config/.
There are several different alternatives as on how to organize code. I ended up creating a services folder and a google_calendar.rb class holding the authorization logic.
Here's the code:
app/services/google_calendar.rb
require 'google/api_client/client_secrets'
require 'google/apis/calendar_v3'
class GoogleCalendar
# Attributes Accessors (attr_writer + attr_reader)
attr_accessor :auth_client, :auth_uri, :code
def initialize
# ENV: Development
# Google's API Credentials are in ~/config/client_secret.json
client_secrets = Google::APIClient::ClientSecrets.load( File.join( Rails.root, 'config', 'client_secret.json' ) )
#auth_client = client_secrets.to_authorization
# Specify privileges and callback URL
#auth_client.update!(
:scope => 'https://www.googleapis.com/auth/calendar',
:redirect_uri => 'http://localhost:3000/oauth2callback'
)
# Build up the Redirecting URL
#auth_uri = #auth_client.authorization_uri.to_s
end
end
app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
protect_from_forgery with: :exception
# Starting action in config/routes.rb
def welcome
# Redirect to Google Authorization Page
redirect_to GoogleCalendar.new.auth_uri
end
def token
# Get a auth_client object from Google API
#google_api = GoogleCalendar.new
#google_api.auth_client.code = params[:code] if params[:code]
response = #google_api.auth_client.fetch_access_token!
session[:access_token] = response['access_token']
# Whichever Controller/Action needed to handle what comes next
redirect_to new_event_path()
end
end

Rails: allow download of files stored on S3 without showing the actual S3 URL to user

I have a Rails application hosted on Heroku. The app generates and stores PDF files on Amazon S3. Users can download these files for viewing in their browser or to save on their computer.
The problem I am having is that although downloading of these files is possible via the S3 URL (like "https://s3.amazonaws.com/my-bucket/F4D8CESSDF.pdf"), it is obviously NOT a good way to do it. It is not desirable to expose to the user so much information about the backend, not to mention the security issues that rise.
Is it possible to have my app somehow retrieve the file data from S3 in a controller, then create a download stream for the user, so that the Amazon URL is not exposed?
You can create your s3 objects as private and generate temporary public urls for them with url_for method (aws-s3 gem). This way you don't stream files through your app servers, which is more scalable. It also allows putting session based authorization (e.g. devise in your app), tracking of download events, etc.
In order to do this, change direct links to s3 hosted files into links to controller/action which creates temporary url and redirects to it. Like this:
class HostedFilesController < ApplicationController
def show
s3_name = params[:id] # sanitize name here, restrict access to only some paths, etc
AWS::S3::Base.establish_connection!( ... )
url = AWS::S3::S3Object.url_for(s3_name, YOUR_BUCKET, :expires_in => 2.minutes)
redirect_to url
end
end
Hiding of amazon domain in download urls is usually done with DNS aliasing. You need to create CNAME record aliasing your subdomain, e.g. downloads.mydomain, to s3.amazonaws.com. Then you can specify :server option in AWS::S3::Base.establish_connection!(:server => "downloads.mydomain", ...) and S3 gem will use it for generating links.
Yes, this is possible - just fetch the remote file with Rails and either store it temporarily on your server or send it directly from the buffer. The problem with this is of course the fact that you need to fetch the file first before you can serve it to the user. See this thread for a discussion, their solution is something like this:
#environment.rb
require 'open-uri'
#controller
def index
data = open(params[:file])
send_data data, :filename => params[:name], ...
end
This issue is also somewhat related.
First you need create a CNAME in your domain, like explain here.
Second you need create a bucket with the same name that you put in CNAME.
And to finish you need add this configurations in your config/initializers/carrierwave.rb:
CarrierWave.configure do |config|
...
config.asset_host = 'http://bucket_name.your_domain.com'
config.fog_directory = 'bucket_name.your_domain.com'
...
end

Resources