Change ActiveStorage DirectDisk service configuration at runtime - ruby-on-rails

I am using Rails 5.2 and ActiveStorage 5.2.3 and DirectDiskService.
In order to have user uploads grouped nicely in directories and in order to be able to use CDNs as I please (eg CloudFlare or CloudFront or any other), I am trying to set up a method in ApplicationController that sets the (local) path for uploads, something like:
class ApplicationController < ActionController::Base
before_action :set_uploads_path
# ...
private
def set_upload_paths
# this doesn't work
ActiveStorage::Service::DirectDiskService.set_root p: "users/#{current_user.username}"
# ... expecting the new root to become "public/users/yaddayadda"
end
end
In config/initializers/active_storage.rb I have:
module SetDirectDiskServiceRoot
def set_root(p:)
#root = Rails.root.join("public", p)
#public_root = p
#public_root.prepend('/') unless #public_root.starts_with?('/')
end
end
ActiveStorage::Service::DirectDiskService.module_eval { attr_writer :root }
ActiveStorage::Service::DirectDiskService.class_eval { include SetDirectDiskServiceRoot }
It does let me set the root on a new instance of the service but not on the one the Rails application is using.
What am I doing wrong? Or how do I get this done?

Here's a dirty hack that helps you keep your local (disk) uploads grouped nice under public/websites/domain_name/uploads.
Step 1) install ActiveStorage DirectDisk service from here: https://github.com/sandrew/activestorage_direct_disk
Step 2) in app/models/active_storage/current.rb
class ActiveStorage::Current < ActiveSupport::CurrentAttributes #:nodoc:
attribute :host
attribute :domain_name
end
Step 3) lib/set_direct_disk_service_path.rb
module SetCurrentDomainName
def set_domain_name(d)
self.domain_name = d
end
end
ActiveStorage::Current.class_eval { include SetCurrentDomainName }
module SetDirectDiskServiceRoot
def initialize(p:, public: false, **options)
#root = Rails.root.join("public", p)
#public_root = p
#public_root.prepend('/') unless #public_root.starts_with?('/')
puts options
end
def current_domain_name
ActiveStorage::Current.domain_name
end
def folder_for(key)
# original: File.join root, folder_for(key), key
p = [ current_domain_name, "uploads", "all", key ]
blob = ActiveStorage::Blob.find_by(key: key)
if blob
att = blob.attachments.first
if att
rec = att.record
if rec
p = [ current_domain_name, "uploads", rec.class.name.split("::").last.downcase, rec.id.to_s, att.name, key ]
end
end
end
return File.join p
end
end
ActiveStorage::Service::DirectDiskService.module_eval { attr_writer :root }
ActiveStorage::Service::DirectDiskService.class_eval { include SetDirectDiskServiceRoot }
Step 4) in config/initializers/active_storage.rb
require Rails.root.join("lib", "set_direct_disk_service_path.rb")
Step 5) in app/controllers/application_controller.rb
before_action :set_active_storage_domain_name
# ...
def set_active_storage_domain_name
ActiveStorage::Current.domain_name = current_website.domain_name # or request.host
end
Step 6) in config/storage.yml
development:
service: DirectDisk
root: 'websites_development'
production:
service: DirectDisk
root: 'websites'
Disadvantages:
While ActiveRecord technically "works", it's missing some very important features that make it unusable for most people, so eventually the developer(s) will listen and adjust; at that time you may need to revisit this code AND all of your uploads.
The service attempts to "guess" the class name a blob is attached to since AS doesn't pass that, so it runs extra 2-3 queries against your database. If this bothers you just remove that bit and let it all go under websites/domain_name/uploads/all/
In some cases (eg variants, or a new record with action_text column) it can't figure out the attachment record and its class name, so it will upload under websites/domain/uploads/all/...

Related

Rails 5.2 Import XLSX with ActiveStorage and Creek

I have a model called ImportTemp which is used for store imported XLSX file to database. I'm using ActiveStorage for storing the files.
this is the model code:
class ImportTemp < ApplicationRecord
belongs_to :user
has_one_attached :file
has_one_attached :log_result
end
this is my import controller code:
def import
# Check filetype
case File.extname(params[:file].original_filename)
when ".xlsx"
# Add File to ImportFile model
import = ImportTemp.new(import_type: 'UnitsUpload', user: current_user)
import.file.attach(params[:file])
import.save
# Import unit via sidekiq with background jobs
ImportUnitWorker.perform_async(import.id)
# Notice
flash.now[:notice] = "We are processing your xlsx, we will inform you after it's done via notifications."
# Unit.import_file(xlsx)
else flash.now[:error] = t('shared.info.unknown')+": #{params[:file].original_filename}"
end
end
after upload the xlsx file, then the import will be processed in sidekiq. This is the worker code (still doesn't do the import actually) :
class ImportUnitWorker
include Sidekiq::Worker
sidekiq_options retry: false
def perform(file_id)
import_unit = ImportTemp.find(file_id)
# Open the uploaded xlsx to Creek
creek = Creek::Book.new(Rails.application.routes.url_helpers.rails_blob_path(import_unit.file, only_path: true))
sheet = creek.sheets[0]
puts "Opening Sheet #{sheet.name}"
sheet.rows.each do |row|
puts row
end
units = []
# Unit.import(units)
end
but after i tried it, it gives me error:
Zip::Error (File /rails/active_storage/blobs/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBCZz09IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3960b6ba5b55f7004e09967d16dfabe63f09f0a9/2018-08-10_10_39_audit_gt.xlsx not found)
but if i tried to open it with my browser, which is the link looks like this:
http://localhost:3000/rails/active_storage/blobs/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBCZz09IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3960b6ba5b55f7004e09967d16dfabe63f09f0a9/2018-08-10_10_39_audit_gt.xlsx
it's working and the xlsx is downloaded. My question is what's wrong with it? why the file is not found in the sidekiq?
I ended up using Tempfile as suggested by George Claghorn. I don't know if this is the best solution or best practices, but it works for me now. I'm going to use this solution while waiting Rails 6 stable to come out with ActiveStorage::Blob#open feature.
def perform(file_id)
import = ImportTemp.find(file_id)
temp_unit = Tempfile.new([ 'unit_import_temp', '.xlsx' ], :encoding => 'ascii-8bit')
units = []
begin
# Write xlsx from ImportTemp to Tempfile
temp_unit.write(import.file.download)
# Open the temp xlsx to Creek
book = Creek::Book.new(temp_unit.path)
sheet = book.sheets[0]
sheet.rows.each do |row|
# Skip the header
next if row.values[0] == 'Name' || row.values[1] == 'Abbreviation'
cells = row.values
# Add cells to new Unit
unit = Unit.new(name: cells[0], abbrev: cells[1], desc: cells[2])
units << unit
end
# Import the unit
Unit.import(units)
ensure
temp_unit.close
temp_unit.unlink # deletes the temp file
end
end
Rails.application.routes.url_helpers.rails_blob_path doesn’t return the path to the file on disk. Rather, it returns a path that can be combined with a hostname to produce an URL for downloading the file, for use in links.
You have two options:
If you’d prefer to keep ImportUnitWorker indifferent to the storage service in use, “download” the file to a tempfile on disk. Switch to Rails master and use ActiveStorage::Blob#open:
def perform(import_id)
import = ImportTemp.find(import_id)
units = []
import.file.open do |file|
book = Creek::Book.new(file.path)
sheet = creek.sheets[0]
# ...
end
Unit.import(units)
end
If you don’t mind ImportWorker knowing that you use the disk service, ask the service for the path to the file on disk. ActiveStorage::Service::DiskService#path_for(key) is private in Rails 5.2, so either forcibly call it with send or upgrade to Rails master, where it’s public:
def perform(import_id)
import = ImportTemp.find(import_id)
units = []
path = ActiveStorage::Blob.service.send(:path_for, import.file.key)
book = Creek::Book.new(path)
sheet = creek.sheets[0]
# ...
Unit.import(units)
end
The answer now seems to be (unless I am missing something):
Creek::Book.new file.service_url, check_file_extension: false, remote: true

Rails console - reload! third party services in modules

My app is connected to some third-party APIs.
I have several APIconnector module-singletons that are initialized only once at application start (initialized means the client is instanciated once with the credentials retrieved from secrets)
When I reload! the application in my console, I am losing those services and I have to exit and restart the console from scratch.
Basically all my connectors include a ServiceConnector module like this one
module ServiceConnector
extend ActiveSupport::Concern
included do
#activated = false
#activation_attempt = false
#client = nil
attr_reader :client, :activated
def self.client
#client ||= service_client
end
def self.service_name
name.gsub('Connector', '')
end
def self.activate
#activation_attempt = true
if credentials_present?
#client = service_client
#activated = true
end
end
Here is an example of a service implementation
module My Connector
include ServiceConnector
#app_id = nil
#api_key = nil
def self.set_credentials(id, key)
#app_id = id
#api_key = key
end
def self.credentials_present?
#app_id.present? and #api_key.present?
end
def self.service_client
::SomeAPI::Client.new(
app_id: #app_id,
api_key: #api_key
)
end
end
I use this pattern that lets me reuse those services outside Rails (eg Capistrano, worker without Rails, etc.). In Rails I would load the services that way
# config/initializers/my_service.rb
if my_service_should_be_activated?
my_service.set_credentials(
Rails.application.secrets.my_service_app_id,
Rails.application.secrets.my_service_app_key
)
my_service.activate
end
I guess that executing reload! seems to clear all my instance variables including #client, #app_id, #api_key.
Is it possible to add code to be executed after a reload! ? In my case I would need to re-run the initializer. Or is there a way to make sure the instance variables of my services are not cleared with a reload! ?
So I have come up with a solution involving two initializers
First, a 000_initializer that will report which secrets were loaded successfully
module SecretChecker
module_function
# Return true if all secrets are present
def secrets?(secret_list, under:)
secret_root = Rails.application.secrets
if under
if under.is_a?(Array)
secret_root = secret_root.public_send(under.shift)&.dig(*under.map(&:to_s))
else
secret_root = secret_root.public_send(under)
end
secret_list.map do |secret|
secret_root&.dig(secret.to_s).present?
end
else
secret_list.map do |secret|
secret_root&.public_send(secret.to_s).present?
end
end.reduce(:&)
end
def check_secrets(theme, secret_list, under: nil)
return if secrets?(secret_list, under: under)
message = "WARNING - Missing secrets for #{theme} - #{yield}"
puts message and Rails.logger.warn(message)
end
end
SecretChecker.check_secrets('Slack', %i[martine], under: [:slack, :webhooks]) do
'Slack Notifications will not work'
end
SecretChecker.check_secrets('MongoDB', %i[user password], under: :mongodb) do
'No Database Connection if auth is activated'
end
Then, a module to reload the services with ActiveSupport::Reloader (an example featuring Slack)
# config/initializers/0_service_activation.rb
module ServiceActivation
def self.with_reload
ActiveSupport::Reloader.to_prepare do
yield
end
end
module Slack
def self.service
::SlackConnector
end
def self.should_be_activated?
Rails.env.production? ||
Rails.env.staging? ||
(Rails.env.development? && ENV['ENABLE_SLACK'] == 'true')
end
def self.activate
slack = service
slack.webhook = Rails.application.secrets.slack&.dig('webhooks', 'my_webhook')
ENV['SLACK_INTERCEPT_CHANNEL'].try do |channel|
slack.intercept_channel = channel if channel.present?
end
slack.activate
slack
end
end
end
[
...,
ServiceActivation::Slack
] .each do |activator|
ServiceActivation.with_reload do
activator.activate if activator.should_be_activated?
activator.service.status_report
end
end

How do I set the tag owner before any tag is saved with the acts-as-taggable gem?

I am using acts-as-taggable gem https://github.com/mbleigh/acts-as-taggable-on
and jQueryTokenInput plugin to add tags to my image model . Tags are created and added fine so far. I followed the tutorial http://bloginius.com/blog/2013/12/31/how-integrate-acts-as-taggable-on-with-jquery-token-input-with-rails-3/.
Now however, I want to be able to give ownership of the tag to the current_user at the time the tag is created.
As in the gem' s github page, I have tried
#some_user.owned_taggings
#some_user.owned_tags
with no satisfactory results. I proceeded and added a user_id to the tags table. Is there a tagsController associated with the acts-as-taggable-on gem that I can modify with a before_save to set the user_id for the tag ?
Thanks!!
A section of the acts-as-taggable-on README dedicated to ownership, and is useful to work through with the specifics of your models.
But, I don't think the methods provided are correct -- they will apply all owned tags (owned by anyone, that is), to each owner<>item relationship. Here's how I'd do it:
DEFAULT_ACTSASTAGGABLEON_TYPE = :tag
module TagToOwner
extend ActiveSupport::Concern
private
def add_owned_tag(item, owner, tags_to_add, options = {})
own_tag(item, owner, arrayify(tags_to_add), "add", options)
end
def remove_owned_tag(item, owner, tags_to_add, options = {})
own_tag(item, owner, arrayify(tags_to_add), "subtract", options)
end
def own_tag(item, owner, tags_to_add, direction = "add", opts)
tag_type = (options[:tag_type] || DEFAULT_ACTSASTAGGABLEON_TYPE)
owned_tag_list = item.owner_tags_on(owner, tag_type).map(&:name)
if direction == "subtract"
owned_tag_list = owned_tag_list.reject{|n| n.in?(tags_to_add)}
else
owned_tag_list.push(*tags_to_add)
end
owner.tag(item, with: stringify(owned_tag_list), on: tag_type, skip_save: (options[:skip_save] || true))
end
def arrayify(tags_to_add)
return tags_to_add if tags_to_add.is_a?(Array)
tags_to_add.split(",")
end
def stringify(tag_list)
tag_list.inject('') { |memo, tag| memo += (tag + ',') }.chomp(",")
end
end
And:
class MyModelController < ApplicationController
include TagToOwner
# ...
def create
#my_model = MyModel.new(my_model_params)
add_tags
# ...
end
# ...
def update
add_tags
# ...
end
private
def add_tags
return unless params[:tag_list] && "#{params[:tag_list]}".split(",").any?
return unless validate_ownership_logic # <- e.g. `current_user`
add_owned_tag(#my_model, current_user, params[:tag_list])
end
end
Note I have filed an issue against acts-as-taggable-on, and a corresponding pull request, to correct their README.

Gem to wrap API can't make API key setter work in all classes

I have a Ruby gem which wraps an API. I have two classes: Client and Season with a Configuration module. But I can't access a change to the API Key, Endpoint made via Client in the Season class.
My ApiWrapper module looks like this:
require "api_wrapper/version"
require 'api_wrapper/configuration'
require_relative "api_wrapper/client"
require_relative "api_wrapper/season"
module ApiWrapper
extend Configuration
end
My Configuration module looks like this:
module ApiWrapper
module Configuration
VALID_CONNECTION_KEYS = [:endpoint, :user_agent, :method].freeze
VALID_OPTIONS_KEYS = [:api_key, :format].freeze
VALID_CONFIG_KEYS = VALID_CONNECTION_KEYS + VALID_OPTIONS_KEYS
DEFAULT_ENDPOINT = 'http://defaulturl.com'
DEFAULT_METHOD = :get
DEFAULT_API_KEY = nil
DEFAULT_FORMAT = :json
attr_accessor *VALID_CONFIG_KEYS
def self.extended(base)
base.reset
end
def reset
self.endpoint = DEFAULT_ENDPOINT
self.method = DEFAULT_METHOD
self.user_agent = DEFAULT_USER_AGENT
self.api_key = DEFAULT_API_KEY
self.format = DEFAULT_FORMAT
end
def configure
yield self
end
def options
Hash[ * VALID_CONFIG_KEYS.map { |key| [key, send(key)] }.flatten ]
end
end # Configuration
end
My Client class looks like this:
module ApiWrapper
class Client
attr_accessor *Configuration::VALID_CONFIG_KEYS
def initialize(options={})
merged_options = ApiWrapper.options.merge(options)
Configuration::VALID_CONFIG_KEYS.each do |key|
send("#{key}=", merged_options[key])
end
end
end # Client
end
My Season class looks like this:
require 'faraday'
require 'json'
API_URL = "/seasons"
module ApiWrapper
class Season
attr_accessor *Configuration::VALID_CONFIG_KEYS
attr_reader :id
def initialize(attributes)
#id = attributes["_links"]["self"]["href"]
...
end
def self.all
puts ApiWrapper.api_key
puts ApiWrapper.endpoint
conn = Faraday.new
response = Faraday.get("#{ApiWrapper.endpoint}#{API_URL}/") do |request|
request.headers['X-Auth-Token'] = "ApiWrapper.api_key"
end
seasons = JSON.parse(response.body)
seasons.map { |attributes| new(attributes) }
end
end
end
This is the test I am running:
def test_it_gives_back_a_seasons
VCR.use_cassette("season") do
#config = {
:api_key => 'ak',
:endpoint => 'http://ep.com',
}
client = ApiWrapper::Client.new(#config)
result = ApiWrapper::Season.all
# Make sure we got all season data
assert_equal 12, result.length
#Make sure that the JSON was parsed
assert result.kind_of?(Array)
assert result.first.kind_of?(ApiWrapper::Season)
end
end
Because I set the api_key via the client to "ak" and the endpoint to "http://ep.com" I would expect puts in the Season class's self.all method to print out "ak" and "http://ep.com", but instead I get the defaults set in the Configuration section.
What I am doing wrong?
The api_key accessors you have on Client and on ApiWrapper are independent. You initialize a Client with the key you want, but then Season references ApiWrapper directly. You've declared api_key, etc. accessors in three places: ApiWrapper::Configuration, ApiWrapper (by extending Configuration) and Client. You should probably figure out what your use cases are and reduce that down to being in just one place to avoid confusion.
If you're going to have many clients with different API keys as you make different requests, you should inject the client into Season and use it instead of ApiWrapper. That might look like this:
def self.all(client)
puts client.api_key
puts client.endpoint
conn = Faraday.new
response = Faraday.get("#{client.endpoint}#{API_URL}/") do |request|
request.headers['X-Auth-Token'] = client.api_key
end
seasons = JSON.parse(response.body)
seasons.map { |attributes| new(attributes) }
end
Note that I also replaced the "ApiWrapper.api_key" string with the client.api_key - you don't want that to be a string anyway.
Having to pass client into every request you make is going to get old, so then you might want to pull out something like a SeasonQuery class to hold onto it.
If you're only ever going to have one api_key and endpoint for the duration of your execution, you don't really need the Client as you've set it up so far. Just set ApiWrapper.api_key directly and continue using it in Season.

Papertrail and Carrierwave

I have a model that use both: Carrierwave for store photos, and PaperTrail for versioning.
I also configured Carrierwave for store diferent files when updates (That's because I want to version the photos) with config.remove_previously_stored_files_after_update = false
The problem is that PaperTrail try to store the whole Ruby Object from the photo (CarrierWave Uploader) instead of simply a string (that would be its url)
(version table, column object)
---
first_name: Foo
last_name: Bar
photo: !ruby/object:PhotoUploader
model: !ruby/object:Bla
attributes:
id: 2
first_name: Foo1
segundo_nombre: 'Bar1'
........
How can I fix this to store a simple string in the photo version?
You can override item_before_change on your versioned model so you don't call the uploader accesor directly and use write_attribute instead. Alternatively, since you might want to do that for several models, you can monkey-patch the method directly, like this:
module PaperTrail
module Model
module InstanceMethods
private
def item_before_change
previous = self.dup
# `dup` clears timestamps so we add them back.
all_timestamp_attributes.each do |column|
previous[column] = send(column) if respond_to?(column) && !send(column).nil?
end
previous.tap do |prev|
prev.id = id
changed_attributes.each do |attr, before|
if defined?(CarrierWave::Uploader::Base) && before.is_a?(CarrierWave::Uploader::Base)
prev.send(:write_attribute, attr, before.url && File.basename(before.url))
else
prev[attr] = before
end
end
end
end
end
end
end
Not sure if it's the best solution, but it seems to work.
Adding #beardedd's comment as an answer because I think this is a better way to handle the problem.
Name your database columns something like picture_filename and then in your model mount the uploader using:
class User < ActiveRecord::Base
has_paper_trail
mount_uploader :picture, PictureUploader, mount_on: :picture_filename
end
You still use the user.picture.url attribute to access your model but PaperTrail will store revisions under picture_filename.
Here is a bit updated version of monkeypatch from #rabusmar, I use it for rails 4.2.0 and paper_trail 4.0.0.beta2, in /config/initializers/paper_trail.rb.
The second method override is required if you use optional object_changes column for versions. It works in a bit strange way for carrierwave + fog if you override filename in uploader, old value will be from cloud and new one from local filename, but in my case it's ok.
Also I have not checked if it works correctly when you restore old version.
module PaperTrail
module Model
module InstanceMethods
private
# override to keep only basename for carrierwave attributes in object hash
def item_before_change
previous = self.dup
# `dup` clears timestamps so we add them back.
all_timestamp_attributes.each do |column|
if self.class.column_names.include?(column.to_s) and not send("#{column}_was").nil?
previous[column] = send("#{column}_was")
end
end
enums = previous.respond_to?(:defined_enums) ? previous.defined_enums : {}
previous.tap do |prev|
prev.id = id # `dup` clears the `id` so we add that back
changed_attributes.select { |k,v| self.class.column_names.include?(k) }.each do |attr, before|
if defined?(CarrierWave::Uploader::Base) && before.is_a?(CarrierWave::Uploader::Base)
prev.send(:write_attribute, attr, before.url && File.basename(before.url))
else
before = enums[attr][before] if enums[attr]
prev[attr] = before
end
end
end
end
# override to keep only basename for carrierwave attributes in object_changes hash
def changes_for_paper_trail
_changes = changes.delete_if { |k,v| !notably_changed.include?(k) }
if PaperTrail.serialized_attributes?
self.class.serialize_attribute_changes(_changes)
end
if defined?(CarrierWave::Uploader::Base)
Hash[
_changes.to_hash.map do |k, values|
[k, values.map { |value| value.is_a?(CarrierWave::Uploader::Base) ? value.url && File.basename(value.url) : value }]
end
]
else
_changes.to_hash
end
end
end
end
end
This is what actually functions for me, put this on config/initializers/paper_trail/.rb
module PaperTrail
module Reifier
class << self
def reify_attributes(model, version, attrs)
enums = model.class.respond_to?(:defined_enums) ? model.class.defined_enums : {}
AttributeSerializers::ObjectAttribute.new(model.class).deserialize(attrs)
attrs.each do |k, v|
is_enum_without_type_caster = ::ActiveRecord::VERSION::MAJOR < 5 && enums.key?(k)
if model.send("#{k}").is_a?(CarrierWave::Uploader::Base)
if v.present?
model.send("remote_#{k}_url=", v["#{k}"][:url])
model.send("#{k}").recreate_versions!
else
model.send("remove_#{k}!")
end
else
if model.has_attribute?(k) && !is_enum_without_type_caster
model[k.to_sym] = v
elsif model.respond_to?("#{k}=")
model.send("#{k}=", v)
elsif version.logger
version.logger.warn(
"Attribute #{k} does not exist on #{version.item_type} (Version id: #{version.id})."
)
end
end
end
end
end
end
end
This overrides the reify method to work on S3 + heroku
For uploaders to keep old files from updated or deleted records do this in the uploader
configure do |config|
config.remove_previously_stored_files_after_update = false
end
def remove!
true
end
Then make up some routine to clear old files from time to time, good luck
I want to add to the previous answers the following:
It can happen that you upload different files with the same name, and this may overwrite your previous file, so you won't be able to restore the old one.
You may use a timestamp in file names or create random and unique filenames for all versioned files.
Update
This doesn't seem to work in all edge cases for me, when assigning more than a single file to the same object within a single request request.
I'm using this right now:
def filename
[#cache_id, original_filename].join('-') if original_filename.present?
end
This seems to work, as the #cache_id is generated for each and every upload again (which isn't the case as it seems for the ideas provided in the links above).
#Sjors Provoost
We also need to override pt_recordable_object method in PaperTrail::Model::InstanceMethods module
def pt_recordable_object
attr = attributes_before_change
object_attrs = object_attrs_for_paper_trail(attr)
hash = Hash[
object_attrs.to_hash.map do |k, value|
[k, value.is_a?(CarrierWave::Uploader::Base) ? value.url && File.basename(value.url) : value ]
end
]
if self.class.paper_trail_version_class.object_col_is_json?
hash
else
PaperTrail.serializer.dump(hash)
end
end

Resources