Any idea how to migrate a running project using Refile to the new rails's Active Storage?
Anyone knows any tutorial/guide about how to do that?
Thanks,
Patrick
I wrote a short post about it here which explains the process in detail:
https://dev.to/mtrolle/migrating-from-refile-to-activestorage-2dfp
Historically I hosted my Refile attached files in AWS S3, so what I did was refactoring all my code to use ActiveStorage instead. This primarily involved updating my model and views to use ActiveStorage syntax.
Then I removed the Refile gem and replaced it with ActiveStorage required gems like the image_processing gem and the aws-sdk-s3 gem.
Finally I created a Rails DB migration file to handle the actual migration of existing files. Here I looped through all records in my model with a Refile attachment to find their respective file in AWS S3, download it and then attach it to the model again using the ActiveStorage attachment.
Once the files were moved I could remove the legacy Refile database fields:
require 'mini_magick' # included by the image_processing gem
require 'aws-sdk-s3' # included by the aws-sdk-s3 gem
class User < ActiveRecord::Base
has_one_attached :avatar
end
class MovingFromRefileToActiveStorage < ActiveRecord::Migration[6.0]
def up
puts 'Connecting to AWS S3'
s3_client = Aws::S3::Client.new(
access_key_id: ENV['AWS_S3_ACCESS_KEY'],
secret_access_key: ENV['AWS_S3_SECRET'],
region: ENV['AWS_S3_REGION']
)
puts 'Migrating user avatar images from Refile to ActiveStorage'
User.where.not(avatar_id: nil).find_each do |user|
tmp_file = Tempfile.new
# Read S3 object to our tmp_file
s3_client.get_object(
response_target: tmp_file.path,
bucket: ENV['AWS_S3_BUCKET'],
key: "store/#{user.avatar_id}"
)
# Find content_type of S3 file using ImageMagick
# If you've been smart enough to save :avatar_content_type with Refile, you can use this value instead
content_type = MiniMagick::Image.new(tmp_file.path).mime_type
# Attach tmp file to our User as an ActiveStorage attachment
user.avatar.attach(
io: tmp_file,
filename: "avatar.#{content_type.split('/').last}",
content_type: content_type
)
if user.avatar.attached?
user.save # Save our changes to the user
puts "- migrated #{user.try(:name)}'s avatar image."
else
puts "- \e[31mFailed to migrate the avatar image for user ##{user.id} with Refile id #{user.avatar_id}\e[0m"
end
tmp_file.close
end
# Now remove the actual Refile column
remove_column :users, :avatar_id, :string
# If you've created other Refile fields like *_content_type, you can safely remove those as well
# remove_column :users, :avatar_content_type, :string
end
def down
raise ActiveRecord::IrreversibleMigration
end
end
Related
The guide says that I can save an attachment to disc to run a process on it like this:
message.video.open do |file|
system '/path/to/virus/scanner', file.path
# ...
end
My model has an attachment defined as:
has_one_attached :zip
And then in the model I have defined:
def process_zip
zip.open do |file|
# process the zip file
end
end
However I am getting an error :
private method `open' called
on the zip.open call.
How can I save the zip locally for processing?
As an alternative in Rails 5.2 you can do this:
def process_zip
# Download the zip file in temp dir
zip_path = "#{Dir.tmpdir}/#{zip.filename}"
File.open(zip_path, 'wb') do |file|
file.write(zip.download)
end
Zip::File.open(zip_path) do |zip_file|
# process the zip file
# ...
puts "processing file #{zip_file}"
end
end
That’s an edge guide (note edgeguides.rubyonrails.org in the URL); it applies to the master branch of the rails/rails repository on GitHub. The latest changes in master haven’t been included in a released version of Rails yet.
You’re likely using Rails 5.2. Use edge Rails to take advantage of ActiveStorage::Blob#open:
gem "rails", github: "rails/rails"
I'm receiving a file in a request params through a standard file input
def create
file = params[:file]
upload = Upload.create(file: file, filename: "img.png")
end
However, for large uploads, I'd like to do this in a background job.
Popular background jobs options like Sidekiq or Resque depend on Redis to store the parameters, so I can't just pass a file object through redis.
I could use a Tempfile, but on some platforms such as Heroku, local storage is not reliable.
What options do I have to make it reliable on "any" platform ?
I would suggest uploading directly to a service like Amazon S3 and then processing the file as you see fit in a background job.
When the user uploads the file, you can rest assure it will be safely stored in S3. You can use a private bucket for prohibiting public access. Then, in your background task you can process the upload by passing the file's S3 URI and let your background worker download the file.
I don't know what your background worker does with the file, but it goes without saying that downloading it again might not be necessary. It's stored somewhere after all.
I've used the carrierwave-direct gem in the past with success. Since you're mentioning Heroku, they have a detailed guide for uploading files directly to S3.
No tempfile
It sounds like you want to either speed up image uploading or push it into background. Here are my suggestions from another post. Maybe they'll help you if that's what you're looking for.
The reason I found this question is because I wanted to save a CSV file and have my background job add to the database with the info in that file.
I have a solution.
Because you the question is a bit unclear and I'm too lazy to post my own question and answer my own question, I'll just post the answer here. lol
Like the other dudes said, save the file on some cloud storage service. For Amazon, you need:
# Gemfile
gem 'aws-sdk', '~> 2.0' # for storing images on AWS S3
gem 'paperclip', '~> 5.0.0' # image processor if you want to use images
You also need this. Use the same code but different bucket name in production.rb
# config/environments/development.rb
Rails.application.configure do
config.paperclip_defaults = {
storage: :s3,
s3_host_name: 's3-us-west-2.amazonaws.com',
s3_credentials: {
bucket: 'my-bucket-development',
s3_region: 'us-west-2',
access_key_id: ENV['AWS_ACCESS_KEY_ID'],
secret_access_key: ENV['AWS_SECRET_ACCESS_KEY']
}
}
end
You also need a migration
# db/migrate/20000000000000_create_files.rb
class CreateFiles < ActiveRecord::Migration[5.0]
def change
create_table :files do |t|
t.attachment :import_file
end
end
end
and a model
class Company < ApplicationRecord
after_save :start_file_import
has_attached_file :import_file, default_url: '/missing.png'
validates_attachment_content_type :import_file, content_type: %r{\Atext\/.*\Z}
def start_file_import
return unless import_file_updated_at_changed?
FileImportJob.perform_later id
end
end
and a job
class FileImportJob < ApplicationJob
queue_as :default
def perform(file_id)
file = File.find file_id
filepath = file.import_file.url
# fetch file
response = HTTParty.get filepath
# we only need the contents of the response
csv_text = response.body
# use the csv gem to create csv table
csv = CSV.parse csv_text, headers: true
p "csv class: #{csv.class}" # => "csv class: CSV::Table"
# loop through each table row and do something with the data
csv.each_with_index do |row, index|
if index == 0
p "row class: #{row.class}" # => "row class: CSV::Row"
p row.to_hash # hash of all the keys and values from the csv file
end
end
end
end
In your controller
def create
#file.create file_params
end
def file_params
params.require(:file).permit(:import_file)
end
First you should save the file on storage(either local or AWS S3).
Then pass filepath or uuid as a parameter to background job.
I strongly recommend avoiding passing Tempfile on parameters. This stores object in memory which can get out of date, causing stale data problems.
I am trying to generate a csv file in a rake task and...
Email it
Upload it to Amazon s3.
Here is the task.
desc "This task is called by the Heroku scheduler add-on"
require 'csv'
task :send_report => :environment do
file = Baseline.to_csv
ReportMailer.database_report(file).deliver_now
Report.create!(:data => file)
end
The generation of the csv file and attachment to the email works fine (not shown). Its the carrierwave upload that isn't working. Please note that I have other uploaders for other models and they work fine so my bucket settings are correct.
Here are the other files.
class Report < ActiveRecord::Base
mount_uploader :data, ReportUploader
end
and
class ReportUploader < CarrierWave::Uploader::Base
storage :fog
def store_dir
"uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
def extension_white_list
%w(jpg jpeg gif png csv xls)
end
end
I have tried various permutations such as store! with not luck. I should add that if I look at the database, the new report is being created (and the data attribute is "nil", with no upload in sight)
Thanks
I'm using the carrierwave and carrierwave-aws gems to upload to S3 from a Rails application. I can upload files without any issue, but I can't remove them from S3.
My ActiveRecord model is called Episode and its video property is set by my CarrierWave::Uploader::Base uploader (which includes CarrierWave::Video).
Here are the steps I've followed:
Create an episode
Upload a video, verify it exists on S3 (this works!)
Call episode.remove_video!; episode.save! from the console
At this point, the video still exists on S3, despite receiving no error message
I've also tried:
episode.video.remove!
episode.save!
Which produces this output, but doesn't delete the file from S3:
[180] pry(main)> episode.video.remove!
=> [:remove_versions!]
I've also tried:
episode.destroy!
which should call the ActiveRecord callback added by Carrierwave, but doesn't. I added my own before_destroy method (destroy_assets below), but this also doesn't work.
Here is my setup:
carrierwave.rb
CarrierWave.configure do |config|
config.storage = :aws
config.aws_bucket = 'BUCKET_NAME'
config.aws_acl = 'public-read'
config.aws_authenticated_url_expiration = 60 * 60 * 24 * 7
config.aws_attributes = {
expires: 2.week.from_now.httpdate,
cache_control: 'max-age=604800'
}
config.aws_credentials = {
access_key_id: ENV['aws_access_key'],
secret_access_key: ENV['aws_secret'],
region: 'us-east-1'
}
config.remove_previously_stored_files_after_update = true
end
video_uploader.rb
class VideoUploader < CarrierWave::Uploader::Base
include CarrierWave::Video
storage :was
def store_dir
"uploads/videos/#{model.id}"
end
end
episode.rb
class Episode < ActiveRecord::Base
mount_uploader :video, VideoUploader
before_destroy :destroy_assets
def destroy_assets
self.video.remove! if self.video
self.save!
end
end
The AWS credentials I'm using are for an IAM user with the AmazonS3FullAccess policy, if that makes any difference here.
I would update your store_dir to be:
"uploads/videos/#{mounted_as}/#{model.id}"
So that carrierwave, while utilizing the mounting system, removes the file, it knows where to find it when it calls the full name method internally!
I want to upload multiple images to cloudinary by association of a carrierwave active record. How can I do this in a seeds file, using an array of remote urls?
Ive read countless articles how to use carrierwave/cloudinary helper tags to target an image upload within an html form, but nothing on doing this directly within code, any ideas?
That's not so hard to do at all.
So what I did:
# Gemfile
gem 'cloudinary'
gem 'carrierwave'
#config/cloudinary.yml
development:
cloud_name: "carrierwave-example"
api_key: "YOUR CLOUDINARY CREDENTIALS"
api_secret: "YOUR CLOUDINARY CREDENTIALS"
# in your uploader
class ImageUploader < CarrierWave::Uploader::Base
include Cloudinary::CarrierWave #include cloudinary lib
# storage :file - comment this out
# def store_dir - comment this too
# "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
# end
end
# You model, that uses uploader
class Picture < ActiveRecord::Base
mount_uploader :image, ImageUploader
end
After writting this simple staff you can create Pictures, that will store images in clodinary like this:
Picture.create(image: "/path/to/image")
or if you have remote links of images, U just itterate through them
["http://image1.jpg","http://image2.jpg","http://image3.jpg"].each do |link|
Picture.create(remote_image_url: link)
end
Just remember to use remote_#{your_column_name}_url if you have remote link