Using carrierwave to upload one image to multiple storage location - ruby-on-rails

I would like to be able to upload one image into two different locations: one location would be on the local filesystem (of the server) and the other would be Amazon S3 (the Amazon S3 location would be optional).
My current environment is Rails 3.2.8, Ruby 1.9.3, with Carrierwave used for uploading the file.
I've had some success using the following method:
Model
class Image < ActiveRecord:Base
attt_accessor :remote
before_save :configure_for_remote
mount_uploader :image, ImageUploader #stores images locally
mount_uploader :image_remote, ImageRemoteUploader #store images on S3
def configure_for_remote
if self.remote=="1"
self.image_remote = self.image.dup
end
end
end
Relevant view form fields (simple form syntax)
<p><%= f.input :image, as: :file %></p>
<p><%= f.input :remote, as: :boolean %></p>
The user checks the "remote" checkbox in the form and chooses the image to upload. The before_save callback stores a duplicate of image into image_remote, the file is processed by their respective uploaders, and I have my desired result.
However, I'm starting to run into problems when I want to update that field. For example, if the user chooses to first upload the file locally and not to S3 (does not check the remote checkbox), then later comes back to the form and checks the remote checkbox. In this case, the before_save callback does not get run because no real active record column has been changed (only the remote flag). I've tried to use before_validation, but this fails to work (the image_remote uploader stores the proper filename in the image_remote column, but the image does not get uploaded to S3). Obviously something is changing between the before_validation and the before_save (image attribute is being converted to and uploader?) but I can't seem to figure out why this doesn't work.
With all this being said, I think my approach with using dup is a bit of a hack, and I'm hoping someone can advise me in a more elegant way of reaching my goal.
Thanks for your help.

I was to solve this, although I'm still not sure if it's the most elegant solution.
First off, I mentioned in my question that when I registered config_for_remote_upload with the before_validation callback, the file was not uploaded to S3, but the image_remote column was populated. Upon further inspection, the situation is even worse. When initializing the image_remote uploader within the before_validation callback, all files were deleted on the S3 storage bucket! I replicated this a couple times. I only tested when the store_dir was set to nil in the uploaded, thus putting the files at the root of the bucket.
Initializing the image_remote column in during the before_save callback does not have this problem. In order force the record to save (it wouldn't save, because only a non db column attribute was being changed) I added a before_validation that changed the update_at field of the record.
before_validation: :change_record_updated_at
...
def change_record_updated_at
self.update_at=Time.current
end
I also moved away from using dup, not because it didn't work, but rather because I didn't know why it worked. Instead I created a StringIO object for the file and assigned that to the image_remote column.
def config_for_remote_upload
if self.remote.to_i==1
#self.image_remote = self.image.dup
#this will open the file as binary
img_binary = File.open(self.image.file.path){ |i| i.read }
img_encoded = Base64.encode64(img_binary)
io = FilelessIO.new(Base64.decode64(img_encoded))
io.original_filename = self.image.file.original_filename
self.image_remote = io
elsif self.remote.to_i==0
#delete remote image and clear field
self.remove_image_remote = true
end
end
See here for further info on FilelessIO (StringIO with original_filename).
With this configuration, the file can be uploaded to the second storage location (S3 in my case) after the initial upload.
Hope this helps someone else out.

Related

When using activestorage in Rails 6, how do I retain a file when redisplaying a form WITHOUT uploading?

What I'd like to know is very simple:
How to retain uploaded files on form resubmission in Rails6 with ActiveStorage?
I have checked the below as a similar question.
When using activestorage in Rails 6, how do I retain a file when redisplaying a form?
The summary of suggested solution in it is like this:
Active Storage store attachments after the record is saved rather than
immediately. So, if you want to persist assigned file after validation
error, you must upload and save the file.
For example,
def update
...
# `obj` is your model that using `has_one_attached`.
if(obj.update)
redirect_to ...
else
obj.attachment_changes.each do |_, change|
if change.is_a?(ActiveStorage::Attached::Changes::CreateOne)
change.upload
change.blob.save
end
end
...
end
end
https://medium.com/#TETRA2000/active-storage-how-to-retain-uploaded-files-on-form-resubmission-91b57be78d53
or, use direct_upload:
= f.file_field :doc, direct_upload: true
= f.hidden_field :doc, value: f.object.doc.signed_id if f.object.doc.attached?
By using these solutions, yeah, I managed to retain a file when redisplaying a form.
However, these are against this pr intention. pr says
It’s of little use to identify an invalid file after it’s already been
shipped off to storage:
you might use a size validation to limit the cost that a single file
can add to your AWS bill, but if the file is stored before
validations run, you incur its cost regardless.
So, I don't want to upload file to persist it after validation.
How can I retain the file without uploading file and saving blob?
I cannot use CarrierWave.

Carrierwave, creating a duplicate attachment when duplicating its containing model

I would like to duplicate a model. The original model contains an attachment through Carrierwave. Ideally, a new attachment would be created, that is a copy of the original image, for the new model object.
I have looked through the Carrierwave documentation, and googled this problem, but have not found a solution that creates a new duplicate of the original image. Is this reasonable? Possible?
I don't believe Carrierwave has this option. However, you can make use of the *_remote_url= method to set the new model's picture to be a duplicate of the first.
Here's a brief example
Say I have a model which has_one :photo attached with carrierwave. I can duplicate, the model, set the photo to the previous one and save it. Example:
first_model = User.first
duplicate_model = first_model.dup #(where the dup code duplicates everything else you need)
duplicate_model.remote_photo_url = first_model.photo_url
duplicate_model.save
This would then "copy" the photo from the first object into your second as a new carrierwave attachment.
While copy_carrierwave_file is a neat gem it is not nescessary as long as you use local storage.
carrierwave can use local files as source of attachments and you can use this to duplicate the attachment:
first_user = User.first
duplicate_user = first_user.dup
duplicate_user.photo = File.open(first_user.photo.file.file) if first_user.photo.present?
duplicate_user.save
This is more efficient than routing the image twice through your web server.
Try this gem https://github.com/equivalent/copy_carrierwave_file , it handles both local storage and Fog storage
original_resource = User.last
new_resource = User.new
CopyCarrierwaveFile::CopyFileService.new(original_resource, new_resource, :avatar).set_file
new_resource.save
nev_resource.avatar.url # https://...image.jpg
For me with CarrierWave 0.10 this works just fine:
user = User.first
dup_user = user.dup
dup_user.photo = user.photo
dup_user.save
Although I'm not sure how this works out when using cloud storage like S3
Extracted from the Carrierwave wiki page:
YourModel.find_each do |ym|
begin
ym.process_your_uploader_upload = true # only if you use carrierwave_backgrounder
ym.your_uploader.cache_stored_file!
ym.your_uploader.retrieve_from_cache!(ym.your_uploader.cache_name)
ym.your_uploader.recreate_versions!(:version1, :version2)
ym.save!
rescue => e
puts "ERROR: YourModel: #{ym.id} -> #{e.to_s}"
end
end
I needed to fully duplicate the whole version set on S3, while some of the versions were cropped.
Unfortunately, remote_#{column}_url= method was of no help, because by the time the versions are recreated, there are no crop params on the model:
I used RailsCasts approach using attr_accessor to crop the avatar, and those params weren't stored in the DB.
After some research and a lot of failures, I found this answer and noticed that copy_to method.
It turned out that both SanitizedFile and Storage::Fog have it, so it's possible to use it for local and S3 files. I didn't however investigate how it literally works and decided to let Carrierwave a chance to take care of it.
class AvatarUploader
…
def duplicate_to(target)
return unless file.present? && target.logo.file.present?
versions.keys.each do |version|
public_send(version).file.copy_to(target.avatar.public_send(version).path)
end
end
end
That's all it takes to fully duplicate the images, no matter if they are cropped or not.
There's a catch, however: you should only call duplicate_to after the model is already saved with other avatar, or the target path would be nil. Thus, one useless round of processing takes place for the new record.
new_user.assign_attributes(old_user.slice(:avatar, :avatar_alignment))
# Won't work!
old_user.avatar.duplicate_to(new_user) # => as the `new_user` hasn't persisted yet, its avatar files are Tempfiles
new_user.save # => will recreate the versions from the original image, losing the cropped versions!
# But this works
new_user.save # => the avatar will be stored as a set of versions created from the original (useless processing)
old_user.avatar.duplicate_to(new_user) # => the avatar files will be rewritten by the copies of old_user files
I think it's a good idea to store the crop params somewhere in the DB in a JSON-like object for such cases (and to be protected from losing cropping data when you have to recreate_versions!), but if that's not an option, this solution might be what you seek.
As this thread is the first G-link when searching for carrierwave duplicate, I decided to post this answer exactly here.
Carrierwave 1.3.2 with fog-aws 1.2.0.
Hope this helps someone or the future me!
This worked for me:
user = User.first
dup_user = user.dup
dup_user.photo = user.photo
dup_user.save
Reference: https://codeutility.org/ruby-on-rails-carrierwave-creating-a-duplicate-attachment-when-duplicating-its-containing-model-stack-overflow/

Carrierwave Rails 3 S3, save the file size to the database

Using Carrierwave with Rails 3.2.6. All fine, except I need to sort a table where some attachments are displayed by file size. I'm using S3 for storage with fog.
Let's say I have a Carrierwave showing like this:
<%= #project.attachment %>
I am able to show the size of the file by using '.size' after the field name:
<%= #project.attachment.size %>
shows the file size in bytes, but as I need to use an order clause when getting the records from the database, I cannot sort on this.
Is there any way to write the file size to a particular column in the database after it has been uploaded so I can sort on this??
many thanks
this worked for me
before_save :update_project_attributes
private
def update_project_attributes
if project.present? && project_changed?
self.file_size = project.file.size
end
end
You should add a virtual attribute to the model and define a custom getter method that returns the file size. You can then sort with respect to this virtual attribute as you usually would. Let me know if you need more details and I will try to provide them!
Ok,
got this to work with before_save
before_save :set_size
def set_size
self.size = self.upload.size
end
where upload is the mounted field and size is a new db column to store the size.

Rails: Preventing Duplicate Photo Uploads with Paperclip?

Is there anyway to throw a validation error if a user tries to upload the same photo twice to a Rails app using Paperclip? Paperclip doesn't seem to offer this functionality...
I'm using Rails 2.3.5 and Paperclip (obviously).
SOLUTION: (or one of them, at least)
Using Beerlington's suggestion, I decided to go with an MD5 Checksum comparison:
class Photo < ActiveRecord::Base
#...
has_attached_file :image #, ...
before_validation_on_create :generate_md5_checksum
validate :unique_photo
#...
def generate_md5_checksum
self.md5_checksum = Digest::MD5.hexdigest(image.to_file.read)
end
def unique_photo
photo_digest = self.md5_checksum
errors.add_to_base "You have already uploaded that file!" unless User.find(self.user_id).photos.find_by_md5_checksum(photo_digest).nil?
end
# ...
end
Then I just added a column to my photos table called md5_checksum, and voila! Now my app throws a validation error if you try to upload the same photo!
No idea how efficient/inefficient this is, so refactoring's welcome!
Thanks!
What about doing an MD5 on the image file? If it is the exact same file, the MD5 hash will be the same for both images.
For anyone else trying to do this. Paperclip now has md5 hashing built in. If you have a [attachment]_fingerprint in your model, paperclip will populate this with the MD5.
Since I already had a column named hash_value, I made a 'virtual' attribute called fingerprint
#Virtual attribute to have paperclip generate the md5
def picture_fingerprint
self.hash_value
end
def picture_fingerprint=(md5Hash)
self.hash_value=md5Hash
end
And, with rails3, using sexy_validations, I was able to simply add this to the top my my model to ensure that the hash_value is unique before it saves the model:
validates :hash_value, :uniqueness => { :message => "Image has already been uploaded." }
You might run into a problem when your images have amended EXIF metadata. This happened to me, and I had to extract pixel values and calculate MD5s out of them, to ignore changes made by Wordpress etc. You can read about it on our blog: http://www.amberbit.com/blog/2013/12/20/similar-images-detection-in-ruby-with-phash/ but essentially you want to get the pixel data out of image with some tool (like RMagick), concatinate it to string, and calculate MD5 out of that.
As Stephen indicated, your biggest issue is how to determine if a file is a duplicate, and there is no clear answer for this.
If these are photos taken with a digital camera, you would want to compare the EXIF data. If the EXIF data matches then the photo is most likely a duplicate. If it is a duplicate then you can inform the user of this. You'll have to accept the upload initially though so that you examine the EXIF data.
I should mention that EXIFR is a nice ruby gem for examining the EXIF data.

Rails non-image file upload to DB without using server-side temp files?

I'm looking into the feasibility of adding a function to my Rails-based intranet site that allows users to upload files.
Two purposes:
My users are widely distributed geographically and linking to documents on the shared network storage doesn't always work (different addresses, DNS entries and stuff outside my control or interest) so I'm thinking about providing a database-oriented alternative.
We have a number of files from which we parse data at the client end. I'd rather like to be able to push that up to the server.
I've looked at attachment_fu, Paperclip and another one (forgotten the name!) all of which seem very image-oriented, although attachment_fu at least can work without a image processing library present, thank goodness.
The big problem is that my server does not permit my application to write files locally, and these plugins all seem to want to create a Tempfile.
The questions (finally!)
Is there a reasonable way to upload binary data and process it in memory and/or store it as a BLOB without any server-side file saves?
Or should I give up on the file distribution idea and give the users a second-best option of copy-and-paste text boxes where possible?
(Closest I could find on SO was this which doesn't really help)
You could read the data from the params object, and write it straight to your model.
For example, you could have a form like this.
<% form_for :upload, :url => {:action=>:upload}, :html=>{:multipart=>true} do |f| %>
<%= f.file_field :file %>
<%= f.submit 'Upload' %>
<% end %>
Then you can easily get the original filename and the binary data.
class TestController < ApplicationController
def upload
file_param = params[:upload][:file]
filename = file_param.original_filename
filedata = file_param.read
#data = UploadedFile.create(:name => filename, :data => filedata)
render :text => "created #{#data.id}"
end
end
Of course your model needs to have the proper columns.
class CreateUploadedFiles < ActiveRecord::Migration
def self.up
create_table :uploaded_files do |t|
t.string :name
t.binary :data
t.timestamps
end
end
def self.down
drop_table :uploaded_files
end
end
Hope this helps!
The big problem is that my server does not permit my application to write files locally, and these plugins all seem to want to create a Tempfile.
Yes it does, or you wouldn't be able to upload the files at all.
Rails itself creates tempfiles if the uploaded file is larger than 15k or so.
<%= f.file_field :file %>
....
file_param = params[:upload][:file]
As soon as you upload something bigger than 15k, params[:upload][:file] is going to be an ActionController::UploadedTempFile.
What's the difference? Rails is likely writing it's tempfiles to the global temp directory (which everyone can write to) but the plugins are probably trying to write to RAILS_ROOT/tmp, which your server disallows. The good news is you can just configure those things to use a different temp dir so they can write their tempfiles, and it should all work.
For example, attachment_fu's default temp path is under rails root.. You should be able to change it like this:
Technoweenie::AttachmentFu.tempfile_path = Dir::tmpdir
**PS: pulling the file data straight out of the params and putting it into the database may still be the best way to go. I personally dislike attachment_fu and it's ilk, as they try to do too many things, but either way, it's very useful to know about how the whole uploaded file/temp file thing works in rails :-)
This HowTo for Rails includes a section (near the end of the page) on how to upload directly to the database. That section is sort of messed up, but the gist of it is that you just read the uploaded file contents into your BLOB field on your ActiveRecord object and save as normal. Since I don't know how you use the file inside your application, I can't really give any advice on how to use it from the database, though there is also a section on downloading from the DB in the HowTo.
It may be easier just to see if you can get permission to write to a single directory, perhaps inside your web app folder, on the server.
So this code in my controller:
def upload
file_content = params[:upload][:file]
render :text => [
[:original_path, :content_type, :local_path, :path, :original_filename].collect {|m| file_content.send(m)},
file_content.class,
file_content.size.to_s].flatten.join("<br/>")
end
gives this for a smaller file:
b_wib.xls
application/vnd.ms-excel
b_wib.xls
ActionController::UploadedStringIO
13824
and this for a larger one:
a_wib.xls
application/vnd.ms-excel
/tmp/CGI.10029.1
/tmp/CGI.10029.1
a_wib.xls
ActionController::UploadedTempfile
27648
...which is exactly as Orion described.
For anyone else reading this while just saving the File/IO in params to the database is a nice solution (why complicate matters) Paperclip, and I would suspect attachment_fu, are not image specific. Since uploading images and resizing are very common Paperclip comes bundled with a processor to resize images, but it is not enabled by default and you can easily add your own processors.

Resources