Copy images from one S3 bucket to another with specified path prefixes - ruby-on-rails

I have an S3 bucket setup already, it contains images of a photo gallery, uploaded from my Rails app using PaperClip. Images are accessible via some arbitrary URLs like: http://s3.amazonaws.com/oldbucket/images/files/000/001/920/original/40a6885fc09c8ed4e1e3745d7f7fb770.jpg?1415766995.
Kindly advice me the best option considering following requirements:
I have to copy those images to another S3 bucket in another AWS
account
I want to make the new image URLs according to specific
patterns, like: .../newbucket/{userid}/{galleryid}/{image-size}.jpg
I want to create multiple versions of each image, according to size
(original, thumbnail and icon)
Any options using Rails gem or software that would do above would be helpful.
Thanks

For this you need to add carrierwave gem for saving image from remote url. You can also do with paperclip.
First create a seed file without adding carrierwave uploader to your app. I am considering User as model and avatar as image.
User.all.each{|u| puts user.avatar.url}`
Now remove paperclip and add carrierwave This will give you list of all images. Now add it to seed file for model you want to add this images.
class Modelx
mount_uploader :avatar, AvatarUploader
end
So your seed file should have entries like.
Modelx.create([{:avatar_remote_url => image_url1}, {:avatar_remote_url => image_url2},.....])
You can set specific path and also create multiple dimension images using carrierwave.
REfrence url for carrierwave here.

Related

How to upload or use existing asset image using Carrierwave uploader?

User has to options:
choose one of three images from my assets folder
upload his own
I am wondering if I can set up CarrierWave uploader to handle both cases.
So I don't want to upload file for chosen one. So I tried to update column for that.
user.update_column(:avatar, 'assets/images/default_avatar_1.jpg')
But then when I am using user.avatar.url uploader looks for this file in public folder as specified.
Is possible to override this behaviour?

Rails s3_direct_upload how to disable it

In Rails, I am using s3_direct_upload gem for asset(doc file) upload. Right now if I try to upload an image file as a different asset, it is directly uploading to s3. I need to disable this option for image upload and it should be enable only for document upload.
s3_direct_upload is provides form helper methods to upload images to s3 directly. like this s3_uploader_form. It just reduces your jquery_file_upload and s3 configuration.
But you can still upload image to your local file system. Using simple form_tag. i.e. When you want to upload images to s3 use s3_uploader_form syntax and when you want to upload images to local file system then use simple form_tag or any other rails provided form syntax.
For uploading images using ajax use remotipart gem with simple form syntax.

How can I migrate CarrierWave files to a new storage mechanism?

I have a Ruby on Rails site with models using CarrierWave for file handling, currently using local storage. I want to start using cloud storage and I need to migrate existing local files to the cloud. I am wondering if anyone can point out a method for doing this?
Bonus points for using a model attribute that would allow me to do this row-by-row in the background without interrupting my site for extended downtime (in other words, some model rows would still have local storage while others used cloud storage).
My first instinct is to create a new uploader for each model that uses cloud storage, so I have two uploaders on each model, then transferring the files from one to the other, setting an attribute to indicate which file should be used until they are all transferred, then removing the old uploader. That seems a little excessive.
Minimal to Possibly Zero Donwtime Procedure
In my opinion, the easiest and fastest way to accomplish what you want with almost no downtime is this: (I will assume that you will use AWS cloud, but similar procedure is applicable to any cloud service)
Figure out and setup your assets bucket, bucket policies etc for making the assets publicly accessible.
Using s3cmd (command line tool for interacting with S3) or a GUI app, copy entire assets folder from file system to the appropriate folder in S3.
In your app, setup carrierwave and update your models/uploaders for :fog storage.
Do not restart your application yet. Instead bring up rails console and for your models, check that the new assets URL is correct and accessible as planned. For example, for a video model with picture asset, you can check this way:
Video.first.picture.url
This will give you a full cloud URL based on the updated settings. Copy the URL and paste in a browser to make sure that you can get to it fine.
If this works for at least one instance of each model that has assets, you are good to restart your application.
Upon restart, all your assets are being served from cloud, and you didn't need any migrations or multiple uploaders in your models.
(Based on comment by #Frederick Cheung): Using s3cmd (or something similar) rsync or sync the assets folder from the filesystem to S3 to account for assets that were uploaded between steps 2 and 5, if any.
PS: If you need help setting up carrierwave for cloud storage, let me know.
I'd try the following steps:
Change the storage in the uploaders to :fog or what ever you want to use
Write a migration like rails g migration MigrateFiles to let carrierwave get the current files, process them and upload them to the cloud.
If your model looks like this:
class Video
mount_uploader :attachment, VideoUploader
end
The migration would look like this:
#videos = Video.all
#videos.each do |video|
video.remote_attachment_url = video.attachment_url
video.save
end
If you execute this migration the following should happen:
Carrierwave downloads each image because you specified a remote url for the attachment(the current location, like http://test.com/images/1.jpg) and saves it to the cloud because you changed that in the uploader.
Edit:
Since San pointed out this will not work directly you should maybe create an extra column first, run a migration to copy the current attachment_urls from all the videos into that column, change the uploader after that and run the above migration using the copied urls in that new column. With another migration just delete the column again. Not that clean and easy but done in some minutes.
When we use Heroku, most of people suggest to use cloudinary. Free and simple setup.
My case is when we use cloudinary service and need move into aws S3 for some reasons.
This is what i did with the uploader:
class AvatarUploader < CarrierWave::Uploader::Base
def self.set_storage
if ENV['UPLOADER_SERVICE'] == 'aws'
:fog
else
nil
end
end
if ENV['UPLOADER_SERVICE'] == 'aws'
include CarrierWave::MiniMagick
else
include Cloudinary::CarrierWave
end
storage set_storage
end
also, setup the rake task:
task :migrate_cloudinary_to_aws do
profile_image_old_url = []
Profile.where("picture IS NOT NULL").each do |profile_image|
profile_image_old_url << profile_image
end
ENV['UPLOADER_SERVICE'] = 'aws'
load("#{Rails.root}/app/uploaders/avatar_uploader.rb")
Profile.where("picture IS NOT NULL OR cover IS NOT NULL").each do |profile_image|
old_profile_image = profile_image_old_url.detect { |image| image.id == profile_image.id }
profile_image.remote_picture_url = old_profile_image.picture.url
profile_image.save
end
end
The trick is how to change the uploader provider by env variable. Good luck!
I have migrated the Carrier wave files to Amazon s3 with s3cmd and it works.
Here are the steps to follow:
Change the storage kind of the uploader to fog.
Create a bucket on Amazon s3 if you already dont have one.
Install s3cmd on the remote server sudo apt-get install s3cmd
Configure s3cmd s3cmd --configure.
You would need to enter public and secret key here, provided by Amazon.
Sync the files by this command s3cmd sync /path_to_your_files ://bucket_name/
Set this flag --acl-public to upload the file as public and avoid permission issues.
Restart your server
Notes:
sync will not duplicate your records. It will first check if the file is present on remote server or not.

Where do you store uploaded user images

I am not yet using a service such as Amazon S3, so where in the file structure should I store uploaded user images? I want to avoid the public directory as the images are private.
Are you using a plugin to handle your uploads? Many of them allow you to specify a path to store files, if you want to avoid the public folder a reasonable suggestion would be "#{RAILS_ROOT}/uploads/images/"
It's very much a matter of personal taste though.
For example in a carrierwave uploader this will place items in an uploads folder below RAILS_ROOT which is not publicly accessible.
def store_dir
"#{RAILS_ROOT}/uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
end
https://github.com/carrierwaveuploader/carrierwave#changing-the-storage-directory

Uploading Images without using ImageMagick

I don't want to have to configure imagemagick but want to use a gem to be able to upload a picture from a form and put it into a database. Are there other options out there? I tried dragonfly and carrierwave but they require ImageMagick.
I often use Blitline which allows we to me to make any edits to the image (if needed) and then I have Blitline store it in an Amazon S3 bucket and place the image url in my database to save space in the database.

Resources