I have several paperclip attachments that live in one monolithic S3 bucket named foo
I want to split them up into several smaller, more logical buckets (foo1, foo2, etc...) based on some internal criteria.
I know which attachments need to move to which new buckets, and I've created the new buckets on S3 already.
Is there a good way to physically move the paperclip attachment to the new buckets programmatically using a rake task?
I can think of two options -
Loop through all attachments and download them temporarily to the filesytem, then re-upload them to the new bucket. This might be difficult on Heroku since there's no real local filesystem
Manually move the files on S3 while preserving the same folder structure. This is pretty laborious for hundreds of attachments though.
Thanks!
Related
Using ActiveStorage in Rails, variants are added "on demand" as you create them. They are now persisted in some storage (disk, S3, google cloud, etc).
If you realize some variants are un-needed after all and change your code to not invoke them... I think they'll still be sitting persisted in storage.
How does one clean these up, so they're not taking up storage space? I can't find any ActiveStorage API to remove variants.
You can delete a file from ActiveStorage services (disk, s3 etc.) with its key . And a variant's key is identified by its blob and transformation. Therefore you can delete a specific variant like this:
avatar = user.avatar
variant = avatar.variant(resize: '100x100')
avatar.service.delete(variant.key)
If the cost of generating all variants again is acceptable or if you need to invalidate most variants, then you can simply delete the variants folder. Rails will generate the variants again when needed.
This question already has answers here:
Storing Images in DB - Yea or Nay?
(56 answers)
Closed 10 years ago.
in my website, user can upload a profile image.
I wanted to know what is the best way to store those images.
my thought is simply dedicated directory. the image name will be the user_id.
is that a good solution, or there's a smarter one?
You have two options, store the images or use an external source (gravatar).
If you're going to store the images, do you want these images to be publically available or are they private? If they are publically available, then you can store them in your public folder.
You can use something like carrierwave to handle the uploading, versioning and storing of the images.
For public stuff, I'll store the file in the public directory under the uploader/model name/field name/id location. This is more for organizational purposes on my part.
Check out http://railscasts.com/episodes/253-carrierwave-file-uploads for a good tutorial.
For private images, I'll set the store directory to something outside of the public folder and will create a download action within the controller with the file. This way, the user cannot download the file unless it goes through the controller action. With authorization (cancan) I can allow or disallow a user to access the download action for that particular file (hence making it somewhat secure). If you are going to be using a production server like apache or nginx, make sure that you set the appropriate handlers for sending the file (ie x_sendfile).
Its very common to store images in a directory for small applications. However there are a few of things to take into consideration here:
Do you have anticipate a lot of users? If you have a million users, storing everyone's photo in your directory will take up a lot of memory when running your application
Are you deploying on Heroku? Many RoR apps are, and if you deploy on Heroku it will destroy any files you store locally when your app is moved to a different dyno (and you generally have no way of predicting when this will happen). You can read about the Ephemeral filesystem here https://devcenter.heroku.com/articles/dynos#isolation-and-security
In general I would advise against storing all your images locally because rewriting the code as you scale will become painful. I recommend you upload to an Amazon S3 Bucket and download the images as you need (and cache them for when your user is logged in). Its helpful becasue you might have to deal with image processing (for example resizing the images that are uploaded, creating thumbnail versions of the uploaded images) and its easier to do this when you have background processes that have persistent access to these files. I've used the 'aws' gem and S3 libraries for this, and its really easy to use, you can read more about it here: http://amazon.rubyforge.org/
However, if you intend for this to be a small app and are not deploying on Heroku, just saving it to a local directory is a lot easier and pain-free
I am working on a Rails web application, running on a Heroku stack, that handles looking after some documents that are attached to a Rails database object. i.e. suppose we have an object called product_i of class/table Product/products, and product_i_prospectus.pdf is the associated product prospectus, where each product has a single prospectus.
Since I am working on Heroku, and thus do not have root access, I plan to use Amazon S3 to store the static resource associated with product_i. So far, so good.
Now suppose that product_i_attributes.txt is also a file I want to upload, and indeed I want to actually fill out information in the product_i object (i.e. the row in the table corresponding to product_i), based on information in the file product_i_attributes.txt.
In a sentence: I want to create, or alter, database objects, based on the content of static text files uploaded to my S3 bucket.
I don't actually have to be able to access them once they are in the bucket strictly speaking, I just need to create some stuff out of a text file.
I have done something similar with csv files. I would not try to process the file directly at upload as it can be resource intensive.
My solution was to upload the file to s3 and then call a background job method(delayed_job, resque, etc.) that processed the csv after upload. You could then call a delete after the job processed to remove the file from s3 if you no longer needed it after processing.
For Heroku this will require that you add a worker (if you don't already have one) to process the background jobs that will process the text files.
Take a look at the aws-sdk-for-ruby gem. This will allow you to access your S3 bucket.
I'm working on a Rails project that currently receives uploaded files using attachment_fu and stores files in the database. I'd like to move them to use the filesystem. The problem is that there are currently several thousand uploaded files in the database, and we need to migrate them out. I can't seem to find anything to help with this; it seems the only migration anyone is posting tips for is filesystem -> S3. How would I go about migrating my files out of the database?
If you are ultimately attempting to serve these static files via S3/CloudFront to reduce load on your web/app servers one thing I may suggest is utilizing the new Custom Origin functionality of CloudFront which would allow you to keep your source files where they are. After being set up the process would basically be:
Your app tells the browser to retrieve the file from http://your-coudfront-host/path/to/file
The browser requests the file
If CloudFront has that file it returns it
If CloudFront does not have the file it retrieves it from your application and caches it for future requests (I believe up to 24 hours.)
This is what I am doing for product images that are dynamically generated on the fly in an application I am currently writing.
The upside of this is that you do not have to get into the overhead of constantly synchronizing data to S3 and if you decide to remove the whole setup you can still service your asset directly like nothing happened.
Im desiging an app which allows users to upload images (max 500k per image, roughly 20 images) from their hard drive to the site so as to be able to make some custom boardgames (e.g. snakes and ladders) in pdf formate. These will be created with prawn instantly and then made available for instant download.
Neither the images uploaded nor the pdfs created need to be saved on my apps side permanently. The moment the user downloads the pdf they are no longer needed.
Heroku doesn't support saving files to the system (it does allow to the tmp directory but says you shouldnt rely on it striking it out for me). I'm wondering what tools / services I should be looking into to get round this. Ive looked into paperclip, I'm wondering if this is right for this type of job.
Paperclip is on the right track, but the key insight is you need to use the S3 storage backend (Paperclip uses the FS by default which as you've noticed is no good on Heroku). It's pretty handy; instead of flushing writes out to the file system, it uses the AWS::S3 gem to upload them to S3. You can read more about it in the rdoc here: http://github.com/thoughtbot/paperclip/blob/master/lib/paperclip/storage/s3.rb
Here's how the flow would work:
I'd let your users upload their multiple source images. Here's an article on allowing multiple attachments to one model with paperclip: http://www.cordinc.com/blog/2009/04/multiple-attachments-with-vali.html.
Then when you're ready to generate the PDF (probably in a background job, right?), what you do is download all the source images to somewhere in tmp/ (make sure the directory is based on your model id or something so if two people do this at once, the files don't get stepped on). Once you've got all the images downloaded, you can generate your PDF. I know this is using the file system, but as long as you do all your filesystem interactions in one request or job cycle, it will work, your files will still be there. I use this method in a couple production web apps. You can't count on tmp/ being there between requests, but within one it's reliably there.
Storing your generated PDF on S3 with paperclip makes sense too, since then you can just hand your users the S3 URL. If you want you can make something to clear the files off every so often if you don't want to pay the S3 costs, but they should be trivial.
Paperclip sounds like an ideal candidate. It will save images in RAILS_ROOT/public/system/, which is both persistent and private (shouldn't be able to be enumerated on shared hosting).
You can configure it to produce thumbnails of your images if you wish.
And it can remove the images it manages when the associated model is destroyed - after your user downloads their PDF, and you delete the record from the database.
Prawn might not be appropriate, depending on the complexity of the PDFs you need to generate. If you have $$$, go for PrinceXML and the princely gem. I've had some success with wkhtmltopdf, which generates PDFs from a Webkit render of HTML/CSS - but it doesn't support any of the advanced page manipulation stuff that Prince does.