I need to be able to encrypt files before storing them on S3, and then decrypt them when accessing them. The files will be images, documents, PDF, etc.
I am using Carrierwave to handle the file upload and storage (this is with Ruby on Rails). I am storing them in Amazon S3.
Has anyone done this, or have any ideas how this would be achieved?
Thanks.
Amazon has now released functionality that lets you encrypt/decrypt files automatically in S3. The need to do this yourself is no longer there. Details are here http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?UsingEncryption.html
To handle the encryption, you should look into implementing a processor. If you are using any other processors, you may have to look at extending the Carrierwave gem and adding a processor ordering mechanism so you can be sure encryption happens last.
For the decryption, you can either override the existing accessor to make the decryption transparent, or add a new method that returns the decrypted file and use that in place of the accessor. The latter approach is probably more resilient to upstream changes.
I know this post is a few months old, but if you're still looking for answers, check out the carrierwave_securefile gem I wrote. It's still new and probably a bit buggy on other setups, but it uses Crypt19 for Blowfish encryption on files prior to upload.
http://github.com/dougc84/carrierwave_securefile
Related
Active Storage stores uploads in a file structure like so:
This is great if the rails app is the only software that needs to use these files.
But what if the app simply serves as a means to upload the images to S3 so that some other (completely separate) service can consume them?
The problem being, other developers wouldn't be able to make any sense of the directory and files, since they're labelled in a way rails can understand, but which a human cannot (e.g. what does folder named "O2" mean?).
Is there some way to ensure uploads are stored in a human-friendly way? e.g. each folder could be a user_id, with their assets inside? (or similar)
As far as I remember, you would have to implement an own service that somehow replace the key used to something else (e.g. S3Service), or patch ActiveStorage to create the key itself in a different way. I am not sure that this would suffice though, as the key is most likely used for other critical functionality outside of the Service.
Alternatively you might poke in ActiveStorage::Blob to fiddle with the key. That said, ActiveStorage does not support what you'd like out of the box and you would have to take the risk of messing with its internals.
A solution to the problem (but not answer to you question) might be implementing an API as outlined in the comments to your question.
I have a pre-existing CDN with images for Spree::Products. I already imported the products but the pictures are yet to be somehow taken care of.
Sure, I can import all 400000 images, but that would take really, really long and caused duplication.
Is there any way to replace paperclip with some simple mechanism that would just serve existing pictures? I want all the pictures to be transferred through the Rails app, because otherwise it would cause unsecure content warnings on a SSL site...
So I'm thinking this: Can I just override the paperclip gem and serve existing images instead? Or can I replace paperclip entirely? Or can I do something about this at all?
Additional info: I have a database of all urls of all images with their PN/SKU/ID relations.
There is no simple mechanism to replace Paperclip. You should just be able to code your own solution, override any calls to paperclips methods to call your method pretty easily enough though. It just may be a bit time consuming, and possibly more time consuming than it would be for you to just deal with importing the images instead.
I've looked at the available options and it seems like everything is optimized for image uploading as display. I just need simple file upload and retrieval. Are there any good options?
Paperclip is a popular choice for uploading and sizing images, but you can upload any type of file with it (doc, zip, txt, pdf... anything). Highly recommended. https://github.com/thoughtbot/paperclip
I like carrierwave. It has built in support for s3, has no workaround for setting up apps on heeroku unlike paperclip.
I use Carrierwave for mine and have been happy with it. I am just uploading general files, not specifically images. It is easy to implement and has good advanced features if you need them later. It also integrates with Fog to make using remote storage sources (like s3 or rackspace cloud files) easy.
Carrierwave benefits:
With carrierwave, the attachment is a seperate model instead of an attribute on an existing model, which might make things cleaner to work with.
It comes with the ability to attach a file via url (user passes in a url to a file) instead of uploading with a form).
It comes with some sort of way to remember files across form validation failures, although I've never used this and I'm not sure how it's done... maybe with two forms and ajax?
It seems to have a more engaged and enthusiastic community around it, with more projects extending it.
For S3, they use fog instead of aws-s3, and fog has much more active development.
That said, paperclip is pretty great and is actively maintained, and might come with handier default image manipulation stuff, I'm not sure.
I need to POST a big string (>2mb) to my heroku app from the client using javascript.
Then I need to store the string in cloudfiles or s3.
What is the best way to do this, taking into account the limitations of Heroku?
The best way to deal with large file and heroku is to use javascript to post it directly to s3 (or your final destination), then use a callback to hook it up to your model.
The main reason is this -- heroku will timeout on any request taking longer than 30s, so if there is any risk that you uploading will take more than that, then you HAVE to bypass heroku (note: I have learned this from experience)
There a bunch of options to accomplish this depending on what you requirements are:
d2s3 -- Direct to S3
plupload -- flash uploader with progress bar, and lots of goodies, supports bulk uploading (this is what I currently use)
uploadify -- good alternative to plupload -- maybe simpler when not working bulk uploading
swfupload -- another option
Note: none of these are super simple to setup
I wish that there was a simpler alternative, it is amazing to me that something as fundamental as file uploading is this difficult.
Heroku's section on S3 has good advice about how to use the paperclip gem to manage these issues -- see http://devcenter.heroku.com/articles/s3
I need to add the ability to upload and store any kind of file, PDF, XLS, DOC, etc. What is the best way to do this in a ruby on rails application?
I think this is exactly what you're looking for.
Upload files.
I'd recommend you to use paperclip or carrierwave both are really good libs and work out of the box in most cases.
you can also look at attachment_fu rails.
I've worked with two of the big players when it comes to file uploads. carrierwave and paperclip.
They provide a good solution for a common task with support for different storage alternatives. Both support filesystem and S3. Carrierwave also supports Rackspace Cloud Files and MongoDB’s GridFS.
I would recommend carrierwave because of one aspect where they are different to use. It uses a separate upload class that you mount on your model. This separates your code related to the file upload from the model code. I find this approach cleaner and easier to test.