I need to POST a big string (>2mb) to my heroku app from the client using javascript.
Then I need to store the string in cloudfiles or s3.
What is the best way to do this, taking into account the limitations of Heroku?
The best way to deal with large file and heroku is to use javascript to post it directly to s3 (or your final destination), then use a callback to hook it up to your model.
The main reason is this -- heroku will timeout on any request taking longer than 30s, so if there is any risk that you uploading will take more than that, then you HAVE to bypass heroku (note: I have learned this from experience)
There a bunch of options to accomplish this depending on what you requirements are:
d2s3 -- Direct to S3
plupload -- flash uploader with progress bar, and lots of goodies, supports bulk uploading (this is what I currently use)
uploadify -- good alternative to plupload -- maybe simpler when not working bulk uploading
swfupload -- another option
Note: none of these are super simple to setup
I wish that there was a simpler alternative, it is amazing to me that something as fundamental as file uploading is this difficult.
Heroku's section on S3 has good advice about how to use the paperclip gem to manage these issues -- see http://devcenter.heroku.com/articles/s3
Related
Multiple question around the same issue, the way active storage returns file urls
For now with the default setup, the following (cloud or local), returns somehow the following :
_domain/_path/_superlong_hash/_original_filename._ext
Given paperclip or many other existing gems, the _path/_superlong_hash/_original_filename._ext part is in hand to be customised, could end up in a clean url for any files
Meaning by that :
is there a way to "proxy" the _path with something more custom ?
is there a way to avoid the _superlong_hash ?
is there a way to customize the filename on the fly (or on uploads) ?
To make it a one-liner, how one would customize the files urls ?
I've seen here and there people ending up creating custom controllers to serve file with decent urls, but let's admit this is a no go (IMHO)
I hope that ActiveStorage proves me wrong soon, but at the time of writing Rails 5.2, the straight answer seems to be that you have to go with your 'no go' option, hacking your own controllers together and heavily patching ActiveStorage to expose files.
For proxying see:
https://github.com/rails/rails/issues/31419
https://github.com/rails/rails/pull/30465
especially georgeclaghorn responses are interesting
For renaming file:
#user.avatar.blob.update(filename: 'NewFilename.jpg')
Manipulating the _superlong_hash / url
I have no good answer for this one. Although ActiveStorage makes it breathtakingly easy to upload (and somewhat easy to manipulate) files it takes Rails opinionated software philosophy to the edge, making it quite difficult to bypass it's obscurity by abstraction approach to url generation. ActiveStorage provides no built-in methods to do rudimentary things like permanent or direct links to files and variants once generated. File/image caching and nice urls seem therefore not to be possible to accomplish out of the box with ActiveStorage at this point in time.
I have a Ruby on Rails application where users can upload files. We run this application in production on home-rolled linux servers. (No Heroku or anything like that.) Using the carrierwave gem, they get stored on AWS S3. Very simple and straight-forward.
If someone wanted to view or modify those files that are stored on S3 without having to download them, open them in Word or Excel, and then reupload- how could that be done?
The only people who have seemed to implement something like this are Dropbox and Citrix ShareFile. They did it using Office Online.
I don't mind moving file storage off S3 to somewhere else (or even something home-rolled) to implement this feature- but it needs to be as reliable and robust as S3.
We've tried to something with WebDAV, but it is fragile and seems to be deprecated.
We would consider any solution, paid or free.
You can indeed use the WOPI framework for this. There are quite a lot of resources and code samples for this, of which the most useful might be This Github Repo. The framework allows you to programmatically integrate into the Word Online viewer/editor.
Hope this helps,
Jesse
I like to use heroku and paperclip for image uploads. Usually my users don't need to upload many images.
I'm now embarking on a website where many users will be trying to upload images at the same time. Unfortunately on heroku, it seems I need a separate dyno for every image upload, otherwise the site becomes unresponsive. Or am I missing something?
What is an optimal way to set up a rails application (not necessarily on Heroku) where the site can easily deal with (in a scalable way, ideally) multiple uploads at the same time?
This is a shortcoming with Heroku - it doesn't handle file uploads very efficiently. See this article that discusses this point, among others. The author suggests using the carrierwave_direct gem or the Cloudinary service. Neither concept will work well for Paperclip as it doesn't provide cloud storage support. You might want to move to CarrierWave for easier implementation of these concepts.
I've looked at the available options and it seems like everything is optimized for image uploading as display. I just need simple file upload and retrieval. Are there any good options?
Paperclip is a popular choice for uploading and sizing images, but you can upload any type of file with it (doc, zip, txt, pdf... anything). Highly recommended. https://github.com/thoughtbot/paperclip
I like carrierwave. It has built in support for s3, has no workaround for setting up apps on heeroku unlike paperclip.
I use Carrierwave for mine and have been happy with it. I am just uploading general files, not specifically images. It is easy to implement and has good advanced features if you need them later. It also integrates with Fog to make using remote storage sources (like s3 or rackspace cloud files) easy.
Carrierwave benefits:
With carrierwave, the attachment is a seperate model instead of an attribute on an existing model, which might make things cleaner to work with.
It comes with the ability to attach a file via url (user passes in a url to a file) instead of uploading with a form).
It comes with some sort of way to remember files across form validation failures, although I've never used this and I'm not sure how it's done... maybe with two forms and ajax?
It seems to have a more engaged and enthusiastic community around it, with more projects extending it.
For S3, they use fog instead of aws-s3, and fog has much more active development.
That said, paperclip is pretty great and is actively maintained, and might come with handier default image manipulation stuff, I'm not sure.
I need to be able to encrypt files before storing them on S3, and then decrypt them when accessing them. The files will be images, documents, PDF, etc.
I am using Carrierwave to handle the file upload and storage (this is with Ruby on Rails). I am storing them in Amazon S3.
Has anyone done this, or have any ideas how this would be achieved?
Thanks.
Amazon has now released functionality that lets you encrypt/decrypt files automatically in S3. The need to do this yourself is no longer there. Details are here http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?UsingEncryption.html
To handle the encryption, you should look into implementing a processor. If you are using any other processors, you may have to look at extending the Carrierwave gem and adding a processor ordering mechanism so you can be sure encryption happens last.
For the decryption, you can either override the existing accessor to make the decryption transparent, or add a new method that returns the decrypted file and use that in place of the accessor. The latter approach is probably more resilient to upstream changes.
I know this post is a few months old, but if you're still looking for answers, check out the carrierwave_securefile gem I wrote. It's still new and probably a bit buggy on other setups, but it uses Crypt19 for Blowfish encryption on files prior to upload.
http://github.com/dougc84/carrierwave_securefile