So, I'm using Paperclip and AWS-S3, which is awesome. And it works great. Just one problem, though: I need to upload really large files. As in over 50 Megabytes. And so, nginx dies. So apparently Paperclip stores things to disk before going to S3?
I found this really cool article, but it also seems to be going to disk first, and then doing everything else in the background.
Ideally, I'd be able to upload the file in the background... I have a small amount of experience doing this with PHP, but nothing with Rails as of yet. Could anyone point me in a general direction, even?
You can bypass the server entirely and upload directly to S3 which will prevent the timeout. The same thing happens on Heroku. If you are using Rails 3, please check out my sample projects:
Sample project using Rails 3, Flash and MooTools-based FancyUploader to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-FancyUploader
Sample project using Rails 3, Flash/Silverlight/GoogleGears/BrowserPlus and jQuery-based Plupload to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-Plupload
By the way, you can do post-processing with Paperclip using something like this blog post (that Nico wrote) describes:
http://www.railstoolkit.com/posts/fancyupload-amazon-s3-uploader-with-paperclip
Maybe you have to increase the timeout in the ngix configs?
You might be interested in my post here:
http://www.railstoolkit.com/posts/fancyupload-amazon-s3-uploader-with-paperclip
Its about uploading multiple files (with progress bars, simultaneously) directly to S3 without hitting the server.
I was having a similar problem but with using paperclip, passenger and apache.
Like nginx, apache has a Timeout directive in apache which I increased to solve my problem.
Also there's an interesting thing passenger does when uploading large files.
Anything over 8k is written to /tmp/passenger. and if apache doesn't have
permissions to write there you get 500 errors also.
Here's the article.
http://tinyw.in/fwVB
Related
Thanks!
I'm a complete beginner but I'd like to know even in a generic way so that my users could upload a file to another user's directory, the file gets saved on to a server and the receiving user could download it when the upload gets finished.
Since you're a complete beginner, I won't give you a lot of code; instead, I'll show you the overview of what you can do, and hopefully that will give you a much more refined approach to adopt:
Rails Uploads Using Paperclip
What makes Rails so powerful is the gem ecosystem - these are little plugins which give you functionality outside the scope of the Rails framework. You can get Gems for everything from file uploads to CSS frameworks - all for free
One of the most important & popular gems for Rails is Paperclip (and you have Carrierwave which also does the same job). These are what you use to process image uploads in Rails
Since you're a total beginner, you should really watch this Railscast about how to upload files with Rails using Paperclip:
There are also a lot of other tutorials on Paperclip:
Heroku's Paperclip Tutorial
Simple Paperclip tutorial
Another Paperclup tutorial
I would write some more updates about Paperclip & JQuery, but I think you should get your head around the upload process first
Like it says on the tin, I'm trying to upload an image with my Ember.js app to a Rails backend that's using Paperclip to manage file uploads. I had a look around and couldn't see any simple way to do this, does anyone know of a good solution here?
I faced similar recently, and it turns out that there are lots of complications with file uploading - does the device support it, do you want to be able to style the input that triggers the upload, etc.
We opted for Jquery File Upload: https://github.com/blueimp/jQuery-File-Upload
The approach I took was to upload directly to S3 from the browser, and then set the token that S3 returns as a property on a model, then save that to the server. Then on the server, you set off a background job to pull in that file from S3 and put it where it should be.
I wrote a fairly simple ember.js file upload example a few months back that shows how you can write a custom view + a custom adapter that allows you to post a multipart form back to the server. The example I did is built for python / django but the concepts should apply
https://github.com/toranb/ember-file-upload
I recently upgraded this to RC1 (like 5 minutes ago) and it appears to still work :D
There is now an Ember Uploader plugin for Ember. I'm just in the process of integrating it right now.
I have a couple kinks I'm ironing out, but it seems pretty legit. Probably less configuration than using the jquery file upload.
Normally I'd use carrierwave, but they do not officially support jruby, and I've been running in to bugs, possibly related. (https://github.com/jnicklas/carrierwave/issues/620, Image corruption on upload to s3, production only. (carrierwave, engineyard))
Have others had success here?
I'm considering trying out paperclip, but it looks like that may not be perfect either-- https://github.com/thoughtbot/paperclip/issues/100
Try dragonfly it's a really great library to manage you file attachment.
check out this guide, it's cake
http://webtempest.com/how-to-allow-image-uploads-in-rails-on-heroku/
It uses paperclip and amazon web s3 gems
I'm having issues with CKEditor. I can upload and insert pictures without issues, but when I try to do the same with files, the link to my file is set to something like javascript:void(0)/*130*/, with the number changing. This is happening on FF/Safari/Chrome.
My app runs on rails 3.1.3, using MongoDB/Mongoid as database/ODM, with paperclip for handling attachments and using S3 for hosting assets. When I explore my bucket I can see that the files are uploaded correctly, so the problem (probably) come from somewhere else. I'm using this gem, and both the rc2 & the master branch doesn't fix that.
Thanks for your time.
Well it's been a while. I solved it by forking the gem (cf ksol/ckeditor), but the diff is too obfuscated to remember what was wrong. Hopefully the original gem is working now.
One client project will be online two months later. One of the requirements changed is to support large files (10 to 15MB per RAW camera file, expected 1000 to 5000 files download per day) download worldwide for their customers. The process will be:
there is upload screen via paperclip to the rails local public folder
a hourly task to upload to web storage (S3?)
update the download url from paperclip url to the web url
Questions:
is there a gem/plug-in for this
purpose?
if no, any gem/plug-in
for S3 to recommend?
Questions about the storage provider:
is S3 recommended?
or other service to recommend?
The baseline is: the client's web server does not and will not have the bandwidth to handle the downloads.
Thanks
I don't think there is anything that will do all of this out of the box for you. Paperclip will push files sychronousy to S3 on upload, so you will need to make this ansychronous yourself.
S3 is rock-solid, I have used it in production on a number of projects. Totally recommended.
You can upload files directly to S3 which may help by reducing the double handling of the file (no longer need to upload to your app before pushing to Amazon):
http://developer.amazonwebservices.com/connect/entry.jspa?categoryID=139&externalID=1434
The aws-s3 and delayed_job gems are probably what you want.
gem install aws-s3
S3 is popular and widely used as far as I am aware.
If you end up going the route of uploading directly to S3 which offloads the work from your Rails server and makes it asynchronous, please check out my sample projects:
Sample project using Rails 3, Flash and MooTools-based FancyUploader to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-FancyUploader
Sample project using Rails 3, Flash/Silverlight/GoogleGears/BrowserPlus and jQuery-based Plupload to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-Plupload