(Rails 3.2 on Heroku)
For handling image uploads in Rails I switched from Paperclip to Dragonfly because I like to be able to generate thumbnails dynamically, when they are requested for the first time.
However, it seems that uploading of attached files to S3 (using S3DataStore) is much slower than with Paperclip
This is how an upload looks in a NewRelic transaction trace:
Anyone have experience in speeding this up?
That's a really surprising benchmark; is the server doing the file uploading on EC2 and also in the same region as your S3 bucket? How big are the generated thumbnails?
Those questions aside, doing any kind of thumbnail generation during a response is probably not a great idea: it'll add some time to each page load where thumbnails need to be generated, and even if that time isn't 3 seconds it'll still be something. I'd process images asynchronously with a gem like Delayed Paperclip. Though you won't save on storage space, as you would with CarrierWave, your response times will be vastly improved.
Related
I want to have my users be able to upload audio files.
I've done some research on this and found Paperclip. However, I've also read that Paperclip performs poorly if the file is over 4MB, and as you probably know most audio files are over 4MB.
So my question is, how can I have users upload audio files without using Paperclip? I plan on storing everything in Amazon S3.
I plan on users being able to stream this audio and download the files, if that makes any difference.
I want to be able to routinely upload about 3x 20mb files to a specific type of record in a Rails app. So far I'm using Paperclip and I get a pretty average success rate and a lot of EOF (bad content body) errors.
What can I do to improve the situation? Googling rails large upload doesn't turn much up.
Uploading large files, while slow, should succeed. The EOF (bad content body) error seems unrelated.
What are you using to upload the files? A standard multipart web form? And is anything unusual happening with the server configuration? Or are you using POW? Apparently POW users have reported similar issues.
For large files, I typically use S3 for storage and would recommend uploading directly to S3.
When I upload an image (on Amazon S3 servers) to a Heroku app from a camera, where the photos have let's say more than 2.5MB, sometimes the app is not processed within 30 seconds and I see on the screen the warning message about Application Error.
This is not very user-friendly behavior, how to avoid this acting? I now I can buy an additional dyno, but I am not sure that this is a solution. For file upload I use Paperclip gem.
How do you solve this situation, when users uploads let's say images bigger than 3MB?
There's a couple things you could do: (in order from best to worst bet)
If you have a need to do a lot of post-procesing on images (like resizing them) you can have all of that processing done on a worker dyno using "Delayed Jobs." This way you get a response back much faster, but your alternate or resized versions of the image aren't immediately available, only :original. There's a tutorial on it here: http://madeofcode.com/posts/42-paperclip-s3-delayed-job-in-rails
You could use Unicorn or one of it's cousins. While it likely won't fix the image upload issue by itself, it allows you to adjust how long it takes for a request to timeout. As an added bonus it should also speed up the rest of your app.
You could try using Carrier Wave with CarrierWaveDirect instead of paperclip. This is kind of a shot in the dark as I've never personally used it, but it's supposedly 'better,' which could mean faster? maybe? It sounds like it works in a similar way as Paperclip with delayed jobs.
I use resque to process pictures and upload to s3 with paperclip in background. But every worker will load whole rails instance about 120MB~150MB.
I found this tutorial could deal with background job without loading rails.
http://railscasts.com/episodes/243-beanstalkd-and-stalker
How could I manipulate paperclip with stalker, or the other better solutions?
Thanks.
I made a lightweight class that can be used for Paperclip.
That a look a it here : gist
Do large file uploads block an applications request/response cycle? I have an app that allows users to upload multiple large files (images in particular). These files are stored on a remote host. I cannot use async background jobs to upload these images as these have to be immediately accessible to the user once the upload finishes. How best should i handle such large uploads? Does it affect concurrency? This is my first time with uploads on a large scale. What should i be wary of other than the huge bills of course? Any input from developers who have created apps which use large file uploads will be greatly appreciated.
Why can't you use an async upload, and just handle the event that signifies that it's done? That's generally how async operations work - you kick them off and then store the pointer somewhere, and then either handle the "Complete" event, or just periodically iterate through he pointers for uploads you've started and check each one to see if it's complete.
It's an old question but still, I was worried about the same problem with large files uploads thinking that the processes get blocked while the file is getting uploaded but it turned out, if I got it right, that the nginx and probably other servers as well buffer the content of the file while its being sent so no rails processes get blocked, only when the upload is finished and rails is processing it, like resizing images or something.