When I upload an image (on Amazon S3 servers) to a Heroku app from a camera, where the photos have let's say more than 2.5MB, sometimes the app is not processed within 30 seconds and I see on the screen the warning message about Application Error.
This is not very user-friendly behavior, how to avoid this acting? I now I can buy an additional dyno, but I am not sure that this is a solution. For file upload I use Paperclip gem.
How do you solve this situation, when users uploads let's say images bigger than 3MB?
There's a couple things you could do: (in order from best to worst bet)
If you have a need to do a lot of post-procesing on images (like resizing them) you can have all of that processing done on a worker dyno using "Delayed Jobs." This way you get a response back much faster, but your alternate or resized versions of the image aren't immediately available, only :original. There's a tutorial on it here: http://madeofcode.com/posts/42-paperclip-s3-delayed-job-in-rails
You could use Unicorn or one of it's cousins. While it likely won't fix the image upload issue by itself, it allows you to adjust how long it takes for a request to timeout. As an added bonus it should also speed up the rest of your app.
You could try using Carrier Wave with CarrierWaveDirect instead of paperclip. This is kind of a shot in the dark as I've never personally used it, but it's supposedly 'better,' which could mean faster? maybe? It sounds like it works in a similar way as Paperclip with delayed jobs.
Related
I'm developing a chat app with Firebase. Am currently still in development phase.
Profile pictures of test users are uploaded to Firebase Storage, and are downloaded in the home screen (with all the pictures). I realized that with that I very quickly used up storage download requests (easily hit 3,000 requests in one night, and hit the free plan quota!).
What are some best practices I could use to minimize download requests? Just to be sure I'm doing it right - I'm sending a GET request to the Firebase Storage url directly: https://firebasestorage.googleapis.com/... to download the image. Is that the right way to do it?
Two suggestions that might help:
Cache your images! If you keep requesting the same images over and over again over the network, that's going to use up your quota pretty fast. Not to mention your user's battery and network traffic. After you retrieve an image from the network, save it locally, and then the next time you need an image, look for it locally before you make another network request. Or consider using a library like PINRemoteImage that does most of the work for you. (Both on the retrieving as well as the caching side)
Consider uploading smaller versions of your image if you think you might be using them often. If your chat app, for instance, saves profile pictures as 1024x768 images, but then spend most of its time showing them as 66x50 thumbnails, you're probably downloading a lot of data you don't need. Consider saving both the original image and a thumbnail, and then grabbing the larger one only if you need it.
Hope that helps...
In my Rails 4 app, I am using the metainspector gem to allow users to display meta data from a URL they post to a form.
Since I installed this gem, each time I try to visit the page of my own app where metadata is pulled from another website, load time increases significantly.
The load time increase goes from an imperceptible delay for small and local websites, to almost freezing the app for larger and foreign websites.
To give you an idea, a regular page usually loads under 400 ms: when we pull data with metainspector, it can go beyond a 30,000 ms (I measured these load times with rack-mini-profiler).
I did not find much about similar issues online.
Here is what I am trying to figure out:
Does this sound normal or did I setup something in the wrong way?
Is there a way to speed up load time with metainspector? For instance by caching responses?
If there is no way to speed up load time, should I implement a timeout limit and display an error message?
That is perfectly normal, to be exact it's not metainspector what is slowing your app down, it's the fact that you're requesting external URLs.
You should try to cache the responses using the built-in caching mechanism in metainspector, but also if possible move this to an asynchronous job using a background queue, and save or cache the result.
I want to upload a bunch of files into my server using paperclip. Currently it is too slow and hence i want to allocate this to my background workers. I am already using backburner for a bunch of tasks. I tried the following which does not work
upload = UploadedFile.async.create(params[:file])
The async function works will all other normal jobs but does not work with paperclip.
I read that i could use delayed_job . However since i am already using backburner which seems to do the same worker allocation, i ideally want to use that. If it is not possible, is it wise to use both backburner and delayed_job at the same time? Will there be any conflict in worker allocationwhen both are called at the same time for different processes on the server?
I'm not experienced with backburner, but.
Uploading files in background is quite strange requirement.
Just imagine, while you are uploading file - browser keeps connection with server to transfer data.
Processing in background means, that you are immediately break this connection. It does not make sense, because you could not transfer file data in such way.
So uploading is synchronous operation, and can't be done in background.
(Rails 3.2 on Heroku)
For handling image uploads in Rails I switched from Paperclip to Dragonfly because I like to be able to generate thumbnails dynamically, when they are requested for the first time.
However, it seems that uploading of attached files to S3 (using S3DataStore) is much slower than with Paperclip
This is how an upload looks in a NewRelic transaction trace:
Anyone have experience in speeding this up?
That's a really surprising benchmark; is the server doing the file uploading on EC2 and also in the same region as your S3 bucket? How big are the generated thumbnails?
Those questions aside, doing any kind of thumbnail generation during a response is probably not a great idea: it'll add some time to each page load where thumbnails need to be generated, and even if that time isn't 3 seconds it'll still be something. I'd process images asynchronously with a gem like Delayed Paperclip. Though you won't save on storage space, as you would with CarrierWave, your response times will be vastly improved.
Do large file uploads block an applications request/response cycle? I have an app that allows users to upload multiple large files (images in particular). These files are stored on a remote host. I cannot use async background jobs to upload these images as these have to be immediately accessible to the user once the upload finishes. How best should i handle such large uploads? Does it affect concurrency? This is my first time with uploads on a large scale. What should i be wary of other than the huge bills of course? Any input from developers who have created apps which use large file uploads will be greatly appreciated.
Why can't you use an async upload, and just handle the event that signifies that it's done? That's generally how async operations work - you kick them off and then store the pointer somewhere, and then either handle the "Complete" event, or just periodically iterate through he pointers for uploads you've started and check each one to see if it's complete.
It's an old question but still, I was worried about the same problem with large files uploads thinking that the processes get blocked while the file is getting uploaded but it turned out, if I got it right, that the nginx and probably other servers as well buffer the content of the file while its being sent so no rails processes get blocked, only when the upload is finished and rails is processing it, like resizing images or something.