I use resque to process pictures and upload to s3 with paperclip in background. But every worker will load whole rails instance about 120MB~150MB.
I found this tutorial could deal with background job without loading rails.
http://railscasts.com/episodes/243-beanstalkd-and-stalker
How could I manipulate paperclip with stalker, or the other better solutions?
Thanks.
I made a lightweight class that can be used for Paperclip.
That a look a it here : gist
Related
We have a rails app that does some user-driven/filtered data representation over a large dataset. So we're calculating things on the fly and it takes longer than the 15s Unicorn gives us!
What's the best option here? I was thinking of using a pub/sub model (like a Node/Faye setup) to allow the rails app to send data that the browser could then render.
I guess another option is to try to pre-generate the data, but as we have a lot of clients and very few would be looking at the data it seems like we'd be wasting a lot of time on preparing data that would never be used.
You're on the right track with pre-generating the data.
If you're concerned about needless number crunching and want to do it on-demand, you could kick off a background job to process the data, and poll periodically to see if the background generation is done or not.
If you're looking for a library to do this for you:
Alternatively, if you're using ActionCable already, get_schwifty was built for this very purpose (shameless plug, I'm the author).
render_async is another option if you're not using ActionCable, however, I beleive it still does the processing in a Unicorn process instead of a background job.
I want to upload a bunch of files into my server using paperclip. Currently it is too slow and hence i want to allocate this to my background workers. I am already using backburner for a bunch of tasks. I tried the following which does not work
upload = UploadedFile.async.create(params[:file])
The async function works will all other normal jobs but does not work with paperclip.
I read that i could use delayed_job . However since i am already using backburner which seems to do the same worker allocation, i ideally want to use that. If it is not possible, is it wise to use both backburner and delayed_job at the same time? Will there be any conflict in worker allocationwhen both are called at the same time for different processes on the server?
I'm not experienced with backburner, but.
Uploading files in background is quite strange requirement.
Just imagine, while you are uploading file - browser keeps connection with server to transfer data.
Processing in background means, that you are immediately break this connection. It does not make sense, because you could not transfer file data in such way.
So uploading is synchronous operation, and can't be done in background.
I want to be able to routinely upload about 3x 20mb files to a specific type of record in a Rails app. So far I'm using Paperclip and I get a pretty average success rate and a lot of EOF (bad content body) errors.
What can I do to improve the situation? Googling rails large upload doesn't turn much up.
Uploading large files, while slow, should succeed. The EOF (bad content body) error seems unrelated.
What are you using to upload the files? A standard multipart web form? And is anything unusual happening with the server configuration? Or are you using POW? Apparently POW users have reported similar issues.
For large files, I typically use S3 for storage and would recommend uploading directly to S3.
When I upload an image (on Amazon S3 servers) to a Heroku app from a camera, where the photos have let's say more than 2.5MB, sometimes the app is not processed within 30 seconds and I see on the screen the warning message about Application Error.
This is not very user-friendly behavior, how to avoid this acting? I now I can buy an additional dyno, but I am not sure that this is a solution. For file upload I use Paperclip gem.
How do you solve this situation, when users uploads let's say images bigger than 3MB?
There's a couple things you could do: (in order from best to worst bet)
If you have a need to do a lot of post-procesing on images (like resizing them) you can have all of that processing done on a worker dyno using "Delayed Jobs." This way you get a response back much faster, but your alternate or resized versions of the image aren't immediately available, only :original. There's a tutorial on it here: http://madeofcode.com/posts/42-paperclip-s3-delayed-job-in-rails
You could use Unicorn or one of it's cousins. While it likely won't fix the image upload issue by itself, it allows you to adjust how long it takes for a request to timeout. As an added bonus it should also speed up the rest of your app.
You could try using Carrier Wave with CarrierWaveDirect instead of paperclip. This is kind of a shot in the dark as I've never personally used it, but it's supposedly 'better,' which could mean faster? maybe? It sounds like it works in a similar way as Paperclip with delayed jobs.
(Rails 3.2 on Heroku)
For handling image uploads in Rails I switched from Paperclip to Dragonfly because I like to be able to generate thumbnails dynamically, when they are requested for the first time.
However, it seems that uploading of attached files to S3 (using S3DataStore) is much slower than with Paperclip
This is how an upload looks in a NewRelic transaction trace:
Anyone have experience in speeding this up?
That's a really surprising benchmark; is the server doing the file uploading on EC2 and also in the same region as your S3 bucket? How big are the generated thumbnails?
Those questions aside, doing any kind of thumbnail generation during a response is probably not a great idea: it'll add some time to each page load where thumbnails need to be generated, and even if that time isn't 3 seconds it'll still be something. I'd process images asynchronously with a gem like Delayed Paperclip. Though you won't save on storage space, as you would with CarrierWave, your response times will be vastly improved.