I am currently working on a rails application in which the user upload a video file on the server, after what I send it to S3 where it's encoded by another service.
The thing is that before uploading it to S3 I need to know the quality of this file, the resolution, the bitrate, ...
I saw that the rvideo gem can do that but I'd like to avoid having to install ffmpeg on my rails server.
So my question is do you see a way to do that without much dependency ?
Thanks,
UPDATE: I also saw 2 gems wrapping the MediaInfo CLI but same problem I'd like to avoid installing MediaInfo CLI on my rails servers
You should be able to use the inspect command available in unix. I know paperclip uses it with their photo processing.
Related
I am trying to create an application that crawls a website providing free financial data in .xlsx format. They upload files once a month and not always on the same day.
Is it possible to download any new files from a specific URL and dump it into my S3 bucket, before reading it into a database? I have read up about creating a worker using Sidekiq. I expect that this will play a crucial part in the process.
Can anybody perhaps give some advice or point me to a tutorial that can help?
Yes, you can, and you don't even need Sidekiq.
Take a look at AWS SDK for Ruby, and do the following things:
Just write a ruby script that downloads the xlsx files then upload to S3. Be sure the script starts with #!/usr/bin/env ruby, and give it execute permission.
Add this script to your crontab jobs, and make it run everyday.
Normally I'd use carrierwave, but they do not officially support jruby, and I've been running in to bugs, possibly related. (https://github.com/jnicklas/carrierwave/issues/620, Image corruption on upload to s3, production only. (carrierwave, engineyard))
Have others had success here?
I'm considering trying out paperclip, but it looks like that may not be perfect either-- https://github.com/thoughtbot/paperclip/issues/100
Try dragonfly it's a really great library to manage you file attachment.
check out this guide, it's cake
http://webtempest.com/how-to-allow-image-uploads-in-rails-on-heroku/
It uses paperclip and amazon web s3 gems
I have a requirement of uploading a file to my disk through my webpage. Seems like I have two options
My requirement is specific that I will upload ONLY text files.
Using default rails methods to upload a file.
Ex: http://www.tutorialspoint.com/ruby-on-rails/rails-file-uploading.htm
Using a plugin like 'PaperClip'
Following are my concerns:
I want to keep the file upload as simple as possible
Keep as away as dependencies like Imagemagic etc
I'm using rails 2.8.3
concurrent file uploads can be happen by multiple users
please can someone tell me what are the pros and cons of having
writing a simple file upload (option 1)
using a plugin/gem to upload a files
Writing your own file uploader is an option, but using a pre-built gem provides you with all of the code you need, straight after install.
Gems will usually have all of the functionality packaged into them that handles all of the cross-platform issues and security headaches your likely to run into by writing something from scratch. A well maintained gem will also have a good community behind it, keeping things up to date.
The popular Gems out there are really easy to use, and unless you are resizing images etc, you shouldn't need ImageMagick installed. Have a look at these:
http://railscasts.com/episodes/134-paperclip
https://github.com/technoweenie/attachment_fu/wiki
Paperclip is far easier to build a simple upload form with, but I'm not sure if it works on Rails 2. Attachment_fu is an old favorite from the Rails 2 days and will definitely be able to handle your problem, it just requires a little more configuration.
One client project will be online two months later. One of the requirements changed is to support large files (10 to 15MB per RAW camera file, expected 1000 to 5000 files download per day) download worldwide for their customers. The process will be:
there is upload screen via paperclip to the rails local public folder
a hourly task to upload to web storage (S3?)
update the download url from paperclip url to the web url
Questions:
is there a gem/plug-in for this
purpose?
if no, any gem/plug-in
for S3 to recommend?
Questions about the storage provider:
is S3 recommended?
or other service to recommend?
The baseline is: the client's web server does not and will not have the bandwidth to handle the downloads.
Thanks
I don't think there is anything that will do all of this out of the box for you. Paperclip will push files sychronousy to S3 on upload, so you will need to make this ansychronous yourself.
S3 is rock-solid, I have used it in production on a number of projects. Totally recommended.
You can upload files directly to S3 which may help by reducing the double handling of the file (no longer need to upload to your app before pushing to Amazon):
http://developer.amazonwebservices.com/connect/entry.jspa?categoryID=139&externalID=1434
The aws-s3 and delayed_job gems are probably what you want.
gem install aws-s3
S3 is popular and widely used as far as I am aware.
If you end up going the route of uploading directly to S3 which offloads the work from your Rails server and makes it asynchronous, please check out my sample projects:
Sample project using Rails 3, Flash and MooTools-based FancyUploader to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-FancyUploader
Sample project using Rails 3, Flash/Silverlight/GoogleGears/BrowserPlus and jQuery-based Plupload to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-Plupload
So, I'm using Paperclip and AWS-S3, which is awesome. And it works great. Just one problem, though: I need to upload really large files. As in over 50 Megabytes. And so, nginx dies. So apparently Paperclip stores things to disk before going to S3?
I found this really cool article, but it also seems to be going to disk first, and then doing everything else in the background.
Ideally, I'd be able to upload the file in the background... I have a small amount of experience doing this with PHP, but nothing with Rails as of yet. Could anyone point me in a general direction, even?
You can bypass the server entirely and upload directly to S3 which will prevent the timeout. The same thing happens on Heroku. If you are using Rails 3, please check out my sample projects:
Sample project using Rails 3, Flash and MooTools-based FancyUploader to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-FancyUploader
Sample project using Rails 3, Flash/Silverlight/GoogleGears/BrowserPlus and jQuery-based Plupload to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-Plupload
By the way, you can do post-processing with Paperclip using something like this blog post (that Nico wrote) describes:
http://www.railstoolkit.com/posts/fancyupload-amazon-s3-uploader-with-paperclip
Maybe you have to increase the timeout in the ngix configs?
You might be interested in my post here:
http://www.railstoolkit.com/posts/fancyupload-amazon-s3-uploader-with-paperclip
Its about uploading multiple files (with progress bars, simultaneously) directly to S3 without hitting the server.
I was having a similar problem but with using paperclip, passenger and apache.
Like nginx, apache has a Timeout directive in apache which I increased to solve my problem.
Also there's an interesting thing passenger does when uploading large files.
Anything over 8k is written to /tmp/passenger. and if apache doesn't have
permissions to write there you get 500 errors also.
Here's the article.
http://tinyw.in/fwVB