Rails + X-Sendfile serving large video to Ipad - ruby-on-rails

I have an application serving large (some hundreds of MB) video files and it is working perfectly on desktop browsers, using Rails + X-Sendfile on Apache.
An important requirement is that these videos must be private and visible only to logged users, so that's why i'm using Rails to serve them.
Everything works perfectly with other devices. I serve the videos in this way:
response.headers["X-Sendfile"]= filename
send_file filename, :disposition => :inline, :stream => true, :x_sendfile => true
But Ipad's requests need the byte range header. A solution (that does not work perfectly) is something like this:
size = File.size(filename)
bytes = Rack::Utils.byte_ranges(request.headers, size)[0]
offset = bytes.begin
length = bytes.end - bytes.begin
response.header["Accept-Ranges"]= "bytes"
response.header["Content-Range"] = "bytes #{bytes.begin}-#{bytes.end}/#{size}"
send_data IO.binread(filename,length, offset), :type => "video/mp4", :stream => true, :disposition => 'inline', :file_name => filename
With this solution I have problems with larger than 50mb videos, and, more important, I'm giving to rails a responsibility that it shouldn't have. It should be apache to handle the heavy load of the streaming, through the x-sendfile module. But I dont know how. the send_data method does not have the x-sendfile parameter, and solutions involving the send_file method are not working.
I found these 2 questions similar to mine but they are not working: rails media file stream accept byte range request through send_data or send_file method, What is the proper way to serve mp4 files through rails to an Ipad?
Any clue on what's going on? I'm struggling with this since weeks and i need to make it work. Other feasible solutions are welcome.

This could be completely unrelated as I am using nginx for the server, but if its only not working for ios, take a look at this blog post. There could be a similar solution for Apache.
In a sense, I had to add a proxy header that internally redirects to a folder path. How ever stupid this may seem, Apple has some sort of privacy issues that make this necessary for playback with audio and video files. Again not sure if this is the solution for you but for nginx this did wonders and cured my month long headache.

Have you enabled X-Sendfile config in your environment? Include the line config.action_dispatch.x_sendfile_header = "X-Sendfile" for Apache. The server will then use that header for sending files.

Check that the apache request body size is large enough too.

Related

rails 3 and PDFkit

I'm trying to follow this tutorial.
When I'm adding .pdf to my url it does nothing. My controller has:
respond_to :html, :pdf.
My mime type has been declared.
I tried this too:
respond_to do |format|
format.html
format.pdf {
html = render_to_string(:layout => false , :action => "www.google.fr")
kit = PDFKit.new(html)
send_data(kit.to_pdf, :filename => "candidats.pdf", :type => 'application/pdf')
return # to avoid double render call
}
end
but it does not work and I don't get errors. My browser keep waiting for localhost, but nothing happens.
So how should I try to use pdfkit ?
edit 2 :
According to my rails' logs, rails render successfully the HTML. I saw this in the .log, rails doesn't send it to webrick nor to my browser. And my browser keeps waiting, and waiting and nothing happen. I only have little pictures here.
edit 3 : My webrick server seem unable to respond to other request, once he starts getting a .pdf version of my url, any ideas ?
edit 4 :
I am using rails 3.1, wkhtmltopdf 0.9.5 (windows installer) and pdfkit 0.5.2
I found a better way to access my .pdf urls even in developpement mode.
# Enable threaded mode
config.threadsafe!
# Code is not reloaded between requests
#config.cache_classes = true
Config.cache_classes is a comment, cause i had some problems when it wasn't. This way pdfkit works wonder even with rails 3.1. But, you don't reload code between requests.
That's not really a problem, cause you first work on your html, and you switch configuration, in order to check the pdf result. This way you don't have to bother about your production database.
Thanks to another stack overflow answer I got part of solution.
This works:
html = '<html><body>Toto de combats</body></html>'
#pdf = PDFKit.new(html)
send_data #pdf.to_pdf, :filename => "whatever.pdf",
:type => "application/pdf",
:disposition => "attachement"
You can replace attachement by inline, so the pdf is displayed in your browser.
In the stack overflow answer I spoke about (I don't remember the link), the .to_pdf was missing, but is mandatory. Otherwise PDF reader doesn't recognize it.
I'm trying to get this work with a .pdf url.
Edit 3:
My problem with .pdf urls is solved. known issue with rails 3.1, but google was unable to find them.
explanation : explanation
Workaround (hadn't tryied yet). workaround

Cache-Control Headers and Heroku Dynamic Images

I'm trying to understand HTTP Caching on Heroku. After reading their article, I'm curious about how the Cache-Control HTTP header is working.
In the sample application mentioned in the article the header is set in a controller action:
def image
#qrimage = QRImage.find_by_md5(params[:md5])
if #qrimage
headers['Cache-Control'] = 'public; max-age=2592000' # cache image for a month
send_data #qrimage.data, :filename => #qrimage.filename, :disposition => 'inline', :type => "image/png"
else
render :nothing => true, :status => 404
end
end
The code for #qrimage.data is like:
def data
qrcode = RQRCode::QRCode.new(self.message, :size => self.version, :level => self.ecc.to_sym)
qrcode.to_s
end
So to me it looks like the image is being generated on the server every time. And then cached by the browser for a month. So the only savings here is when the same visitor tries to view the same image.
If different visitors try to view the same image, it will still be generated and sent. Not really all that helpful if you ask me.
Is my understanding correct, or will the same image not be regenerate for each site visitor?
Heroku apps on the Aspen and Bamboo stacks are fronted by Varnish, an
HTTP accelerator. Varnish will cache output from your application
according to cues provided by standard HTTP headers to describe a
page’s cacheability. These headers are the same ones used by browsers,
so setting these headers correctly gives your app a double boost of
speed when on Heroku: at the Varnish layer, and again at the user’s
browser.
If you don't know, Varnish is a really fast cache that sits between your application and the internet, essentially. When headers say it's safe to cache, Varnish does so and responds to additional requests with the cached object without ever hitting your application.

What is the difference between send_data and send_file in Ruby on Rails?

Which one is best for streaming and file downloads?
Please provide examples.
send_data(_data_, options = {})
send_file(_path_, options = {})
Main difference here is that you pass DATA (binary code or whatever) with send_data or file PATH with send_file.
So you can generate some data and send it as an inline text or as an attachment without generating file on your server via send_data. Or you can send ready file with send_file
data = "Hello World!"
send_data( data, :filename => "my_file.txt" )
Or
data = "Hello World!"
file = "my_file.txt"
File.open(file, "w"){ |f| f << data }
send_file( file )
For perfomance it is better to generate file once and then send it as many times as you want. So send_file will fit better.
For streaming, as far as I understand, both of this methods use the same bunch of options and settings, so you can use X-Send or whatever.
UPD
send_data and save file:
data = "Hello World!"
file = "my_file.txt"
File.open(file, "w"){ |f| f << data }
send_data( data )
send_file may be faster than send_data
As fl00r mentioned, send_file takes a path, and send_data the data.
Therefore send_file is a subset of send_data, as you need a file on the filesystem: you could of course just read the file and use send_data on it. But send_file can be faster, so it is a performance / generality trade-off.
send_file can be faster because it can send the X-Sendfile header on Apache (X-Accel-Redirect on Nginx) instead of the file content, since it knows the path.
This header is consumed by the reverse proxy (Apache or Nginx) which normally runs in front of Rails in a production setup.
If X-Sendfile is present on the response, the reverse proxy ignores most of the current response, and builds a new one that returns the file at the given path.
Client <---> Internet <---> Reverse proxy <---> Rails
This is much more efficient since the reverse proxy is highly specialized at serving static files, and can do it much faster than Rails (which does not send the file data if X-Sendfile will be sent).
The typical use case of send_file is when you want to control the access permission of static files: you cannot put them under /public or else they would get served before Rails has a chance to decide. This is discussed at: Protecting the content of public/ in a Rails app
In order to use the X-Sendfile headers, you have to add:
config.action_dispatch.x_sendfile_header = "X-Sendfile" # for apache
config.action_dispatch.x_sendfile_header = 'X-Accel-Redirect' # for nginx
to config/initializers/production.rb (or config/environment/production.rb in Rails 5.x), not application.rb, since in development you don't have a proxy server and you want send_file to actually send the data.
X-Sendfile is discussed on the Asset Pipeline Guide.

Rails - Paper_Clip - Support for Multi File Uploads

I have paper_clip installed on my Rails 3 app, and can upload a file - wow that was fun and easy!
Challenge now is, allowing a user to upload multiple objects.
Whether it be clicking select fileS and being able to select more than one. Or clicking a more button and getting another file upload button.
I can't find any tutorials or gems to support this out of the box. Shocking I know...
Any suggestions or solutions. Seems like a common need?
Thanks
Okay, this is a complex one but it is doable. Here's how I got it to work.
On the client side I used http://github.com/valums/file-uploader, a javascript library which allows multiple file uploads with progress-bar and drag-and-drop support. It's well supported, highly configurable and the basic implementation is simple:
In the view:
<div id='file-uploader'><noscript><p>Please Enable JavaScript to use the file uploader</p></noscript></div>
In the js:
var uploader = new qq.FileUploader({
element: $('#file-uploader')[0],
action: 'files/upload',
onComplete: function(id, fileName, responseJSON){
// callback
}
});
When handed files, FileUploader posts them to the server as an XHR request where the POST body is the raw file data while the headers and filename are passed in the URL string (this is the only way to upload a file asyncronously via javascript).
This is where it gets complicated, since Paperclip has no idea what to do with these raw requests, you have to catch and convert them back to standard files (preferably before they hit your Rails app), so that Paperclip can work it's magic. This is done with some Rack Middleware which creates a new Tempfile (remember: Heroku is read only):
# Embarrassing note: This code was adapted from an example I found somewhere online
# if you recoginize any of it please let me know so I pass credit.
module Rack
class RawFileStubber
def initialize(app, path=/files\/upload/) # change for your route, careful.
#app, #path = app, path
end
def call(env)
if env["PATH_INFO"] =~ #path
convert_and_pass_on(env)
end
#app.call(env)
end
def convert_and_pass_on(env)
tempfile = env['rack.input'].to_tempfile
fake_file = {
:filename => env['HTTP_X_FILE_NAME'],
:type => content_type(env['HTTP_X_FILE_NAME']),
:tempfile => tempfile
}
env['rack.request.form_input'] = env['rack.input']
env['rack.request.form_hash'] ||= {}
env['rack.request.query_hash'] ||= {}
env['rack.request.form_hash']['file'] = fake_file
env['rack.request.query_hash']['file'] = fake_file
if query_params = env['HTTP_X_QUERY_PARAMS']
require 'json'
params = JSON.parse(query_params)
env['rack.request.form_hash'].merge!(params)
env['rack.request.query_hash'].merge!(params)
end
end
def content_type(filename)
case type = (filename.to_s.match(/\.(\w+)$/)[1] rescue "octet-stream").downcase
when %r"jp(e|g|eg)" then "image/jpeg"
when %r"tiff?" then "image/tiff"
when %r"png", "gif", "bmp" then "image/#{type}"
when "txt" then "text/plain"
when %r"html?" then "text/html"
when "js" then "application/js"
when "csv", "xml", "css" then "text/#{type}"
else 'application/octet-stream'
end
end
end
end
Later, in application.rb:
config.middleware.use 'Rack::RawFileStubber'
Then in the controller:
def upload
#foo = modelWithPaperclip.create({ :img => params[:file] })
end
This works reliably, though it can be a slow process when uploading a lot of files simultaneously.
DISCLAIMER
This was implemented for a project with a single, known & trusted back-end user. It almost certainly has some serious performance implications for a high traffic Heroku app and I have not fire tested it for security. That said, it definitely works.
The method Ryan Bigg recommends is here:
https://github.com/rails3book/ticketee/commit/cd8b466e2ee86733e9b26c6c9015d4b811d88169
https://github.com/rails3book/ticketee/commit/982ddf6241a78a9e6547e16af29086627d9e72d2
The file-uploader recommendation by Daniel Mendel is really great. It's a seriously awesome user experience, like Gmail drag-and-drop uploads. Someone wrote a blog post about how to wire it up with a rails app using the rack-raw-upload middleware, if you're interested in an up-to-date middleware component.
http://pogodan.com/blog/2011/03/28/rails-html5-drag-drop-multi-file-upload
https://github.com/newbamboo/rack-raw-upload
http://marc-bowes.com/2011/08/17/drag-n-drop-upload.html
There's also another plugin that's been updated more recently which may be useful
jQuery-File-Upload
Rails setup instructions
Rails setup instructions for multiples
And another one (Included for completeness. I haven't investigated this one.)
PlUpload
plupload-rails3
These questions are highly related
Drag-and-drop file upload in Google Chrome/Chromium and Safari?
jQuery Upload Progress and AJAX file upload
I cover this in Rails 3 in Action's Chapter 8. I don't cover uploading to S3 or resizing images however.
Recommending you buy it based solely on it fixing this one problem may sound a little biased, but I can just about guarantee you that it'll answer other questions you have down the line. It has a Behaviour Driven Development approach as one of the main themes, introducing you to Rails features during the development of an application. This shows you not only how you can build an application, but also make it maintainable.
As for the resizing of images after they've been uploaded, Paperclip's got pretty good documentation on that. I'd recommend having a read and then asking another question on SO if you don't understand any of the options / methods.
And as for S3 uploading, you can do this:
has_attached_file :photo, :styles => { ... }, :storage => :s3
You'd need to configure Paperclip::Storage::S3 with your S3 details to set it up, and again Paperclip's got some pretty awesome documentation for this.
Good luck!

Problems with x_sendfile in Rails

I’m having some problems with John Guenin's x_sendfile (http://john.guen.in/past/2007/4/17/send_files_faster_with_xsendfile/).
When coding the download of a PDF file, I’m using the following code:
def send_the_file(filename)
xsendfile (“#{Rails.root}/doc/” + filename, :type => ‘application/pdf’)
end
but I only get 1 byte downloaded. This usually happens if the filename is not absolute (hence the #{Rails.root} being added. I’ve also checked that the file has the necessary permissions. This is failing both on localhost and my prod site.
Any ideas what I'm doing wrong?
TIA,
Urf
What version of Rails are you using? If you're on 2.1 or later, the X-Sendfile option is built into Rails' send_file method.
send_file 'filename', :x_sendfile => true
Otherwise, are you sure that mod_xsendfile has been installed and configured correctly?
You may want to make sure that your are actually using a web server that supports xsendfile. If you are dev mode you probably aren't and it may fail.
Try to set in apche httpd.conf file
XSendFileAllowAbove on

Resources