I am making a rails app with video uploads. In testing any medium to big sized video totally wrecks my server logs since it tries to print out the video file. It sent the response but 10 minutes later it is still trying to print out the video file. Can I prevent rails from logging the params?
There's a way of suppressing arbitrary parameters in your configuration, so just include the name you want to suppress. For example:
Rails.application.config.filter_parameters += [ :video ]
This is described in the documentation and is by default loaded in config/initializers/filter_parameter_logging.rb. You could either edit that and add your params, or to make it modular, add another initializer file of the same format.
Note this won't block logging of SQL queries, so if you're inserting large BLOB type objects your log will be noisy. This is why it's usually better to dump the video on an object-store like Amazon S3.
Related
I'm building a webapp which is offline available.
Therefore I need urls to images and videos were not expiring at any time.
ActiveStorage is not build for that case.
So I build my own media controller for serving images and videos based on the url params
get '/media/:kind/:type/:size/:token'
# /media/video/mp4/middle/ABC123
# /media/image/logo/large/ABC124
There is a MediaObject with a unique token. With the params I get the right variant of the image or the converted video file, look for the path of the blob file, seasoned with a quick lookup what filetype it is and serving it.
path = ActiveStorage::Blob.service.send(:path_for, variant_by(size, type).key)
send_file path, type: content_type, disposition: 'inline'
everything is working good, very good. The images and videos got a cacheable (better looking) permanent url.
Now there is an issue with Safari. Every browser can handle the mp4/webm file the controller returns. Except of safari. The video cannot be watched in the browser.
I'm guessing, there is something missing in the headers.
But I cannot find the place in active storage where the videos served.
When I use the default way
# example
Rails.application.routes.url_helpers.url_for(video.mp4)
the video can be played inline.
I've tried to add the content-length to the response header, add range infos and status :partial_content, add the file extension to the url, but nothing works.
The mime type is correct.
Rails 6 introduced a public option in storage.yml. Set it to true to override the expiring URL stuff: https://edgeguides.rubyonrails.org/active_storage_overview.html#public-access
Also, if you're still on Rails 5.2, you should be able to use:
rails_blob_url(imageable.image, disposition: :attachment)
I'm managing to record audio using the recorderjs. It creates the audio. I'm using ruby rails. I save it. It sends the file correctly to my server. When I go to test and see if the audio is the same. It doesn't play and has a play time of 0 seconds.
I want to record an audio file that is 2-5 seconds long. then save that file on the server and it will be linked to a word.
File.open('public/mp3s/' + filename, 'wb') do |f|
f.write(Base64.decode64(params[:mp3data]))
end
Is there something wrong with my write method here?
It is likely that you have an encoding error, or a typo generating an empty file. The param could be a binary octet stream instead of base64 for instance.
Or it could be coming in as a different key, in which case you'd make an empty file.
If you're on mac or linux, try using the file command on the written file. The command should look at the magic byte, and if it is an MP3 you should see something along those lines.
Likewise, if you cat, hexdump, or notepad the file that rails saved, and the generated one you should be able to see any differences.
params[:mp3data] may also be nil
in development mode, the rails server should list the incoming params.
thank you. it was coming back as nil. so it brought me to check the file and if it had any data in it. so i created a size for it along with params. I figured it out and it is now working! '
metadata_size = "data:audio/mp3;base64,".length data = params[:mp3data][metadata_size, params[:mp3data].length] File.open('public/mp3s/' + filename, 'wb') do |f| f.write(Base64.decode64(data)) end
#EnabrenTane
The flow is:
The user selects an image on the client.
Only filename, content-type and size are sent to the server. (E.g. "file.png", "image/png", "123123")
The response are fields and policies for upload directly to S3. (E.g. "key: xxx, "alc": ...)
The case is that if I change the extension of "file.pdf" to "file.png" and then uploads it, the data sent to the server before uploads to S3 are:
"file.png"
"image/png"
The servers says "ok" and return the S3 fields for upload .
But the content type sent is not a real content type. But how I can validate this on the server?
Thanks!
Example:
Testing Redactorjs server side code (https://github.com/dybskiy/redactor-js/blob/master/demo/scripts/image_upload.php) it checks the file content type. But trying upload fake image (test here: http://imperavi.com/redactor/), it not allows the fake image. Like I want!
But how it's possible? Look at the request params: (It sends as image/jpeg, that should be valid)
When I was dealing with this question at work I found a solution using Mechanize.
Say you have an image url, url = "http://my.image.com"
Then you can use img = Mechanize.new.get(url)[:body]
The way to test whether img is really an image is by issuing the following test:
img.is_a?(Mechanize::Image)
If the image is not legitimate, this will return false.
There may be a way to load the image from file instead of URL, I am not sure, but I recommend looking at the mechanize docs to check.
With older browsers there's nothing you can do, since there is no way for you to access the file contents or any metadata beyond its name.
With the HTML5 file api you can do better. For example,
document.getElementById("uploadInput").files[0].type
Returns the mime type of the first file. I don't believe that the method used to perform this identification is mandated by the standard.
If this is insufficient then you could read the file locally with the FileReader apis and do whatever tests you require. This could be as simple as checking for the magic bytes present at the start of various file formats to fully validating that the file conforms to the relevant specification. MDN has a great article that shows how to use various bits of these apis.
Ultimately none of this would stop a malicious attempt.
In my web page (rendered by Rails), I'd like to let the user right-click on a photo to bring up the browser's Save As dialog, to let the user save the photo to their hard drive.
However, the photos on my server have unusual filenames (long hex names) with no file extension. The filename prompt in the Save As dialog has this ugly filename. If the user hits save, they'll end up with a poorly-named file, with no file extension.
The web page is aware of the photo's real file name (the name that came off the camera, for example). Is there a way for me to programmatically override the Save As dialog's filename prompt with a filename of my choosing?
I'm aware of the Content-Dispostion header, and that via this header a filename can be specified. However, I think that in order to be able to make use of this header, I need to load/render the entire file to the browser. If the asset to be made available for download is a movie, that loading of the file could timeout the browser...like, if it's a 100meg video.
Thoughts?
-A
I think I understand the problem here because I encountered (and resolved) at least part of it myself not too long ago.
I have some large mp3's and I link to them on my website
A few problems
I needed to set my content-disposition header to attachment in order to prevent files from automatically streaming whenever a user clicked the download button
my files are on a remote server
my files are large (100MB)
large files can tie up rails controllers if not handled properly
Now, Michael Koziarsky advises in this article that the best way to keep your rails processes free when serving large files, is to create a download action in your controller, and the do something like this (note the use of x_sendfile=>true):
def download
send_file '/path/to/podcast.mp3', :type => 'application/octet-stream', :disposition => 'attachment', :filename=>'something.mp3', :x_sendfile=>true
end
:x_sendfile tells apache to let the file through without tying up a rails controller process. The rest of the code sets the filename and the content-disposition header.
Great, but I'm on heroku, like everyone else nowadays. So I can't use x_sendfile.
I found that I couldn't modify the nginx configuration file either as it's locked down by heroku so it was not possible to get x-accel-redirect (nginx equivalent of x-sendfile) working
So, I decided to add a perl script (see below) to the cgi-bin on our asset-host and this script sets the content-disposition to attachment and gives our file a name too.
Instead of doing a restful download like this:
link_to "download", download_podcast_path(#podcast.mp3)
we just link to the mp3 making sure that we go in through the cgi-bin so that the perl script gets called on every mp3 that leaves the server
# I'm using haml
%a{:href=>"http://afmpodcast.com/cgi-bin/download.cgi?ID=#{#podcast.mp3}"}
download
The result is that my rails controller is no longer called into action when someone downloads a file
I found the perl script here and chopped it up a bit to work for me:
#!/usr/local/bin/perl -wT
use CGI ':standard';
use CGI::Carp qw(fatalsToBrowser);
my $files_location;
my $ID;
my #fileholder;
$files_location = "../";
$ID = param('ID');
open(DLFILE, "<$files_location/$ID") || Error('open', 'file');
#fileholder = <DLFILE>;
close (DLFILE) || Error ('close', 'file');
print "Content-Type:application/x-download\n";
print "Content-Disposition:attachment;filename=$ID\n\n";
print #fileholder
My code, is on github but you'll likely have all sorts of problems using it on your machine as i make heavy use of ENV variables that I store in bashrc and I have no documentation or tests ^hides^
You could do some smart server side url rewrite, like for example rewriting foo.mpeg to youveryuglyfilenamewithoutextension.
Set the Content-Disposition to "attachment; filename="...that's fine. "attachment" explicitly means it's not to be rendered in the browser, file renaming works nonetheless (or possibly particularly for that case).
Based on your comments, you have a few problems.
You want to set the filename using your Rails app.
The file is on a remote host and your Rails app is acting as a middleman.
The file might be big, so you want the file to be sent out to the browser as you receive it instead of queuing the whole thing.
Streaming only with Rails is tricky for a few reasons.
You would need an HTTP client that lets you access the message body as you receive data instead of blocking until you have everything. Net::HTTP is not that client. I'm not sure what library would be better suited.
Once you have a more event-driven way to get your file in pieces, you can pass a proc to the render:
render :text => proc { |response, output| ... }
output can be used like an IO object. Some servers may buffer before sending anyway, though, so that's something to look out for.
It would be easier not handle the byte-shuffling in Rails.
If your webserver or the proxy in front of your webserver supports the X-REPROXY-URL HTTP header, your application can set that header and your webserver or proxy will stream the file.
Perlbal is the only proxy server I know of that supports that header out of the box.
An Apache2 module is also available.
I'm using restclient for a multipart form to send data to a restful web service (it's Panda video encoding service).
The trick though, is that the file I am passing into restclient (Technoweenie branch) is coming from my own form that a user submits.
So, lets walk through this. A user posts a file to my rails app. In my controller, it receives the file from params[:file]. I then want to pass params[:file] down to Panda using RestClient.
The error I'm getting is on the Panda server follows. I noticed that the file param in the stack trace is in a string as well (which I assume is Panda turning into a string for a nicer stacktrace).
~ Started request handling: Wed Aug 12 18:05:15 +0000 2009
~ Params: {"format"=>"html", "multipart"=>"true", "account_key"=>"SECURE_KEY", "action"=>"upload", "id"=>"SECURE_ID", "controller"=>"videos", "file"=>"#<File:0xcf02ca4>"}
~ 9bfb1750-6998-012c-4509-12313900b0f6: (500 returned to client) InternalServerErrorcan't convert nil into String
/var/local/www/panda/app/models/video.rb:246:in `extname'
/var/local/www/panda/app/models/video.rb:246:in `initial_processing'
/var/local/www/panda/app/controllers/videos.rb:79:in `upload'
I doubt you can really pass a CGI-style upload param from Rails into restclient and expect it to work.
A regular upload in Rails would have quite some extra attributes which do not belong in a posted resource (like the original filename and so on), and a Rails upload contains an IO with the actual file data. Also a file upload object in Rails might be a Tempfile handle and might be a StringIO - depending on the size of the upload.
What you effectively need to do is "repackage" your upload for rest-client to handle it properly, and pass the repackaged and rewound Tempfile object to restclient. Maybe you can get away with just picking the upload object itself instead of the whole params[:file]
Confirm that your restclient action can save locally first. If the action cannot save locally, then you will have a better idea where to look while trouble shooting.
Looks like the problem is with rest-client's posting of the file, check out an alternative method for posting like curb.
Lots of examples for posting multipart form data on this question: Ruby: How to post a file via HTTP as multipart/form-data?