I'm using restclient for a multipart form to send data to a restful web service (it's Panda video encoding service).
The trick though, is that the file I am passing into restclient (Technoweenie branch) is coming from my own form that a user submits.
So, lets walk through this. A user posts a file to my rails app. In my controller, it receives the file from params[:file]. I then want to pass params[:file] down to Panda using RestClient.
The error I'm getting is on the Panda server follows. I noticed that the file param in the stack trace is in a string as well (which I assume is Panda turning into a string for a nicer stacktrace).
~ Started request handling: Wed Aug 12 18:05:15 +0000 2009
~ Params: {"format"=>"html", "multipart"=>"true", "account_key"=>"SECURE_KEY", "action"=>"upload", "id"=>"SECURE_ID", "controller"=>"videos", "file"=>"#<File:0xcf02ca4>"}
~ 9bfb1750-6998-012c-4509-12313900b0f6: (500 returned to client) InternalServerErrorcan't convert nil into String
/var/local/www/panda/app/models/video.rb:246:in `extname'
/var/local/www/panda/app/models/video.rb:246:in `initial_processing'
/var/local/www/panda/app/controllers/videos.rb:79:in `upload'
I doubt you can really pass a CGI-style upload param from Rails into restclient and expect it to work.
A regular upload in Rails would have quite some extra attributes which do not belong in a posted resource (like the original filename and so on), and a Rails upload contains an IO with the actual file data. Also a file upload object in Rails might be a Tempfile handle and might be a StringIO - depending on the size of the upload.
What you effectively need to do is "repackage" your upload for rest-client to handle it properly, and pass the repackaged and rewound Tempfile object to restclient. Maybe you can get away with just picking the upload object itself instead of the whole params[:file]
Confirm that your restclient action can save locally first. If the action cannot save locally, then you will have a better idea where to look while trouble shooting.
Looks like the problem is with rest-client's posting of the file, check out an alternative method for posting like curb.
Lots of examples for posting multipart form data on this question: Ruby: How to post a file via HTTP as multipart/form-data?
Related
So, I'm creating an app that works like a bot, it makes a call to an API from time to time, then it receives a response in a json-like format and saves it like this:
finalResult = RestClient.get( apiUrl, headers = apiHeaders )
jsonData = JSON.parse(ActiveSupport::Gzip.decompress(finalResult))
time = Time.now
File.write("public/#{time}.json", jsonData)
I'm using ActiveSupport to be able to parse this Gzip compressed data, since it's a lot of data, otherwise it takes forever to get it. Then I get the time the data was received, basically, and I use it to name the file so that I can keep good track of it.
What I need to do now is compress this .json file, if possible, into a .zip file(it can be .rar, or .7z, or .tz, whatever) before I upload it to my storage so it takes less space. Is there anyway that I can do something similar to File.write but to save it as a zipped json file? I already checked stuff like zlib and ruby-zip, but they only let me zip files that already "exist", so I can't save it as a zipped .json directly, I'd need to take the .json file and then zip it, but how could I do that if the name of the file is a Time.now and it always change?
I'd appreciate any help, thanks in advance :)
EDIT¹
Giving some more details that may help you to help me:
I created a controller and model to handle this, since I'll be using ActiveStorage. It's ResponsesController and Response model, and the only parameter that the Response model has is has_one_attached :json_file. I intend to use Heroku to handle the CRON job of calling the API and I'll upload the .json files(or .zip files) to an AWS storage.
The flow is:
The user selects an image on the client.
Only filename, content-type and size are sent to the server. (E.g. "file.png", "image/png", "123123")
The response are fields and policies for upload directly to S3. (E.g. "key: xxx, "alc": ...)
The case is that if I change the extension of "file.pdf" to "file.png" and then uploads it, the data sent to the server before uploads to S3 are:
"file.png"
"image/png"
The servers says "ok" and return the S3 fields for upload .
But the content type sent is not a real content type. But how I can validate this on the server?
Thanks!
Example:
Testing Redactorjs server side code (https://github.com/dybskiy/redactor-js/blob/master/demo/scripts/image_upload.php) it checks the file content type. But trying upload fake image (test here: http://imperavi.com/redactor/), it not allows the fake image. Like I want!
But how it's possible? Look at the request params: (It sends as image/jpeg, that should be valid)
When I was dealing with this question at work I found a solution using Mechanize.
Say you have an image url, url = "http://my.image.com"
Then you can use img = Mechanize.new.get(url)[:body]
The way to test whether img is really an image is by issuing the following test:
img.is_a?(Mechanize::Image)
If the image is not legitimate, this will return false.
There may be a way to load the image from file instead of URL, I am not sure, but I recommend looking at the mechanize docs to check.
With older browsers there's nothing you can do, since there is no way for you to access the file contents or any metadata beyond its name.
With the HTML5 file api you can do better. For example,
document.getElementById("uploadInput").files[0].type
Returns the mime type of the first file. I don't believe that the method used to perform this identification is mandated by the standard.
If this is insufficient then you could read the file locally with the FileReader apis and do whatever tests you require. This could be as simple as checking for the magic bytes present at the start of various file formats to fully validating that the file conforms to the relevant specification. MDN has a great article that shows how to use various bits of these apis.
Ultimately none of this would stop a malicious attempt.
In our rails application we have a many actions that do regular webapp actions. But, we have a single action that accepts a large XML file. I would like to keep rails from parsing the XML into params. Instead, I would like to be able to get the URL params ( /documents/{id}/action ) and then write out the xml file to a specific directory. How do I keep Rails from processing it?
How would I define the action to handle this?
def handle_xml
# what to put here
end
The upload is done using Content-Type: application/xml It is a single file, and not part of a multipart form. The sample curl statement would be:
curl-H 'Accept: application/xml' -H 'Content-Type: application/xml' -X POST -d '<?xml version="1.0" encoding="UTF-8"?><test></test>' http://0.0.0.0:3000/controller/handle_xml
If you want to prevent rails from automatically parsing the XML data into a hash of parameters, you'll have to replace the ParamsParser middleware with your own custom version.
When a file is posted to rails, the ParamsParser middleware modifies the request parameters and turns it into a Hash if the data format is xml. You can find the details in the params_parser.rb file in rails.
Here's a RoR mailing list message similar to the question that you've asked
Unfortunately, as a new user I can't post any more links, but you should search google with "Sanitizing POST params with custom Rack middleware" for some more details on writing custom rack middleware.
I too have come across this problem recently. However mine is in an internal application where I have full control over both the Rails app and the clients connecting to it.
In my app the client POSTs a large XML data set to the Rails app. I wanted to process the XML document in a delayed job (resque). My workaround was to make the client use an alternate content-type. I used application/octet-stream. This prevents Rails from parsing the POST data. The data is available in request.raw_post.
The action should receive it as a file (through way of multipart form upload) and then store it as a temporary file for you.
Have you tried sending the xml file has one variable in the http uri request? So something like
#xml_file = xml..xml...xml...
parameters => {
query => {
xml_file => #xml_file
}
}
Httparty.post("url", parameters)
Then in your method:
def handle_xml
#xml_file = params[:xml_file]
#xml_file.save (or whatever you want here..)
end
i started using nginx upload module (which creates upload files like /tmp/000121545) but i need paperclip to use original filename while saving files (like /public/avatars/LuckyLuke.jpg)
previously in the parameters Rails were passing just
"avatar"=>#<File:/tmp/RackMultipart20100413-6151-t3ecq0-0>
no original filename as well, so i am wondering where from does it come in paperclip? i tried looking through plugin code but it's currently a bit too complex for me.
The browser sends a http header with the file name. ("Content-Disposition: filename=original_file.jpg")
Rails makes this available as a instance method of the temp file object: params[:avatar].original_filename, and paperclip uses that.
In detail, Rack parses the multipart form in Rack::Utils::Multipart::UploadedFile and puts a hash in the parameters that includes :tempfile and :filename. Then ActionDispatch::Http::Upload comes along and replaces that hash by the File object (value of :tempfile), extending it with the module ActionDispatch::Http::UploadedFile, which adds a instance variable for original_path and the method original_filename.
I am able to 'POST' to a Rails application (with Paperclip) using XML instead of the standard web form (trying to do it from another Ruby script). However, I would like to include a binary file.
Is there any way to include the binary data within an XML tag? Or can I do something like B64 encode the data on the client and then decode it before it hits the Paperclip plugin?
UPDATE:
The browser sends a POST with this data (among others):
Content-Disposition: form-data; name="upload[upload]"; filename="foo.jpg"
Content-Type: image/jpeg
ÿØÿà�JFIF��`�`��ÿþ�Created by AccuSoft Corp.ÿÛ�C�...
I'd like to replicate that, but within XML
The short version is: use type="file", base64-encode the file, and put it inside a CDATA block. I originally found an explanation at this link:
http://techblog.floorplanner.com/2010/02/15/restful-uploading-of-files-using-xml/
That link appears to have died, so I recommend checking out the Internet Archive copy of the blog post:
http://web.archive.org/web/20100825030057/http://techblog.floorplanner.com/2010/02/15/restful-uploading-of-files-using-xml/
Also linked from that post is a gem that implements an encoder for files posted to Rails as XML: https://github.com/nragaz/encoded_attachment