I'm trying to load-test a Rails application using JMeter. A critical part of the application involves a form that includes both text inputs and file uploads. It works fine in a browser, but when I try to post that page in JMeter, Rails is saving all of the parts of the multipart form as temp files, which causes things to break when it's looking for a string and gets a tempfile instead.
It appears that the difference is that, from a browser, the piece of the multipart request that contains a text input looks like this:
-----------------------------7d93b4186074c
Content-Disposition: form-data; name="field_name"
test
-----------------------------7d93b4186074c
while from JMeter it looks like this:
-----------------------------7d159c1302d0y0
Content-Disposition: form-data; name="field_name"
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
test
-----------------------------7d159c1302d0y0
So apparently Rails sees the former and interprets it as a plain text value and treats it as a string, but sees the latter and saves it to a temp file.
I have not been able to find a setting to convince JMeter not to send the additional headers in the multipart form for non-file fields.
Is there a way to convince Rails to ignore those headers and treat the text/plain text as strings instead of text files? Or a quick way to put a filter in front of my controller that will strip the extra headers?
Alternately, is there a better tool to load-test a Rails application that includes file upload?
Turns out these days you can just tick "use browser compatible headers" in JMeter. Could've saved myself a hell of a lot of time there :-)
So, I have customized JMeter's multipart request posting part in the source code to put out the request that rails understand. The change is easy as shown below but to create compiling Java/JMeter environment took time. :(
Anyways, now I can successfully upload a file by multipart post via JMeter.
in src/protocol/http/org/apache/jmeter/protocol/http/sampler/PostWriter.java
writeStartFileMultipart()
//writeln(out, "Content-Transfer-Encoding: binary"); // $NON-NLS-1$
writeFormMultipart()
/*****
writeln(out, "Content-Type: text/plain; charset=" + charSet); // $NON-NLS-1$
writeln(out, "Content-Transfer-Encoding: 8bit"); // $NON-NLS-1$
*****/
P.S.
A tip tip to create the build environment for 2.4 is
to comment out the 3rd party libraries check in build.xml file.
copy lib/xstream-1.3.1.jar from binary archive into lib/ directory
There may be a better way, but I ended up adding a quick filter to turn the text/plain tempfiles into strings within the parameter hash:
def change_text_files_to_strings
params.each_pair do |key, value|
params[key] = value.read if (value.class.to_s=='Tempfile' && value.content_type.start_with?('text/plain') )
end
end
By the way, it turns out that jmeter is correct here, and rails incorrect: according to RFC 2388, each item in a multipart request should have a content type (not just files), so Rails really shouldn't be using the presence of a content-type header to determine whether it's a file. Ah well.
I also used the solution above as ColdFusion was sending similar headers (minus the Content-Transfer-Encoding) with each piece of form data. I wonder if there's a better way.
EDIT: Anyone know if this has been fixed in Rails 3?
What kind of error do you get? Something like
NoMethodError (undefined method `rewind' for "1":String):
There is an issue with Rack that could explain your problem. See https://github.com/rack/rack/issuesearch?state=open&q=rewind#issue/116
We were also having a similar issue, In addition to the above answers we also correlate the X-CSRF-Token of HTTP Header Manager in that request and were
successfully able to upload the required media as many as times we wanted.
Related
I have a Rails/Ember one-page app. Burp reports that
The value of the 'content_type' JSON parameter is copied into the HTML
document as plain text between tags. The payload
da80balert(1)4f31e was submitted in the content_type
JSON parameter. This input was echoed unmodified in the application's
response.
I can't quite parse this message referring to "is copied into" and "was submitted" in, but basically what is happening is:
A PUT or POST from the client contains ...<script>...</script>... in some field.
The server handles this request, and sends back the created object in JSON format, which includes the string in question
The client then displays that string, using the standard Embers/Handlebars {{content_type}}, which HTML-escapes the string and inserts it into the DOM, so the browser displays it on the screen as originally entered (and of course does NOT execute it).
So yes, the input was indeed echoed unmodified in the application's response. However, the application's response was not HTML, in which case there would indeed be a problem, but JSON, containing strings which when referred to by Handlebars will always be escaped properly for proper display in the browser.
So my question is, is this in fact a vulnerability? I have taken great care with my Ember app and can prove that no data from JSON objects is ever inserted "raw" into the DOM. Or is this a false positive given rise to by the mere fact the unescaped string may be found in the response if looked for using an unintelligent string comparison, not taking into account the fact that the JSON will be processed/escaped by the client-side framework?
To put it a different way, in a classic webapp spitting out HTML from the server, we know that user input such as the above must be escaped/sanitized properly. Unsanitized data "on the wire" in and of itself represents a vulnerability. However, in a one-page app based on JSON coming back from the server, the escaping/sanitization occurs in the client; the JSON on the "wire" may contain unsanitized data, and this is as expected. Am I missing something here?
There are subtle ways in which you can trick IE9 and older into treating JSON as HTML. So even if the server's response has a Content-Type header of application/json, IE will second guess it. This is called content type sniffing, and can be disabled by adding the X-Content-Type-Options: nosniff header.
JSON is not an executable format so your understanding is correct.
I did a demo of this exact problem in my talk on securing single page web apps at OWASP AppSec EU 2013 which someone put up on youtube here: http://m.youtube.com/watch?v=Femsrx0m9bU
In our rails application we have a many actions that do regular webapp actions. But, we have a single action that accepts a large XML file. I would like to keep rails from parsing the XML into params. Instead, I would like to be able to get the URL params ( /documents/{id}/action ) and then write out the xml file to a specific directory. How do I keep Rails from processing it?
How would I define the action to handle this?
def handle_xml
# what to put here
end
The upload is done using Content-Type: application/xml It is a single file, and not part of a multipart form. The sample curl statement would be:
curl-H 'Accept: application/xml' -H 'Content-Type: application/xml' -X POST -d '<?xml version="1.0" encoding="UTF-8"?><test></test>' http://0.0.0.0:3000/controller/handle_xml
If you want to prevent rails from automatically parsing the XML data into a hash of parameters, you'll have to replace the ParamsParser middleware with your own custom version.
When a file is posted to rails, the ParamsParser middleware modifies the request parameters and turns it into a Hash if the data format is xml. You can find the details in the params_parser.rb file in rails.
Here's a RoR mailing list message similar to the question that you've asked
Unfortunately, as a new user I can't post any more links, but you should search google with "Sanitizing POST params with custom Rack middleware" for some more details on writing custom rack middleware.
I too have come across this problem recently. However mine is in an internal application where I have full control over both the Rails app and the clients connecting to it.
In my app the client POSTs a large XML data set to the Rails app. I wanted to process the XML document in a delayed job (resque). My workaround was to make the client use an alternate content-type. I used application/octet-stream. This prevents Rails from parsing the POST data. The data is available in request.raw_post.
The action should receive it as a file (through way of multipart form upload) and then store it as a temporary file for you.
Have you tried sending the xml file has one variable in the http uri request? So something like
#xml_file = xml..xml...xml...
parameters => {
query => {
xml_file => #xml_file
}
}
Httparty.post("url", parameters)
Then in your method:
def handle_xml
#xml_file = params[:xml_file]
#xml_file.save (or whatever you want here..)
end
Certain mail clients allow for the sender to place images directly in the body of their email (instead of as a traditional attachment). When I receive one of these emails in my application, I need to be able to look at only the text/plain message body and determine that the sender embedded an inline image.
I'm trying to craft a RegEx to find image placeholders in the text/plain message body so I can swap them for <img> tags in my own HTML-enabled version of the message. (Wacky, I know, but this is the requirement).
The problem I'm finding is that the placeholders differ based on the sending mail client. For example, when sent from MS Outlook, the text/plain body of the multi-part message looks like this:
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Check out this image:
[cid:image001.jpg#01CB50D4.769583B0]
Isn't it cool??
A similar message sent from Gmail is a little bit different:
Content-Type: text/plain; charset=ISO-8859-1
Check out this image:
[image: image001.jpg]
Isn't it cool??
The text/html body and image/jpeg part with the base64 encoded image follow.
Has anyone done any research on this before and compiled a list or built a RegEx specifically for this purpose?
I realize a more reliable way to achieve my goal is to look at the text/html portion of the message--which seems to be a bit more standardized from the few tests I've done--but unfortunately I don't have access to that in this scenario.
I'm using C#, if that matters to anyone.
Here's a list of text/plain image placeholders I've compiled thus far:
Gmail: [image: filename.jpg]
Outlook 2007: [cid:filename.jpg#01CB50D4.769583B0]
Thunderbird 3.0.7: none
I'd suggest to go with html part. If you want to find just a placeholder in plain text part, this very simple regular expression should be sufficient (PCRE):
^\[.*\]$
At least this is what works for examples above. If you'd like to identify image name, a bit complicated expression would be required. Mind that, this will catch all lines starting with [ and ending with ] no matter what the contents are. If you'd like to limit regexp to some file types, try this:
^\[.*(\.jpg|\.jpeg|\.png|\.gif|\.bmp).*\]$i
Examples will work in Perl, since you didn't mention language...
I am able to 'POST' to a Rails application (with Paperclip) using XML instead of the standard web form (trying to do it from another Ruby script). However, I would like to include a binary file.
Is there any way to include the binary data within an XML tag? Or can I do something like B64 encode the data on the client and then decode it before it hits the Paperclip plugin?
UPDATE:
The browser sends a POST with this data (among others):
Content-Disposition: form-data; name="upload[upload]"; filename="foo.jpg"
Content-Type: image/jpeg
ÿØÿà�JFIF��`�`��ÿþ�Created by AccuSoft Corp.ÿÛ�C�...
I'd like to replicate that, but within XML
The short version is: use type="file", base64-encode the file, and put it inside a CDATA block. I originally found an explanation at this link:
http://techblog.floorplanner.com/2010/02/15/restful-uploading-of-files-using-xml/
That link appears to have died, so I recommend checking out the Internet Archive copy of the blog post:
http://web.archive.org/web/20100825030057/http://techblog.floorplanner.com/2010/02/15/restful-uploading-of-files-using-xml/
Also linked from that post is a gem that implements an encoder for files posted to Rails as XML: https://github.com/nragaz/encoded_attachment
I'm using restclient for a multipart form to send data to a restful web service (it's Panda video encoding service).
The trick though, is that the file I am passing into restclient (Technoweenie branch) is coming from my own form that a user submits.
So, lets walk through this. A user posts a file to my rails app. In my controller, it receives the file from params[:file]. I then want to pass params[:file] down to Panda using RestClient.
The error I'm getting is on the Panda server follows. I noticed that the file param in the stack trace is in a string as well (which I assume is Panda turning into a string for a nicer stacktrace).
~ Started request handling: Wed Aug 12 18:05:15 +0000 2009
~ Params: {"format"=>"html", "multipart"=>"true", "account_key"=>"SECURE_KEY", "action"=>"upload", "id"=>"SECURE_ID", "controller"=>"videos", "file"=>"#<File:0xcf02ca4>"}
~ 9bfb1750-6998-012c-4509-12313900b0f6: (500 returned to client) InternalServerErrorcan't convert nil into String
/var/local/www/panda/app/models/video.rb:246:in `extname'
/var/local/www/panda/app/models/video.rb:246:in `initial_processing'
/var/local/www/panda/app/controllers/videos.rb:79:in `upload'
I doubt you can really pass a CGI-style upload param from Rails into restclient and expect it to work.
A regular upload in Rails would have quite some extra attributes which do not belong in a posted resource (like the original filename and so on), and a Rails upload contains an IO with the actual file data. Also a file upload object in Rails might be a Tempfile handle and might be a StringIO - depending on the size of the upload.
What you effectively need to do is "repackage" your upload for rest-client to handle it properly, and pass the repackaged and rewound Tempfile object to restclient. Maybe you can get away with just picking the upload object itself instead of the whole params[:file]
Confirm that your restclient action can save locally first. If the action cannot save locally, then you will have a better idea where to look while trouble shooting.
Looks like the problem is with rest-client's posting of the file, check out an alternative method for posting like curb.
Lots of examples for posting multipart form data on this question: Ruby: How to post a file via HTTP as multipart/form-data?