I'm using commandline curl to do a POST via a proxy but my form data is vanishing to zero content length. Any ideas what I'm doing wrong?
Here's my command line (uses a public test form so others can try it):
curl -v --proxy-ntlm --proxy proxyserver:proxyport --proxy-user : -d "fname=a&lname=b" http://www.snee.com/xml/crud/posttest.cgi
-v = verbose
next few arguments get us through the proxy using windows authentication
-d = should do a post with the given arguments
However, both the response and the verbose print out suggest the form content is vanishing. The curl prints "Content-Length: 0" and the returned html has both arguments missing and a content length of 0.
The bug doesn't seem to be in the proxy server as curl admits it is sending a content length of 0. Does anyone know a solution to this problem? Has anyone else seen it?
Update: this person appears to have the same bug, but no solution suggested, apart from not using ntlm which I have to
Update 2: This definitely only happens with NTLM authentication, I've tried an alternative authentication method which works. Also, using -F instead of -d (for binary form data) fails in the same way.
Update 3 (workaround): I've had a bit of discussion on the curl-users list about this. A workaround was provided which is to use --proxy-anyauth instead of --proxy-ntlm. I'm still investigating the problem but this workarounf works for me.
NTLM is a challenge-response protocol. When you indicate that you're going to use NTLM, a client will first send a request without the body (to avoid wasting the bandwidth of sending the body only to have it rejected by the HTTP/401 challenge from the server). Only once the Challenge/Response protocol is complete will the body actually be posted.
This causes a number of problems in cases where the client expects NTLM but the proxy or server has no idea (and thus acts on the 0-byte POST, never challenging the client).
I was having this problem making a request using HTTP digest. Eric is correct, curl is trying to be clever and not post any data because it knows it will have to make the request again after it receives the challenge from the server.
It turns out if you provide the --anyauth option (which asks curl to autodetect the authentication method), the initial request will include all the POST data, and (in my case) the server responded as expected.
Related
I'm technical, but not experienced in coding and could use some help. I need to create an http request from Twilio studio. I am setting up a phone survey for my client and I need to log both voice and number inputs from the call to a database. I already have the database set up externally. It logs responses with an http request from CLI curl. I'm using the following curl request successfully:
curl --data "q=1 -X POST localhost/test.php
When I try to duplicate the same curl request using the http request widget (no http parameters), I only get error 500 back. By the way, I know it's collecting the data correctly with the flow because I've had it collect and then read back to me successfully.
Twilio studio http request.
I have tried adding http parameters instead of putting them in the body as well. Nothing I try works. My guess is that I have a fundamental misunderstanding of http requests and thus can't duplicate it in Twilio.
Any help you could give me would be great! Thank you!
When you add http parameters instead of putting them in the body (see the capture below)
there is a big red "Save" button on bottom left corner, and a gray (hard to see and easy to miss) "Save" link on the right, above the trash bin.
The gray "Save" link needs to be clicked when adding each name/value pair of the parameters.
The red "Save" button is to be clicked at the end when you're done with the http widget.
When I enter an URL like this, with ..
http://SERVER:8085/../tn/d9dd6c39d71276487ae798d976f8f629_tn.jpg
I obtain a request in my Web-Server without ..-part
Does Firefox remove it silently? Are the .. not allowed in URLs?
P.S.: wget removes .. also :-(
I have recently begun seeing this and, despite what the marked answer states, adding this to a URL does make sense and is a valid folder path in the world of IT security where we intetionally bypass security measures in mis-configured sites, classified as Directory Traversal attacks.
Web (browsers, wget, curl, etc...) tools silently evaluate the URL path and strip out the "/../" making my job of finding vulnerabilities more difficult. To get around this, I use Firefox along with Burpsuite, a proxying assessment tool that captures the request and allows me to modify it before sending to the server.
Doing this, I can type:
https://example.com/vpn/../vpns/cfg/etc
in my browser URL, and when I capture it using Burpsuite, it looks like:
https://example.com/vpns/cfg/etc
showing me that Firefox has in fact changed my original intended URL string. So within Burpsuite, I modify the request back to say:
GET /vpn/../vpns/cfg/etc HTTP/1.1
send it to the server, and voila, the path remains intact and navigates to the correct location. Yes, in a normal well-configured application with proper request handling, doing this shouldn't be necessary. This particular string acts differently in these 2 formats, so modifying it necessary to cause the server to handle it in the manner we want to show there is a configuration problem with how the application handles the request (a Directory Traversal vulnerability).
This can also be proven using curl. If you send a normal curl command like below, curl will do the same as Firefox and evaluate the path, removing "/vpn/.." from it before sending to the server:
curl -i -s -k "https://example.com/vpn/../vpns/cfg/etc"
However, if you add the "--path-as-is" argument, curl will not modify it and send it as-is, and the "/vpn/.." remains intact:
curl -i -s -k "https://example.com/vpn/../vpns/cfg/etc" --path-as-is
After some additional reading, I found this behavior is due in part to URI Normalization standards (https://en.wikipedia.org/wiki/URI_normalization).
This points to RFC 3986 for defining URI Syntax https://www.rfc-editor.org/rfc/rfc3986#section-5.2.4.
".." means a relative path and used for moving up in the hierarchy. So ".." is not a valid name for a folder therefore you cannot use it in the middle of URL. It just makes no sense.
So to answer your question: ".." is allowed in url but only in the beginning.
Complementary information:
"../" will be stripped by the developer tools as well (up to 54.0.1 at least), meaning you cannot use the "Edit and resend" to hand-craft a valid request like this:
GET /../tn/d9dd6c39d71276487ae798d976f8f629_tn.jpg
... which could potentially result in a directory traversal and the file being retrieved.
I'm looking at HTTP POST method to create a web page.
I'm a tad unfamiliar with it, and was looking for some explanation on how this works.
I'm simply wanting to use their seemingly simplistic page for my own server needs.
My question is this, how does this curl link really work?
curl --data-binary #audio2.flac --header 'Content-type: audio/x-flac; rate=16000' 'https://www.google.com/speech-api/v1/recognize?xjerr=1&client=chromium&pfilter=0&maxresults=1&lang="en-US"'
So I got it to work by using curl, by sending a flac file as you can see, and it responds with
this JSon stuff
{"status":0,"id":"3b7853241a0dded048f84744cc1ab896-1","hypotheses":[{"utterance":"ice cream","confidence":0.88215643}]}
Please correct me if I'm wrong:
So I believe this is a post method as it has the ? mark with it to send the name value pair.
I have no idea how they got the link https://www.google.com/speech-api/v1/recognize
to only accept POST requests without having a script name like this at the end
https://www.google.com/speech-api/v1/recognize/scriptName.pl
How do I write something to accept a post request at that given page without a .pl or .py extension? How can I put it in a certain directory on the server.
In addition for my purposes can I also send a file back to them once I process / zip?
Any ideas on how to do that?
Thanks so much for your time and generous help!
It seems that if you use --data/--data-binary/--form that the request is POSTed.
I first totally misunderstood your question and though you wanted to know how to use the interface, but you got that working.
The serverside can be achieved by multiple means, totally depending on the infrastructure and software used.
With the Webserver apache for example, you could use Rewrite or something in a HtAccess file to obfuscate/change the requested file. So looking at the extension (or the absence of it) is not helpful.
Please also see various options mentioned here
Running PHP without extension without using mod_rewrite?
or
ForceType/htaccess file extension question - extensionless files?
to see how you could achieve the same.
Only accepting POST data is then again dependant on the programming/scripting technique you employ at your server side, PHP e.g. distinguishes between multiple superglobals like $_GET and $_POST.
I'm trying to create a restful api capable of receiving a file.
http://leejava.wordpress.com/2009/07/30/upload-file-from-rest-in-ruy-on-rail-with-json-format/
talks of how to accomplish this simply.
Not being very used to rails or curl, I don't know how to formulate my curl call.
Could anyone help me. Explain how to use curl to send a file, and how to use rails to accept and store the file.
Note.. the file is a 10 meg zip file. For testing I called it.
C:\test.zip..
I've looked at tutorials, but they all seem to be web page based. They also all degrade to resizing pictures.. Like uploading and presenting images are the same topic.
You need to know a bit about how your server-side is expecting the data. If it is a http POST with multipart/form-data (which I think it is), then this will probably help:
curl -F "fileParam=#c:\test.zip" -F "param1=xxx" -F "param2=yyyy" http://my.server/post-service
This will use curl to post form-data (including a file) to the given url with the given parameters. You need to know the parameter names for the file and the other optional parameters, and you have to obviously get the URL correct to reach your service.
Hope that helps.
I am trying to understand how Google authentication works with OAuth 2.0. They give some examples at this link.
I am having trouble with one of the first steps. I can execute this http request in a browser:
https://accounts.google.com/o/oauth2/auth?scope=https://www.google.com/m8/feeds&client_id=21302922996.apps.googleusercontent.com&redirect_uri=https://www.example.com/back&response_type=token
and that works fine.
When I try to convert it to a curl GET operation,
curl https://accounts.google.com/o/oauth2/auth?scope=https://www.google.com/m8/feeds&client_id=21302922996.apps.googleusercontent.com&redirect_uri=https://www.example.com/back&response_type=token
I get a response that tells me there is an OAuth 2.0 error: invalid_request.
Converting it to a POST, I can't get this operation to work.
curl http://accounts.google.com/o/oauth2/auth -d “scope=https://www.google.com/m8/feeds&client_id=21302922996.apps.googleusercontent.com&redirect_uri=https://www.example.com/back&response_type=token”
The response is that the document has moved.
If I try this, I get similar results.
curl http://accounts.google.com/o/oauth2/auth -data-urlencode “scope=https://www.google.com/m8/feeds&client_id=21302922996.apps.googleusercontent.com&redirect_uri=https://www.example.com/back&response_type=token”
I have never used curl or OAuth in my code before, so I want to understand some of these basics. Can someone tell me what is wrong with my GET and POST ?
I don't have a sure answer for you, but I was working on something similar for Instagram and one thing that stumped me for a while was that my redirect URL looked to be the same as Instagram expected, but I kept getting errors.
Like you, when I 'hand coded' the redirect URL, I got the result I expected, but my automated calls to the authentication URL failed. It turned out my ampersands were a url encoded which Instagram compared to the redirect URL and saw they were not identical. I'm not sure if that will get you closer to an answer, but I thought I'd share my experience since your question reminds me of the situation I was in. Good luck.