Ruby Net::HTTP - Stop automatically escaping quotes? - ruby-on-rails

I have the following code:
http = Net::HTTP.new("www.something.com", 80)
http.set_debug_output($stdout)
http.post("/blah", "something", {'random-parameter' => 'value1="something",value2="somethingelse"'})
Then when I read the output from stdout, it looks like this:
<- "POST /blah HTTP/1.1\r\nAccept: */*\r\nContent-Type: application/x-www-form-urlencoded\r\nConnection: close\r\nrandom-parameter: value1=\"something\",value2=\"somethingelse\"\r\nContent-Length: 9\r\nHost: www.something.com\r\n\r\n"
<- "something"
with the quotes being escaped. The problem is that the slashes seem to be sent to the server, which doesn't like that. I'm getting an error that says
Unknown value for random-parameter in header: {value1=\\\"something\\\"value2=\\\"somethingelse\\\"}
So my question is, is there a way to tell Net::HTTP to not insert those slashes, or strip them out before sending the header?
Clarifications:
I'm using Ruby 1.8.7 with Rails 2.0.2.
I think it may be Rails that is escaping the characters, but I'm not sure how to make it stop.

Are you sure you're building up the header correctly? Net::HTTP does not quote quotes when sending the request. You can easily verify it by using for example netcat (nc):
Terminal 1:
> nc -v -l -p 2323
Terminal 2 (in irb):
> http = Net::HTTP.new("localhost", 2323)
> http.post("/blah", "something", {'random-parameter' => ... )
Result (in terminal 1):
listening on [any] 2323 ...
connect to [127.0.0.1] from localhost [127.0.0.1] 37598
POST /blah HTTP/1.1
Connection: close
Accept: */*
Random-Parameter: value1="something",value2="somethingelse"
Content-Type: application/x-www-form-urlencoded
Content-Length: 9
Host: localhost:2323
something
I think what you actually may want to do (not sure, but I'm guessing) is something more along the lines of the HTTP spec:
> http.post("/blah", "something", {
'random-parameter' => 'value1="something"; value2="somethingelse"' })
Rails is probably interpreting your first value1=... as the whole value..when you probably need to separate the values with ';', not ','.
Also note that you don't usually pass parameters through request headers. But maybe that's what you want to do in this case(?) Otherwise you should either pass the parameters in the URL itself like param1=foo&param2=bar, or use x-www-form-urlencoded to pass the parameters.
See here for a cheat sheet:
http://www.rubyinside.com/nethttp-cheat-sheet-2940.html

Isn't the problem that your name/value pairs are the value for "random-parameter", hence they need to be escaped.
I'd expect that when you inspect the random-parameter value in the controller method that it would not have the slashes - can you debug the code or print the parameter to see whats being received?

Related

Why do I need to check both params and request.headers?

I have an api method in a Rails controller like the following:
def login
if !request || !request.headers
render :json => {:error => I18n.t('error_must_provide_api_key_or_token')}, :status => :unauthorized and return
end
api_key = params['X-Api-Key']
if api_key.nil?
api_key = request.headers['X-Api-Key']
end
... rest of method ...
The method first checks to see if we have a request, and whether that request has headers. Then, what I want to do is check for a header variable called X-Api-Key. I first check the params hash, and, if there isn't one found there, I then check the request.headers hash.
What I don't understand is why I have to check both of these. Previously, I had:
api_key = request.headers['X-Api-Key']
This works when I'm debugging on my local machine, but it doesn't work once I push to my production server and run in production mode. Conversely, the following:
api_key = params['X-Api-Key']
Works when I push to the production server, but doesn't work when running locally.
My local machine is running MacOS, and rbenv 1.1.0 with ruby 2.4.0p0 and rails 5.1.1.
My server is Ubuntu 16.04 and running ruby 2.1.4p265 with rails 4.2.5.
Request headers are coming from the http headers.
While params are from the body of the HTTP (if its not GET method), or from the encoded url (e.g: something like http://localhost:3000/cars?a=b, then params[:a] would give 'b').
e.g (using curl, -H stands for header)
using url encoded for the params,
curl -H 'X-Api-Key: 1234' http://localhost:3000/cars?X-Api-Key=abc
or using http request body, with application/json as content-type.
curl -X GET -H 'X-Api-Key: 1234' -H "Content-Type: application/json" -d '{"X-Api-Key": "abc" }' http://localhost:3000/cars
Then, request.headers['X-Api-Key] would returns 1234
while params['X-Api-Key'] would returns abc.
So what happened here is likely, in local you are sending the X-Api-Key, through http header, while in production you are sending it through url encoded or http request body as content-type as json.
To solve this,
if you want to support both ways, to feed X-Api-Key
through http header
http encoded params or json params in the http body
Then what you wrote above make sense.
if you want to only support http header, then change your production client code, to send X-Api-Key through http request header.
if you want to only support params, then change your local code, to send X-Api-Key, through http request body or encoded url.

Ruby Telnet: How to send http request using telnet

I am using ruby telnet library to make HTTP get request(http://127.0.0.1:3000/test) but i am not able to make http get request to my server.
Below is the code that i am trying
require 'net/telnet'
webserver = Net::Telnet::new('Host' => '127.0.0.1', 'Port' => 3000, 'Telnetmode' => false)
size = 0
webserver.cmd("GET / HTTP/1.1\nHost: 127.0.0.1/test") do |c|
print c
end
Please let me know what wrong i am doing here.
You need to end your input HTTP with a carriage return and line ending, otherwise the HTTP server will wait for more headers:
webserver.cmd("GET /test HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n") do |c|
print c
end
But telnet really isn't the right thing to use (unless you're just experimenting). If you want to make HTTP requests for a real-world program, you should definitely use a proper HTTP library (net/http at the least, or something like Faraday would be even better). HTTP seems simple, but there are many hidden complexities that mean creating a writer/parser from scratch is a lot of work.

Content-Type for one file in multipart/form-data with HTTPie

I use HTTPie to POST a
multipart/form-data request (passing the -f option). It includes a
file field (using a # option). The corresponding part of the
multipart request has a pseudo-header Content-Disposition, but does
not have a pseudo-header Content-Type.
How is it possible to add such a Content-Type for a specific file?
For completeness, here is what I send now (as shown by -p B):
--xyz
Content-Disposition: form-data; name="file"; filename="file.txt"
Hello, world!
This is the content of the file.
--xyz
but here is what I need to send:
--xyz
Content-Disposition: form-data; name="file"; filename="file.txt"
Content-Type: text/plain
Hello, world!
This is the content of the file.
--xyz
Or to put it another way, like the CURL equivalent option: -F "file=file.txt;type=text/plain".
This worked for me:
http -f POST :8080/submit-file metadata=#metadata.json files#file1.jpeg files#file1.jpeg
This documentation helped me: https://httpie.org/doc#file-upload-forms
The =# makes httpie send it with content type text/plain.
This isn't possible with httpie. There is an issue for it, with some pull requests to fix it, but none have been merged yet.
I have just noticed that the documentation for multipart/form-data now reads like the following (at the end):
When uploading files, their content type is inferred from the file
name. You can manually override the inferred content type:
$ http -f POST httpbin.org/post name='John Smith' cv#'~/files/data.bin;type=application/pdf'
I do not know when that happened, but it seems that now:
the "part Content-Type" is inferred automatically
it is possible to override it from the command line option
So all I was looking for! :-)
Ref: https://httpie.org/docs#file-upload-forms.

How to debug HTTP AUTH params in Rails?

Rubyists,
something's wrong with my HTTP AUTH params that are coming into my Rails 3 app. The password has some whitespace at the end. I was debugging my client app and it looks like it is sending it correctly.
I am doing this in my app:
params[:auth_username], params[:auth_password] = user_name_and_password(request)
Then I am sending this into Warden.
I would like to see the raw data to see if the whitespace is there. How to do that?
Edit: I have debugged the wire between httpd and thin process and I am pretty sure the data are coming correctly. It must be something wrong in my Rails 3.0.10. I was able to decode the base64 string that is coming in the headers and it did not contain any whitespace.
This really looks like BASE64 decoder issue. Maybe a padding problem. My string is:
Qmxvb21iZXJnOnRjbG1lU1JT
which decodes to
Bloomberg:tclmeSRS
correctly using non-Ruby base64 decoders. Even in Ruby:
>> Base64.decode64 "Qmxvb21iZXJnOnRjbG1lU1JT"
=> "Bloomberg:tclmeSRS"
I don't get it. Searching for a bugreport in Rails or something like that.
Edit: So it turns out our Apache httpd proxy adds something to the header:
Authorization: Basic Qmxvb21iZXJnOnRjbG1lU1JT, Basic
This leads to the incorrect characters at the end of the password, because:
>> Base64.decode64('Basic Qmxvb21iZXJnOnRjbG1lU1JT, Basic'.split(' ', 2).last || '')
=> "Bloomberg:tclmeSRS\005\253\""
The question is now - is this correct? Is it a bug in httpd or rails?
Rails user_name_and_password method makes a call to decode_credentials that performs the following, then splits using ":" :
::Base64.decode64(request.authorization.split(' ', 2).last || '')
Applied to your data :
::Base64.decode64("Qmxvb21iZXJnOnRjbG1lU1JT".split(' ', 2).last || '').split(/:/, 2)
=> ["Bloomberg", "tclmeSRS"]
Everything seems to be ok, the problem sits elsewhere IMO. To dump the authorization data from your controller :
render :text => "Authorization: #{request.authorization}"

Nested model parameters during an HTML post

I'm been playing with calling my rails controller using an HTTP POST. I can get it to work with a curl command such as this, given a model named item and an attribute in that item called name:
curl -X POST -d "<item><name>myname</name></item>" -H "Content-Type: text/xml" http://localhost:3000/items.xml
What I'm curious about is how to make the same call using text instead of xml as my content type.... I tried:
curl -X POST -d "name=myname" http://localhost:3000/items.xml
but that seems to pass the 'name' parameter as the top level scope; so it doesn't end up in my params in the controller....
I'm a noob at this; just want to understand how to do it both ways....
Thanks!
Short answer, you need to
(a) send it as -H "Content-type: application/x-www-form-urlencoded".
(b) specify the parameters as Object[field]=value - for example User[name] would refer to the name field in some user object.
(c) [Not Required because you use curl] encode the parameters and POST/PUT them.
Encoding the parameters
The encoding is simple enough and although curl will do it for you, it's useful to understand it. I've quoted an extract from the W3C spec...
"
application/x-www-form-urlencoded
This is the default content type. Forms submitted with this content type must be encoded as follows:
Control names and values are escaped. Space characters are replaced by +', and then reserved characters are escaped as described in [RFC1738], section 2.2: Non-alphanumeric characters are replaced by%HH', a percent sign and two hexadecimal digits representing the ASCII code of the character. Line breaks are represented as "CR LF" pairs (i.e., `%0D%0A').
The control names/values are listed in the order they appear in the document. The name is separated from the value by =' and name/value pairs are separated from each other by&'.
"
Simple Example (New User Form)
The example below is how to send a simple "New User" form.
If I have fields in the user object for name, password, email etc, I specify them like this ...
user[firstname]=chris&user[login]=cmccauley&user[company_id]=8&user[email]=&user[surname]=mccauley
then curl will escape them to give ...
user%5Bfirstname%5D=chris&user%5Blogin%5D=cmccauley&user%5Bcompany_id%5D=8&user%5Bemail%5D=&user%5Bsurname%5D=mccauley
... before posting them like this extract from a wire dump ...
Wireshark dump
POST /users/19 HTTP/1.1
Host: localhost:3000
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.10) Gecko/2009042523 Ubuntu/9.04 (jaunty) Firefox/3.0.10
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://localhost:3000/users/19/edit
Cookie: _my_session=048d330143de668e027c8cd52654e8c5
Content-Type: application/x-www-form-urlencoded
Content-Length: 259
user%5Bfirstname%5D=chris&user%5Blogin%5D=cmccauley&user%5Bcompany_id%5D=8&user%5Bemail%5D=&user%5Bsurname%5D=mccauley&user_password=********&user%5Bjob_id%5D=14&user%5Bpassword%5D=dd793a64b74e108fcdc5d809040e24afcc21ad2c&authenticity_token=&id=19&_method=PUT
"Content-type: application/x-www-form-urlencoded" is already default with curl -d, no need for anything extra
-X POST is superfluous, as -d implies POST
-d does not URL encode the data. To get curl to do that for you, you need to use --data-urlencode instead

Resources