Related
According to the Section 3.3, Path Component of RFC2396 - Uniform Resource Identifiers,
The path may consist of a sequence of path segments separated by a single slash "/" character. Within a path segment, the characters "/", ";", "=", and "?" are reserved. Each path segment may include a sequence of parameters, indicated by the semicolon ";" character. The parameters are not significant to the parsing of relative references.
However, I have never seen a URL with a query parameters in any segment other than the final one. So, I am not sure if I am reading this correctly.
Is http://www.url.com/segment1?seg1param1=val1/page.html?pageparam1=val2 a valid URL?
What the RFC is referring to is something like this:
http://www.example.com/foo/bar;param=value/baz.html
That could be interpreted as the path /foo/bar/baz.html with the parameter param=value to the bar segment. No question marks are used.
Note that RFC 2396 has been obsoleted by RFC 3986, which omits specification of segment-specific parameters in favor of a general note that implementations can (and do) do different things to embed segment-specific parameters:
Aside from dot-segments in hierarchical paths, a path segment is
considered opaque by the generic syntax. URI producing applications
often use the reserved characters allowed in a segment to delimit
scheme-specific or dereference-handler-specific subcomponents. For
example, the semicolon (";") and equals ("=") reserved characters are
often used to delimit parameters and parameter values applicable to
that segment. The comma (",") reserved character is often used for
similar purposes. For example, one URI producer might use a segment
such as "name;v=1.1" to indicate a reference to version 1.1 of
"name", whereas another might use a segment such as "name,1.1" to
indicate the same. Parameter types may be defined by scheme-specific
semantics, but in most cases the syntax of a parameter is specific to
the implementation of the URI's dereferencing algorithm.
When you look at the grammar which is just below, it is written:
path = [ abs_path | opaque_part ]
path_segments = segment *( "/" segment )
segment = *pchar *( ";" param )
param = *pchar
pchar = unreserved | escaped |
":" | "#" | "&" | "=" | "+" | "$" | ","
A segment is composed of pchar and param, param being itself a pchar.
When we continue to read, there is absolutely no "?" character in the pchar character components. So the parameters cannot have any "?", and there cannot be a "?" in segments.
So I agree with the answer of Edward Thomson, who says that "?" only delimit the query segment, and cannot be used inside a path.
According to my reading of RFC 2396, no. The ? is a reserved character and serves only to delimit the query segment. The ? is not allowed in either the path or the query segment.
In your example, the first ? marks the beginning of the query segment. The second ? is inside the query segment, and is disallowed.
I believe you could do a get with that and most web servers would process it but I don't believe you would get the results you are expecting. That is the pageparam1=val2 would not evaluate.
If you want parameters like that you could always use the # symbol (as a lot of javascript based GUIs do now).
Background (question further down)
I've been Googling this back and forth reading RFCs and SO questions trying to crack this, but I still don't got jack.
So I guess we just vote for the "best" answer and that's it, or?
Basically it boils down to this.
3.4. Query Component
The query component is a string of information to be interpreted by the resource.
query = *uric
Within a query component, the characters ";", "/", "?", ":", "#", "&", "=", "+", ",", and "$" are reserved.
The first thing that boggles me is that *uric is defined like this
uric = reserved | unreserved | escaped
reserved = ";" | "/" | "?" | ":" | "#" | "&" | "=" | "+" | "$" | ","
This is however somewhat clarified by paragraphs such as
The "reserved" syntax class above refers to those characters that are allowed within a URI, but which may not be allowed within a particular component of the generic URI syntax; they are used as delimiters of the components described in Section 3.
Characters in the "reserved" set are not reserved in all contexts. The set of characters actually reserved within any given URI component is defined by that component. In general, a character is reserved if the semantics of the URI changes if the character is replaced with its escaped US-ASCII encoding.
This last excerpt feels somewhat backwards, but it clearly states that the reserved character set depends on context. Yet 3.4 states that all the reserved characters are reserved within a query component, however, the only things that would change the semantics here is escaping the question mark (?) as URIs do not define the concept of a query string.
At this point I've given up on the RFCs entirely but found RFC 1738 particularly interesting.
An HTTP URL takes the form:
http://<host>:<port>/<path>?<searchpart>
Within the <path> and <searchpart> components, "/", ";", "?" are reserved. The "/" character may be used within HTTP to designate a hierarchical structure.
I interpret this at least with regards to HTTP URLs that RFC 1738 supersedes RFC 2396. Because the URI query has no notion of a query string also the interpretation of reserved doesn't really let allow me to define query strings as I'm used to doing by now.
Question
This all started when I wanted to pass a list of numbers together with the request of another resource. I didn't think much of it, and just passed it as a comma separated values. To my surprise though the comma was escaped. The query page.html?q=1,2,3 encoded turned into page.html?q=1%2C2%2C3 it works, but it's ugly and didn't expect it. That's when I started going through RFCs.
My first question is simply, is encoding commas really necessary?
My answer, according to RFC 2396: yes, according to RFC 1738: no
Later I found related posts regarding the passing of lists between requests. Where the csv approach was poised as bad. This showed up instead, (haven't seen this before).
page.html?q=1;q=2;q=3
My second question, is this a valid URL?
My answer, according to RFC 2396: no, according to RFC 1738: no (; is reserved)
I don't have any issues with passing csv as long as it's numbers, but yes you do run into the risk of having to encode and decode values back and forth if the comma suddenly is needed for something else. Anyway I tried the semi-colon query string thing with ASP.NET and the result was not what I expected.
Default.aspx?a=1;a=2&b=1&a=3
Request.QueryString["a"] = "1;a=2,3"
Request.QueryString["b"] = "1"
I fail to see how this greatly differs from a csv approach as when I ask for "a" I get a string with commas in it. ASP.NET certainly is not a reference implementation but it hasn't let me down yet.
But most importantly -- my third question -- where is specification for this? and what would you do or for that matter not do?
That a character is reserved within a generic URL component doesn't mean it must be escaped when it appears within the component or within data in the component. The character must also be defined as a delimiter within the generic or scheme-specific syntax and the appearance of the character must be within data.
The current standard for generic URIs is RFC 3986, which has this to say:
2.2. Reserved Characters
URIs include components and subcomponents that are delimited by characters in the "reserved" set. These characters are called "reserved" because they may (or may not) be defined as delimiters by the generic syntax, by each scheme-specific syntax, or by the implementation-specific syntax of a URI's dereferencing algorithm. If data for a URI component would conflict with a reserved character's purpose as a delimiter [emphasis added], then the conflicting data must be percent-encoded before the URI is formed.
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "#"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
3.3. Path Component
[...]
pchar = unreserved / pct-encoded / sub-delims / ":" / "#"
[...]
3.4 Query Component
[...]
query = *( pchar / "/" / "?" )
Thus commas are explicitly allowed within query strings and only need to be escaped in data if specific schemes define it as a delimiter. The HTTP scheme doesn't use the comma or semi-colon as a delimiter in query strings, so they don't need to be escaped. Whether browsers follow this standard is another matter.
Using CSV should work fine for string data, you just have to follow standard CSV conventions and either quote data or escape the commas with backslashes.
As for RFC 2396, it also allows for unescaped commas in HTTP query strings:
2.2. Reserved Characters
Many URI include components consisting of or delimited by, certain
special characters. These characters are called "reserved", since
their usage within the URI component is limited to their reserved
purpose. If the data for a URI component would conflict with the
reserved purpose, then the conflicting data must be escaped before
forming the URI.
Since commas don't have a reserved purpose under the HTTP scheme, they don't have to be escaped in data. The note from § 2.3 about reserved characters being those that change semantics when percent-encoded applies only generally; characters may be percent-encoded without changing semantics for specific schemes and yet still be reserved.
I think the real question is: "What characters should be encoded in a query string?" And that depends mainly on two things: The validity and the meaning of a character.
Validity according to the RFC standard
In RFC3986 we can find which special characters are valid and which are not inside a query string:
// Valid:
! $ & ' ( ) * + , - . / : ; = ? # _ ~
% (should be followed by two hex chars to be completely valid (e.g. %7C))
// Invalid:
" < > [ \ ] ^ ` { | }
space
# (marks the end of the query string, so it can't be a part of it)
extended ASCII characters (e.g. °)
Deviations from the standard
Browsers and web frameworks do not always strictly follow the RFC standard. Below are some examples:
[, ] are not valid, but Chrome and Firefox do not encode these characters inside a query string. The reasoning given by Chrome devs is simply: "If other browsers and an RFC disagree, we will generally match other browsers." QueryHelpers.AddQueryString from ASP.NET Core on the other hand will encode these characters.
Other invalid characters that are not encoded by Chrome and Firefox are:
\ ^ ` { | }
' is a valid character inside a query string but will be encoded by Chrome, Firefox and QueryHelpers nevertheless. The explanation given by Firefox devs is that they knew that they don't have to encode it according to the RFC standard, but did it to reduce vulnerabilities.
Special meaning
Some characters are valid and also don't get encoded by browsers, but should still be encoded in certain cases.
+: Spaces are normally encoded as %20 but alternatively they can be encoded as +. So + inside a query string means it's an encoded space. If you want to include a character that's actually supposed to literally mean plus, then you have to use the encoded version of + which is %2B.
~: Some old Unix systems interpreted URI parts that started with ~ as a path to a home directory. So it's a good idea to encode ~ if it's not meant to denote the start of a Unix home directory path for an old system (so nowadays probably always encode).
=, &: Usually (although RFC doesn't specify that this is required) query strings contain parameters in the format "key1=value1&key2=value2". If that's the case and =s or &s should be part of the parameter key or the parameter value instead of giving them the role of separating the key and value or separating the parameters, then you have to encode those =s and &s. So if a parameter value should for some reason consist of the string "=&" then it has to be encoded as %3D%26 which then can be used for the full key and value: "weirdparam=%3D%26".
%: Usually web frameworks figure out that %s that are not followed by two hex characters simply mean the % itself, but it's still a good idea to always encode % when it's supposed to only mean % and not indicate the start of an encoded character (e.g. %7C) because RFC3986 specifies that % is only valid when followed by two hex characters. So don't use "percentageparam=%" use "percentageparam=%25" instead.
Encoding guidelines
Encode every character that is otherwise invalid* according to RFC3986 and every character that can have special meaning but should only be interpreted in a literal way without giving it a special meaning. You can also encode things that aren't required to be encoded, like '. Why? Because it doesn't hurt to encode more than necessary. Servers and web frameworks when parsing a query string will decode every encoded character, no matter if it was really necessary to previously encode that character or not.
The only characters of a query string that shouldn't be encoded are those that can have a special meaning and shouldn't lose that special meaning, e.g. don't encode the = of "key1=value1". For that to achieve don't apply an encoding method to the whole query string (and also not to the whole URI) but apply it only and separately to the query parameter keys and values. For example, with JS:
var url = "http://example.com?" + encodeURIComponent(myKey1) + "=" + encodeURIComponent(myValue1) + "&" + encodeURIComponent(myKey2)...;
Note that encodeURIComponent encodes a lot more characters than necessary meaning characters that are valid in a query string and don't have special meaning there e.g. /, ?, ...
The reason is that encodeURIComponent wasn't created for query strings alone but instead encodes characters that have special meaning outside of the query string as well, e.g. / for the path URI component. QueryHelpers.AddQueryString works in a similar manner. Under the hood it uses System.Text.Encodings.Web.DefaultUrlEncoder which is not just meant for query strings but also for isegment, ipath-noscheme and ifragment.
* You could probably get away with only regarding those characters as invalid that are both not allowed by the RFC and that are also always encoded by Chrome for instance. This would be Space " < >. But it's probably better to be on the safer side and encode at least everything that RFC3986 considers invalid.
OP's questions
My first question is simply, is encoding commas really necessary -> No it's not necessary, but it doesn't hurt (except ugliness) and will happen with default encoding methods e.g. encodeURIComponent and decoding and query string parsing should work nevertheless.
My second question, is this a valid URL (page.html?q=1;q=2;q=3)? -> It's RFC valid, but your server / web framework might have a hard time parsing the query string when it might expect the typical "key1=value1&key2=value2" format for query strings.
Where is specification for this? -> There isn't a single specification that covers everything because some things are implementation specific. For instance there are different ways of specifying arrays inside of query strings.
Just use ?q=1+2+3
I am answering here a fourth question :) that did not ask but all started with: how do i pass list of numbers a-la comma-separated values? Seems to me the best approach is just to pass them space-separated, where spaces will get url-form-encoded to +. Works great, as longs as you know the values in the list contain no spaces (something numbers tend not to).
page.html?q=1;q=2;q=3
is this a valid URL?
Yes. The ; is reserved, but not by an RFC. The context that defines this component is the definition of the application/x-www-form-urlencoded media type, which is part of the HTML standard (section 17.13.4.1). In particular the sneaky note hidden away in section B.2.2:
We recommend that HTTP server implementors, and in particular, CGI implementors support the use of ";" in place of "&" to save authors the trouble of escaping "&" characters in this manner.
Unfortunately many popular server-side scripting frameworks including ASP.NET do not support this usage.
I would like to note that page.html?q=1&q=2&q=3 is a valid url as well. This is a completely legitimate way of expressing an array in a query string. Your server technology will determine how exactly that is presented.
In Classic ASP, you check Response.QueryString("q").Count and then use Response.QueryString("q")(0) (and (1) and (2)).
Note that you saw this in your ASP.NET, too (I think it was not intended, but look):
Default.aspx?a=1;a=2&b=1&a=3
Request.QueryString["a"] = "1;a=2,3"
Request.QueryString["b"] = "1"
Notice that the semicolon is ignored, so you have a defined twice, and you got its value twice, separated by a comma. Using all ampersands Default.aspx?a=1&a=2&b=1&a=3 will yield a as "1,2,3". But I am sure there is a method to get each individual element, in case the elements themselves contain commas. It is simply the default property of the non-indexed QueryString that concatenates the sub-values together with comma separators.
I had the same issue. The URL that was hyperlinked was a third party URL and was expecting a list of parameters in format page.html?q=1,2,3 ONLY and the URL page.html?q=1%2C2%2C3 did not work. I was able to get it working using javascript. May not be the best approach but can check out the solution here if it helps anyone.
If you are sending the ENCODED characters to FLASH/SWF file, then you should ENCODE the character twice!! (because of Flash parser)
When is a space in a URL encoded to +, and when is it encoded to %20?
From Wikipedia (emphasis and link added):
When data that has been entered into HTML forms is submitted, the form field names and values are encoded and sent to the server in an HTTP request message using method GET or POST, or, historically, via email. The encoding used by default is based on a very early version of the general URI percent-encoding rules, with a number of modifications such as newline normalization and replacing spaces with "+" instead of "%20". The MIME type of data encoded this way is application/x-www-form-urlencoded, and it is currently defined (still in a very outdated manner) in the HTML and XForms specifications.
So, the real percent encoding uses %20 while form data in URLs is in a modified form that uses +. So you're most likely to only see + in URLs in the query string after an ?.
This confusion is because URLs are still 'broken' to this day.
From a blog post:
Take "http://www.google.com" for instance. This is a URL. A URL is a Uniform Resource Locator and is really a pointer to a web page (in most cases). URLs actually have a very well-defined structure since the first specification in 1994.
We can extract detailed information about the "http://www.google.com" URL:
+---------------+-------------------+
| Part | Data |
+---------------+-------------------+
| Scheme | http |
| Host | www.google.com |
+---------------+-------------------+
If we look at a more complex URL such as:
"https://bob:bobby#www.lunatech.com:8080/file;p=1?q=2#third"
we can extract the following information:
+-------------------+---------------------+
| Part | Data |
+-------------------+---------------------+
| Scheme | https |
| User | bob |
| Password | bobby |
| Host | www.lunatech.com |
| Port | 8080 |
| Path | /file;p=1 |
| Path parameter | p=1 |
| Query | q=2 |
| Fragment | third |
+-------------------+---------------------+
https://bob:bobby#www.lunatech.com:8080/file;p=1?q=2#third
\___/ \_/ \___/ \______________/ \__/\_______/ \_/ \___/
| | | | | | \_/ | |
Scheme User Password Host Port Path | | Fragment
\_____________________________/ | Query
| Path parameter
Authority
The reserved characters are different for each part.
For HTTP URLs, a space in a path fragment part has to be encoded to "%20" (not, absolutely not "+"), while the "+" character in the path fragment part can be left unencoded.
Now in the query part, spaces may be encoded to either "+" (for backwards compatibility: do not try to search for it in the URI standard) or "%20" while the "+" character (as a result of this ambiguity) has to be escaped to "%2B".
This means that the "blue+light blue" string has to be encoded differently in the path and query parts:
"http://example.com/blue+light%20blue?blue%2Blight+blue".
From there you can deduce that encoding a fully constructed URL is impossible without a syntactical awareness of the URL structure.
This boils down to:
You should have %20 before the ? and + after.
Source
I would recommend %20.
Are you hard-coding them?
This is not very consistent across languages, though.
If I'm not mistaken, in PHP urlencode() treats spaces as + whereas Python's urlencode() treats them as %20.
EDIT:
It seems I'm mistaken. Python's urlencode() (at least in 2.7.2) uses quote_plus() instead of quote() and thus encodes spaces as "+".
It seems also that the W3C recommendation is the "+" as per here: http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4.1
And in fact, you can follow this interesting debate on Python's own issue tracker about what to use to encode spaces: http://bugs.python.org/issue13866.
EDIT #2:
I understand that the most common way of encoding " " is as "+", but just a note, it may be just me, but I find this a bit confusing:
import urllib
print(urllib.urlencode({' ' : '+ '})
>>> '+=%2B+'
A space may only be encoded to "+" in the "application/x-www-form-urlencoded" content-type key-value pairs query part of an URL. In my opinion, this is a may, not a must. In the rest of URLs, it is encoded as %20.
In my opinion, it's better to always encode spaces as %20, not as "+", even in the query part of an URL, because it is the HTML specification (RFC 1866) that specified that space characters should be encoded as "+" in "application/x-www-form-urlencoded" content-type key-value pairs (see paragraph 8.2.1. subparagraph 1.)
This way of encoding form data is also given in later HTML specifications. For example, look for relevant paragraphs about application/x-www-form-urlencoded in HTML 4.01 Specification, and so on.
Here is a sample string in a URL where the HTML specification allows encoding spaces as pluses: "http://example.com/over/there?name=foo+bar". So, only after "?", spaces can be replaced by pluses. In other cases, spaces should be encoded to %20. But since it's hard to determine the context correctly, it's the best practice to never encode spaces as "+".
I would recommend to percent-encode all character except "unreserved" defined in RFC 3986, p.2.3
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
The implementation depends on the programming language that you chose.
If your URL contains national characters, first encode them to UTF-8 and then percent-encode the result.
To summarize the (somewhat conflicting) answers here, I think it can be boiled down to:
| standard | + | %20 |
|---------------+-----+-----|
| URL | no | yes |
| query string | yes | yes |
| form params | yes | no |
| mailto query | no | yes |
So historically I think what happened is:
The RFC specified a pretty clear standard about the form of URLs and how they are encoded. In this context the query is just a "string", there is no specification how key/value pairs should be encoded
The HTTP guys put out a standard of how key/value pairs are to be encoded in form params, and borrowed from the URL encoding standard, except that spaces should be encoded as +.
The web guys said: cool we have a way to encode key/value pairs let's put that into the URL query string
Result: We end up with two different ways how to encode spaces in a URL depending on which part you're talking about. But it doesn't even violate the URL standard. From URL perspective the "query" is just a blackbox. If you want to use other encodings there besides percent encoding: knock yourself out.
But as the email example shows it can be problematic to borrow from the form-params implementation for an URL query string. So ultimately using %20 is safer but there might not be out of the box library support for it.
The RFC 3986 URI: Generic Syntax specification lists a semicolon as a reserved (sub-delim) character:
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "#"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="
What is the reserved purpose of the ";" of the semicolon in URIs? For that matter, what is the purpose of the other sub-delims (I'm only aware of purposes for "&", "+", and "=")?
There is an explanation at the end of section 3.3.
Aside from dot-segments in
hierarchical paths, a path segment is
considered opaque by the generic
syntax. URI producing applications
often use the reserved characters
allowed in a segment to delimit
scheme-specific or
dereference-handler-specific
subcomponents. For example, the
semicolon (";") and equals ("=")
reserved characters are often used
to delimit parameters and parameter
values applicable to that segment.
The comma (",") reserved character is
often used forsimilar purposes.
For example, one URI producer might
use a segment uch as "name;v=1.1"
to indicate a reference to version 1.1
of "name", whereas another might
use a segment such as "name,1.1" to
indicate the same. Parameter types
may be defined by scheme-specific
semantics, but in most cases the
syntax of a parameter is specific to
the implementation of the URI's
dereferencing algorithm.
In other words, it is reserved so that people who want a delimited list of something in the URL can safely use ; as a delimiter even if the parts contain ;, as long as the contents are percent-encoded. In other words, you can do this:
foo;bar;baz%3bqux
and interpret it as three parts: foo, bar, baz;qux. If semicolon were not a reserved character, the ; and %3bwould be equivalent, so the URI would be incorrectly interpreted as four parts: foo, bar, baz, qux.
The intent is clearer if you go back to older versions of the specification:
path_segments = segment *( "/" segment )
segment = *pchar *( ";" param )
Each path segment may include a
sequence of parameters, indicated by the semicolon ";" character.
I believe it has its origins in FTP URIs.
Section 3.3 covers this - it's an opaque delimiter a URI-producing application can use if convenient:
Aside from dot-segments in
hierarchical paths, a path segment is
considered opaque by the generic
syntax. URI producing applications
often use the reserved characters
allowed in a segment to delimit
scheme-specific or
dereference-handler-specific
subcomponents. For example, the
semicolon (";") and equals ("=")
reserved characters are often used to
delimit parameters and parameter
values applicable to that segment. The
comma (",") reserved character is
often used for similar purposes. For
example, one URI producer might use a
segment such as "name;v=1.1" to
indicate a reference to version 1.1 of
"name", whereas another might use a
segment such as "name,1.1" to indicate
the same. Parameter types may be
defined by scheme-specific semantics,
but in most cases the syntax of a
parameter is specific to the
implementation of the URI's
dereferencing algorithm.
There are some conventions around its current usage that are interesting. These speak to when to use a semicolon or comma. From the book "RESTful Web Services":
Use punctuation characters to separate multiple pieces of data at the same level of hierarchy. Use commas when the order of the items matters, ... Use semicolons when the order doesn't matter.
Since 2014, path segments are known to contribute to Reflected File Download attacks. Let's assume we have a vulnerable API that reflects whatever we send to it:
https://google.com/s?q=rfd%22||calc||
{"results":["q", "rfd\"||calc||","I love rfd"]}
Now, this is harmless in a browser as it's JSON, so it's not going to be rendered, but the browser will rather offer to download the response as a file. Now here's the path segments come to help (for the attacker):
https://google.com/s;/setup.bat;?q=rfd%22||calc||
Everything between semicolons (;/setup.bat;) will be not sent to the web service, but instead the browser will interpret it as the file name... to save the API response.
Now, a file called setup.bat will be downloaded and run without asking about dangers of running files downloaded from the Internet (because it contains the word "setup" in its name). The contents will be interpreted as a Windows batch file, and the calc.exe command will be run.
Prevention:
sanitize your API's input (in this case, they should just allow alphanumerics); escaping is not sufficient
add Content-Disposition: attachment; filename="whatever.txt" on APIs that are not going to be rendered; Google was missing the filename part which actually made the attack easier
add X-Content-Type-Options: nosniff header to API responses
I found the following use cases:
It's the final character of an HTML entity:
List of XML and HTML character entity references
To use one of these character entity references in an HTML or XML
document, enter an ampersand followed by the entity name and a
semicolon, e.g., & for the ampersand ("&").
Apache Tomcat 7 (or newer versions?!) us it as path parameter:
Three Semicolon Vulnerabilities
Apache Tomcat is one example of a web server that supports "Path
Parameters". A path parameter is extra content after a file name,
separated by a semicolon. Any arbitrary content after a semicolon does
not affect the landing page of a web browser. This means that
http://example.com/index.jsp;derp will still return index.jsp, and not
some error page.
URI scheme splits by it the MIME and data:
Data URI scheme
It can contain an optional character set parameter, separated from the
preceding part by a semicolon (;) .
<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA
AAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO
9TXL0Y4OHwAAAABJRU5ErkJggg==" alt="Red dot" />
And there was a bug in IIS 5 and IIS 6 to bypass file upload restrictions:
Unrestricted File Upload
Blacklisting File Extensions This protection might be bypassed by: ...
by adding a semi-colon character after the forbidden extension and
before the permitted one (e.g. "file.asp;.jpg")
Conclusion:
Do not use semicolons in URLs or they could accidentally produce an HTML entity or URI scheme.
This question already has answers here:
URL encoding the space character: + or %20?
(5 answers)
Closed 6 years ago.
In a URL, should I encode the spaces using %20 or +? For example, in the following example, which one is correct?
www.mydomain.com?type=xbox%20360
www.mydomain.com?type=xbox+360
Our company is leaning to the former, but using the Java method URLEncoder.encode(String, String) with "xbox 360" (and "UTF-8") returns the latter.
So, what's the difference?
Form data (for GET or POST) is usually encoded as application/x-www-form-urlencoded: this specifies + for spaces.
URLs are encoded as RFC 1738 which specifies %20.
In theory I think you should have %20 before the ? and + after:
example.com/foo%20bar?foo+bar
According to the W3C (and they are the official source on these things), a space character in the query string (and in the query string only) may be encoded as either "%20" or "+". From the section "Query strings" under "Recommendations":
Within the query string, the plus sign is reserved as shorthand notation for a space. Therefore, real plus signs must be encoded. This method was used to make query URIs easier to pass in systems which did not allow spaces.
According to section 3.4 of RFC2396 which is the official specification on URIs in general, the "query" component is URL-dependent:
3.4. Query Component
The query component is a string of information to be interpreted by
the resource.
query = *uric
Within a query component, the characters ";", "/", "?", ":", "#",
"&", "=", "+", ",", and "$" are reserved.
It is therefore a bug in the other software if it does not accept URLs with spaces in the query string encoded as "+" characters.
As for the third part of your question, one way (though slightly ugly) to fix the output from URLEncoder.encode() is to then call replaceAll("\\+","%20") on the return value.
This confusion is because URL is still 'broken' to this day
Take "http://www.google.com" for instance. This is a URL. A URL
is a Uniform Resource Locator and is really a pointer to a web page
(in most cases). URLs actually have a very well-defined structure
since the first specification in 1994.
We can extract detailed information about the "http://www.google.com"
URL:
+---------------+-------------------+
| Part | Data |
+---------------+-------------------+
| Scheme | http |
| Host address | www.google.com |
+---------------+-------------------+
If we look at a more
complex URL such as
"https://bob:bobby#www.lunatech.com:8080/file;p=1?q=2#third" we can
extract the following information:
+-------------------+---------------------+
| Part | Data |
+-------------------+---------------------+
| Scheme | https |
| User | bob |
| Password | bobby |
| Host address | www.lunatech.com |
| Port | 8080 |
| Path | /file |
| Path parameters | p=1 |
| Query parameters | q=2 |
| Fragment | third |
+-------------------+---------------------+
The reserved characters are different for each part
For HTTP URLs, a space in a path fragment part has to be encoded to
"%20" (not, absolutely not "+"), while the "+" character in the path
fragment part can be left unencoded.
Now in the query part, spaces may be encoded to either "+" (for
backwards compatibility: do not try to search for it in the URI
standard) or "%20" while the "+" character (as a result of this
ambiguity) has to be escaped to "%2B".
This means that the "blue+light blue" string has to be encoded
differently in the path and query parts:
"http://example.com/blue+light%20blue?blue%2Blight+blue". From there
you can deduce that encoding a fully constructed URL is impossible
without a syntactical awareness of the URL structure.
What this boils down to is
you should have %20 before the ? and + after
Source
It shouldn't matter, any more than if you encoded the letter A as %41.
However, if you're dealing with a system that doesn't recognize one form, it seems like you're just going to have to give it what it expects regardless of what the "spec" says.
You can use either - which means most people opt for "+" as it's more human readable.
When encoding query values, either form, plus or percent-20, is valid; however, since the bandwidth of the internet isn't infinite, you should use plus, since it's two fewer bytes.