This works as expected - it returns one record:
https://terraref.ncsa.illinois.edu/bety/api/v1/search.json?sitename=~Season%204&limit=1
this does not
when I use the base url terraref.org rather than terraref.ncsa.illinois.edu:
https://terraref.org/bety/api/v1/search.json?sitename=~Season%204&limit=1
because in this second case the % is replaced by %25 and the query term is then not found.
Questions
How can I fix it?
Why has this appeared in the last few months and
Update
you can see question history for the red herring related to the R packages I was using
thanks to the comments, I've narrowed down the issue, I think
I've included tags ruby-on-rails and nginx because these are the tools that the api and server are using.
Somehow, you are encoding the value for sitename 2 times. In this example the decoded value is Season 4 when you encode first time Season%204and doing it the second time %20Season%25204.
Investigate the code to verify whether value for sitename already encoded and you doing it again.
Check Chrome DevTools to debug the problem.
Your call is being redirected with the following query parameters.
sitename: ~Season 4
limit: 1
If you choose the encoded view, you'll see the culprit.
So, as one might expect, it's just as simple as double url encoding.
If you need to use terraref.org as a base url, replace space with a plus.
https://terraref.org/bety/api/v1/search.json?sitename=~Season+4&limit=1
For complicated reasons I won't go into, we have some requests that come into our application for /blah/[**Token**] (this is literal, none of this is placeholders).
Currently we get errors from MVC saying it can't find an IController for this every time the request comes in.
We would like the request to either: a) 404 or b) serve up a static placeholder image.
Try as we might, we can't figure out how to escape these characters in an IgnoreRoute/Route call such that they are treated literally. Eg. we've tried:
IgnoreRoute("blah/\[\*\*Token\*\*\]")
IgnoreRoute("blah/[\*\*Token\*\*]")
But the problem still persists.
How do we escape this path, such that actual requests to blah/[**Token**] are ignored and result in 404 (or alternatively, how do we map this exact path to a static image)?
Note: We have edited web.config so that * is not a disallowed character. We can't (currently) easily change this incoming request path, we just want to stop our error logs filling up when it happens :-)
Use a route constraint like this
IgnoreRoute("{*constraint}", new { constraint = #".*\[\*\*ProductImageFile\*\*]" });
http://example.com/something/somewhere//somehow/script.js
Does the double slash break anything on the server side? I have a script that parses URLs and i was wondering if it would break anything (or change the path) if i replaced multiple slashes with a single slash. Especially on the server side, some frameworks like CodeIgniter and Joomla use segmented url schemes and routing. I would just want to know if it breaks anything.
HTTP RFC 2396 defines path separator to be single slash.
However, unless you're using some kind of URL rewriting (in which case the rewriting rules may be affected by the number of slashes), the uri maps to a path on disk, but in (most?) modern operating systems (Linux/Unix, Windows), multiple path separators in a row do not have any special meaning, so /path/to/foo and /path//to////foo would eventually map to the same file.
An additional thing that might be affected is caching. Since both your browser and the server cache individual pages (according to their caching settings), requesting same file multiple times via slightly different URIs might affect the caching (depending on server and client implementation).
The correct answer to this question is it depends upon the implementation of the server!
Preface: Double-slash is syntactically valid according to RFC 2396, which defines URL path syntax. As amn explains, it therefore implies an empty URI segment. Note however that RFC 2396 only defines the syntax, not semantics of paths, including empty path segments, so it is up to your server to decide the semantics of the empty path.
You didn't mention the server software stack you're using, perhaps you're even rolling your own? So please use your imagination as to what the semantics could be!
Practically, I would like to point out some everyday semantic-related reasons which mean you should avoid double slashes even though they are syntactically valid:
Since empty being valid is somehow not expected by everyone, it can cause bugs. And even though your server technology of today might be compatible with it, either your server technology of tomorrow or the next version of your server technology of today might decide not to support it any more. Example: ASP.NET MVC Web API library throws an error when you try to specify a route template with a double slash.
Some servers might interpret // as indicating the root path. This can either be on-purpose, or a bug - and then likely it is a security bug, i.e. a directory traversal vulnerability.
Because it is sometimes a bug, and a security bug, some clever server stacks and firewalls will see the substring '//', deduce you are possibly making an attempt at exploiting such a bug, and therefore they will return 403 Forbidden or 400 Bad Request etc, and refuse to actually do any further processing of the URI.
URLs don't have to map to filesystem paths. So even if // in a filesystem path is equivalent to /, you can't guarantee the same is true for all URLs.
Consider the declaration of the relevant path-absolute non-terminal in "RFC3986: Uniform Resource Identifier (URI): Generic Syntax" (specified, as is typical, in ABNF syntax):
path-absolute = "/" [ segment-nz *( "/" segment ) ]
Then consider the segment declaration a few lines further down in the same document:
segment = *pchar
If you can read ABNF, the asterisk (*) specifies that the following element pchar may be repeated multiple times to make up a segment, including zero times. Learning this and re-reading the path-absolute declaration above, you can see that a potentially empty segment imples that the second "/" may repeat indefinitely, hence allowing valid combinations like ////// (arbitrary length of at least one /) as part of path-absolute (which itself is used in specifying the rule describing a URI).
As all URLs are URIs we can conclude that yes, URLs are allowed multiple consecutive forward slashes, per quoted RFC.
But it's not like everyone follows or implements URI parsers per specification, so I am fairly sure there are non-compliant URI/URL parsers and all kinds of software that stacks on top of these where such corner cases break larger systems.
One thing you may want to consider is that it might affect your page indexing in a search engine. According to this web page,
A URL with the same path repeated 3 times will not be indexed in Google
The example they use is:
example.com/path/path/path/
I haven't confirmed this would also be true if you used example.com///, but I would certainly want to find out if SEO optimization was critical for my website.
They mention that "This is because Google thinks it has hit a URL trap." If anyone else knows the answer for sure, please add a comment to this answer; otherwise, I thought it relevant to include this case for consideration.
Yes, it can most definitely break things.
The spec considers http://host/pages/foo.html and http://host/pages//foo.html to be different URIs, and servers are free to assign different meanings to them. However, most servers will treat paths /pages/foo.html and /pages//foo.html identically (because the underlying file system does too). But even when dealing with such servers, it's easily possible for extra slash to break things. Consider the situation where a relative URI is returned by the server.
http://host/pages/foo.html + ../images/foo.png = http://host/images/foo.png
http://host/pages//foo.html + ../images/foo.png = http://host/pages/images/foo.png
Let me explain what that means. Say your server returns an HTML document that contains the following:
<img src="../images/foo.png">
If your browser obtained that page using
http://host/pages/foo.html # Path has 2 segments: "pages" and "foo.html"
your browser will attempt to load
http://host/images/foo.png # ok
However, if your browser obtained that page using
http://host/pages//foo.html # Path has 3 segments: "pages", "" and "foo.html"
you'll probably get the same page (because the server probably doesn't distinguish /pages//foo.html from /pages/foo.html), but your browser will erroneously try to load
http://host/pages/images/foo.png # XXX
You may be surprised for example when building links for resources in your app.
<script src="mysite.com/resources/jquery//../angular/script.js"></script>
will not resolve to mysite.com/resources/angular/script.js but to mysite.com/resources/jquery/angular/script.js what you probably didn't want
Double slashes are evil, try to avoid them.
Your question is "does it break anything". In terms of the URL specification, extra slashes are allowed. Don't read the RFC, here is a quick experiment you can try to see if your browser silently mangles the URL:
echo '<?= $_SERVER['REQUEST_URI'];' > tmp.php
php -S localhost:4000 tmp.php
I tested macOS 10.14 (18A391) with Safari 12.0 (14606.1.36.1.9) and Chrome 69.0.3497.100 and both get the result:
/hello//world
This indicated that using an extra slash is visible to the web application.
Certain use cases will be broken when using a double slash. This includes URL redirects/routing that are expecting a single-slashed URL or other CGI applications that are analyzing the URI directly.
But for normal cases of serving static content, such as your example, this will still get the correct content. But the client will get a cache miss against the same content accessed with different slashes.
We're using MVC .NET and the RouteCollection class to route URLs in our web app. This functions normally until we pass a URL containing the the text "PRN" anywhere inside the URL. When this happens, the routing will not occur and a 400 Page Not Found error is returned to the client. It's like something is throwing the error before the routing collection is even consulted, because the route the URL should take is never touched (by that I mean the underlying code's break-point is never hit, though the exact same URL without the string "PRN" will hit the break-point).
So I thought it might be a page validation issue, that maybe Microsoft decided to throw exceptions when the URL contains the phrase "PRN" because it is like "print" or "porn" but if that were the case then we'd see the "A potentially dangerous Request.Form value was detected from the client" error, but we don't.
Researching this has been a hassle because Google thinks PRN should return results for "porn", which means 98% of my search results are invalid (and inappropriate). Using the "-porn" clause in Google drops your results down to about 10-30 hits, all useless.
Does anyone know why a URL containing the string "PRN" will not route properly? If you have any posts or threads to point me to, that would be awesome (again, Google has failed me).
Currently, when I try to hit certain pages of my site via something like http://www.domain.com/< (which is a valid URL), I get a blank page with the text "Bad Request" on it (and nothing else). This happens with both the escaped and unescaped version of the URL.
I'm fairly certain this is due to IIS6 not liking the < character (which, in general, is valid). Is there a way to stop IIS6 from filtering these characters and giving me this error page?
(I've found similar solutions for IIS7, but nothing has worked in IIS6 so far.)
UPDATE: The URL is being transformed already, ie. hitting domain.com/%3C will also give the "Bad Request" page.
Not sure if this will work, but this got me out of a similar jam caused by design types forgetting key parts of query strings. Sounds like you might have a similar issue. Anyhow, try making a virtual directory called %3c and then having that redirect to where appropriate.
RFC 1738:
Thus, only alphanumerics, the special characters "$-_.+!*'(),", and reserved characters used for their reserved purposes may be used unencoded within a URL.
< transforms to %3C
https://stackoverflow.com/<