I found this example for using netscaler to rewrite requests to an internal server on a specific port.
set transform action trans_action_RSA_SS -priority 1000 -reqUrlFrom
"https://rsa.domain.public" -reqUrlInto
“https://rsa.domain.local:7004″ -resUrlFrom
"https://rsa.domain.local:7004″ -resUrlInto
"https://rsa.domain.public"
I'd like to expand on the example to point the local destination at a vserver.
Assume my vserver is called INTERNALVSERVER and also assume that it
is configured as a load balancer in front of 3 nodes (I suspect the specifics of that are irrelevant to this situation).
I just want to ensure that my urltransform applies to my vserver properly. Conceptually I'm going for something like this:
set transform action trans_action_RSA_SS -priority 1000 -reqUrlFrom
"https://rsa.domain.public" -reqUrlInto
“https://INTERNALVSERVER:7004″ -resUrlFrom
"https://INTERNALVSERVER:7004″ -resUrlInto
"https://rsa.domain.public"
Apparently the example would work as advertised as long as https://rsa.domain.local points at a vserver/service group. So there's no need to change anything in the rules, just make sure "rsa.domain.local" or the equivalent for your setup is pointing to the right place.
Although above solution would work, from web-server's security perspective it's not good idea to do direct URL transformation of external URL to internal as it would make your internal web server vulnerable to denial of service attack. Also, if your web-admin has not secure web-server, SQL-injection would create big enough of problems for you.
Instead, I would suggest you to do LB and responders to pick/channel valid URL traffic and discard all other web-traffic.
Related
I'm trying to rewrite a URL with Cloudflare Transform Rule but I need it to make two changes and I also want it to behave according to a regular expression.
For example:
https://test1-server.cloud.servers/account/login
Should be rewritten to:
https://test1-machine.cloud.servers/forms/login
And
https://prod2-server.cloud.servers/account/login
Should be rewritten to:
https://prod2-machine.cloud.servers/forms/login
Could this be done with Cloudflare?
You should be able to achieve this, although it may be more complicated than just a transform rule. With the transform rule, you can only change the path of the final request it seems -
https://developers.cloudflare.com/rules/transform/url-rewrite/examples
So, I don't think changing the subdomain is possible. However, in theory if both your subdomains are pointed at the same IP address, you could a "Request Header Modification" rule to intercept and update the host header value -
https://developers.cloudflare.com/rules/transform/request-header-modification/examples
So your condition would be the same for both rules, but in the request header rule, you'd be changing the host header to test1-machine.cloud.servers.
Here are all available functions you can make use of (for all transform rule types) -
https://developers.cloudflare.com/firewall/cf-firewall-language/functions#transformation-functions
http://example.com/something/somewhere//somehow/script.js
Does the double slash break anything on the server side? I have a script that parses URLs and i was wondering if it would break anything (or change the path) if i replaced multiple slashes with a single slash. Especially on the server side, some frameworks like CodeIgniter and Joomla use segmented url schemes and routing. I would just want to know if it breaks anything.
HTTP RFC 2396 defines path separator to be single slash.
However, unless you're using some kind of URL rewriting (in which case the rewriting rules may be affected by the number of slashes), the uri maps to a path on disk, but in (most?) modern operating systems (Linux/Unix, Windows), multiple path separators in a row do not have any special meaning, so /path/to/foo and /path//to////foo would eventually map to the same file.
An additional thing that might be affected is caching. Since both your browser and the server cache individual pages (according to their caching settings), requesting same file multiple times via slightly different URIs might affect the caching (depending on server and client implementation).
The correct answer to this question is it depends upon the implementation of the server!
Preface: Double-slash is syntactically valid according to RFC 2396, which defines URL path syntax. As amn explains, it therefore implies an empty URI segment. Note however that RFC 2396 only defines the syntax, not semantics of paths, including empty path segments, so it is up to your server to decide the semantics of the empty path.
You didn't mention the server software stack you're using, perhaps you're even rolling your own? So please use your imagination as to what the semantics could be!
Practically, I would like to point out some everyday semantic-related reasons which mean you should avoid double slashes even though they are syntactically valid:
Since empty being valid is somehow not expected by everyone, it can cause bugs. And even though your server technology of today might be compatible with it, either your server technology of tomorrow or the next version of your server technology of today might decide not to support it any more. Example: ASP.NET MVC Web API library throws an error when you try to specify a route template with a double slash.
Some servers might interpret // as indicating the root path. This can either be on-purpose, or a bug - and then likely it is a security bug, i.e. a directory traversal vulnerability.
Because it is sometimes a bug, and a security bug, some clever server stacks and firewalls will see the substring '//', deduce you are possibly making an attempt at exploiting such a bug, and therefore they will return 403 Forbidden or 400 Bad Request etc, and refuse to actually do any further processing of the URI.
URLs don't have to map to filesystem paths. So even if // in a filesystem path is equivalent to /, you can't guarantee the same is true for all URLs.
Consider the declaration of the relevant path-absolute non-terminal in "RFC3986: Uniform Resource Identifier (URI): Generic Syntax" (specified, as is typical, in ABNF syntax):
path-absolute = "/" [ segment-nz *( "/" segment ) ]
Then consider the segment declaration a few lines further down in the same document:
segment = *pchar
If you can read ABNF, the asterisk (*) specifies that the following element pchar may be repeated multiple times to make up a segment, including zero times. Learning this and re-reading the path-absolute declaration above, you can see that a potentially empty segment imples that the second "/" may repeat indefinitely, hence allowing valid combinations like ////// (arbitrary length of at least one /) as part of path-absolute (which itself is used in specifying the rule describing a URI).
As all URLs are URIs we can conclude that yes, URLs are allowed multiple consecutive forward slashes, per quoted RFC.
But it's not like everyone follows or implements URI parsers per specification, so I am fairly sure there are non-compliant URI/URL parsers and all kinds of software that stacks on top of these where such corner cases break larger systems.
One thing you may want to consider is that it might affect your page indexing in a search engine. According to this web page,
A URL with the same path repeated 3 times will not be indexed in Google
The example they use is:
example.com/path/path/path/
I haven't confirmed this would also be true if you used example.com///, but I would certainly want to find out if SEO optimization was critical for my website.
They mention that "This is because Google thinks it has hit a URL trap." If anyone else knows the answer for sure, please add a comment to this answer; otherwise, I thought it relevant to include this case for consideration.
Yes, it can most definitely break things.
The spec considers http://host/pages/foo.html and http://host/pages//foo.html to be different URIs, and servers are free to assign different meanings to them. However, most servers will treat paths /pages/foo.html and /pages//foo.html identically (because the underlying file system does too). But even when dealing with such servers, it's easily possible for extra slash to break things. Consider the situation where a relative URI is returned by the server.
http://host/pages/foo.html + ../images/foo.png = http://host/images/foo.png
http://host/pages//foo.html + ../images/foo.png = http://host/pages/images/foo.png
Let me explain what that means. Say your server returns an HTML document that contains the following:
<img src="../images/foo.png">
If your browser obtained that page using
http://host/pages/foo.html # Path has 2 segments: "pages" and "foo.html"
your browser will attempt to load
http://host/images/foo.png # ok
However, if your browser obtained that page using
http://host/pages//foo.html # Path has 3 segments: "pages", "" and "foo.html"
you'll probably get the same page (because the server probably doesn't distinguish /pages//foo.html from /pages/foo.html), but your browser will erroneously try to load
http://host/pages/images/foo.png # XXX
You may be surprised for example when building links for resources in your app.
<script src="mysite.com/resources/jquery//../angular/script.js"></script>
will not resolve to mysite.com/resources/angular/script.js but to mysite.com/resources/jquery/angular/script.js what you probably didn't want
Double slashes are evil, try to avoid them.
Your question is "does it break anything". In terms of the URL specification, extra slashes are allowed. Don't read the RFC, here is a quick experiment you can try to see if your browser silently mangles the URL:
echo '<?= $_SERVER['REQUEST_URI'];' > tmp.php
php -S localhost:4000 tmp.php
I tested macOS 10.14 (18A391) with Safari 12.0 (14606.1.36.1.9) and Chrome 69.0.3497.100 and both get the result:
/hello//world
This indicated that using an extra slash is visible to the web application.
Certain use cases will be broken when using a double slash. This includes URL redirects/routing that are expecting a single-slashed URL or other CGI applications that are analyzing the URI directly.
But for normal cases of serving static content, such as your example, this will still get the correct content. But the client will get a cache miss against the same content accessed with different slashes.
What is the method I need to call to find the root URL for a rails application. For example, I have a site where the address is "https://host:1234/foo/app-main".
What method should I be using to get "https://host:1234/foo/images" to get the absolute url for images in the public url?
image_path(image_name)
Edit: Steve has a good point, this will only get you part of the way there. To get the rest you must be inside of a request (you probably are)
In that case though, you can combine the above with Justice's approach,
"#{request.scheme}://#{request.host_with_port}/#{request.script_name}#{image_path(image_name)}"
This question makes sense only on a per-request basis, since your one process might easily be listening on multiple domain names and on multiple schemes.
"#{request.scheme}://#{request.host_with_port}#{request.script_name}"
See Rack::Request.
I have a relatively simple (I think) use-case but I can't find any examples where someone has done this. We are using Varnish as a cache and reverse proxy in front of two different applications and would like to make things a bit more unified across both as they both do similar things. I was hoping Varnish could help rewrite the URLs as shown below.
Original application URL for pagination (get first 10 items):
http://myapplication.com/products/?startindex=1&endindex=10
Desired URL:
http://myapplication.com/products/?paginate=1:10
This is just one example (the most complex because it combines two parameters), but in all cases the input values for the parameters stay the same, it is just that the parameter names will change.
Another example would be:
http://myapplication.com/search/?query=something
to:
http://myapplication.com/search/?q=something
Does anyone have any experience with varnish and how this could be done?
Thanks
Apparently you can. The answer is that regsub is your friend.
For example:
if (req.url ~ "(.*)(id=)") {
set req.url = regsub(req.url, "(feeds/[a-zA-Z]*/)(.*)([\?|&])(id=)([a-zA-Z0-9]*)(.*)", "\1\2\3byGuid=\5\6");
}
This will convert and incoming "id" parameter into a "byGuid" parameter on the backend. t also does a bunch of stuff with the rest of the URL string but the basics are there. SO if anyone wants to do something similar this is a good starting point.
I have a Rails app that needs to expose values from a database as a web service - since I'm using Rails 2.x, I'm going with REST (or at least try). Assuming my resource is Bananas, for which I want to expose several sub-characteristics, consider this:
- /banana -> give a summary of the first 10 bananas, in full (all characteristics)
- /banana/?name=<name> -> give all characteristics for banana named <name>
- /banana/?number=<number> -> give all characteristics for banana number <number>
- /banana/?name=<name>/peel -> give peel data for banana named <name>
- /banana/?number=<number>/length -> give length data for banana number <number>
I don't want to search for ID, only name or number. And I have about 7 sub-characteristics to expose. Is this RESTful?
Thanks for any feedback!
What Wahnfrieden is talking about is something called Hypermedia as the Engine of Application State (HATEOAS) - a central constraint of REST as defined by Fielding.
In a nutshell, REST application clients never construct URIs themselves. Instead, they follow URIs provided by the application. So, URI templates such as the ones you're asking about are irrelevent at best. You can make them conform to a system if you'd like, but REST says nothing about how your URIs need to look. You could, if you wanted to, arrange it so that every resource in your system was available from http://example.com/{hash}.
Publishing URI templates, such as the ones you're talking about in your question, introduces tight coupling between your application and clients - something REST is trying to prevent.
The problem with understanding hypermedia-driven applications is that almost nobody implements or documents their "RESTful" systems this way.
It might help to think about the interaction between a human and server via a browser. The human only knows about content and links that the server provides through the browser. This is how a RESTful system should be built. If your resources aren't exposing links, they're probably not RESTful.
The advantage is that if you want to change your URI system, for example, to expose the Banana "Peel" attribute through a query parameter instead of a nested URL, you can do it anytime you'd like and no client code needs to be changed because they're not constructing links for themselves.
For an example of a system that embraces the hypertext-driven constraint in REST, check out the Sun Cloud API.
I would use these:
/banana
/banana/blah
/banana/123
/banana/blah/peel (and /banana/123/peel)
/banana/blah/length (and /banana/123/length)
First, common practice for ReSTful URIs is /object_name/id/verb, with some of those absent (but in that order). Of course, this is neither required nor expected.
If all your names aren't made of digits, you don't have to explicitly have name in /banana/name/blah. In fact, if anything, it would be better to have id as identifier: /banana/id/123/peel. Hope this helps.
Parameters should only be used for form submission.
Also, URI naming schemas is totally unrelated to REST. The point of REST is to make related resources discoverable via hypertext, not out-of-band conventions, and only from a limit number of entry points. So your /bananas/ entry point might provide the summary info for 10 bananas, but it must also provide the URI for each of those bananas' details resources, as well as the URI to get the summary for the next 10 bananas. Anything else is just RPC.
It is good practice in REST to not use query parameters because query parameters don´t belong to a URL and in REST all resources should be addressable through a URL.
In your example /banana/?name=name should be /banana/name because you are referring a concrete resource.
Even I think /banana/?number=number/length is not good REST style, because you are selecting an attribute through a URL when you should retrieve the whole state with /banana/name . A difference could be /customers/1024/address to get the Customer 1024 address record.
HTH.
A more opt form for the route in url having query string is the plural form, as it is possible that multiple items are returned in the result. In this case, bananas, like bananas?color=yellow, sounds more appropriate.
On the other hand, the singular form banana, like banana/123, is good when fetching a specific resource's representation when its identifier is known and query string is not required.