I want to configure dynamic timeout for inbound requests. I am scraping timeout value from request body and assigning it to a variable. There are Command Operators which are used "%OPERATOR_NAME%" in access logs. I tried to use the Lua variable in same fashion but no luck.
How can I use Lua variable to set timeout value for inbound request in envoy proxy?
Related
Postman lets you specify path variables which allow you to input variable value on the go in a small table while consuming a GET route.
Is there any way to achieve the same for variables specified within the body? So that I don't have to update environment variables every time I am trying to consume a POST/PATCH/PUT route with different data in the body.
You could set the (environment) variables in the pre-request script programatically, and then use them in the body:
pm.environment.set("variable_key", "variable_value");
Please check out Using variables for more details.
We have a URL https://www.mylink.com/est?myId=4d22b9d0-4ff2-46c3-9343-945304dfea93
The above request also contains post data:
myId=4d22b9d0-4ff2-46c3-9343-945304dfea93
How can I parameterize this as the url and post data both contain same value but its dynamic. Also, how may I store it in a variable to use it somewhere else as well?
In order to be able to use it in URL and request body you need to extract it somewhere somehow, this dynamic ID is probably associated with this or that user account so my expectation is that after logging in you should see this ID somewhere in the response.
The process is known as correlation and there is a plenty of information over the web about handling dynamic requests in JMeter. There are also solutions which provide semi or fully automated correlation of the dynamic parameters like Correlations Recorder Plugin for JMeter or JMeter Proxy Recorder
If you just need to generate an unique GUID-like structure - you can do it using __UUID() function
Context
I've been running an intranet admin panel in Symfony 3.x for several years. The users login with google oauth and the system checks if the email matches a validated one in a lookup-list. The oauth client handling is done with the "HWI OAuth Bundle".
In order to start a clean way to migrate this admin panel into SF4 and later to SF5 we've started breaking our monolyth into microservices running in docker.
Moving to docker behind a reverse proxy
Today we were moving this admin panel into a docker. Then we are having the public apache2 doing a ProxyPass towards the docker running the admin panel. Let's imagine the docker runs in http://1.2.3.4:7540 Let's assume the public address is https://admin-europe.example.com
What happens is that the symfony application has a relative URL, as the route google_login configured in the routing.yml and in the service configuration defined in the security.yml:
routing:
# Required by the HWI OAuth Bundle.
hwi_oauth_redirect:
resource: "#HWIOAuthBundle/Resources/config/routing/redirect.xml"
prefix: /connect
hwi_oauth_connect:
resource: "#HWIOAuthBundle/Resources/config/routing/connect.xml"
prefix: /connect
hwi_oauth_login:
resource: "#HWIOAuthBundle/Resources/config/routing/login.xml"
prefix: /login
# HWI OAuth Bundle route needed for each resource provider.
google_login:
path: /login/check-google
logout:
path: /logout
security:
firewalls:
# disables authentication for assets and the profiler, adapt it according to your needs
dev:
pattern: ^/(_(profiler|wdt)|css|images|js)/
security: false
secured_area:
anonymous: true
logout:
path: /logout
target: /
handlers: [ admin.security.logout.handler ]
oauth:
resource_owners:
google: "/login/check-google"
login_path: /
use_forward: false
failure_path: /
oauth_user_provider:
service: admin.user.provider
So when the application was not dockerized, it run properly because the route requested to be the "redirect route" to google was https://admin-europe.example.com/login/check-google.
Nevertheless, now that it's inside the docker when the HWI bundle is building the data block to send to google it requests for this http://1.2.3.4:7540/login/check-google to be authorised as the "redirect URI" but of course it should not. Of course the redirect URI should continue to be https://admin-europe.example.com/login/check-google.
I naturally get this error message:
The reverse proxy
We already have in the reverse proxy the ProxyPassReverse and, in fact, the very same configuration has been working hassle-free for over a month with another microservice we already successfully moved (but that service did not need auth, was a public site).
This is natural, as ProxyPassReverse will tackle into http data but the google-oauth info-block is not handled by the ProxyPassReverse, as it's natural.
The problem
The problem here is not to have this address validated (put a domain alias into the private IP address, etc.)
The problem here is how to generate the "proper public URL" from inside the docker without creating a hard-dependency for the container contents in function of the environment it's going to run. Doing so would be an anti-pattern.
Exploring solutions
Of course the "easy" solution would be to "hardcode" the "external route" inside the container.
But this has a flaw. If I also want the same docker to be accessed from, say, https://admin-asia.example.com/ (note the -asia instead of the -europe), I'll run into problems as the asia users will be redirected to the europe route. This is a mere example, don't care about the specific europe-asia thing... the point is that the container should not be conscious of the sorrounding architecture. Or at least, conscious to "interact" but definitively not to have "hardcoded" inside the container things that depend on the environment.
Ie: Forget about the -europe and -asia thing. Imagine the access is admin-1111. It does not make sense that I have to "recompile" and "redeploy" the container if one day I want it to be accessible as admin-2222.
Temporal solution
I think it would solve the problem to point both the route in the rounting.yml and the config in the security.yml to a "parameter" (in 3.x in parameters.yml) and then move that into an Environment Variable when updating to SF4, but I'm unsure on how the cache compiler of the symfony would behave with a route that does not have a value, but a route that "changes dynamically".
Then pass the value of the redirecion when the container is started. This would solve the problem only partially: All the container would be bound to a redirect route set at the time of start, but it still would not solve the case of the same container instance accessed via different names thus needing multiple redirect routes. Instead when running non-dockerized that works as it just takes the "hostname" to build the absolute path on a relative-path definition.
Investigation so far
When accessing, the browser shows I'm going to
https://accounts.google.com/o/oauth2/auth
?response_type=code
&client_id=111111111111-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.apps.googleusercontent.com
&scope=email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fplus.profile.emails.read+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fplus.login
&redirect_uri=http%3A%2F%2Fmy.nice.domain.example.com%3A7040%2Fapp_dev.php%2Flogin%2Fcheck-google
Here we see that the redirect_uri parameter is the place where we'll return after passing the control to google momentarily.
So somebody needs to be building this URL.
I seeked for "redirect_uri" within the source code and I found that the involved classes are GoogleResourceOwner which extends GenericOAuth2ResourceOwner.
Both classes seem to belong to the domain as per the tests passing the $redirectUri as a string which needs to be already built by the caller.
The involved method is public function getAuthorizationUrl($redirectUri, array $extraParameters = array()). This receives the redirect URI and builds the auth URI with the redurect URI encoded as a parameter.
So, who are the consumers/clients of getAuthorizationUrl()?
I only found one single client usage in OAuthUtils, in a line that says return $resourceOwner->getAuthorizationUrl($redirectUrl, $extraParameters); within the function public function getAuthorizationUrl(Request $request, $name, $redirectUrl = null, array $extraParameters = array())
I see that mainly this OAuthUtils is acting as an adapter between the Symfony Request and the OAuth domain model. Within this method we mainly find code to create the $redirectUri.
The cleanest solution for me would be to create a child class OAuthUtilsBehindProxy inheriting from OAuthUtils, overwriting the method getAuthorizationUrl() and having it interpret the X-FORWARDED-* headers of the request, and then have the dependency injection to autowire my class everywhere the OAuthUtils is used with the hope that noone is doing a new OAuthUtils and every user of this class is getting it passed on the constructor.
This would be clean and woul work.
But frankly it seems an overkill to me. I'm pretty sure someone before me has put an app that needs Google OAuth made with HWI behind a reverse proxy and I wonder if there's a "config option" that I'm missing or really I have to re-code all this and inject it via D.I.
So, question
How do I have HWI-OAuth bundle to behave properly when running in a docker container behind a reverse proxy in regards on how to build the "redirect route" for the google-oauth service?
Is there any way to tell either the HWI bundle or either symfony to add a "full-host" prefix IN FUNCTION of the the X-FORWARDED-* headers "if available"? This would leave the docker image "fixed" and would run in "any" environment.
The underlying reason is the way Symfony generated the full-addresses from a relative path or route name.
Here's the investigation:
The method HWI/OAuthUtils::getAuthorizationUrl() is the one that generates the OAUth auth URI and consumes the method Symfony/HttpUtils::generateUri() to get the absolute URI of the redirect_to callback that will be encoded inside the Auth URI.
The method Symfony/HttpUtils::generateUri() generates an absolute URI (that in our case will be the callback) and to do so, the method handles 3 general cases:
The parameter is already an absolute URI (the return is the parameter without further processing)
The parameter is a relative URL (the function calls the Request class to build the proto + host + port + project-path prefix to prepend to the relative URI)
The parameter is a route name (the funcion calls the Router class to build the absolute URI)
In my example I was configuring a relative URL (google: "/login/check-google") in the security.yml so HttpUtils was delegating into the Request class.
Looking at the source of the Request class we observe:
The Request class is able to use proxy headers to build the absolute class.
But for security, by default symfony does not trust that a proxy exists merely because there are X-FORWARDED-* headers in it.
Indeed it's more secure plus more flexible.
There are 2 levels of security:
Somewhere we need to tell the Request class what is the list of trusted IPs that are proxies accessing the application.
Somewhere else we need to tell the Request class what specific proxy headers are trusted and what headers are not, even it supports different standards headers (RFC headers, non-RFC apache headers, etc)
Stated here https://symfony.com/blog/fixing-the-trusted-proxies-configuration-for-symfony-3-3 is that you need to configure the trusted proxies in the front-controller by calling the static method Request::setTrustedProxies();
So adding those couple of lines in the front-controller one killing non-nee4ded headers and the other with the IP ranges of the proxies, solved the problem:
# app.php
<?php
use Symfony\Component\HttpFoundation\Request;
$loader = require __DIR__.'/../app/autoload.php';
include_once __DIR__.'/../var/bootstrap.php.cache';
$kernel = new AppKernel('prod', false);
$kernel->loadClassCache();
Request::setTrustedHeaderName( Request::HEADER_FORWARDED, null ); # <-- Kill unneeded header.
Request::setTrustedProxies( [ '192.168.104.0/24', '10.0.0.0/8' ] ); # <-- Trust any proxy that lives in any of those two private nets.
$request = Request::createFromGlobals();
$response = $kernel->handle($request);
$response->send();
$kernel->terminate($request, $response);
With this change:
Symfony Request is able to build correct public absolute addresses from relative addresses if called thru a proxy, by deducting the host from HTTP_X_FORWARDED_HOST and HTTP_X_FORWARDED_PORT instead of HTTP_HOST and SERVER_PORT.
Symfony HttpUtils also, as it was delegating to Request.
HWI is in turn able to build a correct absolute callback redirect_to.
HWI can set the proper callback encoded inside the AuthUri.
The AuthURI that contains the proper absolute URI taking in account the proxy effect is sent to google.
Google sees the "public URI" as the one registered in the google configuration.
The workflow completes and the login process can end successfully.
Let's say I want to create a slash command with the flexibility to define the component as tag name which will be used for deployment.
eg: /dev-deploy comments v1.0.0
I have added the slash command as well as set the parameterized URL in
http://host/buildByToken/buildWithParameters?job=dev-deployment&token=test
All the other commands data is present in JSON object how can I access that data and pass it as a parameter to shell script which is executed when the build is triggered.
The slash command from Slack is sending a POST request to your URL. This request contains a property named text, which in your example would be "comments v1.0.0". To extract the tag you would need to parse it.
I would recommend to use some simple script (e.g. PHP) to receive the POST request, parse it (maybe also do some security checks) and then start the job with a shell command.
I am not sure what JSON object you are referring to. The slash command does not send any JSON. So maybe your question is missing some vital information? If so please add.
Here is an example of the POST request (from the official Slack documentation):
token=gIkuvaNzQIHg97ATvDxqgjtO
team_id=T0001
team_domain=example
enterprise_id=E0001
enterprise_name=Globular%20Construct%20Inc
channel_id=C2147483705
channel_name=test
user_id=U2147483697
user_name=Steve
command=/weather
text=94070
response_url=https://hooks.slack.com/commands/1234/5678
When sending a request via Zuul to a client, Zuul seems to change the query String. More specifically, if the client should receive an url-encoded query String, Zuul decodes the query String once. Here is a concrete example:
If "http://localhost:8080/demo/demo?a=http%3A%2F%2Fsomething/" is sent to the client, the client receives as a query String "a=http://something/".
Looking into Zuul`s code, the function "buildZuulRequestQueryParams" uses "HTTPRequestUtils.getInstance().getQueryParams();" which decodes the query String.
Is this a desired feature or a bug?
Zuul actually offers a flag to disable this behavior.
8.9 Query String Encoding
When processing the incoming request, query params are decoded so that they can be available for possible modifications in Zuul filters. They are then re-encoded the backend request is rebuilt in the route filters. The result can be different than the original input if (for example) it was encoded with Javascript’s encodeURIComponent() method. While this causes no issues in most cases, some web servers can be picky with the encoding of complex query string.
To force the original encoding of the query string, it is possible to pass a special flag to ZuulProperties so that the query string is taken as is with the HttpServletRequest::getQueryString method, as shown in the following example:
application.yml.
zuul:
forceOriginalQueryStringEncoding: true
[Note] This special flag works only with SimpleHostRoutingFilter.
Also, you loose the ability to easily override query parameters with
RequestContext.getCurrentContext().setRequestQueryParams(someOverriddenParameters),
because the query string is now fetched directly on the original
HttpServletRequest.
8. Router and Filter: Zuul
I was facing the same issue yesterday. I think it's related to this pull request. A faster way to solve this issue (without wait for PR get merged) is rewrite the classes in your own project using the same package and class name to override the framework class.
I ran into the same issue recently. Submitted a PR to Netflix/Zuul. Basically adding the same ability that's currently available on spring cloud gateway to Netflix. Hoping it'll get addressed soon.
If accepted, you could pretty much add a config to keep the original uri encoding
zuul.keepOriginalQueryStringEncoding=true