Cloud Run / the "Google Frontend" seems to completely buffer responses from a Cloud Run application, even when using chunked transfer encoding for the response. This is bad for incremental rendering.
I have a Java web app based on com.sun.net.HttpServer which supports chunked encoding for the response. Especially flushing the output stream creates a chunk, so I can do:
write response line
flush
compute for 10s
write more response lines
Locally, this results in a chunked response:
HTTP/1.1 200 OK
Date: Sat, 07 Mar 2020 17:08:10 GMT
Transfer-encoding: chunked
Content-type: text/plain;charset=utf-8
1c
<first output>
14
<next output>
17
<next output>
0
Using curl, I can see the output appear incrementally.
In contrast, when deploying the same app in Cloud Run, the response gets fully buffered and returned in full, no matter how long the pause (many seconds), how many chunks or how much content is returned (I've tested up to several megabytes):
curl -v https://...
< HTTP/2 200
< content-type: text/plain;charset=utf-8
< x-cloud-trace-context: 3872abb809e97a76298f4c46b9217656;o=1
< date: Sat, 07 Mar 2020 17:18:48 GMT
< server: Google Frontend
< content-length: 2450359
(Example has 50k chunks!)
Is there a way to have the GFE pass through the chunked encoding?
Cloud Run does not support streaming responses where data is sent in incremental chunks to the client while a request is being processed. All data from your service is sent as a single HTTP response.
Also, the 'Content-Length' and 'Transfer-Encoding' headers are removed from responses your application may serve. Those headers are then added by the Google Front End and finally, the response is served.
Source of information:
1) Internal conversations with the Cloud Run engineering team.
2) Google Cloud Security Whitepaper that details the GFE (Google Frontend) which sits in front of Cloud Run Managed.
https://services.google.com/fh/files/misc/security_whitepapers_march2018.pdf
Related
You can see in the request URL that it's addressing the URL of Office 365 https://outlook.office365.com/api/v1.0/users/{user-email}/calendarview
Request:
GET /api/v1.0/users/testuser#testapp.onmicrosoft.com/calendarview HTTP/1.1
Host: outlook.office365.com
User-Agent: myappagent/1.0
Authorization: Bearer eyJ**********
Accept: application/json
Prefer: outlook.timezone=Europe/London
Response:
HTTP/1.1 403 Forbidden
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8
Server: Microsoft-IIS/10.0
request-id: d5ae24e8-0c34-43b5-a8a3
X-CalculatedFETarget: *******.internal.outlook.com
X-BackEndHttpStatus: 403
X-FEProxyInfo: *********.PROD.OUTLOOK.COM
X-CalculatedBETarget: *********.prod.outlook.com
X-BackEndHttpStatus: 403
X-RUM-Validated: 1
X-BeSku: WCS6
x-ms-appId: ********
OData-Version: 4.0
X-AspNet-Version: 4.0.30319
X-DiagInfo: ********
X-BEServer: ********
X-Proxy-RoutingCorrectness: 1
X-Proxy-BackendServerStatus: 403
X-FEServer: ********
X-Powered-By: ASP.NET
X-FEServer: ********
Date: Fri, 18 Jun 2021 20:22:02 GMT
{
"error":{
"code":"ErrorAccessDenied",
"message":"Access to OData is disabled."
}
}
I have came across this article:
https://learn.microsoft.com/en-us/archive/blogs/wushuai/how-to-fix-access-to-odata-is-disabled-when-calling-graph-api
But I don't think it applies since I'm using the Office 365 Calendar API.
When I checked my access policy, it's empty. Which I'm assuming means there's no restrictions, and therefore my app shouldn't have any problem.
And these are the permissions of the app (I added Graph API calendar just in case, but still same 403 response)
You are using a very old api endpoint, it is recommended that you use the latest ms graph api.
For your question, you need to use the client credential flow to get the token, and then specify the scope as: https://outlook.office365.com/.default.
Parse the token：
After long research and trying. I realized the solution is in the (ApplicationAccessPolicy).
Reference :
https://learn.microsoft.com/en-us/graph/auth-limit-mailbox-access
https://learn.microsoft.com/en-us/powershell/module/exchange/get-applicationaccesspolicy?view=exchange-ps
Helpful Powershell commands :
Test-ApplicationAccessPolicy
Get-ApplicationAccessPolicy
New-ApplicationAccessPolicy
Remove-ApplicationAccessPolicy
Set-ApplicationAccessPolicy
Hope this is helpful for anyone stuck with this issue ;)
Update:
Did you see this blog post? https://developer.microsoft.com/en-us/outlook/blogs/outlook-rest-api-v1-0-deprecation-update/
The API you're using is announced to be deprecated roughly 40 months ago (October 2017). And is actually deprecated April 30th 2021, so you're in luck if it still works for some tenants, but it will stop working very soon.
You could switch to the v2 endpoint of the outlook api, I guess that will be the most compatible. But that endpoint is also announced to be retired https://developer.microsoft.com/en-us/outlook/blogs/outlook-rest-api-v2-0-deprecation-notice/
My advice, enjoy the outlook api while you still can, but invest a little time to convert to the new Graph endpoints if you want your app to continue to work in the near future.
Old (irrelevant) answer:
Have you checked the aud claim of the token? Copy the token and paste it at https://JWT.ms
It should be https://outlook.office365.com (since you’re using the old api).
May I ask why you would want to create a new application that talks to the old api?
Apart from the obvious stuff, the documentation you’re referring to also talks about an “allow list”, if it’s empty no application has access, that is also possible.
The new api (graph endpoints) has an option to do resource scoping (allow an application with client credentials, to certain mailboxes instead of all). If that is configured in the tenant I’ve seen similar errors.
I am working on an AngularDart web app.
I need to retrieve a cookie from a json server that works with cookie authentication.
So on connection, I have to post credentials in body, then the server give me back a response containing the cookie.
import 'package:http/browser_client.dart';
Response response = await _http.post(_loginUrl, body: json.encode(datas));
The response is ok, I get 200 statuscode and the json response from the server, but impossible to find the cookie while in Postman I do get it.
So how to retrieve cookies from response?
EDIT: Informations returned from the server to postman :
ASP.NET_SessionId=03e95c12-6098-4f07-b9a9-054377b0311d; path=/; domain=.www.xxxxx.fr; HttpOnly; Expires=Tue, 19 Jan 2038 03:14:07 GMT;
EDIT: This is the raw headers from the server to the Dart app :
t=1462088 [st=142] HTTP_TRANSACTION_READ_RESPONSE_HEADERS
--> HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/8.5
Set-Cookie: ASP.NET_SessionId=f229a77e-5eb6-4c6c-9c57-9280947cb434; path=/; HttpOnly
X-Powered-By: ASP.NET
Access-Control-Allow-Origin: *
Date: Fri, 30 Nov 2018 09:52:50 GMT
Content-Length: 22443
So the server sends back the cookie, but the browser does not seem to save it ...
Ok so finally, a problem was found on the API server, the developer did not check correctly the cookie. Generally, only one cookie is sent to this server, so in its code, it took the first... But from a browser, severals cookie can be sent at the same time. So he made some change to check the good one...
Postman should show that this is an HttpOnly cookie that can't be read from JavaScript (and the same applies for Dart of course)
See also https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#Secure_and_HttpOnly_cookies
The cookie will be sent back to the server automatically therefore there is no need to read it from JavaScript. Allowing access from JavaScript only undermines security.
I am trying to send data from an arduino to a web server (LAMP) using the ESP8266 module, when I do a POST to a local network server the server receives the data and returns 200, however, when I post to an external server
(Hosting or google cloud) it registers error 400 in the Apache log and returns nothing, but when I do the same type of request by Postman everything is fine, because of this I do not know if it is my fault when mounting or executing the request or if Is some block on the external servers because the http server in my network works.
I'm using this lib to work with ESP: https://github.com/itead/ITEADLIB_Arduino_WeeESP8266
This is the request string:
POST /data/sensor_test.php HTTP/1.1
Host: xxxxxxxxx.com
Accept: */*
Content-Length: 188
Content-Type: application/x-www-form-urlencoded
Cache-Control: no-cache
temperatureAir1=19.70&humidityAir1=82.30&temperatureAir2=19.40&humidityAir2=78.60&externalTemperature=19.31&illumination05=898&illumination10=408&humiditySoilXD28=6&humiditySoilYL69=5
I found the problem, when I concatenated the strings that make up the request I was doing line breaks with \n, I switched to \r\n and it worked!
The amount of bytes really is with error, I am seeing to correct, but the good thing is that now the request is right.
I'm able to get the status code just fine, but I'm looking for a way to get the "HTTP Reason Phrase" from the response. I've spelunked through RKObjectRequestOperation and don't see it anywhere. I feel like I'm just missing it. Is this something that's supported by RestKit?
In my particular case, the service I'm hitting is sending this back:
HTTP/1.1 409 A user with this phone number already exists
Cache-Control: private
Server: Microsoft-IIS/8.0
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Tue, 24 Feb 2015 22:13:21 GMT
Content-Length: 0
I'm trying to get at the "A user with this phone number already exists" bit.
Thanks very much in advance.
This isn't a RestKit thing, it's a URL handling system thing as it is the HTTP underpinnings that RestKit is built on top of.
So it should be provided by localizedStringForStatusCode: from the NSHTTPURLResponse that you can ferret out of the RestKit operation.
To be honest I've never had cause to use it so I'm not sure if it's purely a local client lookup from the returned status code or if it does link to any server response data. As the docs reference RFC 2616 it should pick up the text if the server sends it out...
In my latest project which is in RC1 I have noticed that I have this browser caching issue that I just can't shake. This is what my header looks like
HTTP/1.1 200 OK
Date: Tue, 03 Mar 2009 15:11:34 GMT
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
X-AspNetMvc-Version: 1.0
Cache-Control: private
Expires: Tue, 03 Mar 2009 15:11:34 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 4614
Now technically if this is private it shouldn't have an expiration date on the content right? I've tried no-cache as well with the same results. Anybody have any experience with this particular issue?
Cache-Control: private only specifies that the response is only intended for a single user and should not be stored in a shared cache (say, in a proxy) and used to serve requests for other users. I don't see anything in the protocol documentation that would preclude the use of an Expires header with a value. In fact, it seems a perfectly reasonable thing to say "use this for subsequent requests for this user only, but not after this time." There are other values for Cache-Control where Expires may not make sense, but I believe that the protocol has a means for disambiguating between conflicting headers (See section 4 of the protocol docs).
Quoting from Section 16.2 of the HTTP 1.1 protocol docs:
private
Indicates that all or part of the response message is intended for
a single user and MUST NOT be cached by a shared cache. This
allows an origin server to state that the specified parts of the
response are intended for only one user and are not a valid
response for requests by other users. A private (non-shared)
cache MAY cache the response.
Note: This usage of the word private only controls where the
response may be cached, and cannot ensure the privacy of the
message content.
There's no reason why private content can't be cached, its just that it should only be cached by the browser in the current users context, it should not be cached server side or by other caches such as a proxy server.