Since Friday all of our users are seeing sporadic 302s when trying to access our in-GCP IAP protected resources. Cookies are valid, and definitely being passed with the request.
This has worked for us for two years and nothing has changed here recently past standard GKE upgrades.
Since Friday we're seeing sporadic 302s from IAP (X-Goog-IAP-Generated-Response: true) as if the cookie is invalid. I can recreate this problem using a simple curl command, with my cookie stored in a file called cookie.test.
`curl -vs -b ./cookie.test https://gitlab.mydomain.com/projects/myapp.git
This succeeds maybe 1 out of 5 times. Behaviour is very recreatable. 2 out of 5 times we'll get a response from gitlab.mydomain.com and the other 3 times we'll see a 303 to accounts.google.com. Same cookie every time, all requests within a few seconds of each other.
This is causing an enormous inconvenience for our team.
Has there been a change to IAP recently that might explain this? Do you have any other reports of similar behaviour?
Folks,
I am from the IAP team at Google. Recently IAP has made some changes to the cookie name. However, this change should have been transparent to the browser users.
For people using GCP_IAAP_AUTH_TOKEN cookie name for programmatic auth, your flows will break. The documented way to send credentials in a programmatic call is to use Authorization / Proxy-Authorization header.
https://cloud.google.com/iap/docs/authentication-howto#authenticating_a_user_account
Cookies are meant to be used for browser flows only and IAP holds complete control of the naming and format of the cookie. If you continue to use cookies to send in credentials to IAP (by reverse engineering the new format), you run a risk of being broken again by future changes in cookie name/format.
One clarification is required though. In the original post, it was mentioned that you are getting a response of 302 to accounts.google.com, is that true for browser flows also? If so, please respond back with a har file and I'll be happy to take a look.
cheers.
I have also started facing this issue since last week and have spent around 2 days troubleshooting it as initially we thought that it must be some problem on our side.
Good to know that I am not the only one facing it.
Would really appreciate some updates from Google Around it.
However, one thing I found:- There was one official blog from google around IAP:- https://cloud.google.com/blog/products/identity-security/getting-started-with-cloud-identity-aware-proxy
they have updated this blog on 19th January and removed the mention of the cookie:- GCP_IAAP_AUTH_TOKEN
However, the line they have changed is still unclear to me and very confusing
It now says :-
That token can come from either a browser cookie or, for programmatic
access, from an Authorization: bearer header.
From where will the browser cookie come, what will be its name, there is no mention around it.
Let me know if someone finds a way to get it work again.
Thanks,
Nishant Shah
Related
At my work, we are finally upgrading our old Identity Server 3 to 4. We just got a very weird problem doing so. Everything works fine in all major browsers, but we also need to support some Electron clients. Here is where the weird part begins. All very old clients using Electron version 3 still work. All newer clients starting at Electron 9 also work. The only clients that don't work are the ones using Electron 6 (Chrome 76).
I already found this very helpful article written by Sebastian Gingter which helped to get the login working. But it only got me one step further. Now the client gets stuck at the connect/authorize/callback endpoint using the response_mode = form_post.
I already found some articles/stackoverflow questions pointing out to check the redirect URIs and to downgrade the CSP to version 1. The redirect URIs are configured correctly since the other clients work. The CSP does not help since I don't even get that far. It seems that the response body is never even loaded by Electron/Chrome.
Devtools Timing Screenshot
The request never finishes. On the server-side, it does though. I debugged through the IS 4 code and the dynamic HTML is written to the response like with all the other clients. I even called CompleteAsync() on the response manually and it still did not finish.
I researched and debugged for quite some time now and am out of ideas. Does anyone out there know this issue and more importantly also knows how to fix it?
We are using the following url for Google OAuth2 v3 now.
https://accounts.google.com/o/oauth2/auth
https://www.googleapis.com/oauth2/v3/token
But, I confirmed version-up in the latest document(https://developers.google.com/identity/protocols/OAuth2WebServer).
https://accounts.google.com/o/oauth2/v2/auth
https://www.googleapis.com/oauth2/v4/token
What was changed?
If possible, we want to update it.
This is the third time i think i have seen them change them in the last five years. Exactly why they change them they never inform us. I would suspect that the new endpoints are OpenIDConnect compliant.
The old ones should continue to work for a while i have not heard anything about them shutting them down.
You can also check the Discovery document
We have two sites with different subdomains. Sometimes our employees lose their cookies (they are just gone) on both domains at the same time so they get logged out.
I don't really see how our app can be responsible, because we have different server configurations (and for each site there're multiple servers btw). I guess only nginx versions (1.10.3) are the same. Plus this does not explain why do they get logged out on both sites at the same time.
If it helps, we use rails (3/5), unicorn (4.8.3/5.3.0), on older app sessions are stored in redis and in the new one in cookies.
So I wonder maybe there're some browser (security) policies when it clears cookies. Maybe on some ssl connection error, ip changes or whatever.
I understand that this is not definitive problem description but it seems like magic to us atm so I hope that someone encountered something like this.
P.S. btw we tried to ask one of our employees to use firefox instead of Chrome (that is used by all of them) but it does not seem to be making any difference (he wasnt logged out for a week but then he was like every 20 minutes)
We have realized that this URL http://Keyword:redacted#example.com/ redirects to http://example.com/ when copied and pasted into the browser's address bar.
As far as I understand this might be used in some ftp connections but we have no such use on our website. We are suspecting that we are targeted by an attack and have been warned by Google that we are passing PII (mostly email addresses) in our URL requests to their Google Adsense network. We have not been able to find the source, but we have been warned that the violation is in the form of http://Keyword:redacted#example.com/
How can we stop this from happening?
What URL redirect method we can use to not accept this and return an error message?
FYI I experienced a similar issue for a client website and followed up with Adsense support. The matter was escalated to a specialist team who investigated and determined that flagged violations with the format http://Keyword:redacted#example.com/ will be considered false positives. I'm not sure if this applies to all publishers or was specific to our case, but it might be worth following up with Adsense support.
There is nothing you can do. This is handled entirely by your browser long before it even thinks about "talking" to your server.
That's a strange URL for people to copy/paste into the browser's address bar unless they have been told/trained to do so. Your best bet is to tell them to STOP IT! :-)
I suppose you could look at the HTTP Authorization Headers and report an error if they come in populated... (This would $_SERVER['PHP_AUTH_USER'] in PHP.) I've never looked at these values when the header doesn't request them, so I'm not sure if it would work or not...
The syntax http://abc:def#something.com means you're sending userid='abc', password='def' as basic authentication parameters. Your browser will pull out the userid & password and send them along as authentication information, leaving the url without them.
As Peter Bowers mentioned, you could check the authorization headers and see if they're coming in that way, but you can't stop others from doing it if they want. If it happens a lot then I'd suspect that somewhere there's a web form asking users to enter their user/password and it's getting encoded that way. One way to sleuth it out would be to see if you can identify someone by the userid specified.
Having Keyword:redacted sounds odd. It's possible Google Adsense changed the values to avoid including confidential info.
I have an MVC 4 app and am using the default authentication provider. I'm not using persistent cookies.
I don't have any problems in development but when hosted at HostGator, I SOMETIMES get logged off when I try to create a new item (HTTP POST). When this happens, I end up at the log on page like I wasn't authenticated.
HostGator does NOT have the app on multiple web servers so I'm thinking I shouldn't have to worry about machinekey stuff. Am I wrong?
When this happens, I just log in again and create the item again and it will succeed. Once this happens, I can't recreate the issue. I try reopening the browser and even different browsers but creating items will always work. It only seems to happen again if I try much later.
Some additional info, the timeout is set to 2880 (the default for an MVC project), which I know is long but I can't see how it would be related. Still, thought I'd mention it.
So I can't look at IIS logs or event viewer to get any idea what could be happening but I can add more logging to the app. Can anyone provide ideas for what to check or what logging to add to diagnose?
Thanks
EDIT
I realized that I could get to the IIS logs so I compared the POST that succeeded and the one that failed and immediately noticed something.
When I first did the GET to load the Item/Create page/view, the cs-username was populated but when I did the POST to create the item, it was gone. I can see that when I logged in again and was able to successfully create the item, that POST did have the cs-username populated.
Why would it disappear between the GET and the POST? There was a 7 minute delay from the GET to the POST but I can see I logged on 1 minute before the GET so the session was only 8 minutes old when the post happened. I've double checked that I don't have sessionstate explicitly configured so the default should be 20 minutes. I feel like I'm onto something but not sure exactly what.
Might be worth adding Glimpse, although running that on deployed code is kinda risky. It would have the benefit, though, of letting you see what's actually happening on the server. I've never used HostGator, so I can't say for certain, but if they recycle app pools aggressively, that would invalid your login, and explain why the logoff seems to happen randomly.