org.jsoup.HttpStatusException: HTTP error fetching URL. Status=401, URL= anyurl
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:449)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:465)
at org.jsoup.helper.HttpConnection$Response.execute(HttpConnection.java:424)
at org.jsoup.helper.HttpConnection.execute(HttpConnection.java:178)
at org.jsoup.helper.HttpConnection.get(HttpConnection.java:167)
What I am trying to do is a redirect from a url that needs authentication. The url works fine when I open it in my browser, but doesn't work while using Jsoup. Anybody who has solved this issue?
doc = Jsoup.connect(url)
.userAgent("Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36")
.timeout(0).get();
Related
I'm trying to refresh the oauth2 token using ClientOAuth2.Token.refresh() but keep getting the error sometimes:
{"error":"invalid_grant","error_description":"Session not active"}
This is the request which I captured by Fiddler
POST [URL]= HTTP/1.1
Host: [URL]
Connection: keep-alive
Content-Length: 2250
Accept: application/json, application/x-www-form-urlencoded
Origin: http://localhost:8080
Authorization: Basic YXNpbW92LWRldi1laGlzLXdlYjo=
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36
Sec-Fetch-Mode: cors
Content-Type: application/x-www-form-urlencoded
Sec-Fetch-Site: cross-site
Referer: http://localhost:8080/
Accept-Encoding: gzip, deflate, br
Accept-Language: en,vi;q=0.9,de;q=0.8,vi-VN;q=0.7,en-US;q=0.6,en-AU;q=0.5
refresh_token=[token]&grant_type=refresh_token
Maybe somethings wrong with my setting on KeyCloak account. Are there any suggestions?
You need to add the scope offline_access when retrieving the original token, otherwise you won't be able to refresh it when the related user session is not active (this is specific to OIDC, not oauth2).
Could be that your refresh token grant message is incomplete - missing a client ID or offline access scope - see the Refresh Token Grant section of my article on OAuth messages.
It is because you are logged out from you client application. That's why it giving you the error.
I'm currently using the form data method to upload a video to youtube api, but I have been receiving the following respond error
"message": "Uploads must be sent to the upload URL. Re-send this request to https://www.googleapis.com/upload/youtube/v3/videos?part=snippet,statistics,contentDetails&mine=true&access_token=ya29.ImGUB0a_t9jFeGl1jxduJWSr6qgOKtzb7UpJNMxjyX-U7tkwCTvBx1nEMWunodpyKoULnQqfBkxgzxvs1S27_LTPRIqkTokWHh7quTMssHpRGRpaQiCJzGxKV3wxWjypqQMk"
Youtube v3 API link
The following is my request
Request URL: https://www.googleapis.com/youtube/v3/videos?part=snippet,statistics,contentDetails&mine=true&access_token=ya29.ImGUB0a_t9jFeGl1jxduJWSr6qgOKtzb7UpJNMxjyX-U7tkwCTvBx1nEMWunodpyKoULnQqfBkxgzxvs1S27_LTPRIqkTokWHh7quTMssHpRGRpaQiCJzGxKV3wxWjypqQMk
Request Method: POST
Status Code: 400
Request Headers:
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryFpMCekYXv6NAPEKN
Origin: http://localhost:8080
Referer: http://localhost:8080/
Sec-Fetch-Mode: cors
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36
X-Requested-With: XMLHttpRequest
Query string:
part: snippet,statistics,contentDetails
mine: true
access_token: ya29.ImGUB0a_t9jFeGl1jxduJWSr6qgOKtzb7UpJNMxjyX-U7tkwCTvBx1nEMWunodpyKoULnQqfBkxgzxvs1S27_LTPRIqkTokWHh7quTMssHpRGRpaQiCJzGxKV3wxWjypqQMk
Can someone tell me what went wrong? very thankful.
Google Natural Language API has been working in my iOS app up until yesterday. The API started returning "permission denied" errors as of this morning. E.g:
{
"error": {
"code": 403,
"message": "The caller does not have permission",
"status": "PERMISSION_DENIED"
}
}
Example request:
POST /v1/documents:analyzeEntities?key=..... HTTP/1.1
Host: language.googleapis.com
Content-Type: application/json
Connection: keep-alive
X-Ios-Bundle-Identifier: .....
Accept: */*
Accept-Language: en-us
Content-Length: 291
Accept-Encoding: gzip, deflate
User-Agent: CardScanner/1 CFNetwork/808.2.16 Darwin/15.6.0
{"encodingType":"UTF8","document":{"type":"PLAIN_TEXT","content":"....."}}
Billing is enabled for the account (with a balance of $0). The account also has 36 days left on the trial period.
The key matches the value in the Google Cloud Platform API dashboard. I have also tried regenerating the key, and using the new key in the app.
I have also tried enabling key restrictions for iOS devices, and included the "X-Ios-Bundle-Identifier" header with the app bundle identifier.
The app also uses the Google Vision API which works without issues. Calls to the Vision API do respond to changes to the key restrictions.
Calls made from the demo page also show a permissions error message. Calls from the API explorer do work however.
Edit:
The error is also happening on the demo on the product web page. Tracing the error in Charles shows the same "permission denied" response being returned to the web page:
Edit:
Below is an example of the HTTP request and response captured from the demo page. The request and resulting error is almost identical to my app, except that the demo seems to be using http 2, whereas my app is using http 1.
HTTP request:
:method: POST
:authority: language.googleapis.com
:scheme: https
:path: /v1/documents:analyzeEntities?key=.....
content-length: 250
origin: https://cloud.google.com
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36
content-type: text/plain;charset=UTF-8
accept: */*
referer: https://cloud.google.com/natural-language/
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.8
{"document":{"type":"PLAIN_TEXT","content":"Google, headquartered in Mountain View, unveiled the new Android phone at the Consumer Electronic Show. Sundar Pichai said in his keynote that users love their new Android phones."},"encodingType":"UTF16"}
HTTP response:
:status: 403
vary: Origin
vary: X-Origin
vary: Referer
content-type: application/json; charset=UTF-8
content-encoding: gzip
date: Sun, 26 Feb 2017 14:52:24 GMT
server: ESF
cache-control: private
content-length: 128
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
access-control-allow-origin: https://cloud.google.com
access-control-expose-headers: content-encoding,date,server,content-length
alt-svc: quic=":443"; ma=2592000; v="35,34"
{
"error": {
"code": 403,
"message": "The caller does not have permission",
"status": "PERMISSION_DENIED"
}
}
We are aware of this issue and should be fixed now. Let us know if you still see the same problem.
I have experienced the same behaviour and I believe it is an undocumented change at Google's end.
I can now only make the call work via OAuth, despite the documentation suggesting that an API key is sufficient. Up until very recently it was possible to make a call with an API key only and no restrictions configured on the key.
My "answer" for what it's worth is to provide feedback on the documentation page to complain that the documentation doesn't match the behaviour. I think the wording on the page may have changed recently, but it still suggests an API key should work for testing purposes.
"TRY NATURAL LANGUAGE API" button from Google Cloud Vision API demo page doesn't work either! Looks like some bug, hope they fix it soon...
I see these invalid HTTP requests in the server log.
The request URI includes scheme+hostname+port.
1.2.3.4 [13/Jan/2017:04:20:01 +0000] GET http://www.DOMAIN.hu:80/munkaugyi-segedanyagok/minimalber-2017-kormanyrendelet HTTP/1.1 403 http://m.facebook.com Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Mobile/14C92 [FBAN/FBIOS;FBAV/75.0.0.48.61;FBBV/45926345;FBRV/46260258;FBDV/iPhone8,1;FBMD/iPhone;FBSN/iOS;FBSV/10.2;FBSS/2;FBCR/TelekomHU;FBID/phone;FBLC/en_US;FBOP/5]
All other requests from the same visitor suggest a legitimate user.
Could it be the Facebook app for iPhone?
It turns out most webservers support absolute request URI-s.
https://www.rfc-editor.org/rfc/rfc2616#section-3.2.1
An HTTP client may send requests with absolute request URI-s like the ones in proxy requests but with the same domain name.
I have build a number of REST interfaces in app engine and while debugging a request on my iPhone client and via Chrome, I noticed that two identical requests had different responses sizes:
"GET /card?omid=2 HTTP/1.1" 200 1468 - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36"
"GET /card?omid=2 HTTP/1.1" 200 10669 - "Null Society 1.83 (iPhone; iPhone OS 6.1.3; en_US)"
Note that the identical request from the iPhone client is 7x larger than the one from Chrome. These requests were a minute apart, and my server-side code does not distinguish between clients for determining responses. In this particular request, the server merely reads the query variable, "omid", and returns the appropriate data.
The only thing I can think of is that app engine natively compresses data by client? Seems weird though. Any help would be appreciated!
In the response section of the GAE docs it notes:
If the client sends HTTP headers with the request indicating that the
client can accept compressed (gzipped) content, App Engine compresses
the response data automatically and attaches the appropriate response
headers. It uses both the Accept-Encoding and User-Agent request
headers to determine if the client can reliably receive compressed
responses. Custom clients can indicate that they are able to receive
compressed responses by specifying both Accept-Encoding and User-Agent
headers with a value of gzip.
https://developers.google.com/appengine/docs/python/#Python_Responses
That sounds like it could explain what you are seeing.