I'm starting to use Microsoft Graph threat assessment API to report Phishing Website URL.
(Ref: https://learn.microsoft.com/en-us/graph/api/informationprotection-post-threatassessmentrequests?view=graph-rest-1.0&tabs=http)
My use-case is automatic reporting and manual reporting via Slack Command.
But throttling is very strict, so I get "429" response immediately.
"code": "ActivityLimitReached",
"message": "The client application has been throttled for reaching an activity limit. The request may be repeated after a delay, the length of which may be specified in a 'Retry-After' header.",
Does anyone know a workaround for the throttling?
As far as I confirmed, throttling is 1 request per 15 minutes (Limit per resource) by default.
(150 requests per 15 minutes (Limit per tenant) though)
Ref: https://learn.microsoft.com/en-us/graph/throttling#information-protection
I would try the following best practice to avoid/handle the throttling.
When you implement error handling, use the HTTP error code 429 to detect throttling. The failed response includes the Retry-After response header. Backing off requests using the Retry-After delay is the fastest way to recover from throttling because Microsoft Graph continues to log resource usage while a client is being throttled.
Wait the number of seconds specified in the Retry-After header.
Retry the request.
If the request fails again with a 429 error code, you are still being throttled. Continue to use the recommended Retry-After delay and retry the request until it succeeds.
The guidelines for throttling is already provided by Microsoft Graph team and it's straightforward. Please go through this doc, look out for best practices to avoid throttling, to handle throttling etc and think about throttling/batching to see if it suits your scenario (so you can optimize the calls).
If retry-after header doesnt exists then it would be tough - thats the way to handle throttling, unless if any alternate way exists provided. If you still believe Microsoft to implement this feature, consider creating a new user voice.
Update: #rung created a new uservoice on this.
I am working on the similar use case. I am planning to submit the Msg Id/URL/file to Microsoft for phishing assessment using API . I am stuck with an error that shows " "code": "Unauthorized",
"message": "Required authentication information is either missing or not valid for the resource.",
"
I would highly appreciate your help!
Related
We have an API published on WSO2. It works perfectly.
When I send my request like the picture below, it responses 200 as I wanted:
I just wanted to test my request by adding more deleted=false query. So, I can send request until the request's size is 5.75 KB. I see stil 200 OK nicely. You can see on picture below:
But, if I reach request size 5.76KB by adding 1 more deleted=false query, I see this error:
As I searched on internet, I see that the REST API supports Uniform Resource Locators (URLs) with a length of up to 6000 characters.
My question is, how can I extend this limit? Is there any way to do that ?
As per the shared screenshot, it seems the Backend itself is responding back with a 400 Bad Request status code. The API Manager doesn't have any restrictions on large query parameters in the URI. So, this error is coming up from your actual Backend service, which is not able to handle a large request.
To confirm this behavior, you can enable the WIRE logs in the API Manager server and troubleshoot the behavior. If the request is dispatched to the Backend and the Backend is responded with 400 Bad Request means, the Backend is only capable of handling requests up to 5.75 KB in your case.
Also, as an alternate check, you can also try invoking the actual Backend service URL from the Postman (direct invocation and not via WSO2) and verify the behavior with large requests.
Given below are few documentations related to enabling WIRE logs and understanding the WIRE logs
WSO2 API Manager v3.1.0: Enable WIRE Logs
WSO2 API Manager v2.6.0: Enable WIRE Logs
How to read and understand WIRE Logs
We have a registered AAD application marked as multi-tenant. We are using this App ID to generate a Token for Microsoft Graph.
The first user is a Global Admin in the Tenant where the app is registered.
The second user is part of another Tenant.
When the second user tried to use Microsoft Graph to get information from OneDrive, we sometimes get an HTTP 429 activityLimitReached error.
We read the guide about throttling and it says to repeat the request after the Retry-After value from the response header. But in our case there is no Retry-After field in the response.
We received this error by executing one request per day. Also, after receiving the 429, we can retry and get a successful result (after several attempts). This error appears only in the OneDrive, the other services are OK.
What can we do to avoid 429 error? How can we check the current limit or increase it?
Example of request
GET https://graph.microsoft.com/v1.0/users/:userId/drives
Example of response
HTTP/1.1 429
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: application/json
request-id: 377d2cdf-7be3-4286-819a-46060330365f
client-request-id: 377d2cdf-7be3-4286-819a-46060330365f
x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"West Europe","Slice":"SliceA","Ring":"4","ScaleUnit":"000","Host":"AGSFE_IN_13","ADSiteName":"AMS"}}
Duration: 170.5668
Strict-Transport-Security: max-age=31536000
Date: Wed, 23 May 2018 11:39:08 GMT
{
"error": {
"code": "activityLimitReached",
"message": "The request has been throttled",
"innerError": {
"request-id": "377d2cdf-7be3-4286-819a-46060330365f",
"date": "2018-05-23T11:39:09"
}
}
}
What can we do to avoid 429 error? How can we check the current limit or increase it?
To avoid the 429 error, we must control our request, don't do too many request within limited time. The limit issue is known issue we canot increase it now.
Setting and publishing exact throttling limits sounds very straightforward, but in fact, it's not the best way to go. We continually monitor resource usage on SharePoint Online. Depending on usage, we fine-tune thresholds so users can consume the maximum number of resources without degrading the reliability and performance of SharePoint Online.
Above reference is from MS documentation about throttling and OneDrive for Business/SharePoint: https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online
I would suggest going to UserVoice for Graph and suggest an improvement (or upvote an existing one). The feedback helps Product Group prioritize future work based on the interest in those suggested improvements. But based on the above official docs, the best solution is still to contorl our request but not the feature request.
I do not have much experience with the onedrive-api but I have certainly experienced throttling when using the onenote-api. This post on how onenote throttles may be more useful than the generic microsoft graph link in your question. In particular, long calls against the api will result in throttling occurring much sooner than more targetted calls (and you need to let calls finish before issuing new calls, be very careful with queueing multiple curl requests). Once you have been throttled trying the same call over and over will likely increase the length of time throttling occurs (I haven't seen as full day but I have seen several hours before).
I presume (but am not sure) that all microsoftgraph-api calls will internally count towards resource limits, but if you direct onedrive calls against the onedrive-api this will not count against the microsoft-graph limits and may allow the requests you need without throttling.
I would definitely recommend getting a second access token for the direct onedrive api (you can use the same refresh token) and try this approach.
GET https://www.onedrive.com/v1.0/users/:userId/drives
If it is one user in particular that has issues maybe they have exceeded their tenant resources?
I'm currently running a web application that uses Microsoft Graph's API and we encountered the following message today which severely impacted our application, for a whole day:
"error": {
"code": "ErrorTooManyObjectsOpened",
"message": "Too many concurrent connections opened., The process failed to get the correct properties.",
"innerError": {
"request-id": "removed",
"date": "2017-12-13T17:01:14"
}
}
please note that the request-id was removed
Let me summarize what our web application does.
Basically, we have 2 email folders that we are actively subscribed to, Junk and Folder A.
If anything hits Folder A, we strip the body of the email message and then move the message to Folder B. The subscription on our Junk folder also strips the body and sends them over to Folder B.
Sometimes the webhook subscription service skips messages that may come at the same time, therefore we have 2 cron jobs in our server that run a script and check Junk/Folder A for any messages every 5 minutes, therefore my assumption is that the cron job runs about 288*2 times per day. Not counting our subscription to the folders, we usually get around 200-300 email messages per day.
Unfortunately Microsoft's Graph error codes page does not provide us with any explanation about this code. I would really appreciate if anyone can explain what this means and how to avoid it from happening.
This is occurring because your application is exceeding the throttling thresholds.
There are several different throttling metrics that can affect Microsoft Graph requests. For a high-level overview, see the Microsoft Graph throttling guidance. Since in this case you're hitting Exchange Online via Graph, you can find more specific information from What throttling values do I need to take into consideration? in the Exchange documentation.
Architecturally, you are making a lot of unnecessary calls into the API. Rather than having both a subscription and a scheduled job, you should use just the webhook subscription and the /delta endpoint.
Each call to the /delta endpoint gives you a token that can be used to fetch any changes to a given resource since the token was originally issued. So regardless of if 1 email came in or 1,000, you only get the new emails.
Once you're using the /delta to find your changes, you then use a webhook only as a "trigger". When you receive the webhook, you can ignore the contents and instead issue a request to /delta. This ensures that you capture every incoming email even if you didn't necessarily receive separate webhook notifications.
There is a bug. After making 500 message move requests, a "cannot copy/move error" occurs. Subsequently, a "429: Too many concurrent connections opened" error occurs. Most applications miss the first error because you continually get the 429 error afterwards.
If you let the application "rest" for 30 minutes, the throttle resets itself and you can continue on. I do not think there is a time limit for hitting the 500 moves. My application did 500 moves after 6.5 hours and then we started getting the error.
And, if you keep trying your move call before the 30 min rest period, it never resets. Also, in the response, the retry-after is null... so, that doesn't help you.
If you find a work around, please let me know. We are trying a few things like setting the category, then manually moving the messages. I am also investigating making a rule the moves them for us or some other job. I cannot find a way to execute a rule from the Graph API.
See this link for more information. Also, the more people who report having this issue, hopefully the sooner it can be resolved. Outlook API Throttling documentation #144
I am using Microsoft Graph for OneNote. I have observed few issues with these endpoints. Sometimes when API's are called concurrently a throttling exception is thrown (HTTP 429). And suddenly next API call returns successful response.
As per documentation, a Retry-After field will be returned in the response header along with HTTP 429 when throttling is active but I have never received this field in the response header.
Is there any other way to get the wait time?
How long I should wait to further avoid this issue? This is a blocker issue for me. Kindly help.
The documentation should be more clearer and note that not all endpoints return a Retry-After. This is because not all endpoints throttle using the same rules.
Some endpoints, such as Exchange, throttle based on a formula that looks at the number of requests within a window time. These endpoints respond with Retry-After to alert you how long you have to wait before you have a new window.
Other endpoints simply throttle based on the number concurrent requests. OneNote is one of these endpoints. Rather than looking at requests over time, it simply limits you to a maximum of 5 concurrent calls. Since throttling isn't dependent on time, it would be impossible to provide a Retry-After value.
This is also why you so often see your initial retry succeed, it means some previous call had completed and there were now 5 or less concurrent requests.
https://blogs.msdn.microsoft.com/onenotedev/2016/01/12/onenote-api-throttling-and-how-to-avoid-it/
This blog post contains more information about how to avoid throttling from OneNote endpoints (and how it works).
I grab data from youtube data api,
image: {
bannerImageUrl: "https://yt3.ggpht.com/-HzM7tqQ-Lbo/V-tnNJuy7JI/AAAAAAAAAnQ/Yb5Fnx29BvcWYYAQ9ReVeKlY8DaSuS8agCL8B/w1060-fcrop64=1,00005a57ffffa5a8-nd-c0xffffffff-rj-k-no/Channel-Art-Template-%2528Photoshop%2529.png",
...
}
as for image banner of the channel, I directly use it in <img> tag.
since I am in development (alot of refresh page happening), then this error kick,
403. That’s an error.
Your client does not have permission to get URL /-HzM7tqQ-Lbo/V-tnNJuy7JI/AAAAAAAAAnQ/Yb5Fnx29BvcWYYAQ9ReVeKlY8DaSuS8agCL8B/w1060-fcrop64=1,00005a57ffffa5a8-nd-c0xffffffff-rj-k-no/Channel-Art-Template-%2528Photoshop%2529.png from this server. (Client IP address: xxx.xxx.xxx.xxx)
Rate-limit exceeded That’s all we know.
So my question is, am I allowed to directly use the image from yt3.ggpht.com and show it in my webpage ?
I search around mention that bandwidth limit, any solution for this ?
Thank you.
Based from Google's Standard Error Responses,
Error code 403 rateLimitExceeded, indicates that the global or overall project rate limits have been exceeded. Recommended action for this is to retry using exponential back-off. You need to slow down the rate at which you are sending the requests.
And with regards to using images from YouTube in your website, I suggest that you please check YouTube's Terms of Service especially permissions and restrictions on the use of content.
You may also check given workaround in this forum. Hope it helps!