I am currently working on an application that makes requests using the .NET SDK for the Microsoft Graph API. Specifically to retrieve information about users and their OneDrives.
Microsoft throttles API requests by returning an HTTP 429 status code, and I have implemented a back-off using the Retry-After header. I have noticed however that I seem to be getting throttled after only a handful of requests.
I have also been using Microsoft Graph Explorer to test some of my API calls and have noticed that I never seem to get a 429 response when accessing the API via that method. After also seeing reports of people having issues with the OneDrive client on Linux that they managed to work around by changing their User-Agent header, I thought maybe I needed to set a User-Agent for my requests.
The result is that it seems that if I set the User-Agent header to be something like Mozilla/5.0 then all the throttling issues seem to disappear. I have searched high and low and so far haven't managed to find any documentation on what a valid User-Agent should be, and I would prefer to avoid making my app impersonate a browser, so I wondered whether there was any guidance or documentation around that I might have missed?
For instance, a User-Agent of Mozilla/5.0 seems to result in no throttling, but MyApp/1.0 results in throttling.
There isn't any guidance on the User-Agent header and to be honest, I'm not sure why this would have any effect on throttling.
Throttling in Microsoft Graph is handled by the underlying service you're interacting with. For example, the /notes/ endpoint throttling is governed by OneNote while /messages is governed by Exchange.
In most cases, OneDrive will throttle by number of concurrent requests, per app, per user. So using Delegated permissions, your app should generally be able to concurrently upload 4 files without a problem. Any more than that and you will start to see 429 responses.
there are some best-practices from Microsoft to handle throttling:
https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic-to-avoid-throttling
you should decorate your http user-agent as following:
NONISV|CompanyName|AppName/Version
Identify as NONISV and include Company Name, App Name separated by a pipe character and then adding Version number separated with a slash character
or
ISV|CompanyName|AppName/Version
Identify as ISV and include Company Name, App Name separated by a pipe character and then adding Version number separated with a slash character
Related
We have an API published on WSO2. It works perfectly.
When I send my request like the picture below, it responses 200 as I wanted:
I just wanted to test my request by adding more deleted=false query. So, I can send request until the request's size is 5.75 KB. I see stil 200 OK nicely. You can see on picture below:
But, if I reach request size 5.76KB by adding 1 more deleted=false query, I see this error:
As I searched on internet, I see that the REST API supports Uniform Resource Locators (URLs) with a length of up to 6000 characters.
My question is, how can I extend this limit? Is there any way to do that ?
As per the shared screenshot, it seems the Backend itself is responding back with a 400 Bad Request status code. The API Manager doesn't have any restrictions on large query parameters in the URI. So, this error is coming up from your actual Backend service, which is not able to handle a large request.
To confirm this behavior, you can enable the WIRE logs in the API Manager server and troubleshoot the behavior. If the request is dispatched to the Backend and the Backend is responded with 400 Bad Request means, the Backend is only capable of handling requests up to 5.75 KB in your case.
Also, as an alternate check, you can also try invoking the actual Backend service URL from the Postman (direct invocation and not via WSO2) and verify the behavior with large requests.
Given below are few documentations related to enabling WIRE logs and understanding the WIRE logs
WSO2 API Manager v3.1.0: Enable WIRE Logs
WSO2 API Manager v2.6.0: Enable WIRE Logs
How to read and understand WIRE Logs
I'm running an IIS web site (built using ASP.NET/MVC) that among other things collects files from multiple agents that anonymously upload the files via BITS.
I need to make sure that only files uploaded from known sources as well as matching certain predefined file name pattern will be accepted by IIS. All other BITS upload attempts must be cancelled.
As I understand, BITS uses an ad hoc protocol over HTTP 1.1 using "BITS_POST" verb. So, ideally, I'd like to hook into IIS, analyze a BITS_POST request info and if it does not satisfy my pre-conditions, drop the request.
I've tried to create and register a filter implementing IActionFilter.OnActionExecuting, but it seems that my filter does not receive BITS_POST requests.
I'd be glad to hear if somebody have implemented similar BITS related solutions and how this was done. Anyway, other ideas are welcome too.
Regards,
Natan
I have never worked with BITS, frankly i dont know what is it.
What i usually do is such situations is implement an HTTP module. On its begin request event, you can iterate through incoming HTTP request data and decide to stop processing the request if data is not complying with requirements. You have full access to HttpContext.Current.Request object from HTTP module code.
With HTTP modules, you can execute .NET code even before entering the ASP.NET pipeline.
I am making an api backend that makes use of another api,for example Twitter. (Please note the actual api isn't twitter but I am using them as an example).
Let's say that Twitter has a limit on the number of calls that can be made to their api and above this limit, it starts to charge my credit card. This is why it is very important to me that no one misuses my api.
I want to prevent people from looking at my frontend code and seeing which endpoint it hits, because if a malicious person were to do this, I would very quickly go over the limit and have to pay $$$.
My frontend code uses a get call to mybackend.com/twitter/api
Is it enough to simply add an Access-Control-Allow-Origin header to my backend?
headers['Access-Control-Allow-Origin'] = 'myfrontend.com'
The reason I am asking this is because I noticed that typing mybackend.com/twitter/api directly into the browser worked, which is not what I would expect if I had access-control-allow-origin set to a specific website.
Am I doing something wrong? How do I prevent someone from simply writing a script to hit my backend since it is clear that just typing it into the url of my browser works, despite me having an access-control-allow-origin header.
There are two possible solutions for your problem. You can try to implement a request signature for your API, to know exactly the source of it on your backend. You can take a look on how this works here.
The second option, and for me, a one witch fits your problem better, is to set up a Denial of service approach on your server Load Balancer to prevent multiple requests from a same origin, and so, don't let those kind of malicious requests hit your backend.
I am making Google API request through application using RestClient library to get address.
Sample request code-
require 'rest-client'
require 'json'
gmaps_api_href = "https://maps.googleapis.com/maps/api/geocode/json?latlng=18.56227673,73.76804232&language=ar"
response = RestClient.get gmaps_api_href
result = JSON.parse(response)['results']
This request works fine on my local machine and it completes within 1-2secs. But on production instance it takes 20secs to finish one request.
Due to some security measures, we can not access production instance directly. So I am unable to find pin point for this delay.
After doing trial and error, we found that
If we make request using CURL, it takes 1 sec on server
If we make request using Net::HTTP, it takes 20sec to complete same as we were observed for RestClient.
If we make request using WebRequest in small .net app, that request complete within 1 secs.
Its difficult for me to get difference between above observations.
Please let me know why it is so? and what changes I have to do to make it work in my Rails App?
Are you using a Google API key? Your example does not show use of an API key. if not, I'd guess you are getting rate-limited by Google. On your server, you've probably already deployed a version of this app, which made lots of requests to Google without an API key in the fairly recent past, and Google noticed and it's rate-limiting software may be slowing down your requests made from that server. While your local machine hasn't in the past made an enormous amount of requests to the google api, so is not being rate-limited by google's servers.
It's possible Google's rate-limiting is paying some attention (for now!) to User-Agent, and the different user-agent sent by Curl somehow evades Google's rate-limiting that was triggered by the requests sent by RestClient with it's User-Agent (and RestClient may use net-http under the hood, and have the same User-Agent as it).
While one would hope that if you were rate-limited you'd get a "429 Too Many Requests" error response instead of just a slow response, it's possible RestClient hides this from you (I haven't used RestClient), and I've also seen some unpredictable behavior from Google rate-limiting defenses, especially when not using an API key on a service that requires one for all but a few sample requests. I have seen things similar to what you report in that case.
My guess is you're being rate limited because you are not using an API key. Get and use an API key from Google. Google still has rate limits when you are using an API key, but they are clearly advertised (for free? 2500 per-day, and no more than 10 per second. more if you pay) and should give more clear and predictable error messages when exceeded. That's part of why Google requires the api key, so they can reliably rate-limit you in clear ways.
https://developers.google.com/maps/documentation/geocoding/usage-limits
https://developers.google.com/maps/documentation/geocoding/intro#BYB
The Gmail iOS API under the hood appears to be creating 'POST' requested with a payload that specifies an action. So for example if I want to get a single message by ID, I would expect (at least since the documentation on the API specifies it) that I would create a GET request with the correct URL schema. Instead the Gmail iOS implementation does not and utilizes more of an RPC approach. Unfortunately this means all POST requests have a default timeout of 240 seconds and cannot be changed, apparently an iOS bug/weird implementation. I can create a timer and cancel the request but why not make the implementation RESTful?
Yes, this is a quark with the Google iOS client libraries in general--they do not follow the standard REST semantics of the APIs like all the other client libraries use. I imagine due to some limitation in iOS. In any case, https://code.google.com/p/google-api-objectivec-client/ would be the best place to read more about it, discuss and file bugs. (Not specific to the gmail-api.)
The library was written to create JSON-RPC requests because those have always supported a batch request model. A mechanism for batch REST requests was added much later to Google's JSON API servers.
The timeout issue on POSTs was resolved in iOS 6, according to discussions online like this.
The project site does have links to the library's discussion group and issue tracker.