I am making Google API request through application using RestClient library to get address.
Sample request code-
require 'rest-client'
require 'json'
gmaps_api_href = "https://maps.googleapis.com/maps/api/geocode/json?latlng=18.56227673,73.76804232&language=ar"
response = RestClient.get gmaps_api_href
result = JSON.parse(response)['results']
This request works fine on my local machine and it completes within 1-2secs. But on production instance it takes 20secs to finish one request.
Due to some security measures, we can not access production instance directly. So I am unable to find pin point for this delay.
After doing trial and error, we found that
If we make request using CURL, it takes 1 sec on server
If we make request using Net::HTTP, it takes 20sec to complete same as we were observed for RestClient.
If we make request using WebRequest in small .net app, that request complete within 1 secs.
Its difficult for me to get difference between above observations.
Please let me know why it is so? and what changes I have to do to make it work in my Rails App?
Are you using a Google API key? Your example does not show use of an API key. if not, I'd guess you are getting rate-limited by Google. On your server, you've probably already deployed a version of this app, which made lots of requests to Google without an API key in the fairly recent past, and Google noticed and it's rate-limiting software may be slowing down your requests made from that server. While your local machine hasn't in the past made an enormous amount of requests to the google api, so is not being rate-limited by google's servers.
It's possible Google's rate-limiting is paying some attention (for now!) to User-Agent, and the different user-agent sent by Curl somehow evades Google's rate-limiting that was triggered by the requests sent by RestClient with it's User-Agent (and RestClient may use net-http under the hood, and have the same User-Agent as it).
While one would hope that if you were rate-limited you'd get a "429 Too Many Requests" error response instead of just a slow response, it's possible RestClient hides this from you (I haven't used RestClient), and I've also seen some unpredictable behavior from Google rate-limiting defenses, especially when not using an API key on a service that requires one for all but a few sample requests. I have seen things similar to what you report in that case.
My guess is you're being rate limited because you are not using an API key. Get and use an API key from Google. Google still has rate limits when you are using an API key, but they are clearly advertised (for free? 2500 per-day, and no more than 10 per second. more if you pay) and should give more clear and predictable error messages when exceeded. That's part of why Google requires the api key, so they can reliably rate-limit you in clear ways.
https://developers.google.com/maps/documentation/geocoding/usage-limits
https://developers.google.com/maps/documentation/geocoding/intro#BYB
Related
I am currently working on an application that makes requests using the .NET SDK for the Microsoft Graph API. Specifically to retrieve information about users and their OneDrives.
Microsoft throttles API requests by returning an HTTP 429 status code, and I have implemented a back-off using the Retry-After header. I have noticed however that I seem to be getting throttled after only a handful of requests.
I have also been using Microsoft Graph Explorer to test some of my API calls and have noticed that I never seem to get a 429 response when accessing the API via that method. After also seeing reports of people having issues with the OneDrive client on Linux that they managed to work around by changing their User-Agent header, I thought maybe I needed to set a User-Agent for my requests.
The result is that it seems that if I set the User-Agent header to be something like Mozilla/5.0 then all the throttling issues seem to disappear. I have searched high and low and so far haven't managed to find any documentation on what a valid User-Agent should be, and I would prefer to avoid making my app impersonate a browser, so I wondered whether there was any guidance or documentation around that I might have missed?
For instance, a User-Agent of Mozilla/5.0 seems to result in no throttling, but MyApp/1.0 results in throttling.
There isn't any guidance on the User-Agent header and to be honest, I'm not sure why this would have any effect on throttling.
Throttling in Microsoft Graph is handled by the underlying service you're interacting with. For example, the /notes/ endpoint throttling is governed by OneNote while /messages is governed by Exchange.
In most cases, OneDrive will throttle by number of concurrent requests, per app, per user. So using Delegated permissions, your app should generally be able to concurrently upload 4 files without a problem. Any more than that and you will start to see 429 responses.
there are some best-practices from Microsoft to handle throttling:
https://learn.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online#how-to-decorate-your-http-traffic-to-avoid-throttling
you should decorate your http user-agent as following:
NONISV|CompanyName|AppName/Version
Identify as NONISV and include Company Name, App Name separated by a pipe character and then adding Version number separated with a slash character
or
ISV|CompanyName|AppName/Version
Identify as ISV and include Company Name, App Name separated by a pipe character and then adding Version number separated with a slash character
I am making an api backend that makes use of another api,for example Twitter. (Please note the actual api isn't twitter but I am using them as an example).
Let's say that Twitter has a limit on the number of calls that can be made to their api and above this limit, it starts to charge my credit card. This is why it is very important to me that no one misuses my api.
I want to prevent people from looking at my frontend code and seeing which endpoint it hits, because if a malicious person were to do this, I would very quickly go over the limit and have to pay $$$.
My frontend code uses a get call to mybackend.com/twitter/api
Is it enough to simply add an Access-Control-Allow-Origin header to my backend?
headers['Access-Control-Allow-Origin'] = 'myfrontend.com'
The reason I am asking this is because I noticed that typing mybackend.com/twitter/api directly into the browser worked, which is not what I would expect if I had access-control-allow-origin set to a specific website.
Am I doing something wrong? How do I prevent someone from simply writing a script to hit my backend since it is clear that just typing it into the url of my browser works, despite me having an access-control-allow-origin header.
There are two possible solutions for your problem. You can try to implement a request signature for your API, to know exactly the source of it on your backend. You can take a look on how this works here.
The second option, and for me, a one witch fits your problem better, is to set up a Denial of service approach on your server Load Balancer to prevent multiple requests from a same origin, and so, don't let those kind of malicious requests hit your backend.
The Gmail iOS API under the hood appears to be creating 'POST' requested with a payload that specifies an action. So for example if I want to get a single message by ID, I would expect (at least since the documentation on the API specifies it) that I would create a GET request with the correct URL schema. Instead the Gmail iOS implementation does not and utilizes more of an RPC approach. Unfortunately this means all POST requests have a default timeout of 240 seconds and cannot be changed, apparently an iOS bug/weird implementation. I can create a timer and cancel the request but why not make the implementation RESTful?
Yes, this is a quark with the Google iOS client libraries in general--they do not follow the standard REST semantics of the APIs like all the other client libraries use. I imagine due to some limitation in iOS. In any case, https://code.google.com/p/google-api-objectivec-client/ would be the best place to read more about it, discuss and file bugs. (Not specific to the gmail-api.)
The library was written to create JSON-RPC requests because those have always supported a batch request model. A mechanism for batch REST requests was added much later to Google's JSON API servers.
The timeout issue on POSTs was resolved in iOS 6, according to discussions online like this.
The project site does have links to the library's discussion group and issue tracker.
I need to do a keyword based data fetching on Twitter. I looked up the documentation and "POST statuses/filter" seemed like the best option. However, I do not understand how the rate limiting works. Does this mean that I can fire this request repeatedly? If yes, at what rate should I do so? Or do I have to fire the request only once and keep on getting data continuously? They have given clear explanations for the REST API. There's even a page showing the number of requests permissible in a 15 minute window for each REST API method. I was unable to find something similar for "POST statuses/filter".
From what I've been researching about using the Streaming API there aren't any rate limits because you just make the request once to open the connection, then you keep it open and you are sent a stream (hence the name) of tweets.
Once applications establish a connection to a streaming endpoint, they
are delivered a feed of Tweets, without needing to worry about polling
or REST API rate limits.
https://dev.twitter.com/docs/streaming-apis/streams/public
I have a Twitter app that works fantastic locally - it searches for keywords then for each user it grabs their info using Hpricot to parse the xml e.g.
Hpricot(open("http://twitter.com/users/show/"+myuser+".xml"))
Works fine locally but when I go love it fails. Looking at my log I get this error:
OpenURI::HTTPError (400 Bad Request):
The weird thing is though, sometimes it works.
This has been a recurring problem for a few days now and driving me nuts. Will hug anyone with a solution :)
It's almost definitely rate-limiting - http://apiwiki.twitter.com/HTTP-Response-Codes-and-Errors . Haven't seen 400s returned for anything other than rate limit before, though the docs say there could be an accompanying message that tells you more exactly what's wrong.
You might be able to get whitelisted for more queries, see http://twitter.com/help/request_whitelisting .
You are probably making too many requests. You are allowed to make 150 REST API calls per hour unauthenticated.
REST API Rate Limiting
Unauthenticated calls are permitted 150 requests per hour. Unauthenticated calls are measured against the public facing IP of the server or device making the request.
OAuth calls are permitted 350 requests per hour and are measured against the oauth_token used in the request.