When using the search endpoint, I am only getting a max of 15 results, no matter how popular of a search query I use. Setting count to 100 does not make a difference, however it does when sorting by most recent. Does anybody else experience this? Is it a possible bug or on purpose?
Here's an example call:
https://api.twitter.com/1.1/search/tweets.json?q=pluto&result_type=popular&count=100
Docs: https://dev.twitter.com/rest/public/search
I have actually the same problem. I can just tell you, if your request has more result than 15, that you can "repeat" the request checking the last block "search_metadata" in the json file. You get directly the next request to do under "next_results". If there are non more results you will not see this part of code.
Related
My application makes a lot of calls to the graph API to get the properties I need. It is impossible to reduce the number of requests in my case. And for this, I need to understand when the number of requests approaches the limit and that I need to stop doing them so as not to get 429)
The documentation says that the parameter "x-ms-throttle-limit-percentage" should come in the header when the number of requests approaches the limit from 0.8. As I understand it, 0.8 is a coefficient from 1, where 1 is the upper limit of the limit:
https://learn.microsoft.com/en-us/graph/throttling?view=graph-rest-1.0#regular-responses-requests
But I didn’t get this parameter in the header, although Retry-After with TooManyRequests.
How can I get this parameter in the response? Perhaps you need to specify additional parameters for this? Or set up Tenant for this?
Or is there another way to view throttle-limit?
Thanks in advance for your reply)
If you haven't got "x-ms-throttle-limit-percentage" parameter in header response, this means that you haven't consumed more than 0.8 of its limit, its mentioned in docs. please check the screenshot.
You can check service specific throttle limit ,please follow docs ,
We were curious to know, what service you were hitting ?
I'm looking at building a caching layer on top of the YouTube API and making use of the HTTP standard ETag functionality to do this as described here https://developers.google.com/youtube/v3/getting-started#etags
I've done some direct testing of this against the API in most cases it seems to be working - I can get 304 responses etc.
However I'm seeing a few places where the API is returning different ETags when the response has not changed.
In these cases the ETags seem to cycle between a set of values instead of being a single consistent value.
When I pick one of the ETags and use it to send a conditional GET I will sometimes get a 304 back (when it matches) or sometimes get a 200 with a full response (when it was one of the other values) even though the actual response data is the same.
I've found this behaviour in at least two places:
1) youtube/v3/channels part=brandingSettings
In the response here the brandingSettings has a "hints" value which is an array of size 3.
The order of the elements in this array is random and varies on each request however it seems to affect the etag, meaning I get 6 (permutations for 3 items) different possible ETags values for the same data.
Either the array order should be fixed or the ETag generation algorithm should account for this?
2) youtube/v3/channels part=contentDetails
The ETag for the response here seems to vary between 3 different values, despite there being no other differences in the data. In particular the other "etag" value within "items" remains constant.
Is this a bug in the YouTube API Etag implementation? Surely this behaviour will effectively break any caching layer trying to reduce data retrieval from the YouTube API?
I am using rsolr gem to integrate solr search with my RoR app. Now for each search, I need to specify the rows parameter, which is the number of results I want to retrieve. In order to retrieve all results corresponding to a query, I set the rows parameter to a high value as mentioned in this post.
But doing that makes the processing really really slow and I am getting the following error in the rails logs:
[2014-01-11 15:51:08] ERROR WEBrick::HTTPStatus::RequestURITooLarge
[2014-01-11 15:51:08] ERROR TypeError: can't convert nil into an exact number
/home/nish/.rvm/gems/ruby-1.9.2-p320#voylla/gems/activesupport-3.1.10/lib/active_support/core_ext/time/calculations.rb:266:in `-'
/home/nish/.rvm/gems/ruby-1.9.2-p320#voylla/gems/activesupport-3.1.10/lib/active_support/core_ext/time/calculations.rb:266:in `minus_with_duration'
/home/nish/.rvm/gems/ruby-1.9.2-p320#voylla/gems/activesupport-3.1.10/lib/active_support/core_ext/time/calculations.rb:277:in `minus_with_coercion'
/home/nish/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/webrick/accesslog.rb:42:in `setup_params'
/home/nish/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/webrick/httpserver.rb:164:in `access_log'
/home/nish/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/webrick/httpserver.rb:87:in `run'
/home/nish/.rvm/rubies/ruby-1.9.2-p320/lib/ruby/1.9.1/webrick/server.rb:183:in `block in start_thread'
How can I fix this issue? Thanks
Your error is related to RoR, not Solr. It's telling you the problem -- the requested URI is too large. WEBrick is not a production-caliber web server, and v1.9.3 appears to limit http request length to 2083 (per this other SO question.)
The short-term fix? Use a web server that doesn't limit your requested URI length to something so short.
However, that's just one part of the fix -- the process you're engaging in will grow in linear or worse fashion in terms of execution time relative to the number of results. Not only does the number of results affect performance, but also the size of the documents being retrieved.
Can you share your requirements that led to an implementation where all results are returned with each query?
From the Solr FAQ:
This is impractical in most cases. People typically only want to do
this when they know they are dealing with an index whose size
guarantees the result sets will be always be small enough that they
can feasibly be transmitted in a manageable amount -- but if that's
the case just specify what you consider a "manageable amount" as your
rows param and get the best of both worlds (all the results when your
assumption is right, and a sanity cap on the result size if it turns
out your assumptions are wrong)
I have a YQL query that returns data that I know for sure will not ever change.
In order to avoid rate limits, I was thinking of adding a maxage header to the yql response.
Now I'm wondering what a reasonable value would be (in the case where I know for certain that the response will never ever change): a year ? 10 years ? more ?
Are there any specificities as to the way yql would treat the maxage header ?
Nice article on maxAge and how to use it: http://www.yqlblog.net/blog/2010/03/12/avoiding-rate-limits-and-getting-banned-in-yql-and-pipes-caching-is-your-friend/ . This should answer most of your queries about max age.
For your second question, if the response will never ever change, why even make an API call in the first place? You could eliminate the network latency altogether and have a conf/property file having the response on your server itself.
Am not quite sure if I understood what you meant by if there were any specifications to the way YQL would treat the header but will try to answer it to best of my knowledge. From the link I shared earlier, following are a few lines:
Secondly you can just ask YQL to cache the response to a statement for longer – just append the _maxage query parameter to your call and the result will be stored in cache for that length of time (but not shorter than it would have been originally):
http://query.yahooapis.com/v1/public/yql?q=select * from weather.forecast where location=90210&_maxage=3600
This is really useful when you’re using output from a table that’s not caching enough or an XML source without having to do any open table work
Hope this helps.
I am using the NWS REST API as my weather service for an app I am making. I was initially reluctant to use NWS because of its bad documentation, but I couldn't resist as it is offered completely free.
Now that I am trying to use it, I am running into some difficulty. When making a request for multiple days, the minimum temperature appears nil for several days.
(EDIT: As I have been testing the API more I have found that it is not always the minimum temperatures that are nil. It can be a max temp or a precipitation, it seems completely random. If you would like to make test calls using their web interface, you can do so here: http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdBrowserByDay.htm
and here: http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdXML.htm)
Here is an example of a request the minimum temperatures are empty: http://graphical.weather.gov/xml/sample_products/browser_interface/ndfdBrowserClientByDay.php?listLatLon=40.863235,-73.714780&format=24%20hourly&numDays=7
Surprisingly, on their website, the minimum temperatures are available:
http://forecast.weather.gov/MapClick.php?textField1=40.83&textField2=-73.70
You'll see under the Minimum temperatures that it is filled with about 5 (sometimes less, it is inconsistent) blank fields that say <value xsi:nil="true"/>
If anybody can help me it would be greatly appreciated, using the NWS API can be a little overwhelming at times.
Thanks,
The nil values, from what I can understand of the documentation, here and here, simply indicate that the data is unavailable.
Without making assumptions about NOAA's data architecture, it's conceivable that the information available via the API may differ from what their website displays.
Missing values are represented by an empty element and xsi:nil=”true” (R2.2.1).
Nil values being returned seems to involve the time period. Notice the difference between the time-layout keys (see section 5.3.2) in 1 in these requests:
k-p24h-n7-1
k-p24h-n6-1
The data times are different.
<layout-key> element
The key is derived using the following convention:
“k” stands for key.
“p24h” implies a data period length of 24 hours.
“n7” means that the number of data times is 7.
“1” is a sequential number used to keep the layout keys unique.
Here, startDate is the factor. Leaving it off includes more time and might account for some requested data not yet being available.
Per documentation:
The beginning day for which you want NDFD data. If the string is empty, the start date is assumed to be the earliest available day in the database. This input is only needed if one wants to shorten the time window data is to be retrieved for (less than entire 7 days worth), e.g. if user wants data for days 2-5.
I'm not experiencing the randomness you mention. The folks on NOAA's Yahoo! Groups forum might be able to tell you more.