I'm using the Twitter API to collect tweets with access through the Academic Research Developer account. I've been trying to pull tweets using the API Search Full Archive feature of Tweepy and read that there is a rate limit of 300 tweets/15 mins. In practice, however, this does not seem to be the case, as I was able to pull ~6000 tweets over a couple minutes before getting a rate limit error. After 15 minutes, I don't see any reset and cannot pull another batch of tweets.
I was wondering whether anyone knows how the rate limit actually works, so that I can better understand how to design my program. Thanks
I was curious about this a short while ago, and took a stab after reading some other posts on here to determine rate limit / its consumption.
Here's what I have for you (assuming you are using tweepy).
print(api.rate_limit_status()['resources'])
gives you all the rate limits for your set up. If you are curious specifically around searches, you can leverage this. I took the liberty to drop in the one I built, feel free to modify it to your liking.
print("Searches Remaining: ", api.rate_limit_status()['resources']['search']['/search/tweets']['remaining'],
"/" ,api.rate_limit_status()['resources']['search']['/search/tweets']['limit'],"")
In my case, it looks like this right now. Searches Remaining: 162 / 180
If you choose to play around with it, and peel it back a bit, you will see the JSON payload containing this in the search section (it is a total of over 1400 lines)
"search": {
"/search/tweets": {
"limit": 180,
"remaining": 162,
"reset": 1660874618
}
Hope this helps!
Related
I was wondering if I could get some help understanding the Youtube Data Api, specifically Google's website says: "Each project starts with 10,000 units per day, an amount sufficient for the overwhelming majority of our API users."
Is this 10,000 per person who signs in through your project then? Because 10,000 does not in any way seem sufficient! A youtube Search alone takes 100 units if I understand the documentation, so 5 people each doing 20 searches and your project is done for the day. This seems to make building a robust Youtube browser all but impossible for a user base bigger than 2.
As you can tell I'm a bit frustrated and sure I must be missing something. Any clarifications you can offer would be appreciated.
If you check the following screen
only one of the quotas says "per user" the others are project based quotas.
You have a quota of 10000 per day for the total project.
Quota cost is calculated by the request resource.
A youtube Search alone takes 100 units per request. if I understand the documentation, so 5 people each doing 20 searches and your project is done for the day.
10000 (limit) / 100 (quota cost) = 100 search requests pre day.
It sounds to me like you have understood the quota system perfectly. If you intend to make more requests than that you will need to request additional quota.
I have an application where I need to get complete, realtime search results from twitter (preferably polling every 500ms or less). Based on my understanding, doing this using the search API will run into rate limits very quickly. However, the streaming API doesn't seem to support getting complete anything (only a 5% sample).
More specifically, I have a search query term which typically comes up with <20 matching tweets per hour, and I would like to be informed of these new tweets within 1-2 seconds, and it is considered a failure if I am not notified within 5 seconds. Due to the relatively low frequency of posting, missing even one tweet is very undesirable.
Is there any way I can realistically do this using twitter API, or is my only choice to write a browser extension to repeatedly refresh the search page?
The answer is "yes". Although you are rate limited (the limit is closer to 1% than 5%), that is only a cutoff based on your query results. Very roughly, you can stream about 60 tweets per second max. In your case, you say you expect under 20 tweets per hour, so you should have no problem getting all those tweets.
You also require a latency less than 5 seconds. In my experience latency has always been a second or two. I think you should be fine.
So I'm currently using NodeXL to get search for a particular Twitter hashtag, and I'm having trouble on understand how exactly the rate-limiting works. I looked it up in Twitter's API Rate Limits page, and also this SO post, but even after reading both, I don't really understand. The API page says:
Search will be limited at 180 queries per 15 minute window for the time being.
and also
Rate limiting in version 1.1 of the API is primarily considered on a per-user basis — or more accurately described, per access token in your control. If a method allows for 15 requests per rate limit window, then it allows you to make 15 requests per window per leveraged access token.
But I'm totally confused... probably because I've never really worked with anything database, or social network analysis before.
When it says that it always 180 queries per 15 minutes, what exactly constitutes a query? The way the search works on NodeXL is that you limit the amount of tweets you are searching for. So if I search once and set my tweet limit to 1000 tweets, is that only 1 query?
Sorry if this seems like a stupid or really elementary question, but I just don't have any experience with this stuff at all, and any help would be much appreciated, thanks!
When it says that it always 180 queries per 15 minutes, what exactly
constitutes a query?
Whenever you make one request to Twitter, its considered as one query. For Search API you can make 180 calls per 15 minute.
So if I search once and set my tweet limit to 1000 tweets, is that
only 1 query?
Yes, but you can't set count to 1000 since the maximum tweets you can return per request is 100 as it mentioned here.
You can retrieve the latest 100 tweets with the normal search query and for pagination you should use since_id and max_id to retrieve the next 100 tweets for fresh tweets.
The number of queries you can make per 15 min windows varies by API. For example, you can query 180 requests per 15 min window if you use Search API. But, if you use API like GET friends/ids, it's limited to 15 query per 15 min windows. i.e you can make call only 15 times per 15 minutes.
Here's the Rate limits chart where you can find how many requests you can make per 15 window for each API.
I'm using the Twitter search API (for example: http://search.twitter.com/search.rss?q=%23juventus&rpp=100&page=4)
I read here: http://search.twitter.com/api/ this:
We do not rate limit the search API under ordinary circumstances, however we have put measures in place to limit the abuse of our API. If you find yourself encountering these limits, please contact us and describe your app's requirements.
The limit seems random: sometimes I do 150 requests sometimes 300, generally, after 5 minutes I can do other requests.
I was wondering if is it possible do more requests
They'll detect floods and throttle accordingly rather than publisher defined limits, which is why it appears random. It'll also no doubt be based on load from other sources at the time.
If you need lots more, then they gave you the answer - contact them telling them why.
For a research project, I need to download the top 100 most used words on Twitter, multiple times per hour. However, as far as I can tell, the Twitter API only supports downloading the top 10 most used words ("trends").
My questions therefore are:
Am I missing something in the API? Is there another way to fetch more than 10 trends?
If there isn't, does anybody know of a workaround for this problem?
Put ?count=50 at the end of the URL to get the top 50. I haven't been able to get more than 50.
http://api.twitter.com/1/trends/current.json?count=50
you should monitor your timeline get all tweets , save in database, analyze via NLP, and save words(for example person names), after aggregate and get counts, for example "Obama 50 times, Java 10 times, linux 5 times"