I am trying to extract data from Twitter. The data includes the tweets and people who retweeted a particular tweet. I have 46,000 tweets and I need to find retweeters for each of the tweet. Further, using Twitter call: retweet/id, you can pass only one id at a time, limiting 15 requests per 15 minutes.
Is there any way to surpass this limit and make unlimited calls?
Not through the REST API, no.
You may want to investigate Twitter's Streaming API to see if the functionality it provides meets your needs. Accessing it is a little more complex than the REST API, but it may be able to help you meet your needs.
You will find people who will tell you to do things like set up dummy accounts and dummy applications. Don't do this. Twitter actively monitors the API for use patterns like this and you will find your applications and IP addresses blacklisted.
Related
I want to get the complete like list of a specific tweet, but the Twitter API only provided an API that can retrieve the 100 most recent users who liked the specific tweet. I also looked for Twitter crawlers on Github, but they all worked in a user-oriented manner, ie they can only get a list of liked tweets of a user, not a list of liking users of a specific tweet.
I also tried to crawl the list using selenium, but maybe due to my limited skill, it didn't work well. I don't want to spend a lot of time studying selenium and front-end knowledge just to accomplish a simple thing, so are there any open source codes or twitter APIs that can do this?
Yes. This has just been announced in the Twitter API v2.
Previously, you were limited to the 100 most recent Likes or Retweets with these endpoints. We heard your feedback that this was too limiting and have updated these endpoints to now return all results. To retrieve a complete list of Likes and Retweets, you can now use pagination.
Use the v2 Likes lookup endpoint: GET /2/tweets/:id/liking_users
I am building a Twitter app and I'll be pulling a big amount of data from the user's timeline. For speed, I need to query the timeline in parallel. My aim is to pull 1000 of user's tweets from the API, but the upper limit of number of tweets per request is set to 200 by the Twitter API. Pagination works by specifying the last (oldest) tweet's ID from the previous request, so I need to know the result of the previous API call to make the next call. This method is not parallelizable. Is there any alternative method for getting the user timeline from the Twitter API where I can make parallel requests (there is the page property, but is deprecated and will be nonfunctional in the near future).
What you have to remember, is that Twitter have a difficult relationship with external developers. Using their API for anything interesting like this is simply not allowed by them.
What you need is access to the Firehose.
However, even if you're willing to pay a million dollars a year - Twitter aren't interested.
You could try getting it from a third party like Gnip but - again - likely to be expensive.
So, essentially, you can't. Twitter just aren't interested in amateur developers doing anything innovative with their platform. Sorry.
I have IDs from many tweets, and I'd like to fetch their full information from Twitter in order to do some data analysis. The obvious API method (https://dev.twitter.com/docs/api/1/get/statuses/show/:id) appears to take only one ID at a time. This is a problem because the number of tweets we need to analyze is well more than the API limit of 350 calls per hour.
Thus: is there some way to get full information for a set of tweet IDs, not just one, or alternately to submit many REST calls in the same HTTP request and have it count only once against the API limit?
There's unfortunately no bulk lookup offered for Tweets. You'll need to perform requests one at a time and scope your project to cope with the rate limitations. If you have friends who would like to help you, you could potentially ask them to authorize your application and leverage their permission to gain access to more requests.
There is a large number of sites like Twitaholic or Twittergrader that offer rankings of Twitter users depending on the number of followers, influence, etc. I haven't found much information, though, on how do they compute these rankings.
My guess is that they begin with a handful users and keep exploring the followers' graph, while periodically updating the information of the users they already know of.
So the question is: is this the right approach or is there a more trivial way of doing it?
The sites you mention started years ago, and at that time they were given whitelisting by Twitter, which means that they can make tens of thousands of API requests per hour. Twitter no longer gives out new whitelisted accounts, so this type of analysis cannot be done by new sites. New accounts are only allowed to make 350 API requests per hour.
It is in fact possible just to use the Twitter API to examine and remember everything about every user, which is what quite a few sites do. twitter streaming api
I am writing a small app, building stats for twitter users (no of tweets, friends etc). I am using this api
http://api.twitter.com/1/users/show.json?user_id=12345
I can only make 150 calls per hour, which is very very small, given the size of twitter. How do companies that rely on Twitter's API manage to overcome this rate limit?
The 150 API calls is per user per application. Larger companies likely broker deals with Twitter.
You need to get whitelisted to get a far higher rate limit. They are open to all sorts of developers, as long as you give a good reason for what you are developing:
http://twitter.com/help/request_whitelisting
You will easily get whitelisted, just apply. They will accept more or less any reasonable application, but just don't want to leave it 'wide open'. If they dont accept you, and you still want to get your hands on the data, just scrape it.