I am building a Twitter app and I'll be pulling a big amount of data from the user's timeline. For speed, I need to query the timeline in parallel. My aim is to pull 1000 of user's tweets from the API, but the upper limit of number of tweets per request is set to 200 by the Twitter API. Pagination works by specifying the last (oldest) tweet's ID from the previous request, so I need to know the result of the previous API call to make the next call. This method is not parallelizable. Is there any alternative method for getting the user timeline from the Twitter API where I can make parallel requests (there is the page property, but is deprecated and will be nonfunctional in the near future).
What you have to remember, is that Twitter have a difficult relationship with external developers. Using their API for anything interesting like this is simply not allowed by them.
What you need is access to the Firehose.
However, even if you're willing to pay a million dollars a year - Twitter aren't interested.
You could try getting it from a third party like Gnip but - again - likely to be expensive.
So, essentially, you can't. Twitter just aren't interested in amateur developers doing anything innovative with their platform. Sorry.
Related
I've got a bunch of free online HTML, CSS, and JS tutorials under my belt and I want to try using them to make a browser extension. But I want to make sure that the data I want to use is actually accessible before getting started.
My goal is to make a browser extension for twitter.com that shows the number of impressions of any tweet next to the likes, retweets, and replies. My basic idea is to get the status URL of any given tweet, poll the Twitter API for the number of impressions of that tweet, store that in a variable, and then use CSS to display a little eye icon and the number stored in the impressions variable.
I know that I can find the number of impressions of all of my tweets, both through Twitter Analytics, and also just going to my profile page and clicking the little bar chart icon next to views, retweets, etc. But I'm not clear on whether I can do that for other people's tweets via Twitter's API or anything else. Can you?
For the record, I'm not too concerned about the varying definition of "impression," since it will be consistently applied across all tweets and I'm mostly interested in giving users a comparison between tweets. This is part of a research project to see how this might change how people engage with social media if they know how many views a given post has. If there's a simpler way to go about that using existing platforms, I'm open to suggestions.
Thanks for the advice!
No, impressions data is private. If you are authenticated to the Twitter API then you can use the new Twitter Developer Labs Tweets API to get private metrics like impressions, but you cannot get that for other people's Tweets. Also, the Twitter API does not support CORS, so I don't think you'll be successful trying to use it from a browser extension.
So since we no longer can get share counts using Twitter's API anymore, are there any workarounds to get the result, no matter how convoluted?
The only thing I'm thinking of is using the search/tweets endpoint to get tweets page by page and then using the next_results object to get the next page, counting each time.
Obviously this has massive flaws, for a popular search term the next iteration of the loop will probably have duplicates, not to mention too many API calls will invoke the rate limiter.
There's no good way to do it from the API. There are third party services which track shares.
I use http://newsharecounts.com/ - once you've signed up your domain, you can call a URL like:
http://public.newsharecounts.com/count.json?url=https://shkspr.mobi/blog/2015/03/this-is-what-a-graph-of-8000-fake-twitter-accounts-looks-like/
And get back a JSON count
{
"url":"https://shkspr.mobi/blog/2015/03/this-is-what-a-graph-of-8000-fake-twitter-accounts-looks-like/",
"count":739,
"tracked":6,
"historic":733
}
There's also http://opensharecount.com/ which works in a similar way - although I've not had much success with it.
I am trying to extract data from Twitter. The data includes the tweets and people who retweeted a particular tweet. I have 46,000 tweets and I need to find retweeters for each of the tweet. Further, using Twitter call: retweet/id, you can pass only one id at a time, limiting 15 requests per 15 minutes.
Is there any way to surpass this limit and make unlimited calls?
Not through the REST API, no.
You may want to investigate Twitter's Streaming API to see if the functionality it provides meets your needs. Accessing it is a little more complex than the REST API, but it may be able to help you meet your needs.
You will find people who will tell you to do things like set up dummy accounts and dummy applications. Don't do this. Twitter actively monitors the API for use patterns like this and you will find your applications and IP addresses blacklisted.
I have IDs from many tweets, and I'd like to fetch their full information from Twitter in order to do some data analysis. The obvious API method (https://dev.twitter.com/docs/api/1/get/statuses/show/:id) appears to take only one ID at a time. This is a problem because the number of tweets we need to analyze is well more than the API limit of 350 calls per hour.
Thus: is there some way to get full information for a set of tweet IDs, not just one, or alternately to submit many REST calls in the same HTTP request and have it count only once against the API limit?
There's unfortunately no bulk lookup offered for Tweets. You'll need to perform requests one at a time and scope your project to cope with the rate limitations. If you have friends who would like to help you, you could potentially ask them to authorize your application and leverage their permission to gain access to more requests.
I have seen numerous companies doing like Twitter lotteries where users got to retweet their tweet and then one of retweeters will get the prize or whatever.
I was wondering now how do they get the list of all retweeters or pick the winner?
I checked Twitter API and found only this thing: GET statuses/retweets/:id but it returns maximum of 100 retweeters. Is that the only way?
It looks likes there's a couple services out there doing almost exactly this. A quick google pulls up http://onekontest.com/ and there's a few other Twitter contest services, but they all seem to be different levels of broken since they haven't kept up with changes to the API.
As far as the Twitter API itself is concerned, if you were expecting more than 100 responses, I think using GET statuses/mentions makes the most sense. That API call returns any mentions of a user, and you can pass the flag include_rts to include any retweets of your tweets. Then, if you wanted to list RTs of a specific tweet, you could check the in_reply_to_status_id field in the returned data to see if it matches the original tweet ID. This API call only returns the last 800 status, 200 at a time, so if you expect a bunch of data, you would need to poll the API repeatedly over time to get all the tweets. I imagine services like favstar are doing exactly this, just on a larger scale.
If you're actually looking for code to do something like this, I wrote a sinatra app called twitter-rss-digest which handles querying Twitter over time to track different sorts of queries. It's pretty rough, and doesn't quite handle this specifically, but it might point you in the right direction if you want to code something.
The Twitter API has an endpoint that returns up to 100 retweeter IDs for a given tweet.
Note that for historical reasons, that endpoint only returns up to 100 results and the documentation about the cursor is useless. Twitter refused to fix the situation.
2013 note
That said, the first thread on the Developers site that surfaced in a quick google has #episod, a Twitter employee saying:
You can't likely get to all of them. The data is spread out. Your best bet is to use the REST API to determine many of the users who performed the retweet, but the data will still be constrained.
It's easiest to track retweets as they happen rather than try to find them from the past. Use the Streaming API for that.
I like muffinista's method, but I think if you want a 100% complete list of retweets, simply enable the retweet email notifications and write a script that polls the email box for those matching the subject "retweeted one of your Tweets!" and put the data into a table. Do this right from the start.
The site https://twren.ch/ enlists all the retweeters for a given tweet (note that it only enlists retweeters who are direct followers of the source tweeter.) Nevertheless its probably the only public source available.