I need to do some comparison on twitter trends and therefore I need to get past statistics.
for example, how many followers user X had 2 years ago.
Any idea of how to achieve this? or any point to the right direction?
thanks.
Don't think you can. Haven't been keeping up with twitter much lately but I know there are some sites that track your stats but they only go as far back as when you signed up for the site. Meaning you'd have to keep track of the data yourself and, only in two years could you could see how many followers I had two years ago (today).
Twitter has a list of partner providers on their site that you might want to contact for historical data:
https://dev.twitter.com/docs/twitter-data-providers
Twitter will only return a certain number of recent days in search queries, so the Twitter API isn't too helpful with historical data.
Related
I want to buy data from twitter or any other source that can sell twitter data legally. I want all tweets that mention a specific keyword in the last six months. I want them to make analysis for a research purpose.
It seems that plan were changed in recent days for twitter and Gnip.
I'd recommend going to https://developer.twitter.com/en/docs/tutorials/one-time-historical-powertrack-jobs.html.
It'll ask you to submit the query rules you're looking for and the duration of time. Once that's set up, a Twitter employee should reach out with an estimate.
I am working since last 3 days to find solution about how we can get old data from twitter api around 3 month over limitation of twitter api about 1 week. can some one help me to know best solution.
You cannot get any tweet for the last 3 months apart if you store them in your own database.
The only solution for you would be to pay Twitter GNIP to access historical data.
The Twitter API only provides the most recent tweets for a given search, up to ~3200.
The so-called "firehose" and some datasets are available mostly for research or developer purposes.
There are some services which sell custom datasets, including Twitter itself (which purchased Gnip). The overview by Justin Littman at GWU is rather comprehensive.
For an academic usage, I would like to analyze about three months of tweets. However, it seems the official Twitter search API doesn't provide tweets older than one week.
I've tried to write a self crawler, however, given a search keyword, Twitter page will not show tweets older than about one week.
Is there any trick that I can get older tweets? Or my best bet is to hit the API once a week and do it for the following three months?
From the Twitter API documentation regarding limitations:
- The Search API is not complete index of all Tweets, but instead an index of recent Tweets.
- At the moment that index includes between 6-9 days of Tweets.
- You cannot use the Search API to find Tweets older than about a week.
So, yes, if you need to collect a certain span of time, it will require multiple queries, as you suggested.
(You should also read this answer: retrieving tweets from specific user older than 7 days)
There are also currently two commercial companies that have access to the Twitter firehose and can provide this data (they are called "licensed re-syndicators"):
Gnip - offers 30 days of Twitter data
DataSift - up to two years of Twitter data
I am developing an application to guess locations of tornadoes by analyzing twitter data. For this, I would first need to train a neural network on some manually annotated tweets. I am trying to get tweets from last year which have the word 'tornado' in them. This is my code below :-
Query query = new Query("tornado");
query.setRpp(100);
query.setSince("2010-11-01");
query.setUntil("2011-01-13");
QueryResult queryResult = instance.search(query);
tweetList = queryResult.getTweets();
I am able to retrieve tweets from periods closer to now such as last week and such, but am unable to get any results for periods such as the one listed above. Any clues, suggestions would be help. Thanks in advance.
I just found out the reason through a different medium, thought i'd share the answer in case there are other people with the same issue.
It turns out that the twitter search api does not return tweets older than around a week and also, depending on the server load, at times this could be as low as 24 hours ! Hence, any 3rd party libraries (such as twitter4j) which have a wrapper for the twitter search api will behave similarly.
The best way to go about this would be to use third party search and indexing sites such as snapbird, topsy, etc..
There is a large number of sites like Twitaholic or Twittergrader that offer rankings of Twitter users depending on the number of followers, influence, etc. I haven't found much information, though, on how do they compute these rankings.
My guess is that they begin with a handful users and keep exploring the followers' graph, while periodically updating the information of the users they already know of.
So the question is: is this the right approach or is there a more trivial way of doing it?
The sites you mention started years ago, and at that time they were given whitelisting by Twitter, which means that they can make tens of thousands of API requests per hour. Twitter no longer gives out new whitelisted accounts, so this type of analysis cannot be done by new sites. New accounts are only allowed to make 350 API requests per hour.
It is in fact possible just to use the Twitter API to examine and remember everything about every user, which is what quite a few sites do. twitter streaming api