I am working since last 3 days to find solution about how we can get old data from twitter api around 3 month over limitation of twitter api about 1 week. can some one help me to know best solution.
You cannot get any tweet for the last 3 months apart if you store them in your own database.
The only solution for you would be to pay Twitter GNIP to access historical data.
The Twitter API only provides the most recent tweets for a given search, up to ~3200.
The so-called "firehose" and some datasets are available mostly for research or developer purposes.
There are some services which sell custom datasets, including Twitter itself (which purchased Gnip). The overview by Justin Littman at GWU is rather comprehensive.
Related
I need to do some comparison on twitter trends and therefore I need to get past statistics.
for example, how many followers user X had 2 years ago.
Any idea of how to achieve this? or any point to the right direction?
thanks.
Don't think you can. Haven't been keeping up with twitter much lately but I know there are some sites that track your stats but they only go as far back as when you signed up for the site. Meaning you'd have to keep track of the data yourself and, only in two years could you could see how many followers I had two years ago (today).
Twitter has a list of partner providers on their site that you might want to contact for historical data:
https://dev.twitter.com/docs/twitter-data-providers
Twitter will only return a certain number of recent days in search queries, so the Twitter API isn't too helpful with historical data.
For an academic usage, I would like to analyze about three months of tweets. However, it seems the official Twitter search API doesn't provide tweets older than one week.
I've tried to write a self crawler, however, given a search keyword, Twitter page will not show tweets older than about one week.
Is there any trick that I can get older tweets? Or my best bet is to hit the API once a week and do it for the following three months?
From the Twitter API documentation regarding limitations:
- The Search API is not complete index of all Tweets, but instead an index of recent Tweets.
- At the moment that index includes between 6-9 days of Tweets.
- You cannot use the Search API to find Tweets older than about a week.
So, yes, if you need to collect a certain span of time, it will require multiple queries, as you suggested.
(You should also read this answer: retrieving tweets from specific user older than 7 days)
There are also currently two commercial companies that have access to the Twitter firehose and can provide this data (they are called "licensed re-syndicators"):
Gnip - offers 30 days of Twitter data
DataSift - up to two years of Twitter data
I am developing an application to guess locations of tornadoes by analyzing twitter data. For this, I would first need to train a neural network on some manually annotated tweets. I am trying to get tweets from last year which have the word 'tornado' in them. This is my code below :-
Query query = new Query("tornado");
query.setRpp(100);
query.setSince("2010-11-01");
query.setUntil("2011-01-13");
QueryResult queryResult = instance.search(query);
tweetList = queryResult.getTweets();
I am able to retrieve tweets from periods closer to now such as last week and such, but am unable to get any results for periods such as the one listed above. Any clues, suggestions would be help. Thanks in advance.
I just found out the reason through a different medium, thought i'd share the answer in case there are other people with the same issue.
It turns out that the twitter search api does not return tweets older than around a week and also, depending on the server load, at times this could be as low as 24 hours ! Hence, any 3rd party libraries (such as twitter4j) which have a wrapper for the twitter search api will behave similarly.
The best way to go about this would be to use third party search and indexing sites such as snapbird, topsy, etc..
There is a large number of sites like Twitaholic or Twittergrader that offer rankings of Twitter users depending on the number of followers, influence, etc. I haven't found much information, though, on how do they compute these rankings.
My guess is that they begin with a handful users and keep exploring the followers' graph, while periodically updating the information of the users they already know of.
So the question is: is this the right approach or is there a more trivial way of doing it?
The sites you mention started years ago, and at that time they were given whitelisting by Twitter, which means that they can make tens of thousands of API requests per hour. Twitter no longer gives out new whitelisted accounts, so this type of analysis cannot be done by new sites. New accounts are only allowed to make 350 API requests per hour.
It is in fact possible just to use the Twitter API to examine and remember everything about every user, which is what quite a few sites do. twitter streaming api
is there any service from where we can download tweets?
UPDATE!!!
Googling for sometime gave me this result
a.) http://snap.stanford.edu/data/twitter7.html
b.) http://140kit.com/datasets
Yes, there is. It's called the Twitter API.
As we have access to limited tweets by Twitter-API, we should make use of third-party resellers like Topsy for just the past data, GNIP just for streaming data, or DataSift for both streaming data as well as past data.
You might also want to check the following sites:
http://www.infochimps.com/collections/twitter-census
http://www.tweetarchivist.com/
Twitter API allows provides partial results, it gives you the last 100 or even 500 tweets fo every search. If you need to keep tweets long term, twitter API shows its limits.
I had same need as you apparently hae and I developed a tool that queries twitter API periodically and stores search results on a Wordpress database.
I called the tools twittcorder and you can find a live demo on twittcorder.com
I hope this helps.
These other data sources are probably shared against the Twitter TOS. I wouldn't want to invest my time and effort building something on datasets that are non-repeatable. The Twitter Streaming API allows collection of a sample of Tweets.
There's also Gnip: http://gnip.com/.
Sysomos is there for complete data analysis including twitter, faecbook and various boards and forums