I have a particular keyword that I want to search, and scrape all tweets containing it in the last 10 years.
I used tweepy but it can only provide last 7 days of tweets containing the keyword.
I have access to Twitter Academic API but dont know how to use it.
Related
i'm University students of South korea
I'm developing analysis application using bigdata of twitter with my advisor professor. So i'm gathering tweets contains specific keyword(relevant word of crime) at period. I use 'streaming api' and 'search api' now. I have seen that using search api and streaming api result is return tweets of only one week.
I should be get the old data that have keyword of crime and since 2006 until 2016
do you have any idea?
Sadly you can't get tweets from that time range.
From the documentation:
The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets.
So, you can only get recent tweets from the search API. Be careful too with the data beacuse it's about relevance not completeness, from the same documentation:
Before getting involved, it’s important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead.
If you really need older tweets you will have to get them from other sources like Gnip. Otherwise you will have to approach differently your problem.
If you have the names (or id's) of all the users that you want to get info you could get the timelines from each user getting up to 3200 tweets.
For example how could search for tweets using the #oldtweets sent on 5-29-16, from 8pm to 9pm.
According to the The Twitter Search API documentation, the query you want is not possible: https://dev.twitter.com/rest/public/search
The Twitter Search API searches against a sampling of recent Tweets
published in the past 7 days.
Beyond the last 7 days, what you want to achieve can only be done through manually searching an account on Twitter.
Another alternative would be to use https://webrecorder.io/
Scroll to the timeframe you want to record or you can attempt to capture the entire feed. Note the auto scrolling option as well.
I am working since last 3 days to find solution about how we can get old data from twitter api around 3 month over limitation of twitter api about 1 week. can some one help me to know best solution.
You cannot get any tweet for the last 3 months apart if you store them in your own database.
The only solution for you would be to pay Twitter GNIP to access historical data.
The Twitter API only provides the most recent tweets for a given search, up to ~3200.
The so-called "firehose" and some datasets are available mostly for research or developer purposes.
There are some services which sell custom datasets, including Twitter itself (which purchased Gnip). The overview by Justin Littman at GWU is rather comprehensive.
This question already has answers here:
How to search twitter for keywords
(3 answers)
Closed 9 years ago.
I have used twitter api. I want to fetch tweets that contain a specific word for my application. like if a user types "iphone" then all the tweets containing that keyword should be fetched from twitter. How can i do it ? and should i use streaming api or the REST api? please help. The main reason for asking is the REST only gives the top 1500 tweets , but i need more. And the streaming gives real time tweets. so i don't know which one to use. or is there a way to get more then 1500 tweets using REST?
The Twitter API supports doing searches, by querying the /search/tweets endpoint.
Reference documentation
If you are worried about running into the API limit, then offload the calls to your client by making the calls via JSONP.
For an academic usage, I would like to analyze about three months of tweets. However, it seems the official Twitter search API doesn't provide tweets older than one week.
I've tried to write a self crawler, however, given a search keyword, Twitter page will not show tweets older than about one week.
Is there any trick that I can get older tweets? Or my best bet is to hit the API once a week and do it for the following three months?
From the Twitter API documentation regarding limitations:
- The Search API is not complete index of all Tweets, but instead an index of recent Tweets.
- At the moment that index includes between 6-9 days of Tweets.
- You cannot use the Search API to find Tweets older than about a week.
So, yes, if you need to collect a certain span of time, it will require multiple queries, as you suggested.
(You should also read this answer: retrieving tweets from specific user older than 7 days)
There are also currently two commercial companies that have access to the Twitter firehose and can provide this data (they are called "licensed re-syndicators"):
Gnip - offers 30 days of Twitter data
DataSift - up to two years of Twitter data