i'm University students of South korea
I'm developing analysis application using bigdata of twitter with my advisor professor. So i'm gathering tweets contains specific keyword(relevant word of crime) at period. I use 'streaming api' and 'search api' now. I have seen that using search api and streaming api result is return tweets of only one week.
I should be get the old data that have keyword of crime and since 2006 until 2016
do you have any idea?
Sadly you can't get tweets from that time range.
From the documentation:
The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets.
So, you can only get recent tweets from the search API. Be careful too with the data beacuse it's about relevance not completeness, from the same documentation:
Before getting involved, it’s important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead.
If you really need older tweets you will have to get them from other sources like Gnip. Otherwise you will have to approach differently your problem.
If you have the names (or id's) of all the users that you want to get info you could get the timelines from each user getting up to 3200 tweets.
Related
TL;DR: I want to be able to retrieve the N most popular tweets for any arbitrary country within the last X hours (up to 24 hours)
More detail
I want to show the details of the most popular tweets by geographic region (country) over the past few hours (adjustable up to 24 hours). How can I use the Twitter REST API to achieve this (v1.1 or v2)?
There are endpoints for querying tweets and filtering by popularity, but they require a search string (e.g. "NASA") and return the most popular tweets matching that search string. I am not interested in the contents of the tweets, I just want to know what is most popular.
I plan on using this functionality to show a world map (using Leaflet) to summarise the most popular tweets by country for the past day.
I am using Twit in NodeJS but not looking for answers specific to Node, rather how to leverage the capabilities of the API.
I am not aware of a way that this can be done directly through the API itself (V1 or V2). I also do not think that this is going to be a trivial task at all.
What I would suggest is using the search endpoint...
V1: Reference
V2: Reference Note that to use geolocation search parameters (see below) you'll need academic access.
... in conjunction with one of the geolocation search parameters. For example, you could pull some subset of tweets from within a country (you will not be able to download all tweets within a single country on any given day, not to mention all countries). After you get this data, you'll need to do some of your own data processing based on how you want to define "popular" (e.g. retweets, likes, etc.) and then go from there.
As I said earlier, this seems like a very large project and not something that can be solved simply with the Twitter API.
I am trying to search keywords in twitter through tweepy.
However, I found it seems like that I can not search the tweets one week before, the code below is the main search code.
for searched_tweets in tweepy.Cursor( API.search,
q = "python",
since = 2014-02-03,
until = 2014-02-04,
lang='en' ).items()
I am not sure whether there is any limited or any better way to search by time, thanks for your help!!
:)
Unfortunately, you can't get tweets older than a week with the twitter search API.
Also note that the search results at twitter.com may return historical results while the Search API usually only serves tweets from the past week. - Twitter documentation.
You can get specific tweets older than a week by looking at individual users or using specific post ids (if you have them) but it's not reasonable to index every single tweet ever to be searchable using the API.
If you need a large time range, you can collect them yourself using the streaming API or check out a service that does (see this dev twitter thread for examples).
I have list of users
['foo','bar']
I want to search whether they have checked in somewhere or not (using 4sq api)..
So basically, all I am looking for is that whether their tweets contain "\4sq.com\" or not?
I get very confused looking into their api?
Bonus points if the steps can be implemented in python.
Thanks
You have two options for checking a user's tweets:
One option is to look through the tweets in their User Timeline (accessible in Tweepy through api.user_timeline). However, you may have to search through a lot of unrelated Tweets before you come across the one your looking for. Given how many tweets some users have, you might want to only look thorough tweets more recent than a certain date (you can look at the created_at attribute of returned tweets).
The other option is to use the Search API (accessible in Tweepy though 'api.search'). This has the advantage of allowing you specify a search query, giving you relevant tweets. However, you will need to search through the tweets until you find one by the user you're looking for. Again, you might want to limit the date range of tweets that you search through.
For an academic usage, I would like to analyze about three months of tweets. However, it seems the official Twitter search API doesn't provide tweets older than one week.
I've tried to write a self crawler, however, given a search keyword, Twitter page will not show tweets older than about one week.
Is there any trick that I can get older tweets? Or my best bet is to hit the API once a week and do it for the following three months?
From the Twitter API documentation regarding limitations:
- The Search API is not complete index of all Tweets, but instead an index of recent Tweets.
- At the moment that index includes between 6-9 days of Tweets.
- You cannot use the Search API to find Tweets older than about a week.
So, yes, if you need to collect a certain span of time, it will require multiple queries, as you suggested.
(You should also read this answer: retrieving tweets from specific user older than 7 days)
There are also currently two commercial companies that have access to the Twitter firehose and can provide this data (they are called "licensed re-syndicators"):
Gnip - offers 30 days of Twitter data
DataSift - up to two years of Twitter data
I want to gather samplings of all tweets from the past year. Being able to request tweets from a specific date would be great, but I'll take what I can get.
I do not want to find tweets by a specific user or containing a specific term, just a sampling of all tweets. The Twitter search API claims that a query term is optional, but if I try an empty query like
http://search.twitter.com/search.atom
as opposed to giving a search term,
http://search.twitter.com/search.atom?q=twitter
the response is
<hash>
<error>
You must enter a query.
</error>
</hash>
If the API really doesn't provide any functionality for this type of query, how can I hack around it? Are tweet ids roughly sequential by date and can I somehow use this info to grab bunches of tweets centered around an id of a tweet whose date I know?
You are referencing the obsolete documentation. If you read the current version you will find that a query is required.
You should also know that the Search API only provides results going back about two weeks. You might be able to find historical data from sites like infochimps.
Not useful for historical data, but in case someone stumbles across this question looking for a sampling of all current tweets, you want the streaming API. (This is my first foray into Twitter and I hadn't noticed it. I only saw the public timeline method in the normal API.)