I have the Academic Research access to Twitter's API and have been using Tweepy to access the API. My problem is I cannot retrieve the tweets from older tweets
This is the code attempting to retrieve the tweets using the conversation_id, from 2014
# https://twitter.com/NintendoAmerica/status/535462600294035456
start_time = '2014-11-01T00:00:00Z'
end_time = '2014-12-12T00:00:00Z'
tweets = client.search_all_tweets(query = 'conversation_id:535462600294035456', max_results = 500, start_time=start_time, end_time=end_time)
and the output is:
Response(data=[<Tweet id=535465221679489024 text='#NintendoAmerica #Pokemon [this was a link I had to remove]'>], includes={}, errors=[], meta={'newest_id': '535465221679489024', 'oldest_id': '535465221679489024', 'result_count': 1})
which is only one seemingly random tweet amongst many.
However, when I tried running the same code on a more recent tweet, it retrieved all the tweets. I do not have to specify a start/end time because it's a tweet from the past 30 days.
# https://twitter.com/380kmh/status/1545477360916373504
tweets = client.search_all_tweets(query = 'conversation_id:1545477360916373504', max_results = 500)
the output was complete (shortened, I removed the tweets):
Response(data=[...], meta={'newest_id': '1546465585093087235', 'oldest_id': '1545477768229670912', 'result_count': 18})
I followed Tweepy's documentation here, using Client.search_all_tweets:
https://docs.tweepy.org/en/stable/client.html#tweepy.Client.search_all_tweets
I also tried using Postman to retrieve the tweets but it came out empty, even though I followed the documentation here:
https://developer.twitter.com/en/docs/twitter-api/tweets/search/api-reference/get-tweets-search-all#tab0
Query: https://api.twitter.com/2/tweets/search/all?query=conversation_id%3A537923834557771776&start_time=2014-11-01T00:00:00.000Z&end_time=2014-12-18T00:00:00.000Z&tweet.fields=in_reply_to_user_id,text
Output:
{
"meta": {
"result_count": 0
}
}
What am I doing wrong?
I need to only get the followers I have not fetched before. Currently I can only get the top 200 items or given count but I don't want to get the same data more than once.
The only way I know how is to cycle through them and follow them if they haven't already been followed. I don't believe it's possible by looking at the API:
https://www.geeksforgeeks.org/python-api-followers-in-tweepy/
Make sure you have the third line for the API:
auth = tweepy.OAuthHandler(CONSUMER_KEY, CONSUMER_SECRET)
auth.set_access_token(ACCESS_KEY, ACCESS_SECRET)
api = tweepy.API(auth, wait_on_rate_limit = True)
Here's my snippet for following:
followers = tweepy.Cursor(api.get_followers).items()
for follower in followers:
if follower.id not in friends:
user = api.get_user(follower.id)
follower.follow()
followers = tweepy.Cursor(api.get_followers).items()
I just discovered that .items() can be passed a number. So you could do something like:
followers = tweepy.Cursor(api.get_followers).items(50)
Additionally, looking at the API documentation, API.get_followers() method, you can also set the number of followers to go through by passing a value to the count variable.
API.get_followers(*, user_id, screen_name, cursor, count, skip_status, include_user_entities)
API.get_followers(count=50)
The followers are returned in the number that they were added.
I am using this rails gem to access the Mailchimp API.
https://bitbucket.org/mailchimp/mailchimp-api-ruby
mailchimp = Mailchimp::API.new(ENV['MAILCHIMP-API-KEY'])
My list has 59 members and I can confirm that from
newsletter_members = mailchimp.lists.members(listID)['total']
but when I try to access the data hash, it only returns 25 objects?
newsletter_members = mailchimp.lists.members(listID)['data'].size
Any ideas as to why?
I am basically trying to find if a email exists in my mail chimp list and this code is breaking because I am not getting all the members
mailchimp = Mailchimp::API.new(ENV['MAILCHIMP-API-KEY'])
listID = 'my-list-id'
email_array = []
newsletter_members = mailchimp.lists.members(listID)['data']
# the email array is cut short..not getting all members
newsletter_members.each do |member|
email_array << member['email']
end
#returns true or false
member_exists = email_array.include?(user_email)
Mailchimp's API defaults to return 25 items at a time. You can request more (though the limit is 100), or you can make multiple requests for "pages" of results.
To request 100 results:
mailchimp.lists.members(listID, 'subscribed', {limit: 100})
To request the next page of results (keep in mind the first page (results 1-25) is 0, second (26-50) is 1, etc.):
mailchimp.lists.members(listID, 'subscribed', {start: 1}) # Then to get 51-75 you'd start at 2
See the source code for an overview of the available options.
You can also look at Mailchimp's API docs for the lists/members endpoint to see available options, defaults, etc.
I'm facing an issue using Koala 1.7.0rc1 and the new Facebook graph api. I'm trying to retrieve number of likes for a post using this request [object_id]/likes?summary=1. This query works in Facebook Graph Explorer but I cannot access 'summary' using Koala :
likes = graph.get_object("5718732097_10151698726822098", summary: 1){|data| data['likes']}
# OR
likes = graph.get_object("5718732097_10151698726822098/likes?summary=1")
You should do:
graph.get_object(your_post_id, :fields => "likes.summary(true)")
The api documentation that Facebook gave is kind of misleading here:
https://developers.facebook.com/docs/reference/api/post/
It says summary = 1 which should be summary = true in rails
You'll need to get the summary data from the raw response in Koala, like so:
likes = graph.get_object("5718732097_10151698726822098/likes?summary=1").
raw_response["summary"]["total_count"]
Is there a way in the Twitter API to get the replies to a particular tweet? Thanks
Here is the procedure to get the replies for a tweets
when you fetch the tweet store the tweetId ie., id_str
using twitter search api do the following query
[q="to:$tweeterusername", sinceId = $tweetId]
Loop all the results , the results matching the in_reply_to_status_id_str to $tweetid is the replies for the post.
From what I understand, there's not a way to do that directly (at least not now). Seems like something that should be added. They recently added some 'retweet' capabilities, seem logical to add this as well.
Here's one possible way to do this, first sample tweet data (from status/show):
<status>
<created_at>Tue Apr 07 22:52:51 +0000 2009</created_at>
<id>1472669360</id>
<text>At least I can get your humor through tweets. RT #abdur: I don't mean this in a bad way, but genetically speaking your a cul-de-sac.</text>
<source>TweetDeck</source>
<truncated>false</truncated>
<in_reply_to_status_id></in_reply_to_status_id>
<in_reply_to_user_id></in_reply_to_user_id>
<favorited>false</favorited>
<in_reply_to_screen_name></in_reply_to_screen_name>
<user>
<id>1401881</id>
...
From status/show you can find the user's id. Then statuses/mentions_timeline will return a list of status for a user. Just parse that return looking for a in_reply_to_status_id matching the original tweet's id.
The Twitter API v2 supports this now using a conversation_id field. You can read more in the docs.
First, request the conversation_id field of the tweet.
https://api.twitter.com/2/tweets?ids=1225917697675886593&tweet.fields=conversation_id
Second, then search tweets using the conversation_id as the query.
https://api.twitter.com/2/tweets/search/recent?query=conversation_id:1225912275971657728
This is a minimal example, so you should add other fields as you need to the URL.
Twitter has an undocumented api called related_results. It will give you replies for the specified tweet id. Not sure how reliable it is as its experimental, however this is the same api call that is called on twitter web.
Use at your own risk. :)
https://api.twitter.com/1/related_results/show/172019363942117377.json?include_entities=1
For more info, check out this discussion on dev.twitter:
https://dev.twitter.com/discussions/293
Here is my solution. It utilizes Abraham's Twitter Oauth PHP library: https://github.com/abraham/twitteroauth
It requires you to know the Twitter user's screen_name attribute as well as the id_str attribute of the tweet in question. This way, you can get an arbitrary conversation feed from any arbitrary user's tweet:
*UPDATE: Refreshed code to reflect object access vs array access:
function get_conversation($id_str, $screen_name, $return_type = 'json', $count = 100, $result_type = 'mixed', $include_entities = true) {
$params = array(
'q' => 'to:' . $screen_name, // no need to urlencode this!
'count' => $count,
'result_type' => $result_type,
'include_entities' => $include_entities,
'since_id' => $id_str
);
$feed = $connection->get('search/tweets', $params);
$comments = array();
for ($index = 0; $index < count($feed->statuses); $index++) {
if ($feed->statuses[$index]->in_reply_to_status_id_str == $id_str) {
array_push($comments, $feed->statuses[$index]);
}
}
switch ($return_type) {
case 'array':
return $comments;
break;
case 'json':
default:
return json_encode($comments);
break;
}
}
Here I am sharing simple R code to fetch reply of specific tweet
userName = "SrBachchan"
##fetch tweets from #userName timeline
tweets = userTimeline(userName,n = 1)
## converting tweets list to DataFrame
tweets <- twListToDF(tweets)
## building queryString to fetch retweets
queryString = paste0("to:",userName)
## retrieving tweet ID for which reply is to be fetched
Id = tweets[1,"id"]
## fetching all the reply to userName
rply = searchTwitter(queryString, sinceID = Id)
rply = twListToDF(rply)
## eliminate all the reply other then reply to required tweet Id
rply = rply[!rply$replyToSID > Id,]
rply = rply[!rply$replyToSID < Id,]
rply = rply[complete.cases(rply[,"replyToSID"]),]
## now rply DataFrame contains all the required replies.
You can use twarc package in python to collect all the replies to a tweet.
twarc replies 824077910927691778 > replies.jsonl
Also, it is possible to collect all the reply chains (replies to the replies) to a tweet using command below:
twarc replies 824077910927691778 --recursive
Not in an easy pragmatic way. There is an feature request in for it:
http://code.google.com/p/twitter-api/issues/detail?id=142
There are a couple of third-party websites that provide APIs but they often miss statuses.
I've implemented this in the following way:
1) statuses/update returns id of the last status (if include_entities is true)
2) Then you can request statuses/mentions and filter the result by in_reply_to_status_id. The latter should be equal to the particular id from step 1
As states satheesh it works great. Here is REST API code what I used
ini_set('display_errors', 1);
require_once('TwitterAPIExchange.php');
/** Set access tokens here - see: https://dev.twitter.com/apps/ **/
$settings = array(
'oauth_access_token' => "xxxx",
'oauth_access_token_secret' => "xxxx",
'consumer_key' => "xxxx",
'consumer_secret' => "xxxx"
);
// Your specific requirements
$url = 'https://api.twitter.com/1.1/search/tweets.json';
$requestMethod = 'GET';
$getfield = '?q=to:screen_name&sinceId=twitter_id';
// Perform the request
$twitter = new TwitterAPIExchange($settings);
$b = $twitter->setGetfield($getfield)
->buildOauth($url, $requestMethod)
->performRequest();
$arr = json_decode($b,TRUE);
echo "Replies <pre>";
print_r($arr);
die;
I came across the same issue a few months ago at work, as I was previously using their related_tweets endpoint in REST V1.
So I had to create a workaround, which I have documented here:
http://adriancrepaz.com/twitter_conversations_api Mirror - Github fork
This class should do exactly what you want.
It scrapes the HTML of the mobile site, and parses a conversation. I've used it for a while and it seems very reliable.
To fetch a conversation...
Request
<?php
require_once 'acTwitterConversation.php';
$twitter = new acTwitterConversation;
$conversation = $twitter->fetchConversion(324215761998594048);
print_r($conversation);
?>
Response
Array
(
[error] => false
[tweets] => Array
(
[0] => Array
(
[id] => 324214451756728320
[state] => before
[username] => facebook
[name] => Facebook
[content] => Facebook for iOS v6.0 ? Now with chat heads and stickers in private messages, and a more beautiful News Feed on iPad itunes.apple.com/us/app/faceboo?
[date] => 16 Apr
[images] => Array
(
[thumbnail] => https://pbs.twimg.com/profile_images/3513354941/24aaffa670e634a7da9a087bfa83abe6_normal.png
[large] => https://pbs.twimg.com/profile_images/3513354941/24aaffa670e634a7da9a087bfa83abe6.png
)
)
[1] => Array
(
[id] => 324214861728989184
[state] => before
[username] => michaelschultz
[name] => Michael Schultz
[content] => #facebook good April Fools joke Facebook?.chat hasn?t changed. No new features.
[date] => 16 Apr
[images] => Array
(
[thumbnail] => https://pbs.twimg.com/profile_images/414193649073668096/dbIUerA8_normal.jpeg
[large] => https://pbs.twimg.com/profile_images/414193649073668096/dbIUerA8.jpeg
)
)
....
)
)
since statuses/mentions_timeline will return the 20 most recent mention this won't be that efficient to call, and it has limitations like 75 requests per window (15min) , insted of this we can use user_timeline
the best way: 1. get the screen_name or user_id parameters From status/show.
2. now use user_timeline
GET https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=screen_name&count=count
(screen_name== name which we got From status/show)
(count== 1 to max 200)
count: Specifies the number of Tweets to try and retrieve, up to a maximum of 200 per distinct request.
from the result Just parse that return looking for an in_reply_to_status_id matching the original tweet's id.
Obviously, it's not ideal, but it will work.
If you need all replies related to one user for ANY DATE RANGE, and you only need to do it once (like for downloading your stuff), it is doable.
Make a Twitter development application
Apply for elevated credentials. You will instantly get them after filling out the forms. At least I did on two separate accounts today.
Your development account now has access to the v1.1 API search in the "Sandbox" tier. You get 50 requests against the tweets/search/fullarchive endpoint maxing out at 5000 returned tweets.
Make an environment for your development application.
Make a script to query https://api.twitter.com/1.1/tweets/search/fullarchive/<env name>.json where <env name> is the name of your environment. Make your query to:your_twitter_username and fromDate when you created your account, toDate today.
Iterate over the results
This will not get your replies recursively