I'm making a program that takes in a Twitter username and returns all of the external links a user shared. So far the program returns some links, but not others, even among tweets linking to the same website.
I've recognized that the problem is that for the tweets that do return external links as desired, the external link is the "expanded_url" attribute of the tweet. For the tweets that don't return the external link, the "expanded_url" attribute is just the link to the tweet itself.
Anyone know why this is? Is there any way around this?
I'm using Tweepy and the "expanded URL" attribute.
For example, these two similar tweets:
https://twitter.com/KamalaHarris/status/1079137488378183681
https://twitter.com/KamalaHarris/status/1077337876227084290
The first one returns the URL to the article.
The second one does not.
Update for anyone curious: Adding tweet_mode="extended" to my search term seemed to do the trick!
Related
Is it possible to create a Zap that will look-up URL mentions on Twitter?
Been scratching my head with this one, since simply entering URL into Zapier's Twitter Search field doesn't seem to do the trick.
Is it at all possible? Twitter uses its URL shortening service t.co on all links posted. Might this be a reason for me not being able to find any mentions of my URL even when I post one myself? Twitter's native search function finds URL just fine, but not Zapier.
OK, I think I figured it out.
First, use Twitter's special "url" prefix for URL search:
url:amazon - will find URL with the word “amazon” anywhere within it.
Second, most importantly, it looks like the Twitter account I used for testing got ignored after a couple of same URL posts. So be aware of that too.
I am interested in finding all tweets that share a given url (e.g. this story). I am not looking for how many re-tweets a single tweet gets or how many times a page is shared on Twitter (i.e. look at the page's social shares counter). Instead, I am trying to figure out how many unique tweets shared a story where the url in the tweets might be different but each tweet is pointing to the same webpage/story. So for the story I linked to above, can I get a list of all tweets that link to that NY Times article?
Short answer: no.
I can link to that article with shortened URLs from several sites, and it will still point to the same article.
In order to find all of the links that point to the same story, you would have to parse EVERY tweet with a shortened URL. Then, you would have to write code that calls each of the URLs to see which point to the page you're talking about.
You would have to parse millions of tweets a day, and access to Twitter's stream in those volumes is restricted.
There have been quite a few number of start-up pertaining to analyzing Twitter data. There is CrowdBooster, then there is Klout, which use Twitter data to tell the user their True reach.
I have got the following two questions:
1) Is there a way to find out who has viewed one's tweet, or the number of people that have viewed a tweet. Crowdbooster claims to tell you how many impression one received per tweet. How do they do it?
2) Thousands and thousands of links are shared each day on Twitter. Can we find out which user has clicked the link in a tweet?
I have looked through Twitter API and some of the companies that have licensed Twitter's Firehose, but have not found anything that meet my needs.
Also, to give you a short answer to your 2nd question. Now that we've established that view analysis is impossible. Can you find out which user has clicked on that link, absolutely. And depending on what your talking about, user as far as the user who has clicked on the link or the user that has the link on their Twitter stream. Both are possible,
in the case of A, you would get the referring users IP address. Methods vary depending on language.
But what I think your asking for is scenario B, finding out which user has the link in their Twitter stream. This can be done by querying the link, the API response you will get can include tweet entities which will list all this information out for you and more. Open up a firehose with your link and watch what comes in.
https://dev.twitter.com/docs/streaming-api/methods
1) Is there a way to find out who has viewed one's tweet, or the
number of people that have viewed a tweet. Crowdbooster claims to tell
you how many impression one received per tweet. How do they do it?
No, in the case of a view - this would be impossible. The tweet impression can happen in multiple silos. On the website, in a widget, in a mobile app. You can imagine that it's simply not possible to get the impression of a tweet on a view because of this reason and because unlike a click, there is no I viewed this tweet identifier sent when a view has been enacted. I spent a great deal of time researching for a way to get the tweet impression even based on a similar clicked link and this is not even possible. (edit: it's possible see the last paragraph) This brings us to question 2.
2) Thousands and thousands of links are shared each day on Twitter.
Can we find out which user has clicked the link in a tweet?
Yes, what these websites are mainly doing is analyzing links that you process through their website. If you can have a unique hash marker on a link then analysis becomes possible. Without a unique hash marker, Twitter will re-interrupt two of the same links in a exactly the same way, even in the case that it shortens your link to it's custom t.co wrapper.
This means the only reliable way to do tweet analysis is by including a unique link marker code on your tweet and analyze the the fact that somebody that has hit your server has clicked on that link.
There is a somewhat hidden Twitter API feature that helps you understand how popular a particular link is. That being the link count API .. http://urls.api.twitter.com/1/urls/count.json?url=
Something really outside of the box you can do if your set on analyzing multiple versions of exactly the same link without using markers and if your also using the Streaming (firehose) would be to analyze the tweet views (using the link count API) on similar links that hit your server. The link that got the +1 boost in view is the one that hit your server. But that's about the extent of creative analysis you can get with your tweets and more specifically the links, as mentioned links are the only thing your really able to analyze when it comes to Twitter.
1) Is there a way to find out who has viewed one's tweet, or the number of people that have viewed a tweet. Crowdbooster claims to tell you how many impression one received per tweet. How do they do it?
Yes, sign up for Twitter Analytics https://analytics.twitter.com (free service provided by Twitter) and you can see how many people view (impressions) for each tweet and totals for specific dates or a date range.
2) Thousands and thousands of links are shared each day on Twitter. Can we find out which user has clicked the link in a tweet?
Yes, you can do this. Using a URL shortening service like Bitly.com you can track how many clicks you had from Twitter (only give out that Bitly link on Twitter to do this). But if you want more indept information you may need to create a tracking software, as I don't know of any available. To do that you would need the tracking software to track the link and find out the refer header and see if it's from Twitter (or better yet, just give out a unique URL for your tweets), then you would need to use the Twitter API to find out the handle (username) of that visitor who clicked your link. Lastly store this information in a database so you can review who clicked what link.
I'm trying to use Twitter search to find all mentions of a word that DIDN'T come from a "tweet this" button. I'm trying to build a stream of all brand mentions, but I don't want to sift through all of the retweets coming from our blog, which uses Twitter's button.
As a first step, I tried looking searching for all of the tweets that came from my personal account, filtering by source:txt. It showed all of the tweets I texted in. Then source:web. It showed all of the tweets I entered on twitter.com.
But then I tried searching with source:twitterbutton. Nothing. Then source:tweetbutton. Nothing. Same with source:tweet button. I tried searching through all of Twitter's documentation on the Tweet button to see what the source: attribute is, but couldn't find anything.
When I go through my stream of tweets without filtering for the source, I see some that say via Tweet Button so I know they exist, but I can't seem to find the right source: operator to find them automatically.
FYI: I'm using the Twitter Search API Method.
Bonus points to whoever can tell me how to negate a source (return everything EXCEPT that source)
The source is Tweet Button. The correct way to search for it will be source:tweet_button. Searching by source is very unreliable though (especially for multiword sources). Filtering by source requires that you include a keyword so you can't search just by source.
To negate a source you add a - in front like this google -source:web.
Update: Multi-word sources are now _ (underscore) separated.
I started tweeting with the fine folks over at Tweet Deck since their application lets you look for ones that came from the Tweet Button. It turns out I was only one character away. D'oh!
Use source:tweet_button to get only tweets from Tweet Buttons. And, as Abraham mentioned, all you need is a - before the parameter for it to negate that source. Example search.
Thanks for all the assistance! Especially from #DesktopDeck!
Basically I want to know how many people have tweeted a link to a url, but since there are dozens of link shortener out there I don't see any way to do this without having access to all of their url maps. I found a previous question here but it was over a year old and didn't have any new answers.
So #1, does anyone know of a service/API for doing this?
And #2, can anyone think of a way to accomplish this task other than submitting the long url in question to all the popular link shortening sites?
ps- I'm also open to comments about why this is impossible or impractical.
You could perform a Google search (or the equivalent via API) for any pages that link to your page. This is done with the link: keyword. So if you're trying to figure out how many people link to www.example.com (regardless of whether it's through a link shortner URL), then you would just do a Google search for link:www.example.com.
e.g.: http://www.google.com/search?q=link:www.example.com
Note that this will only find pages that have been indexed, so pages that haven't been crawled, or pages that get crawled infrequently, will not show up in the results until a later date (if at all).
Since all sites have different algorithms for shortening the URLs, and these are different sites that most likely do not share their data with each other, how can you hope to find all of them in a single or small number of queries?
All you can do is brute-force it, and even then this might not be any good if a site is content to create a new value for the same long-form URL (especially if you send a different long-form URL that maps to the same place, like http://www.stackoverflow.com/ rather than http://stackoverflow.com/).
In order to really get this to work, there would have to be a site that ALREADY automatically collects all of this information from every site, which the URL shortening sites voluntarily call. And even if you wrote such a site, that doesn't account for the URL-shortening sites already out there who already have data!
In short, I do not see how this is remotely possible, unless I'm wrong about there being such a database somewhere out there.
So months after asking this question I came across a solution to a similar question, that is how to tell how many times a link has been shared on facebook. The solution, via a simple new API call:
http://graph.facebook.com/http://stackoverflow.com
returns the following json data:
{
"id": "http://stackoverflow.com",
"shares": 1627
}