Let's say I need to obtain the text (post and all comments) from the sub-reddits that were most popular during December 2014 and January 2015. How can that be done using PRAW? I was wondering, just like Twitter doesn't let us crawl tweets which are more than a week old, is there any such time constraint in place in case of Reddit?
PRAW has a limit of 1,000 for how far back you can get threads from a subreddit based on score, newness, "hotness" (I'm not sure if that one still works) or "controversy" (I haven't tried that one in a while, as well). Unfortunately, the most popular subreddits have had too many posts to look that far back.
You can try using search() http://praw.readthedocs.org/en/latest/pages/code_overview.html#praw.init.UnauthenticatedReddit.search to get the values you want, but it requires a bit more work.
Related
I have a small program that fetch tweets and displays them in google maps, it works perfect but I would like to know if there is a way to put a timeframe, for instance lets say I would like to fetch the tweets written in the last 4 hours, is it possible? Thanks for your help!
The program is written in PHP and javaScript.
It may be possible using the max_id and since_id parameters. How you would calculate finding an ID within that 4 hour window, I am unsure, but this post might provide some answers: https://stackoverflow.com/a/6416836/5884189
I just coded a solution to the same problem in a similar question. Check the answer and change the WINDOW constant to 4 to get the tweets in the last four hours. Good luck!
PS: I think the mods don't want you to repeat similar answers in 2 questions so I let you the link, but I could edit the answer later and paste the code here if I'm wrong.
Since 20 November, its' not possible to get Twitter share counts in this way:
http://cdn.api.twitter.com/1/urls/count.json?url=
Is there alternative to do that or it's not possible? Thanks.
Check out http://opensharecount.com, it provides a drop-in replacement for that URL. Only downside is it only can 'see' tweets from up to seven days ago, but it will include those in future counts so it is a good way to get approximate counts for your new posts.
Good evening, StackOverflow!
I am working with twitter API to retrieve tweets from the twitter. I need to receive tweets by a time range, for example, all tweets from #someUser for the last week, or last 10 days... or from March to April.
I am looking at https://dev.twitter.com/rest/reference/get/statuses/user_timeline documentation, and it doesn't really specify how to use the date to narrow down the list of receiving tweets. I mean, I can use since_id or max_id, but it is not really what I'm looking for.
I mean, I can receive tweets by one using max_id and since_id, check the date, and see if it falls into my time frame, and continue until I get to the edge. Anyway, it is not a good practice.
So, is there a way or maybe I missed something in the documentation?
I really would appreciate any help (just a tip would work).
I've been using the old url api(v1) to get the count of a given url, lately I needed to get also the re-tweets and started searching about that.
this is the exact url I'm using right now:
http://urls.api.twitter.com/1/urls/count.json?url=http://google.com
As I viewed with some reading the v1 api is deprecated but at least it's still working.
I found some questions on the dev page of twitter:
https://dev.twitter.com/discussions/12643
those are a little old questions and have no specific solving to the problem. I mean, the most near solution was using the search api(search/tweets) which could be good but not a exactly replacement for the urls/count method.
Please note that Twitter's search service and, by extension, the
Search API is not meant to be an exhaustive source of Tweets. Not all
Tweets will be indexed or made available via the search interface.
also it has a limit for 100 results at maximum per 'page', even it throws the link to get the next set of objects, thats good but when the search reaches 1 million of results I'll need to get page over page to now how much tweets I got and having to do to much request to the api...
I sought some question over the dev page on twitter suggested using the stream api, I've tried using (statuses/filter) but that don't work very well given a URL as track param(which they said that is the keyword to track).
So, anyone who's been using the old urls/count has found a reliable alternative with the new apiv1.1, especiffically to get the tweets and re-tweets for a given url ?
The official suggestion by Twitter staff is that either the search/tweets endpoint (having just the last 7 days data) or the Streaming API be used (handling yourself the counters, making everything just too complicated for a d*mn counter).
As an extra warning, the old endpoint (http://urls.api.twitter.com/1/urls/count.json?url=YOUR_URL) will stop working on November 20th, and according to this blog post from Twitter there are no plans to replace it with anything in the short term and they are even removing the count from their own buttons.
I am developing an application to guess locations of tornadoes by analyzing twitter data. For this, I would first need to train a neural network on some manually annotated tweets. I am trying to get tweets from last year which have the word 'tornado' in them. This is my code below :-
Query query = new Query("tornado");
query.setRpp(100);
query.setSince("2010-11-01");
query.setUntil("2011-01-13");
QueryResult queryResult = instance.search(query);
tweetList = queryResult.getTweets();
I am able to retrieve tweets from periods closer to now such as last week and such, but am unable to get any results for periods such as the one listed above. Any clues, suggestions would be help. Thanks in advance.
I just found out the reason through a different medium, thought i'd share the answer in case there are other people with the same issue.
It turns out that the twitter search api does not return tweets older than around a week and also, depending on the server load, at times this could be as low as 24 hours ! Hence, any 3rd party libraries (such as twitter4j) which have a wrapper for the twitter search api will behave similarly.
The best way to go about this would be to use third party search and indexing sites such as snapbird, topsy, etc..