So, I'm using Twitter's user timeline method in a bookmarklet. As you may know, the user timeline method is rate limited; you can only call it 150 times in an hour. My question is, how would it count against my limit when the call happens within the bookmarklet?
The corny mess is here:
We Still At War?
If you are embedding your credentials in the bookmarklet, then everyone will be contributing to the rate limiting, because I believe it is by token per application. But that would be silly.
Unauthenticated timeline REST requests count against the IP they are called from. So these would count against the person browsing, not you. This is the same as how people view twitter.com when they aren't logged in.
Related
I'm designing a web application that needs to retrieve and organize emails from at least 100 mailboxes from our company's domain. Something like:
abc#company.com
cde#company.com
efg#compnay.com
...
My web app needs to check each of these emails every couple of seconds to retrieve the new emails and index them. However, I don't want to hit the API limit for them.
Reading the official documentation, it seems like I can have a single app ID and use it to retrieve all these informations and won't hit the API limit.
If I have 100 mail boxes, and let's say, check each mailbox every 10 seconds (with maximum 4 concurrent threads), is it safe to say that I won't hit any kind of rate limit?
It might be worth mentioning that I'm going to use the Delta Link feature to check for new emails. This will make things faster and I'm not sure if it has any effect on the rate limits.
You really need to decrease the frequency of direct hits and only do that once you receive a notification from a particular mailbox/folder.
See https://learn.microsoft.com/en-us/graph/api/subscription-post-subscriptions?view=graph-rest-1.0&tabs=http for details on Graph event subscription.
This question is very similar to this one, however there are no answers on that one. I posted this one with more clarity in hopes of receiving an answer.
According to this presentation, Twitter incorporates a fanout method to push Tweets to each individual user's timeline in Redis. Obviously, this fanout only takes place when a user you're following Tweets something.
Suppose a new user, who has never followed anyone before (and conversely has no Tweets in their timeline), decides to follow someone. Using just the above method, they would have to wait until the user they followed Tweeted something for anything to show up on their timeline. After some observation, this is not the case. Twitter pulls in the latest Tweets from the user.
Now suppose that a new user follows 5 users, how does Twitter organize and push those Tweets into the user's timeline in Redis?
Suppose a user already follows 5 users and they have a fair amount of Tweets from these users in their timeline. When they follow another 5 users, how are these user's individual Tweets pushed into the initial user's timeline in Redis in the correct order? More importantly, how is it able to calculate how many to bring in from each user (seeing that they cap timelines at 800 Tweets).
Here is a way of how I would try to implement it this if I understand well your question.
Store each tweet in a hash. The key of the hash could be something like: tweet:<tweetID>.
Store the IDs of the tweets of a given user in a sorted set named user:<userID>:tweets. You set the score of the tweet as a unix timestamp, so they appear in the correct order. You can then get a list of the 800 most recent tweet IDs for the user with the instruction ZREVRANGEBYSCORE
ZREVRANGEBYSCORE user:<userID>:tweets +inf -inf LIMIT 0 800
When a user follows a new person, you copy the list of ids returned by this instruction in the timeline of the follower (either in the application code, or using a LUA script). This timeline is once again represented by a sorted set, with unix timestamps as scores. If you do the copy in the application code, which is perfectly acceptable with Redis, don't forget to use pipelining to perform your multiples writes in the sorted set in a unique network operation. It will greatly improve the performances.
To get the timeline content, use pipelining too. Request the tweets ID, using ZREVRANGEBYSCORE with a limit option and/or a timestamp as lower limit if you don't want tweets posted before a certain date.
So I am working on a Rails application, and the person I am designing it for has what seem like extremely hefty data volume requirements. They want to gather ALL posts by a user that logs into the application, and all of the posts for each of their friends for the past year.
Before this particular level of detail was communicated to me, I built the thing using the fb_graph gem and would paginate through posts. I am running into the fact that first it takes a very long time to do this, even when I change the number of posts requested per page. Second, I frequently run into the Oauth error #613, more than 600 requests per 600 seconds. After increasing each request to 200 posts I run into this limit less, but it still takes an incredibly long time to get all of this data.
I am not particularly familiar with the FQL alternative, but it seems to me that we are going to have to either prioritize speed or volume of data. Is there a way that I am missing that would allow me to quickly retrieve this level of information?
Edit: I do save all posts to the database as I retrieve them. What is required is to make one pass through and grab all of the posts for the past year, for the user and friends. This process takes a long time and I am basically wondering if there is any way that it can be sped up.
One thing that I'd like to point out here:
You should implement some kind of local caching for user's posts. I mean, instead of querying FB each time for the posts, you should save the posts in your local database and only check for new posts (whenever needed).
This is faster and saves you many API requests.
I have a Twitter "widget" on my site where I allow users to login and pull in their twitter details and home timeline. I'm using OAuth for this and I have a little concern over rate limiting. In my widget, I'm having to make 2 requests every time I need the latest data. One request for the users details (and latest status) (the top section of my widget), and a separate request for their home timeline (the bottom section of my widget). What I'm wondering is if there is a way to get both with a single request. It just seems like it's a waste of requests to have to do it twice. Any way to do this or is this something I need to make a feature request for to Twitter? Does anyone else think that it would make sense for Twitter to always return the latest user details with each API request/result?
Your best bet is not to pull the user details with every refresh; cache it locally and only update the display of user details intermittently. Or give your use an option so they can control their own rate limit usage.
There's no single API to return what you're looking for. You can ask them to add it but don't hold your breath :).
The functionality I'm trying to implement is used in a site called Wefollow ( http://wefollow.com/ ):
On this WeFollow each account is checked for
# of followers
# of statuses
How can reliably update the information for each account without bumping into the 2000 queries/hour limit imposed by Twitter?
I'm trying to build a directory and update the same details. How can I deal with this?
Help would be very much appreciated.
EDIT: I'm trying to understand how that site works, not promoting it. I'm sorry if I wasn't clear enough.
If there's a hard limit, there's a hard limit. I would do this by putting all accounts in a rotating queue and updating them in that order up to a given maximum. If you can't cover all accounts, that's too bad. You could also calculate activity values based on deltas per user and prioritise the updating of those users. If a user hasn't changed in a month then there's no point checking that user more frequently than every week. Likewise, if a given user is very active, they can be pushed to the front of the queue.
BTW I would say this is verging on not programming related.
You can apply to have your IP address and account whitelisted which will increase your rate limit to 20,000/hour if you are approved. (http://apiwiki.twitter.com/Rate-limiting)
At FanPageList.com, we update information for our accounts every 2-4 hours. If you look closely at wefollow.com, some of their counts are outdated. Even at TwitterCounter.com, they only claim to update their counts daily (unless you pay for their paid service, they will start tracking your account hourly).