So basically I want to allow my users to connect to my website with Twitter in order to fetch their friends (followings), save these relationships to a table and keep this table updated when there is a change on Twitter (new follow, unfollow).
For the fetch part, I handle it with https://github.com/sferik/twitter. But I don't really know how to get started with the "update" part considering I could have a large users base and Twitter's rate limits. I thought about using a background job and play with rate limits, but it doesn't look like a viable or scalable option.
Any ideas to put me on tracks?
I don't know the twitter api, but it would be magnificent if you could have a webhook that gets to notifications form twitter every time a user of your app gets or looses a follower so that every 100 notifications or so you trigger a background job using active job; although I don't think that's very convenient for twitter either.
If that's not an option maybe you could have a background job for now and have it running on a special dyno or dynos when your users base grows large enough (I can explain myself in heroku terms).
Related
I'm designing a web application that needs to retrieve and organize emails from at least 100 mailboxes from our company's domain. Something like:
abc#company.com
cde#company.com
efg#compnay.com
...
My web app needs to check each of these emails every couple of seconds to retrieve the new emails and index them. However, I don't want to hit the API limit for them.
Reading the official documentation, it seems like I can have a single app ID and use it to retrieve all these informations and won't hit the API limit.
If I have 100 mail boxes, and let's say, check each mailbox every 10 seconds (with maximum 4 concurrent threads), is it safe to say that I won't hit any kind of rate limit?
It might be worth mentioning that I'm going to use the Delta Link feature to check for new emails. This will make things faster and I'm not sure if it has any effect on the rate limits.
You really need to decrease the frequency of direct hits and only do that once you receive a notification from a particular mailbox/folder.
See https://learn.microsoft.com/en-us/graph/api/subscription-post-subscriptions?view=graph-rest-1.0&tabs=http for details on Graph event subscription.
So I am working on a Rails application, and the person I am designing it for has what seem like extremely hefty data volume requirements. They want to gather ALL posts by a user that logs into the application, and all of the posts for each of their friends for the past year.
Before this particular level of detail was communicated to me, I built the thing using the fb_graph gem and would paginate through posts. I am running into the fact that first it takes a very long time to do this, even when I change the number of posts requested per page. Second, I frequently run into the Oauth error #613, more than 600 requests per 600 seconds. After increasing each request to 200 posts I run into this limit less, but it still takes an incredibly long time to get all of this data.
I am not particularly familiar with the FQL alternative, but it seems to me that we are going to have to either prioritize speed or volume of data. Is there a way that I am missing that would allow me to quickly retrieve this level of information?
Edit: I do save all posts to the database as I retrieve them. What is required is to make one pass through and grab all of the posts for the past year, for the user and friends. This process takes a long time and I am basically wondering if there is any way that it can be sped up.
One thing that I'd like to point out here:
You should implement some kind of local caching for user's posts. I mean, instead of querying FB each time for the posts, you should save the posts in your local database and only check for new posts (whenever needed).
This is faster and saves you many API requests.
I have a Twitter "widget" on my site where I allow users to login and pull in their twitter details and home timeline. I'm using OAuth for this and I have a little concern over rate limiting. In my widget, I'm having to make 2 requests every time I need the latest data. One request for the users details (and latest status) (the top section of my widget), and a separate request for their home timeline (the bottom section of my widget). What I'm wondering is if there is a way to get both with a single request. It just seems like it's a waste of requests to have to do it twice. Any way to do this or is this something I need to make a feature request for to Twitter? Does anyone else think that it would make sense for Twitter to always return the latest user details with each API request/result?
Your best bet is not to pull the user details with every refresh; cache it locally and only update the display of user details intermittently. Or give your use an option so they can control their own rate limit usage.
There's no single API to return what you're looking for. You can ask them to add it but don't hold your breath :).
I need to develop an application which should help me in getting all the status,messages from different servers like Twitter,facebook etc in my application and also when i post a message it should gets updated in all the services. I am using authlogic for authentication. Can anyone suggest me what gems/plug-ins i can use..
I need API help to get all the tweets/messages to be displayed in my application and also ways to post the messages to the corresponding services by posting it from my application. Can anyone help me from design point.
Walk through what you'd want to do in your head. Imagine the working site, imagine your webapp working before you start. So your user logs in (handled by authlogic) and sees a textbox called "What are you doing right now?". The user fills in a status message and clicks "post". The status message appears at the top of their previously posted messages.
Start with the easy part. Create a class that posts to two services. Use the twitter gem and rfacebook to post to two already defined services. In the future, you'll want to let the user associate services to their account and you would iterate through the associated services and post the message to each. Once you have this working, you can refactor or polish the UI a bit to round out this feature. I personally would do the "add a social media account to my profile" feature towards the end.
Harder is the reading of the data (strangely enough) because you're going to have to figure out how to store it. You could store nothing but I suspect you'd run into API limits just searching all the time (could design around this). I would keep a little cache of posts associated to the user's social media account. In this way, the data model would look like this:
A user has many social media accounts.
A social media account has many posts. (cache)
Of course, now you need to schedule the caching of the posts. This could be done manually, based on an event (like when they login) or time based. So when the update happens, you load up the posts for that social media account and the user will see the posts the next time they hit the page. For real-time push to the client's browser while they stare at the screen, use faye (non-trivial) and ajax to pull the new posts to the top of the social media stream view.
The time based one is tricky because you'd either have to have a cron job run or have rails handle it all with a gem like clockwork. But then you have to leave rails running. I've also solved this by having a class in /lib do all the work and a simple web call kicks off the update. But it wasn't in a multi-user use case. So that might not work. In any case, you'll want to have some nice reusable code for these problems since update requests can come from many different sources.
You'll also have to deal with the API limits. When pulling down content from twitter, you won't get everything. That will just have to be known by the user or you'll have to indicate a "break in time" somehow.
The UI should be pretty easy (functionally anyway), because you know which source the post/content is coming from. It'd be easy to throw a little icon next to the post to display which social media site it's coming from.
Anyway, good luck, sounds like a fun project.
I'm going to attempt to create an open project which compares the most common MP3 download providers.
This will require a user to enter a track/album/artist name i.e. Deadmau5 this will then pull the relevant prices from the API's.
I have a few questions that some of you may have encountered before:
Should I have one server side page that requests all the data and it is all loaded simultaneously. If so, how would you deal with timeouts or any other problems that may arise. Or should the page load, then each price get pulled in one by one (ajax). What are your experiences when running a comparison check?
The main feature will to compare prices, but how can I be sure that the products are the same. I was thinking running time, track numbers but I would still have to set one source as my primary.
I'm making this a wiki, please add and edit any issues that you can think of.
Thanks for your help. Look out for a future blog!
I would check amazon first. they will give you a SKU (the barcode on the back of the album, I think amazon calls it an EAN) If the other providers use this, you can make sure they are looking at the right item.
I would cache all results into a database, and expire them after a reasonable time. This way when you get 100 requests for Britney Spears, you don't have to hammer the other sites and slow down your application.
You should also make sure you are multithreading whatever requests you are doing server side. Curl for instance allows you to pull multiple urls, and assigns a user defined callback. I'd have the callback send a some data so you can update your page with as the results come back. GETTUNES => curl callback returns some data for each url while connection is open that you parse it on the client side.