What is the best practice if I have 20K twitter user base and I want to track user's specific keywords via statuses/filter?
Should I distribute the processing on multiple nodes, lets say open a streaming connection tracking keywords for 5K users each (on different IPs or same IP with different authenticating users)?
Or just apply for a bigger access level and use a single connection to get the whole thing.
Thanks,
Alam Sher
Bigger access level. Using multiple accounts to circumvent access limits is
Frowned upon by Twitter
Complicates your processing
Related
I'm just starting iOS app development with Swift (and in general) and I'm looking to get some information on popular practices when creating apps that require communication over arbitrary networks (i.e. not necessarily on the same network). I tried searching this on google but the answers weren't entirely what I was looking for; hopefully somebody can point be in the right direction. I wouldn't mind paying for a service, but unfortunately I don't know the first thing about backends and don't want to end up overpaying for services that I don't need. For example, I found an API called Parse, but I think it has far too many features that wouldn't benefit my app. Here's the main premise of the app:
There are two versions of the app - one for Admins and one for Employees
The Admins have the ability to post notes to a central list of notes for the employees to see
The Employees can access this list and scroll through it to pick which one they want to open. After a certain number of time, the notes expire and are removed from the list automatically
It's as simple as that. There likely won't be too many notes getting sent at once, so a large database isn't needed. My questions are as follows:
Do I need a database to store the notes, or can I handle it in some other way?
How is communication generally handled? The only things I've come across are ways to communicate when you're on the same WiFi or Bluetooth, but I haven't seen anything outside of that. How does an app like GroupMe communicate to users?
This is more of a general question, but how can you tell if you need a backend or not? I'm still kinda confused on the interaction between the frontend and backend.
Any help for any of the questions is greatly appreciated. I feel as though I don't even know where to start with a project like this.
EDIT: To clarify, I'm just looking for a place to start, not code or any implementation.
It's as simple as that. There likely won't be too many notes getting sent at once, so a large database isn't needed. My questions are as follows:
Do I need a database to store the notes, or can I handle it in some other way?
Yes you need some kind of database. That could be something complex like MySQL or something simple like writing a txt file for each note to the disk, with the filename being the date of the note.
You could use a service like Parse or run your own PHP server and write the software yourself. Parse is cheaper for a small database, running your own PHP server is cheaper for a big one and it gives you more control.
(You don't have to use PHP, but that is the most popular language for these things and it's what I use).
How is communication generally handled? The only things I've come across are ways to communicate when you're on the same WiFi or Bluetooth, but I haven't seen anything outside of that. How does an app like GroupMe communicate to users?
Usually your the phone sends a HTTP POST request to the server with some text in JSON format in the body of the HTTP request.
The server then responds with more text in JSON format in the response.
On the phone you use NSURLSession to handle to do the network communication and NSJSONSerialization to encode/decode the content. On the server, there will be something equivalent available.
Usually there would be a username and password or some other authentication system in the HTTP POST JSON text that tells the server wether or not the user is allowed to do whatever they're trying to do.
All communication between the phone and the server must be encrypted using SSL to protect your users. Do your homework and make sure you get this part right before you deploy your app to the store.
Parse will handle all of that stuff for you, but it's good to at least understand what's going on.
This is more of a general question, but how can you tell if you need a backend or not? I'm still kinda confused on the interaction between the frontend and backend.
You know you need a backend if you want two devices to communicate without being on the same WiFi/Bluetooth network. This is a security feature that cell network carriers (and home broadband ISPs) enforce to prevent malicious activity.
Generally only a commercial internet connection (and commercial router) will allow anonymous incoming network packets to get through to a phone/computer connected via that internet connection. Consumer internet connections only allow traffic coming in from a known source (for example, if you ask Google for some data, the router will temporarily allow Google to send some data to you. But if Google just sends some data without a phone/computer in your home asking for it, then it will be rejected).
You should be able to take what I've written and do a bunch of research.
If you decide to go with writing your own system in PHP, it comes pre-installed with OS X (just has to be enabled) and you can access it by IP address from the phone as long as you're on the same IP address. That should get you started for testing/development purposes at least.
The only part you won't have is SSL. Starting in iOS 9 (it's almost here!) you will need to disable NSURLSession's built in check for SSL or else it won't let you connect to the test server.
I am building a multiplayer site in Rails, and I need a reliable way of blocking computers from using the site. For example, someone has been banned from the site multiple times, and there should be a way to block their computer from using the site.
I would find a way to do this with IP addresses but they're not always static are they? Also I'm asking about identifying browsers so that if there was unauthorised access from a different browser or machine, then I could implement a security feature to make sure the account belonged to them. I would also need to store these locations as trusted locations in a database, hashed.
I am using Rails 3.1.1 and I would like to redirect users, for example, from the U.S.A. to a proper subdomain us.site.com (this one is hosted on the same server as site.com). I know that I can localize a user by his/her IP address but how can I do that so to redirect he/she to the proper subdomain? There is a technique/gem to geo-locate user IPs and then handle redirection?
P.S.: Maybe, for performance reasons, I should use middlewares...
https://rubygems.org/gems/rack-geoipcity is a rack middleware gem I've published which you could use, or just use the GeoIP gem in your controllers.
With the rack-geoipcity, you would query the X-headers it adds in and make a decision based on that. Something like:
if headers['X_GEOIP_COUNTRY_CODE'] == "IN"
redirect "/india"
end
though I don't currently use Rails, so it might be slightly different.
There are plenty to choose from if you don't fancy using the MaxMind db.
One approach that I've used happily in the past is to perform the geolocation lookup via DNS before users connect to the service; this way, they automatically connect to the server nearest them, you get cheap and easy load balancing and ability to remove servers from active use as you need, and individual sites can go down without influencing other sites.
OFTC uses a self-written oftcdns tool to provide users with nearest servers. During the time I was an administrator on the OFTC network, this tool was a drastic improvement over running a simpler Bind-based DNS server that did not provide geo-location features and complicated bringing servers in and out of the rotation.
Wikipedia uses PowerDNS with a geobackend to provide their geo-ip services. PowerDNS is definitely well-tested, high-demand-capable tool.
I wonder if anyone can help me, I'm getting a little confused as to
which API to use. If anyone can offer some guidance I would really
appreciate it.
I'm trying to create an website where users can monitor Twitter for
certain hashtags. The site will continually search twitter for any new
updates and store any tweet related to that particular hashtag. This process will run for up to 60 days.
As far as I can gather, my two options are:
Using the Search API
The problem with this API is that if I have a 1000 users all
monitoring different hashtags, I am quickly going to reach my API
limit since I will be making a fair few requests, potentially once
every 2-3 minutes. Is there a way to use oauth in conjunction with the
search API so that the limits are user based and not application
based? That way, the limit will be user specific and I won't have to
worry.
Using the Stream API
I thought this might be a better solution, but it seems you are
limited to how many connections you can have open. The documentation
seems unclear as to how this works... is the connection limit per twitter account
or service ip? For example, if my site had 1000 users each of those users was
monitoring a hashtag, would those 1000 stream api connections be
against my servers ip or would they against the user?
You will want to use the Streaming API. You will open a single connection that will track the terms for all of the users. When users add new terms to track you will restart the stream with the new terms. The single stream will be for a bot Twitter account you create and not your users accounts.
I am new to the Twitter API, and I looked at their whitelisting policies and I am a little confused... I'm basically writing a twitter aggregrator that crawls the public tweets of a set of users (not more than 200) hourly. I wanted to apply for whitelisting, and they seem to offer account based and IP based whitelisting. Since I am using a shared hosting, my outbound IP address might vary (and twitter does'nt allow IP ranges for whitelisting). So I am considering using account based whitelisting.
However, while using OAuth, is it possible for me to use account based whitelisting for a background process that crawls the API hourly?
You won't be able to whitelist each of the 200 accounts you want to crawl. However. Assuming each of those 200 has OAuthd with you, you can use their access token to crawl their timelines. This eats into their rate limits, not the one for your service. This has the obvious downside of eating into their rate limits though.