I'm currently living in a country in which YouTube website is banned and you'll need a VPN or a proxy to be able to access the YouTube website and videos using a web browser, but surprisingly, I've found that sometimes (most of the times actually) I can access YouTube and watch YT videos without using any VPN or proxies by using the YouTube Android application on my phone/tablet. This is related to my ISP as far as I know.
How does the YouTube Android application can connect to YouTube servers without the need of any proxies or VPN while at the same time I have to use a proxy program like "Freegate" to be able to watch YouTube on my computer? Is there a trick to do on my computer to be able to watch YT in web browsers without using any proxies?
I used an app called "Packet Capture" which captured the following IP addresses:
74.125.173.200:443
216.58.208.42:443
172.217.22.34:443
173.194.188.70:443
When I type them into a web browser, they redirect to google.com.
Android Oreo 8.1.0 and the latest version of Youtube.
My main goal is to be able to use youtube-dl without any proxies.
My question is similar to this.
Thank you.
The censors can block in a number of ways:
By modifying the ISP's DNS server to make youtube.com resolve to an IP address of their own (or just 0.0.0.0). Due to its minimal impact on the Internet infrastructure, this tends to be the preferred way in democratic countries. It can be easily evaded by using a DNS server not under the censor's control, e.g. 8.8.8.8 or 1.1.1.1.
By blocking ranges of IP addresses, for example 172.217.0.0/16. This is very likely to overblock, especially if the target is a smaller website which shares its IP range (or even IP address with others). Even in this case, outright blocking the whole /16 subnet would prevent access to google.com as well. This involves a lot of work on the censor's part, as IP addresses are routinely and automatically changed.
By reading traffic and blocking it if specific content is included. Since everything is encrypted these days, only DNS names and SNI are in the clear. There are already ongoing works to close these loopholes. The downside for the censor is that they have to read and parse every single packet, i.e. need large equipment, although if restricting to only DNS it is far more feasible. This is also likely to introduce problems for unrelated services which may occasionally be misdiagnosed.
(There are a couple of other ways and finer distinctions I'm listing in my PhD thesis on the topic of censorship, but they don't apply here).
No matter which ways the censors choose, they must feed a list of forbidden items into their equipment. In case 1&3, that's domain names, in case 2, that's IP address (ranges).
Modern services like YouTube are internally split into multiple smaller services. For instance, YouTube may internally consist of:
The web frontend, youtube.com / 172.217.22.46
The thumbnail service, ytimg.l.google.com / 172.217.21.206
The API googlevideo.com / 172.217.22.68
The video service, in multiple subdomains like r3---sn-i5onxoxu-q0n.googlevideo.com / 92.226.0.78
All domain names and addresses are examples; in practice, every service uses many IP addresses and different domain names (for instance there's www.youtube.co.uk .
If the censors in your country block only web frontend, the Android app will continue to work. The censors may also forget some domains - i.e. block youtube.com and youtube.pk and their subdomains, but not youtube.fr.
Alternatively, the censors may have tried to block the Android app by IP address, but the IP address may have been changed by YouTube, either as part of automatic internal reassignment or by explicit censorship avoidance.
The Android app contacts the internal google API, since it wants machine-readable information. So why does youtube-dl use the web interface and not the Android API?
I can't speak for all developers, but as the long-time lead developer of youtube-dl I know why I mostly focused on the web interface: It was simply a matter of convenience, as the tools to debug and observe web applications are far better than those for Android (or iOS apps). Therefore, when I added features, I would observe how the web JavaScript app realizes them, and reimplement that in youtube-dl.
So far, mainly using the web API was sufficient, but you are very welcome to add alternative code that uses the calls the Android app makes. Beware that this relief may be temporary though, as the censors may notice that the YouTube app is unblocked, and update their domain name and IP range lists to include that service as well.
Related
I am trying to build an web app, where a client logs in and wants to send some data to another website (for example he wants to upload a csv file that the application will read and send the read data to the other website).
The website does not have an official API which I can use and they block an IP Address if it is making too many requests.
So, my idea was to use the user's IP Address to send the request in his name so that the website will not block my app from making requests.
It is possible to do this from asp .net core mvc / asp.net mvc?
If not, what can I do to make this possible?
Thanks in advance
No, this is not possible. I would encourage you to actually read up on the TCP/IP protocol (HTTP, too, while you're at it, though it doesn't directly relate to this issue). Just like you would need to understand how something like iOS works before you could write apps for it, you should not how the underlying protocols of the Internet work, if you intend to write web apps.
In particular here, all communication over TCP/IP is by IP address. It is necessarily unique because the client/server needs to be able to "address" its packets there. Based on this, even if you could impersonate the user's IP address, the result would be the server sending its response directly to the user, instead of your server.
FWIW, there is a concept of "shared" IP addresses. For example, unless you have a dedicated IP from your ISP, you along with a lot of its other customers will appear as if you all have the IP address. IP addresses are traditionally limited. IPv6 is virtually limitless, but most everything still uses IPv4, which only exposes a range of ~4 trillion addresses. That may sound like a lot, until you realize that every single client needs a unique address. Just one person likely has a home computer, a work computer, a laptop, a smartphone and perhaps even a second business smartphone. Additionally they could have a range of other devices like a tablet, smart TV, connected thermostat, etc. As a result, your ISP essentially proxies your request to the destination and it then receives the response and forwards it on to you. It uses its internal IP tables to route the response back to the internal IP that requested it. So, while it's technically possible for multiple clients/servers to share the same external IP, it is not the same as what you're talking about here.
I'm just starting iOS app development with Swift (and in general) and I'm looking to get some information on popular practices when creating apps that require communication over arbitrary networks (i.e. not necessarily on the same network). I tried searching this on google but the answers weren't entirely what I was looking for; hopefully somebody can point be in the right direction. I wouldn't mind paying for a service, but unfortunately I don't know the first thing about backends and don't want to end up overpaying for services that I don't need. For example, I found an API called Parse, but I think it has far too many features that wouldn't benefit my app. Here's the main premise of the app:
There are two versions of the app - one for Admins and one for Employees
The Admins have the ability to post notes to a central list of notes for the employees to see
The Employees can access this list and scroll through it to pick which one they want to open. After a certain number of time, the notes expire and are removed from the list automatically
It's as simple as that. There likely won't be too many notes getting sent at once, so a large database isn't needed. My questions are as follows:
Do I need a database to store the notes, or can I handle it in some other way?
How is communication generally handled? The only things I've come across are ways to communicate when you're on the same WiFi or Bluetooth, but I haven't seen anything outside of that. How does an app like GroupMe communicate to users?
This is more of a general question, but how can you tell if you need a backend or not? I'm still kinda confused on the interaction between the frontend and backend.
Any help for any of the questions is greatly appreciated. I feel as though I don't even know where to start with a project like this.
EDIT: To clarify, I'm just looking for a place to start, not code or any implementation.
It's as simple as that. There likely won't be too many notes getting sent at once, so a large database isn't needed. My questions are as follows:
Do I need a database to store the notes, or can I handle it in some other way?
Yes you need some kind of database. That could be something complex like MySQL or something simple like writing a txt file for each note to the disk, with the filename being the date of the note.
You could use a service like Parse or run your own PHP server and write the software yourself. Parse is cheaper for a small database, running your own PHP server is cheaper for a big one and it gives you more control.
(You don't have to use PHP, but that is the most popular language for these things and it's what I use).
How is communication generally handled? The only things I've come across are ways to communicate when you're on the same WiFi or Bluetooth, but I haven't seen anything outside of that. How does an app like GroupMe communicate to users?
Usually your the phone sends a HTTP POST request to the server with some text in JSON format in the body of the HTTP request.
The server then responds with more text in JSON format in the response.
On the phone you use NSURLSession to handle to do the network communication and NSJSONSerialization to encode/decode the content. On the server, there will be something equivalent available.
Usually there would be a username and password or some other authentication system in the HTTP POST JSON text that tells the server wether or not the user is allowed to do whatever they're trying to do.
All communication between the phone and the server must be encrypted using SSL to protect your users. Do your homework and make sure you get this part right before you deploy your app to the store.
Parse will handle all of that stuff for you, but it's good to at least understand what's going on.
This is more of a general question, but how can you tell if you need a backend or not? I'm still kinda confused on the interaction between the frontend and backend.
You know you need a backend if you want two devices to communicate without being on the same WiFi/Bluetooth network. This is a security feature that cell network carriers (and home broadband ISPs) enforce to prevent malicious activity.
Generally only a commercial internet connection (and commercial router) will allow anonymous incoming network packets to get through to a phone/computer connected via that internet connection. Consumer internet connections only allow traffic coming in from a known source (for example, if you ask Google for some data, the router will temporarily allow Google to send some data to you. But if Google just sends some data without a phone/computer in your home asking for it, then it will be rejected).
You should be able to take what I've written and do a bunch of research.
If you decide to go with writing your own system in PHP, it comes pre-installed with OS X (just has to be enabled) and you can access it by IP address from the phone as long as you're on the same IP address. That should get you started for testing/development purposes at least.
The only part you won't have is SSL. Starting in iOS 9 (it's almost here!) you will need to disable NSURLSession's built in check for SSL or else it won't let you connect to the test server.
My iOS app uses a single hard-code URL api.xyz.com to find our REST service. At the moment there are just two servers running this service, and we use Amazon Route 53 DNS. But I've found that the timeout of an hour (or more) is too long incase one of our servers fails; don't want to leave users in the dark that long.
The alternative would be to implement a failover mechanism in the app. To be honest, I don't like the idea of pulling this low level DNS-related logic in the app, but I don't see another solution at the moment.
So my question is: How do I implement such a failover mechanism on iOS? I'm using AFNetworking for my REST API.
Or, are there better alternatives on server side? At the moment the servers are individually rented ones, so no Amazon, Google, ... cloud service.
I am building a multiplayer site in Rails, and I need a reliable way of blocking computers from using the site. For example, someone has been banned from the site multiple times, and there should be a way to block their computer from using the site.
I would find a way to do this with IP addresses but they're not always static are they? Also I'm asking about identifying browsers so that if there was unauthorised access from a different browser or machine, then I could implement a security feature to make sure the account belonged to them. I would also need to store these locations as trusted locations in a database, hashed.
I'm currently building a mobile application (iOS at first), which needs a backend web service to communicate with.
Since this service will be exposing data that I only want to be accessed by my mobile clients, I would like to restrict the access to the service.
However I'm in a bit of a doubt as to how this should be implemented. Since my app doesn't require authentication, I can't just authenticate against the service with these credentials. Somehow I need to be able to identify if the request is coming from a trusted client (i.e. my app), and this of course leads to the thought that one could just use certificates. But couldn't this certificate just be extracted from the app and hence misused?
Currently my app is based on iOS, but later on android and WP will come as well.
The web service I'm expecting to develop in nodejs, though this is not a final decision - it will however be a RESTful service.
Any advice on best practice is appreciated!
Simple answer: You cannot prevent just anybody from acecssing your web site from a non-mobile client. You can, however, make it harder.
Easy:
Send a nonstandard HTTP header
Set some unique query parameter
Send an interesting (or subtly non-interesting) User Agent string
(you can probably think of a few more)
Difficult:
Implement a challenge/response protocol to identify your client
(Ab)use HTTP as a transport for your own encrypted content
(you can probably think of a few more)
Of course anybody could extract the data, decompile your code, replay your HTTP requests, and whatnot. But at some point, being able to access a free Web application wouldn't be worth the effort that'd be required to reverse-engineer your app.
There's a more basic question here, however. What would be the harm of accessing your site with some other client? You haven't said; and without that information it's basically impossible to recommend an appropriate solution.