I understand that the Twitter streaming API only allows for a single connection per user account on Twitter. I was wondering if multiple streaming connections from the same IP address are allowed, if each connection has credentials from a different user account? For example, in a scenario where a single server may process tweets for multiple users. Specifically, I am curious about the filtered public streams endpoint.
I have read the following, and cannot locate an answer:
Twitter Streaming API limits?
https://dev.twitter.com/rest/public/rate-limiting
https://dev.twitter.com/streaming/public
Thank you, -Matt
I can confirm that I have three different Twitter API streams in a Node-RED application running on the same IP with no problem. Whenever I make changes or redeploy I do get rate limit errors for the first minute or so, but they go away and do not appear again.
The standard Streaming API supports up to two connections on a single IP. Any more than that will result in the oldest stream being disconnected. The developer agreement and policy states that you should not use multiple application keys and/or IP addresses to circumvent this limitation.
Note that it is on the roadmap for the standard filter and sample streams to eventually be updated with new versions, but it is not yet clear if they will carry the same connection limitations.
Related
The current youtube API quota is 10,000 units/day.
So what if I created an app and this app has 100 copies installed on different 100 devices and all of them use different IPs, would the quota be 10,000 units/day for every device (1M/day) or the 10,000 units would be distributed among the different copies?
Note that the requests will be executed on different devices. this is an embedded app., not a cloud one.
10,000 units/day is project based, not user based. IP address won't affect anything.
As you can see from the above image on the Queries per minute per user is user based, and detected via ip address, sending the quotauser parameter along with the request or sending an authorization header containing an access token.
In order to get around your 10k quota in this manner you would need to create 100 projects on google developer console and create credietnals for each of those projects and then each project would have 10k quota. However the drawback would be trying to get 100 projects though the google verification process.
You could just request an extension of your quota if you need more quota. 😊
Additional Clarifications
API key Used to access public data only and identifies your application to google
Client id and client secrete used to create a access token, will identify both the application and the user to google once a user is authorized and you are sending an access token authorization header with your request.
The quotaUser and userIp parameters can only be used for anonymous requests with some Google apis this has changed recently i am not sure if it still works with YouTube api private calls. userIp allows you to send any ip address with the request, and QuotaUser allows you to send any string for example your internal user id for the user making the requests. In the past this allowed for some hacking / work around for the per user quotas.
None of the above information will really help you as the quota you are trying to get around is a project based quota and not a user based quota. Its detecting the project associated with your client id / api key (depending upon what you are using) changing the IP or generating a new client under your project isnt going to help.
Really just request more quota
There are two ways an app is identified by the API:
the API key that the app uses when invoking the API endpoints (for reading only public data), or
the credentials data (i.e. access token) the app makes use of when calling in various endpoints (for reading only public data pertaining to any YouTube channel, or, otherwise, for reading/updating/deleting public and/or private data pertaining the YouTube account/channel that granted permissions of operation to the respective app).
Therefore, from the point of view of the API, it doesn't really matter if you are running N different instances of your app (i.e. programs running on concrete computers). The API accounts for the quota consumption of a given project created through Google developers console.
In this context of an app (as a concrete program) running on several machines that share API keys and/or credentials data, I recommend warmly to get familiar with the content of the DTOS specifications.
I was using Youtube Data APIto get youtube video but after getting mail from youtube which says api is disable due to inactive.
Youtube Data api not working since it was disabled by youtube because of inactive for 90 Days. I am not able to call(It is enable in API Console) it even with delete and add new delete back. Please help.
Error shows when i am calling from Web Browsers with api Key i also tried to generate new API key also delete and regenerate new api key but still no luck.
"Access Not Configured. YouTube Data API has not been used in project
264969722993 before or it is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/youtube.googleapis.com/overview?project=264969722993
then retry. If you enabled this API recently, wait a few minutes for
the action to propagate to our systems and retry."
There are things you can do to try and solve this issue. This first option is to apply for an API exception by filling out this out this form. I've filled out this form and am waiting for a response.
The second option is to create a new project and use that project to generate an API key. Because Google has only disabled access to the Youtube Data API for your current project, the new project's API key should work just fine. I've tried this and can confirm this works. This is probably the more hassle-free option if you have extra projects to spare as Google only allows 10 projects at any point in time.
I'm in the same position, disabled for no calls in 90 days. You can request an exemption but after about 25 back and forth emails it's clear that it is just a smokescreen. After explaining what it's used for they then wanted screenshots, then videos of you using the application, how we use the videos we pull up, and then a shot of the room where it was being used. They even asked who made the video we were looking at (we have no idea!). They want your whole business model and what you do. It was just never ending questions and more stuff to send them. After about 2 1/2 months of slow back and forth, I get a message saying I am out of compliance.
Our application uses YouTube to pull up videos for manufacturers product for research and training in a small training/conference room for internal use only. That's all!
Nothing was out of compliance.
This is the email we received:
Hello ,
We have reviewed your quota extension request for project number
#### and have revoked the allocated quota for your API Client(s) as it is out of compliance on the following YouTube API Services -
Developer Policy:
Policy F. 1 (User Experience - YouTube Look and Feel) Please do not
reply to this email, you may complete and submit this form to re-apply
once the above concern has been addressed.
Thanks, YouTube API Services team
The form link in the email is just a link to start the whole process over again.
According to YouTube policy
If your API Client's quota is reduced or eliminated, you may reapply for quota or a quota extension, and YouTube will review that application based on YouTube’s determination of your expected use of the YouTube API Services.
https://developers.google.com/youtube/terms/developer-policies
I suggest reapplying.
To add to avatarhzh's answer:
If you have already used up your initial cap of max. 12 projects and your channel is a brand account and you do not want to go through the process of applying for more quota/getting back your previous quota, you can also create new projects on a different google account and then add that account to your channel as a manager. It is then possible to access your channel through this new account via the youtube data API.
If your channel is not a brand account yet, you can move your channel to a brand account. This allows you to add different google accounts as managers/co-owners of the channel. Warning: certain things (e.g. comments) cannot be transferred when moving to a brand account.
jut for a background, I'm building a poc of twilio based video chat contact center and I've came upon one issue. Most of the people using the service will be anonymous customers, no logging / authentication here. Looking at the twilio api, once I produce a Token with VideoGrant I loose all control of what the javascript client does with it, for example, someone can modify my JS client to create their own room, instead of joining the one assigned by my app.
Is there any way to create VideoGrant which would allow only joining a room, preferably a single, selected room?
There is a mention of Room Events in the console, but I didn't see any docs on it, maybe it would allow to catch rooms creation and kill the unknown ones, btw is there a REST api for rooms? nothing on official REST api docs again.
Sorry if the questions are a bit vauge, I'm aware that they might not meet SO standards, but I know that few Twilio developer evangelists are active so maybe you guys can propose some best practices here or just point out to the right docs if I am dumb enough to have missed them.
Lukasz,
Rob Brazier here, PM for Video. In the next few weeks we'll be adding a new grant to the Video section of our Access Tokens that allow you to lock down access only to a specific Room.
The basic flow will be:
User visits your site.
Your app server decides what Room they should connect to.
Your app server creates a Room using whatever identifier you like as the name (using the upcoming Video REST API).
Your app server generates an Access Token with a grant for the specific Room created in #3.
Your app server returns that Access Token to the client.
Client connects using the provided token and is joined to the specified Room.
Let me know if this is clear. Thanks,
The twitter user stream docs state
Note that User Streams are not intended for server-to-server connections.
To maintain some metrics (when the user is away) I need to run a user stream from my server.
I cant user the public stream API as it does not allow me to track more than 400 words.
What happens if I start user stream(s) from my server for, say, a thousand users?
Although it is not intended but it is technically possible.
Tested with twitter4j on my server.
I have a corporate website that I want to pull in tweets to, but i'm getting a rate limit using the http feed. So, I want to use an authenticated method to get the tweets.
Do I really have to register an application to do this, even though it's not really an application and my users will never be entering or changing the twitter account info.
Also, my corporate site doesn't have a public address, and registering an application through twitter appears to require a public url. So how can I get around this? Do I have to create a "fake" application with a public url, just to generate my keys?
Thanks for any help on this.
If your site is behind a proxy server along with all your users, using Javascript/jQuery won't help. All the requests will still be coming from the same IP and will hit a rate limit, as you're doing now.
The other issue is that you don't need to register an app to request a feed. Apps are only needed for Oauth, and getting a feed doesn't need that.
The best way to deal with this is to get the feed with a server script, store it on the server, and then deliver the server copy to the web pages. If you request the feed less than 150 times per hour, you won't have a limit problem.
If you want more than a single feed, you can use the streaming API to get all the tweets for up to 400 keywords or from up to 5,000 users. This still doesn't need a registered app, since the streaming API still allows Basic Auth.
Just wanted to post this for future reference and in case anyone else has the same question. The solution to my problem, was to register an application on twitter. But since I'm just using a single user, you don't have to do the regular OAuth steps of generating a request for a key, getting the response etc. Every app you register in twitter get's its own "Access Token" that you can use to retrieve tweets etc. So, this is what I ended up doing to solve the problem I was having.
Additional details: My main concern was having to do the OAuth steps of requesting an access code etc... Since my application is only a single user implementaion (just pulling in our company related tweets from company held twitter accounts), it just seemed unneccesary to have to do all of that. But what I found was that when you register an app on twitter, you get a private access token for each app. You can view a little information about that here: https://dev.twitter.com/pages/oauth_single_token.
It sounds like you are pulling the feed down over http on the server? You could just limit the updates so you don't hit the rate limit.
I would recommend instead doing this on the client side. There are a lot of very easy to use embeddable java script twitter clients out there. The rate limiting problem would dissapear as the feed would be coming from the desktop and not the server (unless they just kept refreshing it).
The Twitter developer wiki lists a few.
JQuery plugin for Twitter
Tweet (another JQuery plugin)