YouTube imposes the following quota cost limits (default values listed):
Queries per day = 1Million units
Queries per 100 sec per user = 300,000 units/100sec/user
Queries per 100 seconds = 3,000,000
What is the meaning of the last limit? How can the quota per 100 seconds exceed the total quota per day?
Here are the meaning of the different quota in the YouTube Data API
QPD(quota per day) - meaning the maximum numbers of request over a 24 hour period a client id is able to make to an API.
QPS(quota per second) - meaning a global quota per second for the application, meaning how many calls per second an application can make.
quota per seconds per user - meaning the number of queries a user, in the application can make.
The quota of 3,000,000 per 100 sec did not exceed the 1M QPD because you need to divide the 3M QPS to 100.
So meaning you only have 30,000 QPS or queries per second.
I hope this information helps you.
I believe the "Queries per 100 seconds = 3,000,000" is a inaccurate/left over/a mistake from Google's old Query limits. Clearly 3,000,000 in 100 seconds is 3x your total per day and makes no sense!
The "old" limits used to be much higher:
50,000,000 Queries per day
The per 100 seconds limits, however, did not change (or at least were not updated properly).
I was also looking for this, as the documentation from YouTube is not clear at all. Some answers, such as #nightsurgex2's, hinted at the fact that only the daily limit counts, but I wanted to be sure before sending this to production, so I wrote and ran some custom tests. I'm not going to speak about the "per user" limits, as the application we are developing does not use this.
The test application just sends a lot of dummy requests (each worth 1 quota point) and breaks when API returns an error. Please keep in mind that this will exhaust your application quota for that day, so use a dummy project if you want to try anything similar. The results were:
Finished YouTube Data: 03/17/2022 15:24:13
Took: 51759.2994 ms
Total requests: 10451
Finished YouTube Analytics: 03/17/2022 15:29:29
Took: 16080.7929 ms
Total requests: 892
Finished YouTube Analytics: 03/17/2022 18:29:57
Took: 10478830.055 ms
Total requests: 98927
The limit for YouTube Data v3 is 10k points per day. The number we got is very close to this. Trying to send further requests, even hours later, will fail. There is no per-minute limit.
YouTube Analytics v2 enforces both a per-minute and a per-day limit. These values are consistent with Google's documentation and with the test above: 720 requests per minute, or 100k per day (reset at midnight US West Coast). If you go steadily at the maximum per-minute rate, you will exhaust your daily quota in around 2 hours and 20 minutes.
YouTube Data has a much smaller quota, however, you are not likely to use it as often. You only use it to get playlist ids, video ids, and the like. The bulk of data (viewers, demographics, subscribers, etc) will come from Analytics. Even then, you will possibly have to phase our your requests or cache their result, depending on your needs.
Related
I am using a simple script to pull the total number of views of a particular video onto a webpage.
As I want it as 'realtime' as possible, I have a metatag that automatically refreshes the page every 60 seconds.
My question here is, I guess every time the page refreshes that is seen as a new call and comes from my quota. As this is running 24/7 does that mean I will exceed my quota fairly quickly given I will soon reach the 10,000 mark at this rate?
Or does each page refresh not class as a call?
I want to firstly ensure I don't go over quota and it ends up disabled but more importantly not look like I'm completely taking the mick and get seen as a spammer of some sort.
If a page refresh makes an API call, then that API call is accounted for quota usage.
Now, you say that your page refreshes at a rate of one Video.list API endpoint call per minute.
Therefore, during a full day (24 hours), you'll have 24 * 60 = 1440 calls to Videos.list.
According to the official specification of Videos.list, each call to this endpoint has a quota cost of 1 unit.
Consequently, if only accounting for 1440 calls to Videos.list, the quota cost of your page refreshing amounts to 1440 units. That's well below the allocated daily quota of 10000 units.
This implies also that the API will by no means consider you a spammer.
I'm currently developing a chat bot for one specific YouTube channel, which can already fetch messages from the currently active livechat. However I noticed my quota usage shooting up, so I took the "liberty" to calculate my quota cost.
My API call currently looks like this https://www.googleapis.com/youtube/v3/liveChat/messages?liveChatId=some_livechat_id&part=snippet,authorDetails&pageToken=pageTokenIfProvided, which uses up 5 units. I checked this by running one API call and comparing the quota usage before and after (so apologies, if this is inaccurate). The response contains pollingIntervalMillis set to 5086 milliseconds. Currently, my bot adds that interval to the current datetime and schedules the next fetch at that time (using Celery), so it currently fetches messages at a rate of 4-6 seconds. I'm gonna take the liberty and always wait for 6 seconds.
Calculating my API quota would result in a usage of 72.000 units per day:
10 requests per minute * 60 minutes * 24 hours = 14.400 requests per day
14.400 requests * 5 units per request = 72.000 units per day
This means that if I used the pollingIntervalMillis as a guideline for how often to request, I'd easily reach the maximum quota of 10.000 units by running the bot for 3 hours and 20 minutes. In order to not use up the quota by just fetching chat messages, I would need to run 1 API call per minute (1,3889 approximately). This is very unfeasible for a chatbot, since this is only for fetching messages and not even sending any messages to the chat.
So my question is: Is there maybe a more efficient way to fetch chat messages which won't use up the quota so much? Or will I only get this resolved by applying for a quota extension? And if this is only resolved by a quota extension, how much would I need to ask for reliably? Around 100k units? Even more?
I am also asking myself how something like Streamlabs Chatbot (previously known as AnkhBot) accomplishes this without hitting the quota limit despite thousands of users using their API client, their quota must probably be in the millions or billions.
And another question would be how I'd actually fill out the form, if the bot is still in this "early" state of development?
You pretty much hit the nail on the head. Services like Streamlabs are owned by larger companies, in their case Logitech. They not only have the money to throw around for things like increasing their API quota, but they also have professional relationships with companies like Google to decrease their per unit cost.
As for efficiency, the API costs are easily found in the documentation, but for live chat as you've found, you're going to be hitting the API for 5 units per hit. The only way to improve your overall daily cost with your calls is to perform them less frequently. While once per minute is clearly excessively long, once every 15-18 seconds could reduce the overall cost of your API quota increase, while making the chat bot adequately responsive.
Of course that all depends on your desired usage of the data, but still a recommendation if you're implementing the bot still in the realm of hobbyist usage.
There are 3 rate limit categories in the API explorer:
If I create a client-side web app using the Youtube Data api (in which the implicit OAuth grant flow is followed), am I still limited to 10k quota units total? If this is the case, then the application cannot handle >10 users who make a single youtube-search through the api.
The limitation is done at the Registered Application level. The 10,000 queries per day is really 10,000 units per day and depending on the methods your using and the number of times you use the particular methods the cost can vary. Each user of the application does not get 10,000 units. The users share a pool of 10,000 per day. You need to come up with a math formula based on the different methods you use, and the number of times you use them per day on average and sum them up to figure out how many credits you need and if 10,000 credits will be enough.
Google calculates your quota usage by assigning a cost to each API method request.
Different types of operations have different quota costs.
For example:
A read operation that retrieves a list of resources -- channels, videos, playlists -- usually costs 1 unit.
A write operation that creates, updates, or deletes a resource usually has costs 50 units.
A search request costs 100 units.
A video upload costs 1600 units.
See this link for a complete table of available API methods and the associated cost: https://developers.google.com/youtube/v3/determine_quota_cost
After you complete your formula and sum up the number of units you need per day, if it exceeds 10,000 you can make a request to get an extension and increased quota https://support.google.com/youtube/contact/yt_api_form
My application allows users to "link" their YouTube accounts to our system and then we allow them to upload videos to their channels both automatically in some cases, and on an "upload" action on their part. We have hundreds of videos being uploaded because we have thousands of users that use our system.
Today I saw hundreds of errors in my application when our batch automatic upload job was running. The errors were for:
quotaExceeded, video upload limit reached.
My API quotas are very high (50,000,000 per day, 3,000,000 per 100 seconds, 300,000 per 100 seconds per user), so if a video is 1600 points, I have enough limit to upload 30,000+ per day, 1,800 videos per 100 seconds, and 180+ per 100 seconds per user.
I have seen other questions out there hinting at some newly enforced limit by Google to just 50 videos and then 1 video every 15 minutes? This is a very low limit for my application which has such a large quota! To be clear, at most I have seen my application handle 1,000 videos in a single day (1,600,000 points of my total 50,000,000 quota).
Is there any way I can get this artificial limit of 50 videos and then 1 per 15 minutes removed? This is a major block to my users' functionality for a very popular web app. I could understand if it were 50 videos per user and then 1 every 15 minutes, but I highly doubt my errors this morning were from one user trying to upload 300+ videos at a time. My system only uploads their "newest" videos each day, which most people only have 1-10 videos at max. Hundreds would be a very rare edge case.
You can actually confirm in your Developers Console the available quota for your application. By default, YouTube API have a quota allocation of 1 million units per day as mentioned in Quota usage. If you see that your usage reached your quota limit, you can request additional quota on the Quotas tab.
Note also that, all API requests, including invalid requests, incur a quota cost of at least one point. You may use the Quota Calculator to get an estimate of the quota cost for an API query.
On the other hand, to work efficiently with your quota and if you haven't done so, I suggest that you implement exponential backoff if you're encountering high error ratio. See this sample code which shows an exponential backoff strategy to resume a failed upload. Also, if applicable, subscribe to Push Notifications which is much more efficient than polling-based solutions.
Check the documentation for more information on how PubSubHubbub callback server receives Atom feed notifications when a channel does any of the following activities:
uploads a video
updates a video's title
updates a video's description
Hope that helps!
I keep getting:
Response:
{"errors":[{"message":"You have made too many requests recently. Please, be chill."}]}
When trying to download all my tasks - is there a published QPS or other quota limit so I know how liong I should pause between requests?
(I work at Asana)
As stated in the documentation, the current request limit is around 100 / minute. The error response you are getting back also contains a Retry-After header which contains the number of seconds you must wait until you can make a request again.
We may also be institute a daily limit at some point in the future -- we think 100 / minute is a reasonable burst rate, but not a reasonable sustained rate throughout the day. However, we are not enforcing a daily limit yet.