I've got a program that splits market research videos into 30 minute chunks and uploads them to YouTube, via the API. I know there are API limits, but they don't seem to be published. Is there a set limit to how frequently you can upload videos via the API?
What I'm doing now is uploading a video, and if it takes less than 5 minutes, the program sleeps until 5 minutes have passed. I realize I could just continuously upload them, and catch the "too many recent calls" API exception, but then I have to make the program wait 10 minutes to reset the quota. If there is a limit, I'd just like to be able to adjust the program around that.
The canonical source of information is this blog post. We recently covered this topic as part of a YouTube Developers Live broadcast, and there's a new embedded video at the bottom of that post in which I answer the question in a lot more detail.
To summarize a more specific answer from the video (which I recommend watching, anyway), waiting 10 minutes can often be overkill, but was a handy interval to give out as a best practice. You won't be penalized if you keep uploading as fast you can and then wait for fewer than 10 minutes once you encounter a quota error. An exponential backoff in which you start with waiting a minute and then work your way up to 2, 4, 8, 16, etc. is reasonable, too.
Related
I have reached the Daily Quota Limit, and have submitted the Quota Increase Form.
After seeing the confirmation notice of my submission, I have not heard or received an email from them.
Is there any other solution to this issue? How long does it usually take for them to get back?
With things considered, we may have to increase the daily quota up to 100,000.
Is there a way to collect multiple data from a single quota?
My website mainly involves collecting view counts of videos through video IDs.
I have submitted the YouTube API Services - Audit and Quota Extension Form.
Thank you in advance
The time to get the quota increase varies greatly. It kind of depends on how back logged the team is.
In the beginning when they reduced it to 10k and I applied for mine it took more then three months.
These days I think you should get something in less then two weeks but don't hold me to that I dont work for YouTube this is just my experience.
Oh and just check it now and then they may apply it before the actually send you an email saying that they are going to apply it.
I'm currently developing a chat bot for one specific YouTube channel, which can already fetch messages from the currently active livechat. However I noticed my quota usage shooting up, so I took the "liberty" to calculate my quota cost.
My API call currently looks like this https://www.googleapis.com/youtube/v3/liveChat/messages?liveChatId=some_livechat_id&part=snippet,authorDetails&pageToken=pageTokenIfProvided, which uses up 5 units. I checked this by running one API call and comparing the quota usage before and after (so apologies, if this is inaccurate). The response contains pollingIntervalMillis set to 5086 milliseconds. Currently, my bot adds that interval to the current datetime and schedules the next fetch at that time (using Celery), so it currently fetches messages at a rate of 4-6 seconds. I'm gonna take the liberty and always wait for 6 seconds.
Calculating my API quota would result in a usage of 72.000 units per day:
10 requests per minute * 60 minutes * 24 hours = 14.400 requests per day
14.400 requests * 5 units per request = 72.000 units per day
This means that if I used the pollingIntervalMillis as a guideline for how often to request, I'd easily reach the maximum quota of 10.000 units by running the bot for 3 hours and 20 minutes. In order to not use up the quota by just fetching chat messages, I would need to run 1 API call per minute (1,3889 approximately). This is very unfeasible for a chatbot, since this is only for fetching messages and not even sending any messages to the chat.
So my question is: Is there maybe a more efficient way to fetch chat messages which won't use up the quota so much? Or will I only get this resolved by applying for a quota extension? And if this is only resolved by a quota extension, how much would I need to ask for reliably? Around 100k units? Even more?
I am also asking myself how something like Streamlabs Chatbot (previously known as AnkhBot) accomplishes this without hitting the quota limit despite thousands of users using their API client, their quota must probably be in the millions or billions.
And another question would be how I'd actually fill out the form, if the bot is still in this "early" state of development?
You pretty much hit the nail on the head. Services like Streamlabs are owned by larger companies, in their case Logitech. They not only have the money to throw around for things like increasing their API quota, but they also have professional relationships with companies like Google to decrease their per unit cost.
As for efficiency, the API costs are easily found in the documentation, but for live chat as you've found, you're going to be hitting the API for 5 units per hit. The only way to improve your overall daily cost with your calls is to perform them less frequently. While once per minute is clearly excessively long, once every 15-18 seconds could reduce the overall cost of your API quota increase, while making the chat bot adequately responsive.
Of course that all depends on your desired usage of the data, but still a recommendation if you're implementing the bot still in the realm of hobbyist usage.
I recently began using the Youtube Data v3 API for a program that I'm writing which is purely for personal use. To give a brief summary of what it does, it checks the the live chat from my most recent (usually ongoing) livestream and performs actions based on certain keywords entered in chat (essentially commands for people to use from live chat). In order to do that, however, I have to constantly send requests to get a refreshed livechat. As it is now, it sends requests on 1 second intervals. I recently did a livestream to test out my program and it only took about 25 minutes for me to reach the daily quota limit of 10,000 units/day.
The request is:youtube.liveChatMessages().list(liveChatId=liveChatId,part="snippet")
It seems like every request I make costs 6 units, according to the math. I want to be able to host livestreams at lengths of up to 3 hours, which would require a significant quota increase. I'm aware that there is an option to fill out a form to request additional quota. However, it asks for business information such as a business name, business website, business mailing address, etc. Like I said before, I'm doing this for my own use only. I'm in no way part of a business, and just made my program as a personal project. Does anyone know if there's any way to apply for additional quota as an individual/hobbyist? If not, do you think just putting n/a in those fields would be acceptable? I did find another post where someone else had the exact same problem, but no one was able to give a helpful answer. Any advice would be greatly appreciated.
Unfortunately, and although only related, it seems as Google is for the money here. I also tried to do something similar myself (a very basic chat bot just reading the chat messages), and, although some other users on the net got some different results, they all have in common that, according to the doc how it should be done, all poll at this interval of about once a second (that's the timeout one get as part of the answer to a poll for new messages). I, along with a few others, got as most as about 5 minutes with polling once a second, some others, like you, got a few more minutes out of it. I changed the interval by hand in incrementing intervals of 5 seconds each: 5, 10, 15, etc... you get the picture. I can't remember on which value I finally tuned in, but I was only able to get about 2 1/2 hours worth with a rather long polling interval of just once every 10 seconds or so - still way enough for a simple chat bot just reading the chat. But also replying would had at least doubled the usage and hence halfed the time.
It's already a pain to get it working as an idividual as just setting up the required OAuth authentication requires one to at least provide basic information like providing a fixed callback and some legal and policy information. I always ended up in had it rejected with this standard reply "Your project seem to be for internal use only.". I even was able to got this G suite working (before it required payment) to set up an "internal" project (only possible if account belongs to a G suite organization account), but after I set up the OAuth login I got the error that my private account I wanted to use the bot on was not part of the organization and hence can't be used. TLDR: Just useless waste of time.
As far as I'm in for this for several months now there's just no way to get it done as a private individual for personal use. Yes, one can just set it up and have the required check rejected (as it uses the YouTube data API scopes), but one still stuck with that 10.000 units / day quota. Building your own powerful tool capable of doing more than just polling once every 10 to 30 seconds with just a minimum of interaction doesn't get you any further than just a few minuts, maybe one or two hours if you're lucky. If you want more you have to set up a business and pay for it - simple and short: Google wants you to pay for that service.
As Mixer got officially announced to be shut down on July 22nd you have exactly these two options:
Use one of the public available services like Streamlabs, Nightbot, etc ... They're backed by their respective "businesses" and by it don't seem to have those quota limits (although I just found some complaints on Streamlabs just from April - so about one month prior to when you posted this question where they admitted to had reached their limits - don't know if they already got it solved).
Don't use YouTube for streaming but rather Twitch - as Twitch doesn't have these limits and anybody is free to set up an API token either on the main account or on a second bot account (which is also explicitly explained in their docs). The downside of this are of course the objective sacrifices one has to suffer: a) viewers only have the quality of the streamer until one reaches at least affiliate b) caped at max 1080p60 with only 6.000kBit/s c) only short time of VOD storage
I myself wanted to use YouTube as my main platform (and currently do, but without my own stuff at the moment) and my own bot stuff and such as streaming on YouTube has some advantages over Twitch, but as YouTube wants me to pay what others (namely: Twitch) offer me for free (although overall not as good quality) it's an easy decision to make. Mixer looked promissing, as it also offered quite some neat features (overall better quality than Twitch, lower latency), but the requirements to get partner status were so high (2.000 followers along with another insane high number to reach) and Mixer itself just so little of a platform (I made the fun to count all the streamers and viewers - only a few hundred streamers with just a few 10.000s viewers the whole platform had less than some big Twitch channels on their own) - and now it's announced soon to be dead anyway.
Hope this may give you some input into what a small streamer has to consider and suffer from when chosing a platform - but after all what I experienced I have these information: Either do it like all the others: Stream on Twitch and use YouTube as an archive to export to from Twitch (although Twitch STILL doesn't have an auto-export of the latest VOD implemented - but I guess that could be done by some small script) - or if you want to stay on YouTube use some existing bot like Nightbot or any of the other services like Streamlabs.
If you get any other information on how to convince Google to increase the limit as an individual please let us know.
When I run the program using the youtube data API, sometimes I get a dailyLimitExceeded error and a quotaExceeded error. In the case of dailyLimitExceeded, if I run it again after a few seconds,
sometimes it works, sometimes it doesn't work. However, in the case of quotaExceeded, quota is not initialized until 5 o'clock of the next day in the Pacific. Only after the quota has been initialized can the program resume normal operation.
I understand quotaExceeded to some extent, but I don't know exactly what dailyLimitExceeded is.
It's because of the latency of the data refresh rate. if you sending a request to Youtube frequently. Youtube protects himself and gives us Daily Limit exceeded results temporarily, but a bit time later it will work properly with a new request.
Check out the report:
https://issuetracker.google.com/150106191
My application allows users to "link" their YouTube accounts to our system and then we allow them to upload videos to their channels both automatically in some cases, and on an "upload" action on their part. We have hundreds of videos being uploaded because we have thousands of users that use our system.
Today I saw hundreds of errors in my application when our batch automatic upload job was running. The errors were for:
quotaExceeded, video upload limit reached.
My API quotas are very high (50,000,000 per day, 3,000,000 per 100 seconds, 300,000 per 100 seconds per user), so if a video is 1600 points, I have enough limit to upload 30,000+ per day, 1,800 videos per 100 seconds, and 180+ per 100 seconds per user.
I have seen other questions out there hinting at some newly enforced limit by Google to just 50 videos and then 1 video every 15 minutes? This is a very low limit for my application which has such a large quota! To be clear, at most I have seen my application handle 1,000 videos in a single day (1,600,000 points of my total 50,000,000 quota).
Is there any way I can get this artificial limit of 50 videos and then 1 per 15 minutes removed? This is a major block to my users' functionality for a very popular web app. I could understand if it were 50 videos per user and then 1 every 15 minutes, but I highly doubt my errors this morning were from one user trying to upload 300+ videos at a time. My system only uploads their "newest" videos each day, which most people only have 1-10 videos at max. Hundreds would be a very rare edge case.
You can actually confirm in your Developers Console the available quota for your application. By default, YouTube API have a quota allocation of 1 million units per day as mentioned in Quota usage. If you see that your usage reached your quota limit, you can request additional quota on the Quotas tab.
Note also that, all API requests, including invalid requests, incur a quota cost of at least one point. You may use the Quota Calculator to get an estimate of the quota cost for an API query.
On the other hand, to work efficiently with your quota and if you haven't done so, I suggest that you implement exponential backoff if you're encountering high error ratio. See this sample code which shows an exponential backoff strategy to resume a failed upload. Also, if applicable, subscribe to Push Notifications which is much more efficient than polling-based solutions.
Check the documentation for more information on how PubSubHubbub callback server receives Atom feed notifications when a channel does any of the following activities:
uploads a video
updates a video's title
updates a video's description
Hope that helps!