I am trying to fetch all the subscriptions ids of a youtube channel that has 100k+ subscribers. When fetching the first page of results, youtube returns properly the total amount of subscriptions and the next page token.
After a few hundred calls (because you can only fetch 50 results per call), the api doesn't provide anymore the nextPageToken, and the listing stops with only ~20k subscriptions listed.
I tried this on several big youtube channels, and always the same behaviour when I reach around 20k subscriptions listed.
In the documentation I couldn't find anything about any limit on listing subscriptions...
Anybody encountered the same issue ? :-)
Thanks
I think you need to check for an error, If you have reached the quota limit you should get an error when you try and make another request.
Daily quota is 50,000,000 units/day
depending upon which part you request from subscriptions.list some of them like snippet count double against the quota.
Math time
If you have 100000 subscriptions and you have to fetch them in 50 subscription bites that's going to take you 2000 requests. Even if you are using one of the double parts its still should only be around 4000 requests.
I don't think the problem is quota check to see if you are getting an error.
Googling found
issue request might be related. Youtube api impossible to get all results
YouTube api page tokens
possible hack
Related
I have reached the Daily Quota Limit, and have submitted the Quota Increase Form.
After seeing the confirmation notice of my submission, I have not heard or received an email from them.
Is there any other solution to this issue? How long does it usually take for them to get back?
With things considered, we may have to increase the daily quota up to 100,000.
Is there a way to collect multiple data from a single quota?
My website mainly involves collecting view counts of videos through video IDs.
I have submitted the YouTube API Services - Audit and Quota Extension Form.
Thank you in advance
The time to get the quota increase varies greatly. It kind of depends on how back logged the team is.
In the beginning when they reduced it to 10k and I applied for mine it took more then three months.
These days I think you should get something in less then two weeks but don't hold me to that I dont work for YouTube this is just my experience.
Oh and just check it now and then they may apply it before the actually send you an email saying that they are going to apply it.
I have a very simple message and getting the v3 youtube data api to get the list of comments. I am just fetching the list of videos and then fetching the comments (at frequency of 5 sec) to get updated messages. using the page token as needed to minimize the load and computaion.
Today after some time while internally testing the application i started getting the quota exceeded exception. I know the youtube provided by default 10000 units and since reading the comments (and videos as well) is just 1 unit, i should expect to get similar numbers.
However, the data is wrongly calculated.
Following are request details
If you see, there are 2895 total requests LiveChatMessages-> List.
However, when i go to IAM-> Quotas, it showed 14k earlier, then 12.6k in quota usage
There seems to be some problem either with the computation or with the Documentation that defines the units for queries. Can someone help please..
PS: Just using the two apis as mentioned above in screenshot. Both are list.
If you see, there are 2895 total requests LiveChatMessages-> List. However, when i go to IAM-> Quotas, it showed 14k earlier, then 12.6k in quota usage
Yes i can see that there are 2895 requests, but how do you know what the qutoa costs are for those requests. You are using the YouTube Live Streaming api for those requests. Not the YouTube-Data-api
There is no documentation of the quota cost for the YouTube Live Streaming api calls. If Google says you used all your quota then you probably have.
I would post an issue over on the issue forum asking them to document the quota cost for the calls Issue forum
My application allows users to "link" their YouTube accounts to our system and then we allow them to upload videos to their channels both automatically in some cases, and on an "upload" action on their part. We have hundreds of videos being uploaded because we have thousands of users that use our system.
Today I saw hundreds of errors in my application when our batch automatic upload job was running. The errors were for:
quotaExceeded, video upload limit reached.
My API quotas are very high (50,000,000 per day, 3,000,000 per 100 seconds, 300,000 per 100 seconds per user), so if a video is 1600 points, I have enough limit to upload 30,000+ per day, 1,800 videos per 100 seconds, and 180+ per 100 seconds per user.
I have seen other questions out there hinting at some newly enforced limit by Google to just 50 videos and then 1 video every 15 minutes? This is a very low limit for my application which has such a large quota! To be clear, at most I have seen my application handle 1,000 videos in a single day (1,600,000 points of my total 50,000,000 quota).
Is there any way I can get this artificial limit of 50 videos and then 1 per 15 minutes removed? This is a major block to my users' functionality for a very popular web app. I could understand if it were 50 videos per user and then 1 every 15 minutes, but I highly doubt my errors this morning were from one user trying to upload 300+ videos at a time. My system only uploads their "newest" videos each day, which most people only have 1-10 videos at max. Hundreds would be a very rare edge case.
You can actually confirm in your Developers Console the available quota for your application. By default, YouTube API have a quota allocation of 1 million units per day as mentioned in Quota usage. If you see that your usage reached your quota limit, you can request additional quota on the Quotas tab.
Note also that, all API requests, including invalid requests, incur a quota cost of at least one point. You may use the Quota Calculator to get an estimate of the quota cost for an API query.
On the other hand, to work efficiently with your quota and if you haven't done so, I suggest that you implement exponential backoff if you're encountering high error ratio. See this sample code which shows an exponential backoff strategy to resume a failed upload. Also, if applicable, subscribe to Push Notifications which is much more efficient than polling-based solutions.
Check the documentation for more information on how PubSubHubbub callback server receives Atom feed notifications when a channel does any of the following activities:
uploads a video
updates a video's title
updates a video's description
Hope that helps!
For V2 of the YouTube Data API what are the exact limits for the quota?
I am aware that this is a frequent question, however I am yet to find any concrete answers.
Reason for Question:
I am going to querying a large pool of videos for their comments on a regular basis and would like to know when I am coming close to my quota limit, so the system can slow down. In V3 of the YouTube API, the quota limits are clearly documented. However I'm unable to use V3 of the API as it does not support the retrieval of comments (sidenote - does anyone know why?)
In v2 of the data API, the quota was not a fixed number per day as it is in v3, but instead was a limit that prevented too many requests within a short period of time. Unfortunately, I don't believe that there exists anywhere some firm documentation as to how many requests that would be or what the short period of time would be, either; generally, Youtube has always stated that if you get a quota error while making a call to v2 of the data API you should wait "a few minutes" before trying again. Here's the only official statement.
https://developers.google.com/youtube/2.0/developers_guide_protocol_error_responses?hl=en#Quota_errors
It is possible that one of the reasons for this lack of direct documentation is that there isn't a hard and fast number, but it changes in response to the current load.
In answer to your side question, there haven't been any official statements from the YouTube team about why comment retrieval hasn't yet been implemented, but it likely will be in time (as will other pieces of data retrievable via v2 but not yet via v3).
I am storing in a database, every 30 minutes, Twitter's trending topics of a country Y. No problem with that.
Now, I want to get as much tweets as possible matching those trending topics for research purposes.
Since I would like to study the patterns of the trends, I would like continuous tweet data of at least 3 days centered in the day the trend peak was detected, for every trending topic. In order to achieve that, I thought of doing the following:
Suppose I am in day X. I could retrieve the unique trends of day X-2, and for every trend, look for tweets matching the trend in the interval [X-3, X-1], that is 3 days. However, the problem here is Twitter rate limitations. If I have 100 trending topics in day X-2, and I make 20 GET search requests/trend, I would end up doing a total of 2,000 requests, which overpasses Twitter's 350 hourly rate limit. If make 300 req/hour, it would take more than 6 hours to get the data for only one day...
Does anybody know any other (better) way for getting tweets associated with trends?
Thanks in advance
Twitter Streaming API?
Twitter Streaming API doesn't deliver any past tweets. You only receive tweets starting from the time the server connection is established. The search API will return tweets matching the current query up to 7 days old in theory, but that is entirely up to Twitter’s current load. (Note*-At times this interval has been as short as 24 hours. In addition, you are limited by the ability to only receive up to 1,500 tweets regardless of how old they are.)
Is there any way to get more tweets from the streaming?
None that I know. But, do refer the below mentioned information if you are considering to switch among search or streaming API.
Please choose your case:
If you need real time data and your number of requests are high:
Go for Streaming API
The streaming API requires that you keep the connection active. This requires a server process with an infinite loop, to get the latest tweets.
Advantage
1)Lag in retrieving results: Tweets delivered with this method are basically real-time, with a lag of a second or two at most between the time the tweet is posted and it is received from the API
2)Not rate limited.
If you need aggregate data regardless of its time range and your number of requests are not high:
Go for Search API
The search API is the easier of the two methods to implement but it is rate limited .Each request will return up to 100 tweets, and you can use a page parameter to request up to 15 pages, giving you a theoretical maximum of 1,500 tweets for a single query.
Advantage
1)Finding tweets in the past:The search API wins by default in this area, because the streaming API doesn’t deliver any past tweets
2)Easier to implement