Is there a way to get data (especially subscriptions.list and superChatEvents.list) in a more realtime manner?
What we do currently: we send request to endpoints every X seconds, and filter received data to get only the latest. We have X seconds delay, which isn't even close to realtime. Shortening X costs us too much quota, because we need to support a lot of users (lets say 50 000)
What we would like to do is:
send only ore request which gets data from more than one youtube channel (not something that YT API supports)
or
have WebSocket connection (Push notifications doesn't support what we're interested in)
or
?
There is a service online doing that (ex. you connect YT channel with them, channel gets subscription, and they know it immediately, and they support far more than 50 000 users), but they aren't open-source. All I managed to peep out is an enigmatic xhr request to https://www.youtube.com/youtubei/v1/log_interaction?alt=json&key=[devKey] but google doesn't help when I'm searching for details about that, like it wouldn't exists.
Related
I am using https://developers.google.com/youtube/v3/guides/push_notifications to continually monitor ~ 2,000 channels for new videos etc.
Up until last week, I wasn't having any problems, with pubnubsubbub, however, for some reason, I am getting 403 when I am trying to access (via curl or requests) https://pubsubhubbub.appspot.com/subscribe saying nothing more than
<p>
<b>403.</b> <ins>That’s an error.</ins>
<p>Your client does not have permission to get URL <code>/subscribe</code> from this server. <ins>That’s all we know.</ins>
</p>
So my questions are:
is there any limit (either IP or time or quota) that is documented somewhere and I am somehow hitting? As xml calls were "unlimited" and i do not see any difference in the official YouTube Data Api documentation
have somebody found a better pub/sub channel (either paid or not) with YouTube DataApi 3
if somebody had a similar situation, how long my "ban" will stay
As suggested by Marco Aurelio, I am adding my solution.
My VPS provider had some bans from Google on their IPv6 network in the data center where I had my server running. Thus by contacting them and fixing my network settings the pubsubhubbub continued working without issues.
So the tip for future debuggers stumbling upon here is trying to curl any other Google API (e.g. a function) that you are 100% sure it works and if you are still getting 403, then your IP address range is blocked for some reason (like data center ban in my case).
If you continue to have the problem and getting the "notification" with a maximum delay of 6 minutes isn't a problem, here is a workaround.
I don't use pubsubhubbub but you can simulate its advantages. Indeed you have 10 000 quota a day, using PlaylistItems: list with the uploads playlist of the channel found with contentDetails in part by using Channels: list.
In that way you can check for new videos for 50 YouTube channels for 1 quota so if you want to check your 2 000 channels you have to spend 40 of quota.
If you want to uniformly check during the day you can so make 250 global checks a day, so every 346 seconds (almost 6 minutes).
Of course you have to store your last videos found for a YouTube channel in order to check if there is any difference during your new global check.
If you just want a boolean response whether or not your current request response is different from the previous one, you can check the etag.
I have finished creating a bot and got blocked 1 minute after I finally got it to work. The account is now restricted.
this is what I get on my terminal if I try to run the bot now:
tweepy.error.TweepError: [{'message': 'Application cannot perform write actions. Contact Twitter Platform Operations through https://help.twitter.com/forms/platform.', 'code': 261}]
I understand Twitter may restrict new bots for several reasons. I have summited my report to try and get my account unrestricted.
What are good do and don'ts for bots?
In my original bot I was replying every 10s was this too much ? Should I had set a different time frame?
https://help.twitter.com/en/rules-and-policies/twitter-limits
Relevant parts:
Tweets: 2,400 per day. The daily update limit is further broken down into smaller limits for semi-hourly intervals. Retweets are counted as Tweets.
These limits include actions from all devices, including web, mobile, phone, API, etc. API requests from all third-party applications are tracked against the hourly API limit. People who use multiple third-party applications with their account will therefore reach the API limit more quickly.
What happens if I hit a limit?
If you do reach a limit, we'll let you know with an error message telling you which limit you've hit. For limits that are time-based (like the Direct Messages, Tweets, changes to account email, and API request limits), you'll be able to try again after the time limit has elapsed.
10 seconds is too much.
I'm providing management of live streaming to the International Sporting Competition. At the request of the client we're using YouTube as a delivery platform. I've built an interface so the events and stream keys can be easily managed outside of the Youtube API. Ideally I would like to insert all events, so end users can have a direct link to the YouTube event.
After inserting 30 events (inserting at a rate of 1 per second) I seem to have hit an undocumented limit. I get a 403 error "request not authorised". I can still use the credentials for other actions so I have not crossed the global quota for the user or the account.
Unfortunately the Youtube API increase form is broken so I cannot contact Youtube there and request a rate increase.
Looks like there's a limit of 50 liveBroadcast.inserts(). This may be per 24 hour period I will confirm tomorrow. The web gui confirms that it is denying my requests because I have made too many live events and to try again "later".
I'm building a thing that tells people when a YouTube stream goes live. In order to build this, I need to know when a stream goes live, be it via push notification, polling, or some other method.
With Twitch, it's dead easy. All you have to do is send a GET to https://api.twitch.tv/kraken/streams/[STREAMER_USERNAME_HERE] and you get a JSON object with everything you'd ever hope to know (try it in your browser, it's great). I have combed through the YouTube Streaming and Data API's and nothing even close to this feature seems to exist.
I know that something, somewhere is alerting something when a stream goes live, because my phone gets a notification when the people I am subscribed to start streaming. But if there are no public calls in the API, I'm left with no other option than scraping the HTML off a page every couple seconds, which is bad for both me and YouTube.
Further on this, YouTube's push notification API for regular videos famously doesn't work [1] [2] [3] [4], and every dev is left to either use the deprecated XML feeds (which update sometimes up to an hour slow) or scrape HTML off youtube.com pages (the only current solution for instant updates). So, even if there's something off the API that I missed, it's moot if it doesn't actually work.
That said, am I missing something? Is it possible to get a notification if a stream goes live? It's kind of a deal-breaker for YouTube vs Twitch if there isn't.
It is currently not possible to do what you are asking with the public YouTube API.
You can only retrieve information about your own liveStream or liveBroadcast objects, not anyone else's. Additionally, there is no public API to get a notification when another user goes live.
We switched over to YouTube from a previously developed internal video system, and right now I've got it going on public upload feeds for users. The problem comes when we first got only about a dozen out of 150 or so videos available from the primary user's account. Over time of the day that number rose to about 50, and its stayed there. I found one post on the YouTube API forums about delay, but no word on how long I can expect that.
Anyone have experience about this to tell me what to expect?
Expected latencies can vary, and are documented here (http://code.google.com/apis/youtube/2.0/reference.html) - ways you can minimize it:
1) Upload the video as public. If you upload the video as private first, this will leave the video out of the fast-track indexing
2) Make an authenticated request to the user feed, this will guarantee the freshest data that the API has.
ie. http://gdata.youtube.com/api/feeds/users/username/uploads (with a dev key and auth token for 'username') instead of http://gdata.youtube.com/api/feeds/videos?author=username