I'm working on a website to load multiple youtube channels live streams. At first i was trying to figure out a way to do this without utilizing youtube's api but have decided to give in.
To find whether a channel is live streaming and to get the live stream links I've been using:
https://www.googleapis.com/youtube/v3/search?part=snippet&channelId={CHANNEL_ID}&eventType=live&maxResults=10&type=video&key={API_KEY}
However with the minimum quota being 10000 and each search being worth 100, Im only able to do about 100 searches before I exceed my quota limit which doesn't help at all. I ended up exceeding the quota limit in about 10 minutes. :(
Does anyone know of a better way to figure out if a channel is currently live streaming and what the live stream links are, using as minimal quota points as possible?
I want to reload youtube data for each user every 3 minutes, save it into a database, and display the information using my own api to save server resources as well as quota points.
Hopefully someone has a good solution to this problem!
If nothing can be done about links just determining if the user is live without using 100 quota points each time would be a big help.
Since the question only specified that Search API quotas should not be used in finding out if the channel is streaming, I thought I would share a sort of work-around method. It might require a bit more work than a simple API call, but it reduces API quota use to practically nothing:
I used a simple Perl GET request to retrieve a Youtube channel's main page. Several unique elements are found in the HTML of a channel page that is streaming live:
The number of live viewers tag, e.g. <li>753 watching</li>. The LIVE NOW
badge tag: <span class="yt-badge yt-badge-live" >Live now</span>.
To ascertain whether a channel is currently streaming live requires a simple match to see if the unique HTML tag is contained in the GET request results. Something like: if ($get_results =~ /$unique_html/) (Perl). Then, an API call can be made only to a channel ID that is actually streaming, in order to obtain the video ID of the stream.
The advantage of this is that you already know the channel is streaming, instead of using thousands of quota points to find out. My test script successfully identifies whether a channel is streaming, by looking in the HTML code for: <span class="yt-badge yt-badge-live" > (note the weird extra spaces in the code from Youtube).
I don't know what language OP is using, or I would help with a basic GET request in that language. I used Perl, and included browser headers, User Agent and cookies, to look like a normal computer visit.
Youtube's robots.txt doesn't seem to forbid crawling a channel's main page, only the community page of a channel.
Let me know what you think about the pros and cons of this method, and please comment with what might be improved rather than disliking if you find a flaw. Thanks, happy coding!
2020 UPDATE
The yt-badge-live seems to have been deprecated, it no longer reliably shows whether the channel is streaming. Instead, I now check the HTML for this string:
{"text":" watching"}
If I get a match, it means the page is streaming. (Non-streaming channels don't contain this string.) Again, note the weird extra whitespace. I also escape all the quotation marks since I'm using Perl.
Here are my two suggestions:
Check my answer where I explain how you can check how retrieve videos from channels who are livesrteaming.
Another option could be use the following URL and somehow make request(s) each time for check if there's a livestreaming.
https://www.youtube.com/channel/<CHANNEL_ID>/live
Where CHANNEL_ID is the channel id you want check if that channel is livestreaming1.
1 Just notice that maybe the URL wont work in all channels (and that depends of the channel itself).
For example, if you check the channel_id UC7_YxT-KID8kRbqZo7MyscQ - link to this channel livestreaming - https://www.youtube.com/channel/UC4nprx9Vd84-ly7N-1Ce6Og/live, this channel will show if he is livestreaming, but, with his channel id UC4nprx9Vd84-ly7N-1Ce6Og - link to this channel livestreaming -, it will show his main page instead.
Adding to the answer by Bman70, I tried eliminating the need of making a costly search request after knowing that the channel is streaming live. I did this using two indicators in the HTML response from channels page who are streaming live.
function findLiveStreamVideoId(channelId, cb){
$.ajax({
url: 'https://www.youtube.com/channel/'+channelId,
type: "GET",
headers: {
'Access-Control-Allow-Origin': '*',
'Accept-Language': 'en-US, en;q=0.5'
}}).done(function(resp) {
//one method to find live video
let n = resp.search(/\{"videoId[\sA-Za-z0-9:"\{\}\]\[,\-_]+BADGE_STYLE_TYPE_LIVE_NOW/i);
//If found
if(n>=0){
let videoId = resp.slice(n+1, resp.indexOf("}",n)-1).split("\":\"")[1]
return cb(videoId);
}
//If not found, then try another method to find live video
n = resp.search(/https:\/\/i.ytimg.com\/vi\/[A-Za-z0-9\-_]+\/hqdefault_live.jpg/i);
if (n >= 0){
let videoId = resp.slice(n,resp.indexOf(".jpg",n)-1).split("/")[4]
return cb(videoId);
}
//No streams found
return cb(null, "No live streams found");
}).fail(function() {
return cb(null, "CORS Request blocked");
});
}
However, there's a tradeoff. This method confuses a recently ended stream with currently live streams. A workaround for this issue is to get status of the videoId returned from Youtube API (costs a single unit from your quota).
I found youtube API to be very restrictive given the cost of search operation. Apparently the accepted answer did not work for me as I found the string on non live streams as well. Web scraping with aiohttp and beautifulsoup was not an option since the better indicators required javascript support. Hence I turned to selenium. I looked for the css selector
#info-text
and then search for the string Started streaming or with watching now in it.
To reduce load on my tiny server that would have otherwise required lot more resources, I moved this test of functionality to a heroku dyno with a small flask app.
# import flask dependencies
import os
from flask import Flask, request, make_response, jsonify
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
base = "https://www.youtube.com/watch?v={0}"
delay = 3
# initialize the flask app
app = Flask(__name__)
# default route
#app.route("/")
def index():
return "Hello World!"
# create a route for webhook
#app.route("/islive", methods=["GET", "POST"])
def is_live():
chrome_options = Options()
chrome_options.binary_location = os.environ.get('GOOGLE_CHROME_BIN')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--remote-debugging-port=9222')
driver = webdriver.Chrome(executable_path=os.environ.get('CHROMEDRIVER_PATH'), chrome_options=chrome_options)
url = request.args.get("url")
if "youtube.com" in url:
video_id = url.split("?v=")[-1]
else:
video_id = url
url = base.format(url)
print(url)
response = { "url": url, "is_live": False, "ok": False, "video_id": video_id }
driver.get(url)
try:
element = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#info-text")))
result = element.text.lower().find("Started streaming".lower())
if result != -1:
response["is_live"] = True
else:
result = element.text.lower().find("watching now".lower())
if result != -1:
response["is_live"] = True
response["ok"] = True
return jsonify(response)
except Exception as e:
print(e)
return jsonify(response)
finally:
driver.close()
# run the app
if __name__ == "__main__":
app.run()
You'll however need to add the following buildpacks in settings
https://github.com/heroku/heroku-buildpack-google-chrome
https://github.com/heroku/heroku-buildpack-chromedriver
https://github.com/heroku/heroku-buildpack-python
Set the following Config Vars in settings
CHROMEDRIVER_PATH=/app/.chromedriver/bin/chromedriver
GOOGLE_CHROME_BIN=/app/.apt/usr/bin/google-chrome
You can find supported python runtime here but anything below python 3.9 should be good since selenium had problems with improper use of is operator
I hope youtube will provide better alternatives than workarounds.
I know this is a old thread, but i thought i share my way of checking to for example grab the status code to use in an app.
This is for a single Channel, but you could easly do a foreach with it.
<?php
#####
$ytchannelID = "UCd0BTXriKLvOs1ANx3puZ3Q";
#####
$ytliveurl = "https://www.youtube.com/channel/".$ytchannelID."/live";
$ytchannelLIVE = '{"text":" watching now"}';
$contents = file_get_contents($ytliveurl);
if ( strpos($contents, $ytchannelLIVE) !== false ){http_response_code(200);} else {http_response_code(201);}
unset($ytliveurl);
?>
Adding onto the other answers here, I use a GET request to https://www.youtube.com/c/<CHANNEL_NAME>/live and then search for "isLive":true (rather than {"text":" watching"})
In an answer to this 2014 post
Unable to retrieve members of a google group, getting Invalid Input
you read: "There is no API to manage consumer googlegroups.com groups programatically".
Is this still the situation in 2018?
I tried to follow the suggestion in answer 3 of the post How to get the list of members in a Google group in Google app script (Admin SDK)? but I get the following error message:
ReferenceError: "AdminDirectory" is not defined. (line 9, file "Code")
where line 9 (and following) is (are):
page = AdminDirectory.Members.list(groupKey,
{
domainName: 'googlegroups.#com',
maxResults: 500,
pageToken: pageToken,
});
Searching to understand the error I found the reference page for Members: list. Using the "Try this API" form in that page I get the error reported in the first post I mentioned.
If it is NOT true that "There is no API to manage consumer googlegroups.com groups programatically", is there a guide to copy the list of the members of a group I own in a google-sheet sheet? (I mean to copy via a function, non by hand exporting and reimporting the CSV)
Many thanks, Roberto Scotti
It's 2021 and I still can't find any evidence there's an API for #googlegroups.com groups, sadly.
I want to share files from MS OneDrive to a user via MS graph API. And user can view my shared file directly through the link. I have read the Document of Creating a sharing Link for a DriveItem and use this API to create a sharing link for my sharing files.
I wonder how to implement with MS graph API? Any suggestion and tips are welcome. Thanks
According to your description, I assume you want to get the share file by using MS Graph API.
Base on my test, We can create a shareLink for this this file.
Then we can use the following steps to get the file information by converting the shareLink.
Encoding the shareLink by using the following logic:
1)First, use base64 encode the URL.
2)Convert the base64 encoded result to unpadded base64url format by removing = characters from the end of the value, replacing / with _ and + with -.)
3)Append u! to be beginning of the string.
If you want access the shared files, you can use the following API:
GET /shares/{shareIdOrUrl}/driveItem
The shareIdOrUrl parameter is the result in step1.
This API will return all the information about the shared file.
As an example, to encode a URL in C#:
string sharingUrl = "https://onedrive.live.com/redir?resid=1231244193912!12&authKey=1201919!12921!1";
string base64Value = System.Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(sharingUrl));
string encodedUrl = "u!" + base64Value.TrimEnd('=').Replace('/','_').Replace('+','-');
For more detail, we can refer to this document.
I'm working to download a file via the Google Drive API using the gem google-api-client.
x = Google::Apis::DriveV2
drive = x::DriveService.new
drive.authorization = auth
files = drive.list_files
files.items.each_with_index do |file, index|
url_to_index = file.export_links.select { |k, v| v if k == 'text/plain' }
file_content = open(url_to_index["text/plain"]).read
end
The problem is file_content is returning the Google login screen not the file in text/plain format. It appears that when my Rails app opens the URL it does not have access to the text file.
What's the right way to enable my Rails app to grab the file in the text format?
Stated in Download Files
Depending on the type of download you'd like to perform — a file, a Google Document, or a content link — you'll use one of the following URLs:
Download a file — files.get with alt=media file resource
Download and export a Google Doc — files.export
Link a user to a file — webContentLink from the file resource
Downloading the file requires the user to have at least read access. Additionally, your app must be authorized with a scope that allows reading of file content.
You may go through the documentation for more information and examples.
See the log output: Two factor autentifications need to you permit access from you app to your google drive account.
In log you will see needed info to do this: link and secret_key.
tail -f log/*.log
I just realize that mimetype for download is different from the file meta.
Please check this url for available mimetype : https://developers.google.com/drive/api/v3/integrate-open
for example mimeType : 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
'
I created a configuration file (Simple Text File) on my Google Drive and now I would like to read it from my Chrome Packaged Dart Application. But I'm not able to get more information of the file than it's name, size etc.
For accessing Google Drive I use the google_drive_v2_api.
Any suggestion on how to get the contents of my configuration file would be great! Thanks!
I just did some test in my own chrome app, uploading and downloading a simple file:
chrome.identity.getAuthToken(new chrome.TokenDetails(interactive: true ))
.then((token){
OAuth2 auth = new SimpleOAuth2(token);
var drive = new gdrive.Drive(auth)..makeAuthRequests=true;
drive.files.insert({},content:window.btoa('hello drive!')).then((sentMeta){
print("File sent! Now retrieving...");
drive.files.get(sentMeta.id).then((repliedMeta){
HttpRequest request = new HttpRequest()..open('GET', repliedMeta.downloadUrl)
..onLoad.listen((r)=>print('here is the result:'+r.target.responseText));
auth.authenticate(request).then((oAuthReq)=>oAuthReq.send());
});
});
});
It works, but the HttpRequest to get content back seems heavy...
But i really recommend you to a take look to chrome.storage.sync if your config file size is < to 4ko... If not, you could also use the chrome SyncFileSystem API... They are both easier to use, and SyncFileSystem use Drive as backend.
This page on downloading files talks through the process for getting the contents of a file.