The text here: https://developers.google.com/fusiontables/docs/v2/migration_guide implies that the 10MB limit not in effect for API v2, or that an alternative service "Media download" could be used for large responses.
The API Reference here: https://developers.google.com/fusiontables/docs/v2/reference/ does not have any information regarding the 10MB limit, or how you use "media download" to recieve your request.
How do I work around the 10MB limit for Fusion Tables API v2? I can't seem to find documentation that explains it.
To use media-download simply add the parameter alt=media to the URL
For those who use Google's API Client Libraries, the 'media download' is specified by using a specific method. For the Python library, there are two versions of the SQL query methods: sql*() and sql*_media() (and this is very likely true for the other client libraries as well).
Example usage:
# Build the googleapiclient service
FusionTables = build('fusiontables', 'v2', credentials=credentials);
query = 'select * from <table id>';
# "standard" query, returning fusiontables#sqlresponse JSON:
jsonRequest = FusionTables.query().sqlGet(sql = query);
jsonResponse = jsonRequest.execute();
# alt=media query, returning a CSV-formatted bytestring (in Python, at least):
bytestrRequest = FusionTables.query().sqlGet_media(sql = query);
byteResponse = bytestrRequest.execute();
As Kerry mentions here, media format queries that are too large to be sent as a GET request will fail (while regular format queries of the same length succeed provided the query result is less than 10 MB). In the python client, this failure appears as a HTTP 502: Bad Gateway error.
Also note that ROWIDs are currently not returned in the media response format.
Related
In our application we make calls to the Microsoft Graph API to fetch the email header data.
During the initial call to the api we received the following conversationId details for an email
{"conversationId":"AAQkADVhMzU1YWY5LWVkNGQtNGZjOC1hMjJmLTk0MzIxMGQzMzRhMQAQAA8kZ_w6rdxIsHkFk+h7byQ="}
And a few seconds later when we made a similar request for the same email, we received a different conversationId
{"conversationId":"AAQkADVhMzU1YWY5LWVkNGQtNGZjOC1hMjJmLTk0MzIxMGQzMzRhMQAQAA8kZ_w6rdxIsHkFk_h7byQ="}
Now the expectation here is that the value of the conversationId should always remain the same.
In the above scenario the only difference in the 2 conversationId returned is the '+' being replaced with the '_'
AAQkADVhMzU1YWY5LWVkNGQtNGZjOC1hMjJmLTk0MzIxMGQzMzRhMQAQAA8kZ_w6rdxIsHkFk+h7byQ=
AAQkADVhMzU1YWY5LWVkNGQtNGZjOC1hMjJmLTk0MzIxMGQzMzRhMQAQAA8kZ_w6rdxIsHkFk_h7byQ=
Detailed Steps:-
The owa mail dom is parsed to fetch the conversationId
AAQkADVhMzU1YWY5LWVkNGQtNGZjOC1hMjJmLTk0MzIxMGQzMzRhMQAQAA8kZ_w6rdxIsHkFk+h7byQ=
Using this conversationId we make a call to the MS Graph API and get the details including the messageId which we store in our DB as a primary key
A few minutes later, we make another MS Graph API call, this time using the messageId, and in response we get a different conversationId
AAQkADVhMzU1YWY5LWVkNGQtNGZjOC1hMjJmLTk0MzIxMGQzMzRhMQAQAA8kZ_w6rdxIsHkFk_h7byQ=
Question:-
Is it possible by any chance that + and _ are interchangable
You shouldn't assume Exchange identifiers are interchangeable across platforms. Over the years, Exchange has used a crazy number of different formats for identifiers. This is likely the crux of the issue here.
There are some mechanisms for converting between them but I generally advise against it unless you've got no other option.
I'm not sure why/how you're parsing the OWS DOM but I'm frankly more surprised the identifiers are that close than I am that it doesn't work as expected. If you want to pull the message from Graph from within OWA, I would use an Outlook Web Add-in for this. The add-in framework includes a convertToRestId method that returns an identifier you can use with Microsoft Graph:
function getItemRestId() {
if (Office.context.mailbox.diagnostics.hostName === 'OutlookIOS') {
// itemId is already REST-formatted
return Office.context.mailbox.item.itemId;
} else {
// Convert to an item ID for API v2.0
return Office.context.mailbox.convertToRestId(
Office.context.mailbox.item.itemId,
Office.MailboxEnums.RestVersion.v2_0
);
}
}
As for the + vs _, this comes down to the encoding format. There are multiple flavors of Base64. Standard In standard Base64, the 62nd character is + and the 63rd is /. This presents a problem when you want to use Base64 in a URL as + and / are reserved characters. To get around this, you need a URL safe variant of Base64. There are several such variants out there. The most common is base64url (defined in RFC 4648) which swaps out the - for the 62nd and _ for the 63rd.
I'm working on a website to load multiple youtube channels live streams. At first i was trying to figure out a way to do this without utilizing youtube's api but have decided to give in.
To find whether a channel is live streaming and to get the live stream links I've been using:
https://www.googleapis.com/youtube/v3/search?part=snippet&channelId={CHANNEL_ID}&eventType=live&maxResults=10&type=video&key={API_KEY}
However with the minimum quota being 10000 and each search being worth 100, Im only able to do about 100 searches before I exceed my quota limit which doesn't help at all. I ended up exceeding the quota limit in about 10 minutes. :(
Does anyone know of a better way to figure out if a channel is currently live streaming and what the live stream links are, using as minimal quota points as possible?
I want to reload youtube data for each user every 3 minutes, save it into a database, and display the information using my own api to save server resources as well as quota points.
Hopefully someone has a good solution to this problem!
If nothing can be done about links just determining if the user is live without using 100 quota points each time would be a big help.
Since the question only specified that Search API quotas should not be used in finding out if the channel is streaming, I thought I would share a sort of work-around method. It might require a bit more work than a simple API call, but it reduces API quota use to practically nothing:
I used a simple Perl GET request to retrieve a Youtube channel's main page. Several unique elements are found in the HTML of a channel page that is streaming live:
The number of live viewers tag, e.g. <li>753 watching</li>. The LIVE NOW
badge tag: <span class="yt-badge yt-badge-live" >Live now</span>.
To ascertain whether a channel is currently streaming live requires a simple match to see if the unique HTML tag is contained in the GET request results. Something like: if ($get_results =~ /$unique_html/) (Perl). Then, an API call can be made only to a channel ID that is actually streaming, in order to obtain the video ID of the stream.
The advantage of this is that you already know the channel is streaming, instead of using thousands of quota points to find out. My test script successfully identifies whether a channel is streaming, by looking in the HTML code for: <span class="yt-badge yt-badge-live" > (note the weird extra spaces in the code from Youtube).
I don't know what language OP is using, or I would help with a basic GET request in that language. I used Perl, and included browser headers, User Agent and cookies, to look like a normal computer visit.
Youtube's robots.txt doesn't seem to forbid crawling a channel's main page, only the community page of a channel.
Let me know what you think about the pros and cons of this method, and please comment with what might be improved rather than disliking if you find a flaw. Thanks, happy coding!
2020 UPDATE
The yt-badge-live seems to have been deprecated, it no longer reliably shows whether the channel is streaming. Instead, I now check the HTML for this string:
{"text":" watching"}
If I get a match, it means the page is streaming. (Non-streaming channels don't contain this string.) Again, note the weird extra whitespace. I also escape all the quotation marks since I'm using Perl.
Here are my two suggestions:
Check my answer where I explain how you can check how retrieve videos from channels who are livesrteaming.
Another option could be use the following URL and somehow make request(s) each time for check if there's a livestreaming.
https://www.youtube.com/channel/<CHANNEL_ID>/live
Where CHANNEL_ID is the channel id you want check if that channel is livestreaming1.
1 Just notice that maybe the URL wont work in all channels (and that depends of the channel itself).
For example, if you check the channel_id UC7_YxT-KID8kRbqZo7MyscQ - link to this channel livestreaming - https://www.youtube.com/channel/UC4nprx9Vd84-ly7N-1Ce6Og/live, this channel will show if he is livestreaming, but, with his channel id UC4nprx9Vd84-ly7N-1Ce6Og - link to this channel livestreaming -, it will show his main page instead.
Adding to the answer by Bman70, I tried eliminating the need of making a costly search request after knowing that the channel is streaming live. I did this using two indicators in the HTML response from channels page who are streaming live.
function findLiveStreamVideoId(channelId, cb){
$.ajax({
url: 'https://www.youtube.com/channel/'+channelId,
type: "GET",
headers: {
'Access-Control-Allow-Origin': '*',
'Accept-Language': 'en-US, en;q=0.5'
}}).done(function(resp) {
//one method to find live video
let n = resp.search(/\{"videoId[\sA-Za-z0-9:"\{\}\]\[,\-_]+BADGE_STYLE_TYPE_LIVE_NOW/i);
//If found
if(n>=0){
let videoId = resp.slice(n+1, resp.indexOf("}",n)-1).split("\":\"")[1]
return cb(videoId);
}
//If not found, then try another method to find live video
n = resp.search(/https:\/\/i.ytimg.com\/vi\/[A-Za-z0-9\-_]+\/hqdefault_live.jpg/i);
if (n >= 0){
let videoId = resp.slice(n,resp.indexOf(".jpg",n)-1).split("/")[4]
return cb(videoId);
}
//No streams found
return cb(null, "No live streams found");
}).fail(function() {
return cb(null, "CORS Request blocked");
});
}
However, there's a tradeoff. This method confuses a recently ended stream with currently live streams. A workaround for this issue is to get status of the videoId returned from Youtube API (costs a single unit from your quota).
I found youtube API to be very restrictive given the cost of search operation. Apparently the accepted answer did not work for me as I found the string on non live streams as well. Web scraping with aiohttp and beautifulsoup was not an option since the better indicators required javascript support. Hence I turned to selenium. I looked for the css selector
#info-text
and then search for the string Started streaming or with watching now in it.
To reduce load on my tiny server that would have otherwise required lot more resources, I moved this test of functionality to a heroku dyno with a small flask app.
# import flask dependencies
import os
from flask import Flask, request, make_response, jsonify
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
base = "https://www.youtube.com/watch?v={0}"
delay = 3
# initialize the flask app
app = Flask(__name__)
# default route
#app.route("/")
def index():
return "Hello World!"
# create a route for webhook
#app.route("/islive", methods=["GET", "POST"])
def is_live():
chrome_options = Options()
chrome_options.binary_location = os.environ.get('GOOGLE_CHROME_BIN')
chrome_options.add_argument('--disable-gpu')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--remote-debugging-port=9222')
driver = webdriver.Chrome(executable_path=os.environ.get('CHROMEDRIVER_PATH'), chrome_options=chrome_options)
url = request.args.get("url")
if "youtube.com" in url:
video_id = url.split("?v=")[-1]
else:
video_id = url
url = base.format(url)
print(url)
response = { "url": url, "is_live": False, "ok": False, "video_id": video_id }
driver.get(url)
try:
element = WebDriverWait(driver, delay).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#info-text")))
result = element.text.lower().find("Started streaming".lower())
if result != -1:
response["is_live"] = True
else:
result = element.text.lower().find("watching now".lower())
if result != -1:
response["is_live"] = True
response["ok"] = True
return jsonify(response)
except Exception as e:
print(e)
return jsonify(response)
finally:
driver.close()
# run the app
if __name__ == "__main__":
app.run()
You'll however need to add the following buildpacks in settings
https://github.com/heroku/heroku-buildpack-google-chrome
https://github.com/heroku/heroku-buildpack-chromedriver
https://github.com/heroku/heroku-buildpack-python
Set the following Config Vars in settings
CHROMEDRIVER_PATH=/app/.chromedriver/bin/chromedriver
GOOGLE_CHROME_BIN=/app/.apt/usr/bin/google-chrome
You can find supported python runtime here but anything below python 3.9 should be good since selenium had problems with improper use of is operator
I hope youtube will provide better alternatives than workarounds.
I know this is a old thread, but i thought i share my way of checking to for example grab the status code to use in an app.
This is for a single Channel, but you could easly do a foreach with it.
<?php
#####
$ytchannelID = "UCd0BTXriKLvOs1ANx3puZ3Q";
#####
$ytliveurl = "https://www.youtube.com/channel/".$ytchannelID."/live";
$ytchannelLIVE = '{"text":" watching now"}';
$contents = file_get_contents($ytliveurl);
if ( strpos($contents, $ytchannelLIVE) !== false ){http_response_code(200);} else {http_response_code(201);}
unset($ytliveurl);
?>
Adding onto the other answers here, I use a GET request to https://www.youtube.com/c/<CHANNEL_NAME>/live and then search for "isLive":true (rather than {"text":" watching"})
So this is sort of weird. For every 1 request sent from my website using our YouTube API key, the developer console shows 102 queries actually being made. Here is the query format (using Python) -
search_q = '<query-string-here>'
service = build('youtube', 'v3', developerKey='<api-key>')
results = service.search().list(
part='snippet',
channelId='<specific-channel-id-to-search-through>',
type='video',
q=search_q,
).execute()
My logs show only one request being sent using this but my query count on the quotas page increases by 102.
Is there something I'm doing wrong? Or is this a bug on Google's end?
You can use the Quota Calculator to approximate the quota costs your request is using. Sure enough the search API request quota is on 100 range:
I do have a hughe database where some data sets link to certain youtube videos. As we all know some youtube videos disappear after a while from youtube and this leads to my solution and my problem as well --> I'd like to check if the youtube video still exists by simply checking via JSON if there is data to retrieve from a video. If not than I'd simply delete that certain data set.
So the first part of my solution would be to go through each row of my data table and checking for each id if there is data to retrieve from youtube as seen in the following code:
$result = $db->query("SELECT id, link FROM songs");
while($row = $result->fetch_assoc())
{
$number = 1+$rown++;
$id = $row['id'];
$link = $row['link'];
$video_ID = $link;
$JSON = file_get_contents("https://gdata.youtube.com/feeds/api/videos/{$video_ID}?v=2&alt=json");
$JSON_Data = json_decode($JSON);
$views = $JSON_Data->{'entry'}->{'yt$statistics'}->{'viewCount'};
echo $number .' row<br />';
echo $link .' link<br />';
echo $views .' views<br /><br />';
}
This attempt works fine and outputs me the data I need. The only problem is, that it just gets me data from the first 150-190 rows and that's it. Now I am checking for a solution that checks each row for empty youtube data and this lead to two concrete questions I have:
1st) Might youtube be responsible for that due to a restriction in retrieving data from one single query?
2nd) Might this be a server issue of mine that stops queries after x-seconds (but I already expand the time limit by putting a line set_time_limit (10000000); into my php code but without success)?
Hope you can help, thanks in advance.
YouTube, naturally, enforces limits on how many requests you can make per period of time. Unfortunately, there are no clear guidelines on what those limits are ... for v2, the guidelines merely state:
The YouTube API enforces quotas to prevent problems associated with
irregular API usage. Specifically, a too_many_recent_calls error
indicates that the API servers have received too many calls from the
same caller in a short amount of time. If you receive this type of
error, then we recommend that you wait a few minutes and then try your
request again.
If time isn't an issue for you, you could slow down each query so that you only make 1 request per every 10-15 seconds or so. Alternatively, you'd probably have better luck batch processing. With this, you can make up to 50 requests at once (this counts as 50 requests against your overall request per day quota, but only as one against your per time quota). Batch processing with v2 of the API is a little involved, as you make a POST request to a batch endpoint first, and then based on those results you can send in the multiple requests. Here's the documentation:
https://developers.google.com/youtube/2.0/developers_guide_protocol?hl=en#Batch_processing
Batch processing is much easier with v3, as you just have the videoId parameter be a comma delimited list of the videos you want info on -- so in your case, you'd execute file_get_contents on a URL like this:
https://www.googleapis.com/youtube/v3/videos?part=id&id={comma-separated-list-of-IDs}&maxResults=50&key={YOUR_API_KEY}
Any video ID in your list that doesn't come back in the JSON response doesn't exist anymore. IF you do 50 at a time, wait for 15 seconds, do another 50, etc., that should give you better performance.
There exists a DocsClient.get_resource_by_id function to get the document entry for a single ID. Is there a similar way to obtain (in a single call) multiple document entries given multiple document IDs?
My application needs to efficiently download the content from multiple files for which I have the IDs. I need to get the document entries to access the appropriate download URL (I could manually construct the URLs, but this is discouraged in the API docs). It is also advantageous to have the document type and, in the case of spreadsheets, the document entry is required in order to access individual worksheets.
Overall I'm trying to reduce I/O waits, so if there's a way I can bundle the doc ID lookup, it will save me some I/O expense.
[Edit] Backporting AddQuery to gdata v2.0 (from Alain's solution):
client = DocsClient()
# ...
request_feed = gdata.data.BatchFeed()
request_entry = gdata.data.BatchEntry()
request_entry.batch_id = gdata.data.BatchId(text=resource_id)
request_entry.batch_operation = gdata.data.BATCH_QUERY
request_feed.add_batch_entry(entry=request_entry, batch_id_string=resource_id, operation_string=gdata.data.BATCH_QUERY)
batch_url = gdata.docs.client.RESOURCE_FEED_URI + '/batch'
rsp = client.batch(request_feed, batch_url)
rsp.entry is a collection of BatchEntry objects, which appear to refer to the correct resources, but which differ from the entries I'd normally get via client.get_resource_by_id().
My workaround is to convert gdata.data.BatchEntry objects into gdata.docs.data.Resource objects ilke thus:
entry = atom.core.parse(entry.to_string(), gdata.docs.data.Resource)
You can use a batch request to send multiple "GET" requests to the API using a single HTTP request.
Using the Python client library, you can use this code snippet to accomplish that:
def retrieve_resources(gd_client, ids):
"""Retrieve Documents List API Resources using a batch request.
Args:
gd_client: authorized gdata.docs.client.DocsClient instance.
ids: Collection of resource id to retrieve.
Returns:
ResourceFeed containing the retrieved resources.
"""
# Feed that holds the batch request entries.
request_feed = gdata.docs.data.ResourceFeed()
for resource_id in ids:
# Entry that holds the batch request.
request_entry = gdata.docs.data.Resource()
self_link = gdata.docs.client.RESOURCE_SELF_LINK_TEMPLATE % resource_id
request_entry.id = atom.data.Id(text=self_link)
# Add the request entry to the batch feed.
request_feed.AddQuery(entry=request_entry, batch_id_string=resource_id)
# Submit the batch request to the server.
batch_url = gdata.docs.client.RESOURCE_FEED_URI + '/batch'
response_feed = gd_client.Post(request_feed, batch_url)
# Check the batch request's status.
for entry in response_feed.entry:
print '%s: %s (%s)' % (entry.batch_id.text,
entry.batch_status.code,
entry.batch_status.reason)
return response_feed
Make sure to sync to the latest version of the project repository.