I'm trying to scrape all comments of an YouTube video. The comment API is paginated and limited to 100 items per page.
https://developers.google.com/youtube/v3/docs/commentThreads/list?apix_params=%7B%22part%22%3A%5B%22snippet%2Creplies%22%5D%2C%22videoId%22%3A%22_VB39Jo8mAQ%22%7D.
If the page has the attribute nextPageToken, there are more comments on that page. If not, you have reached the last page. I can confirm this with this request to a video that conveniently has 97 comments:
https://youtube.googleapis.com/youtube/v3/commentThreads?part=snippet%2Creplies&videoId=WZePccmYHQ8&key=*********************&maxResults=100
So what I'm trying is: I'm requesting the comments with ...&maxResults=90, this gives me a nextPageToken:
Now I take the nextPageToken, and add it as a pageToken to another request:
https://youtube.googleapis.com/youtube/v3/commentThreads?part=snippet%2Creplies&videoId=WZePccmYHQ8&key=*********************&maxResults=90&PageToken=QURTSl9pM3JDSTJYZzVjZFlDRUhJXzh3aHlpamdjN25ycTl5UkpBTU9GQVRRMFllVU5EbWVyU3Zwa0hRdWxVaGVqVWhlWUxqaUVCelh5Yw==
I would expect to see:
"pageInfo": {
"totalResults": 7,
"resultsPerPage": 90
},
But I'm getting:
Can anyone tell me why? I am very new to this, as you can tell.
Related
I want a way to get YouTube shorts for a specific channel from YouTube API. I looked every where and I couldn't find anything.
Currently I can get a playlist ID for all channel videos with this endpoint:
request = youtube.channels().list(
part="contentDetails",
id=id
)
I also tried these parameters:
request = youtube.channels().list(
part="snippet,contentDetails,statistics,brandingSettings",
id=id
)
So is there a way to get YouTube shorts from a specific channel from YouTube API or any other source if it's available.
One way of detecting if a YouTube video ID is a Short without even using the API is to try a HEAD HTTP request to the /shorts/ version of the URL and see if it redirects you.
https://www.youtube.com/shorts/hKwrn5-7FjQ is a Short and if you visit that URL, you'll get an HTTP status code of 200 and the URL won't change.
https://www.youtube.com/watch?v=B-s71n0dHUk is not a Short, and if you visit https://www.youtube.com/shorts/B-s71n0dHUk, you get a 303 redirect back to the original URL.
Keep in mind, that this behavior might change down the line, but it works as of May 2022.
It seems that once again YouTube Data API v3 doesn't provide a basic feature.
For checking if a given video is a short:
I would recommend you to use my open-source YouTube operational API. Indeed by requesting the JSON document https://yt.lemnoslife.com/videos?part=short&id=VIDEO_ID containing item["short"]["available"] boolean, your problem is solved.
Example of short id: ydPkyvWtmg4
For listing shorts of a channel:
I would recommend you to use my open-source YouTube operational API. Indeed by requesting the JSON document https://yt.lemnoslife.com/channels?part=shorts&id=CHANNEL_ID. The entry item["shorts"] contains the data you are looking for. Note that the pagination works as the one of YouTube Data API v3.
Example of result for channel UC5O114-PQNYkurlTg6hekZw:
{
"kind": "youtube#channelListResponse",
"etag": "NotImplemented",
"items": [
{
"kind": "youtube#channel",
"etag": "NotImplemented",
"id": "UC5O114-PQNYkurlTg6hekZw",
"shorts": [
{
"videoId": "fP8nKVauFwc",
"title": "India: United Nations Counter Terrorism Committee Watch LIVE #shorts",
"thumbnails": [
{
"url": "https:\/\/i.ytimg.com\/vi\/fP8nKVauFwc\/hq720_2.jpg?sqp=-oaymwEYCNAFENAFSFryq4qpAwoIARUAAIhC0AEB&rs=AOn4CLCgJEYgv_msT5pkfWeEEN3VBt4wjg",
"width": 720,
"height": 720
}
],
"viewCount": 3700
},
...
],
"nextPageToken": "4qmFsgLlARIYVUM1TzExNC1QUU5Za3VybFRnNmhla1p3GsgBOGdhU0FScVBBVktNQVFxSEFRcGZRME00VVVGU2IyWnZaMWxqUTJob1ZsRjZWbEJOVkVVd1RGWkNVbFJzYkhKa1dFcHpWa2RqTW1GSFZuSlhibU5SUVZOSlVrTm5PSGhQYWtVeVRtcGplVTE2VlRST2FrVXdUbXBCY1VSUmIweFhWRUl5VGtab1dGSllSbGRNVmtVU0pEWXpOakJoTkRVNUxUQXdNREF0TWpKaE15MDRObUV6TFdRMFpqVTBOMlZqWVRSbFl4Z0I=,CgtuNjFmZlJlR0QxcyiVgICbBg=="
}
]
}
Below is a sample python code to send the HEAD HTTP request.
import requests
def is_short(vid):
url = 'https://www.youtube.com/shorts/' + vid
ret = requests.head(url)
if ret.status_code == 200:
return True
else: # whether 303 or other values, it's not short
return False
I don't know why but I don't get status code 303 whether it's a short or not with axios. So this is another way of checking if it's a short or not.
const isShort = async (videoId) => {
const url = "https://www.youtube.com/shorts/" + videoId
const res = await axios.head(url)
console.log(res.request.res.responseUrl)
// if it's a short it ends with "/shorts/videoId"
// if it's NOT a short it ends "/watch?=videoId"
}
Maybe axios automatically redirects?
You can use the new dimension called 'creatorContentType' from Youtube Analytics and Reports API.
// You can get IDs from PlaylistItems or Search API
const IDs = ["videoID1", "videoID2", "videoID3"];
// Get the analytics data of selected videos based on their IDs
const { data: analyticsData } = await youtubeAnalytics.reports.query({
ids: "channel==MINE",
startDate: "2019-01-01",
// Today's date
endDate: new Date().toISOString().split("T")[0],
metrics: "views",
dimensions: "video,creatorContentType",
filters: `video==${IDs.join(",")}`,
access_token,
});
It basically returns values listed below:
Value
Description
LIVE_STREAM
The viewed content was a YouTube live stream.
SHORTS
The viewed content was a YouTube Short.
STORY
The viewed content was a YouTube Story.
VIDEO_ON_DEMAND
The viewed content was a YouTube video that does not fall under one of the other dimension values.
UNSPECIFIED
The content type of the viewed content is unknown.
Notes:
Don't forget it returns values just for the videos uploaded after 01.01.2019.
Don't forget to add analytics scopes and enable Analytics and Reports API.
We are using YouTube Data API to show product review videos on product pages of our price comparison website. We basically grab a product title and append "review" to it. The problem is that very often we do not get any results from the API, while a normal YouTube website search yields many results.
For example, the following request, when searching for "Reiskussen Intex Downy Opblaasbaar Blauw review" (using the Try this API form in the docs) does not give us any results:
https://www.googleapis.com/youtube/v3/search?part=snippet&q=Reiskussen%20Intex%20Downy%20Opblaasbaar%20Blauw%20review&key=[YOUR_API_KEY]
This is the JSON response:
{
"kind": "youtube#searchListResponse",
"etag": "z1qySaHHjn5TA8Vggil4LrmXKdc",
"regionCode": "NL",
"pageInfo": {
"totalResults": 0,
"resultsPerPage": 5
},
"items": []
}
When we search on Youtube, we get several (relevant) results:
Why does the YouTube website find results and our API call does not? Is there any way to have the API find the same results or any result at all?
Using the youtube API, I get no results for either channels list, search, ... for a channel where there is some content.
There is no error returned, just an empty set. No channels, no video, nothing either in the API explorer or through my code.
Everything was working correctly up until a few days/weeks (hard to tell when since I had no errors)
curl -v 'https://www.googleapis.com/youtube/v3/search?key=[api_key]&channelId=UCKTgGP9lrL5Yjs4f8WuI1Vw&part=snippet,id&order=date&maxResults=20'
is the query I used but I tried to get the uploads channel id as some answers here advised but the channels/list endpoint is empty too. I checked the documentation and I couldn't find anything wrong with my query.
This is the answer I currently get:
{
"kind": "youtube#searchListResponse",
"etag": "\"ksCrgYQhtFrXgbHAhi9Fo5t0C2I/nrqzXB-_ht29Bt6u-f7_lLyFcCw\"",
"regionCode": "BE",
"pageInfo": {
"totalResults": 0,
"resultsPerPage": 20
},
"items": []
}
and you can see here https://www.youtube.com/channel/UCKTgGP9lrL5Yjs4f8WuI1Vw/videos that it's not empty.
Try to add type=video as param, it seems working :)
I'm new to basically all forms of programming, so forgive me if there's an obvious answer, but I was just trying to make a random video chooser by randomly adding characters to the Youtube API's search function under list, but when I run the API with either just id or id and snippet on, it shows this:
{
"kind": "youtube#searchListResponse",
"etag": "\"i don't know if I can post my etag"",
"regionCode": "CA",
"pageInfo": {
"totalResults": 1,
"resultsPerPage": 50
},
"items": [
]
}
.
Its hard to know exactly what you have searched on but by the looks of the result there is just no results for your request. YOu might want to try playing with the api here search.list
When you run it click Execute without OAuth not the authorize and execute button. It will return a random list of videos.
You can use the q parameter to do a text search on videos for example that like toys.
I'm using the playlistItems endpoint to get a list of videos that have been uploaded by an account. The response shows 1 as totalResults, which is expected, but the items array is empty, which is not. It's not clear to me what's going on, since the video is public and has views on it.
I'm not seeing this issue pulling uploads from other accounts, so I'm wondering if it's a caching issue on the YT side, or some setting we need to enable on our account? I'm not able to find any documentation that would explain what's happening here.
The endpoint I'm hitting is https://www.googleapis.com/youtube/v3/playlistItems?key=<api key>&playlistId=UUjQYaIbYceXNNdC63dQ9aTg&maxResults=10&part=snippet, and the full response I'm receiving is:
{
"kind": "youtube#playlistItemListResponse",
"etag": "\"PSjn-HSKiX6orvNhGZvglLI2lvk/eJKYM5x3WquVgzqxNc7NMRuOS4o\"",
"pageInfo": {
"totalResults": 1,
"resultsPerPage": 10
},
"items": []
}
The item is now showing up, so it looks to just take some time before the API reflects what's been uploaded.