I'm writing a script that does stuff with the youtube advertisment videos that people have to watch before the actual videos. (These ad-vids at the start are just simple youtube videos from the brand's channel)
I've searched the whole source code and scripts, but I can't find the video id of those ads anywhere. It must be somewhere, but it seems to be hidden well.
Anybody got an idea where to look?
I did some research on your case.
The video id of the ad video is definitively not part of the initial source code, as you already figured out. Youtube makes an ajax-Request to the http://googleads.g.doubleclick.net/ API to get information about related ad videos.
If you take a look into the source code, you can see a lot of javascript that is related to the google ads part. By looking into the code, you can find URL routes to the API. See screenshot, it is just an excerpt:
But unfortunately you cannot simply copy the url and make a remote-call to it. By doing so, you get a 400 Bad Request response.
As i figured out, there are missing params, which are dynamically added by youtube's javascript.
If you compare the request, that is actually made by youtube, you can see that there are more params sent:
compared to the request, that is directly copied from source code:
The result of the working request looks like this:
I tried several ways to make the invalid API request work, but have not found a way. Debugging the javascript is not that easy, because its obfuscated and minified. But additionally the variables are scoped within the function. So anyways, you would not have access to them.
If you make a javascript breakpoint right before the XHR request, you can see the actual API request:
But it is within the Local scope, without access to it.
Later on, there is even a second request to the youtube data API to fetch information about the ad video
In my opinion, there are only two ways to get the video, both require javascript to run.
Look into the source code AFTER Javascript has run. Then you can get the id directly from your markup:
Hook onto the Ajax Requests and grab the data directly from your network traffic.
Related
I was surfing on youtube and I realized something.
When I hover mouse on a video, then the url will be changed.
Interestingly, this happens in some browsers.What's the matter? Why does string start with &? https://www.youtube.com/?&ab_channel=NASA
What is the benefit to change the URL?
Interestingly, this happens in some browsers.
Different browser different support, a what you see is what you get is a standard we all want and must write our scripts specific to each browser if a feature requires it. In this case the new feature may not be widely supported or their coding wasn't compliant enough to give you this exact result each browser.
What's the matter?
No problem here, the URL is a tiny-bit broken but won't impact site performance unless you happen to error out the server and crash the entire network.
Why does string start with ?& https://www.youtube.com/?&ab_channel=NASA
What is the benefit to change the URL?
A URL alone has no parameters passed to it, so youtube.com. When a parameter is passed through the site on its HTTPS request will check these and determine what it is you want. So the response will return NASA cause ab_channel included it.
Because ? has nothing after it like ?video=asd89sa982 it's treated as undefined and serves no value or importance.
YouTube can fix it if they desire with script adjustment.
the URLs works in a way that when the site has started or reloaded, It's going to check for any element that has a href which has a link that has either https://www.youtube.com/watch?v= or https://youtu.be/ basically a YouTube video and then save those links, when one of them i hovered over, It's tell which. This works fine but one downside that I'm currently facing is that new links that are added in after the site has already started or reloaded won't be counted and therefore when I hover them it won't show the links, I'm reffering to comments; for-example if I make a comment that has a link to a video, after i post that comments and hover over that link I posted, It won't show the link. I could make a function which reloads every like 5 seconds but this doesn't seem to be a good idea. Plus what I'm actually working on, realoding every time won't be good.
I am using Twitter API to post status updates programmatically.
However, when my status contains a link, Twitter automatically adds the preview for that link. Sometimes it's not the desired behaviour, however, I didn't find a property in the API to disable it.
I know we can remove the link preview when posting manually via the Twitter website, as described here Remove Preview Image when Posting a Link on Twitter, so I suspect that it's possible, but API never mentions it.
Also, I am using tweepy library for Python, but happy to resort to the raw POST if that solves my problem.
Any ideas what should I add to the request to prevent the link preview generation?
I actually tried to post to Twitter manually on the web, twice:
With the generated link preview
Explicitly removing the preview (clicking the cross button once the preview is generated)
While doing it, I observed the payload that the website sends to the Twitter servers (with Chrome Dev Tools).
And interestingly enough, the only difference between these two tweets was only one property: card_uri: tombstone://card. More specifically, this property was present in the tweet without the preview.
After that, I actually tried to go and add card_uri property to my status update request, and it actually did the job: the preview was not automatically rendered.
Conviniently, tweepy library has an optional card_uri argument in the update_status menthod.
To sum up, you need to set card_uri='tombstone://card' in your request to prevent Twitter from autogenerating the link previews.
Unfortunately, I didn't find any references to this in the API (or in Google at all), so I am not confident that this is a long-lasting solution, but that's at least something to start with.
I have an abstract question: How can I get comments (Not count of comments!) from Youtube via API fast? I mean in API side - I have a powerful server and fine code. But via API it's really painful long process.
I see only one way. There are steps:
Make API request for get first page of comments;
Save comments from step 1 and get nextPageToken from response;
Make API request with pageToken option from step 2;
So... Loop steps 2 and 3 while have a nextPageToken in response.
If we have 10-20-50 pages of comments it's painful... But if we have 1k+ pages of comments it's a HELL! And we have a limit for API requests... So for 2-3 popular videos we can spend the limit.
It looks like I can't find something :) It can't be so hard... Isn't it?
How do you get a comments from Youtube?
Well, the max comment retrieval limit is 100 each time right? So i guess you will have to go that way anyway. Have you tried any non-API solution? I found this but not sure if working better (or at all)?
Use CommentThreads.list to get all the comments in a certain video. The required parameters you need to provide are part, videoId and fields.
And to test this, go to Youtube API explorer Commenthreads.list and fill in the parameter for videoId (added the others for you). Then, click Authorize and Execute. All the comments of your video will be listed under textOriginal.
It's now up to you to implement this.
I test with browser with the link I get from "http://www.youtube.com/get_video_info?&video_id=xx" to see if it is be able to download.
After remove some parameters from original link, some works but some still don't.
I read some post here but most of are pretty old. A lot of change since then.
I wonder if there are somebody working on this recently.
The purpose I need this is because my youtube view program need a better quality video to display.
This is link doesn't work:
http://r17---sn-tt17rn7e.c.youtube.com/videoplayback?fexp=902529%2C932000%2C906383%2C902000%2C919512%2C929903%2C931202%2C900821%2C900823%2C931203%2C931401%2C908529%2C919373%2C930803%2C906836%2C920201%2C929602%2C930101%2C930603%2C900824%2C910223&ipbits=8&expire=1364854787&sver=3&mt=1364829200&newshard=yes&id=26c94a41dba396f5&key=yt1&upn=GrcnDUPfreQ&cp=U0hVSVhQUl9NUUNONV9QSlZIOm9BbnVkMTJzOXE5&sparams=algorithm%2Cburst%2Ccp%2Cfactor%2Cid%2Cip%2Cipbits%2Citag%2Csource%2Cupn%2Cexpire&ip=173.248.214.165&itag=34&ms=au&source=youtube&mv=m&signature=9593596F58B377FAA4C8F5A4516C7F53CE473340.507CA2EA250CEED2E2B2377FD70EE1A0478EE322&type=video/mp4; codecs="avc1.42001E, mp4a.40.2"&itag=18&
What wrong with it? I removed fallback_path and replace sig with signature.
The working link is for webm. This is for h264. Both have the same kind of parameters.
And this is working now.
http://r17---sn-tt17rn7e.c.youtube.com/videoplayback?fexp=909708%2C912514%2C930802%2C932400%2C916624%2C931009%2C932000%2C906383%2C902000%2C919512%2C929903%2C931202%2C900821%2C900823%2C931203%2C931401%2C908529%2C930807%2C919373%2C906836%2C920201%2C929602%2C930101%2C930603%2C900824%2C910223&ms=au&itag=44&mt=1364825784&ipbits=8&cp=U0hVSVhQT19NUUNONV9QSlNCOlp5ZGoyMXJ3emlq&ip=173.248.214.165&upn=ohH0s8EjPyo&newshard=yes&source=youtube&ratebypass=yes&mv=m&sparams=cp%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&id=26c94a41dba396f5&expire=1364851187&key=yt1&sver=3&signature=798EED35782B846D2B0EA190A17E837A0DBA18EA.AD56BCF6365AAD974C18F09F352F9422084C50AC&type=video/webm; codecs="vp8.0, vorbis"&quality=large,itag=35&
That's not a supported method of interacting with YouTube. You need to use one of the official YouTube Player mechanisms to display YouTube videos.
Please familiarize yourself with the YouTube API Terms of Service if you have any more questions.
I wish to be able to check for the latest videos (in near realtime or at most a couple of minutes out) for a set of users (up to 200 or so) in a single call to the YouTube API and then store the IDs of uploaded videos in my own database. The only solution I believe there is for this is the YouTube SUP API but I'm not entirely clear on how it works and was wondering if someone could please explain it. I have read the entire API documentation on it but am still not completely clear.
I was assuming that one can call the SUP URL (http://gdata.youtube.com/sup) and check if the user hash has had any activity recently and if they have, then do something with that. My issue is I don't understand how you interpret the activity from ["b305e88","afd4"] in the SUP feed and is there any way to specify a subset of users or must you search through the entire feed? It seems to take a fair few seconds to load the SUP feed.
On the SUP API page it also states that you can visit a URL such as https://gdata.youtube.com/feeds/api/users/bbc/events?v=2 to obtain the hash key for a user's feed, but as you can see if you try to visit it, the link appears to be broken. How else could I obtain the hash?
I'm currently wanting to do this in a Rails project while using the youtube_it gem but I don't believe this has support for it. Correct me if I'm wrong.
Edit
My mistake. The developer key is required to obtain the events of a user such as https://gdata.youtube.com/feeds/api/users/bbc/events?v=2&key=YOUR_DEVELOPER_KEY
Still no progress with the SUP method although I'm potentially considering using a channel and just automatically subscribing to each user. Every minute I will then poll for the list of new videos by the users.
I'd suggest using PubSubHubbub: http://apiblog.youtube.com/2010/10/pubsubhubbub-for-youtube-activities.html
A handler in your web application will automatically receive a POST whenever one of the feeds you're watching is updated, and the content of the POST will be the updated feed itself, saving you the trouble of having to fetch it.
There isn't much documentation specific to using PuSH and the YouTube API beyond that blog post, but the general PuSH docs all apply: https://pubsubhubbub.appspot.com/
Failing that, SUP should still work, so we could try to debug that further if you'd rather use that.