Extract all seasons and episode video links from CBS website - parsing

Question

CBS has the TV show "The Challenge", and there are many seasons and many episodes for each season. They are on the CBS website here: https://www.cbs.com/shows/the-challenge/
I would like a list of all the video links, such as this (first 4 episodes of Season 11).
https://www.cbs.com/shows/the-challenge/video/C5NCzTv2qJp1GwhcV5KWxeKWN4p6_Mqt/the-challenge-throwing-down-the-gauntlet/
https://www.cbs.com/shows/the-challenge/video/kXdno68B36gd6s06OhdrUDUvAAYY4q_e/the-challenge-derrick-steps-it-up/
https://www.cbs.com/shows/the-challenge/video/RYA43Dqs2bRJsgAtcZIZhN8zVVQ1FIxf/the-challenge-we-can-work-it-out/
https://www.cbs.com/shows/the-challenge/video/lJvc_Lkns9Q2NYkDfmsNQmeajXP3QjRm/the-challenge-the-10-000-pyramid/
How can I automatically extract the video links for all of the episodes? I was able to "view page source", but it only showed 12/18 episodes for Season 11: (Open with chrome): view-source:https://www.cbs.com/shows/the-challenge/ , search for https://www.cbs.com/shows/the-challenge/video, should show 12 matches.
The page "hides" episodes and seasons inside the main page, so there is not a separate URL for other seasons. The solution I have now is to manually copy the link address for each of the videos...

Answer 1

This page is (partially) dynamically loaded using javascript. For example, the links for episodes 13-18 are loaded that way.
To capture those, you'll need to use your browser's Developer tab (that's a long and complicated story; you can start reading about it here, for example).
Once you have that link, the response is a json, which - treated as a python dictionary, yields the desired output.
So all together:
import requests
cookies = {
'CBS_ADV_VAL': 'c',
'CBS_ADV_SUBSES_VAL': '4',
'ovvuid': '9f064779-4c06-49f1-9cdd-7e64e653145e',
'OptanonConsent': 'isIABGlobal=false&datestamp=Wed+Sep+09+2020+15%3A44%3A13+GMT-0400+(Eastern+Daylight+Time)&version=6.5.0&hosts=&consentId=d1c945ba-78ea-46e6-ba6f-5329085e06d8&interactionCount=1&landingPath=https%3A%2F%2Fwww.cbs.com%2Fshows%2Fthe-challenge%2F&groups=1%3A1%2C2%3A1%2C3%3A1%2C4%3A1%2C5%3A1',
}
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:80.0) Gecko/20100101 Firefox/80.0',
'Accept': 'application/json, text/javascript, */*; q=0.01',
'Accept-Language': 'en-US,en;q=0.5',
'X-Requested-With': 'XMLHttpRequest',
'Referer': 'https://www.cbs.com/shows/the-challenge/',
'Connection': 'keep-alive',
'Cache-Control': 'max-age=0',
'TE': 'Trailers',
}
response = requests.get('https://www.cbs.com/shows/the-challenge/xhr/episodes/page/0/size/18/xs/0/season/11/', headers=headers, cookies=cookies)
links = response.json()
for entry in (links['result']['data']):
print(entry['url'])
Output:
/shows/the-challenge/video/IBWXQxtaPVmI40RnAACOc_zo0u13Ups1/the-challenge-blind-panic/
/shows/the-challenge/video/uFv8wFmvUFRKfiM29HVT3K_gGCZ4IWYS/the-challenge-last-men-standing/
/shows/the-challenge/video/9GP_ASLg9U_MmFvFmXPHO9liRzjdHhwI/the-challenge-don-t-bet-on-it/
etc., all 18 episodes. You can then concatenate each of these links with the base url (https://www.cbs.com) to form the final links.