I want to download a playlist of videos from youtube using youtube-dl for further offline viewing.
Some videos have unavailable fragments (but for some reason there is no problem when viewing through browser). I want to download the playlist, but skip the videos with missing fragments.
My original run was:
youtube-dl -if bestvideo+bestaudio PLEsfXFp6DpzTD1BD1aWNxS2Ep06vIkaeW
But it "freezes" at videos with missed fragments.
Later, in documentation I found this option option --abort-on-unavailable-fragment
But following command just aborts, when it takes 10 retries for attempt to download fragment.
youtube-dl --abort-on-unavailable-fragment -if bestvideo+bestaudio PLEsfXFp6DpzTD1BD1aWNxS2Ep06vIkaeW
I also found parameter --skip-unavailable-fragments, but it just skips fragments, which are unavailable and continue to download next fragment. It results in glitches in video after merging.
youtube-dl --skip-unavailable-fragments -if bestvideo+bestaudio PLEsfXFp6DpzTD1BD1aWNxS2Ep06vIkaeW
Are there any options / parameters to download the whole playlist, but as soon as there is video with missed fragment: skip that video and switch to the next video in playlist?
-i, --ignore-errors
This one works fine for me.
--abort-on-unavailable-fragment Abort downloading when some fragment is not
available
This will skip the video if it has unavailable fragments, taken from the readme: https://github.com/ytdl-org/youtube-dl#download-options tested on the same issue.
I tried the above mentioned answer but it aborts the whole playlist. I tried using -i which ignores all the errors and skips if the video is unavailable in your country. And this method works without any problems.
youtube-dl -i -o "%(title)s.%(ext)s" 'https://www.youtube.com/playlist?list=PLx65qkgCWNJIs3FPaj8JZhduXSpQ_ZfvL'
use --extract-audio if you want only audio
use --audio-format mp3 if you want to specify the audio format
Update in 2021:
You can use
--abort-on-unavailable-fragment --ignore-errors
In this way if a video have a fragment unavaible it will skip to the next whiout aborting, ignoring the abort.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 3 years ago.
The community reviewed whether to reopen this question 6 months ago and left it closed:
Original close reason(s) were not resolved
Improve this question
I want to download a video whose URL is not a simple MP4 file, but rather a blob type for example:
<video id="playerVideo" width="450px" autoplay="autoplay" height="338px"
style="height:100%;width:100%;" class="mejs-rai-e"
src="blob:http://www.example.com/d70a74e1-0324-4b9f-bad4-84e3036ad354">
</video>
Is there any chrome extension or software which can be used to download videos from blob URLs?
I just came up with a general solution, which should work on most websites.
I tried this on Chrome only, but this method should work with any other browser, though, as Dev Tools are pretty much the same in them all.
Steps:
Open the browser's Dev Tools (usually F12, or Ctrl-Shift-I, or right-click and then Inspect in the popup menu) on the page with the video you are interested in.
Go to Network tab and then reload the page. The tab will get populated with a list of requests (may be up to a hundred of them or even more).
Search through the names of requests and find the request with .m3u8 extension. There may be many of them, but most likely the first or largest is the one you are looking for. It may have any name, e.g. playlist.m3u8.
Click its name to open the request. Under the Headers subsection you will see request's full URL in the Request URL field. Copy it.
Extract the video from m3u8. There are many ways to do it, I'll give you those I tried, but you can google more by "download video from m3u8".
Option 1. If you have VLC player installed, feed the URL to VLC using the "Open Network…" menu option. I'm not going to go into details on this part here, there are a number of comprehensive guides in many places, for example, here. If the page doesn't work, you can always google another one by "vlc download online video".
Option 2. If you are more into command line, use FFMPEG or your own script, as directed in this SuperUser question.
Use the HLS Downloader Google Chrome extension to get the link to the M3U playlist. Its icon in the browser bar will show the number of playlists found on the current webpage. Clicking on the icon you can then see a list of the playlist link and then use the copy button next to a link to copy it.
Then use the youtube-dl program to download the file.
youtube-dl --all-subs -f mp4 -o "file-name-to-save-as.mp4" "https://link-from-Google_Chrome-HLS_Downloader_extension"
Explanation of command line options:
-f mp4 = Output format mp4
--all-subs = Download all subtitles
-o "file-name-to-save-as.mp4" = Name of the file to save the video as.
"https://link-from-Google_Chrome-HLS_Downloader_extension" = This is the link to the playlist you copied from the HLS Downloader extension.
If you use the same configuration options all the time for youtube-dl you may want to take a look at the configuration options for youtube-dl, as this can save you a lot of typing.
The HLS Downloader extension is free and open source under the MIT license if you want to see the code it can be found on its project page on Github.
There are a variety of ways to get the URL .m3u8 either by viewing the source of a page, looking at the Network tab in the Developer Tools in Chrome, or using a plugin such as HDL/HLS Video Downloader.
With the .m3u8 URL in hand you can then use ffmpeg to download the video to a file like so:
$ ffmpeg -i 'https://url/to/some/file.m3u8' -bsf:a aac_adtstoasc \
-vcodec copy -c copy -crf 50 file.mp4
This is how I manage to "download" it:
Use inspect-element to identify the URL of the M3U playlist file
Download the M3U file
Use VLC to read the M3U file, stream and convert the video to MP4
In Firefox the M3U file appeared as of type application/vnd.apple.mpegurl
The contents of the M3U file would look like:
Open VLC medial player and use the Media => Convert option. Use your (saved) M3U file as the source:
The process can differ depending on where and how the video is being hosted. Knowing that can help to answer the question in more detail.
As an example; this is how you can download videos with blob links on Vimeo.
View the source code of the video player iframe
Search for mp4
Copy link with token query
Download before token expires
Source & step-by-step instructions here.
If you can NOT find the .m3u8 file you will need to do a couple of steps different.
1) Go to the network tab and sort by Media
2) You will see something here and select the first item. In my example, it's an mpd file. then copy the Request URL.
3) Next, download the file using your favorite command line tool using the URL from step 2.
youtube-dl -f bestvideo+bestaudio https://url.com/destination/stream.mpd
4) Depending on the encoding you might have to join the audio and video files together but this will depend on a video by video case.
Find the playlist/manifest with the developer tools network tab. There is always one, as that's how it works. It might have an m3u8 extension that you can type into the Filter. (The youtube-dl tool can also find the m3u8 tool automatically some time give it a direct link to the webpage where the video is being displayed.)
Give it to the youtube-dl tool (Download). It can download much more than just YouTube. It'll auto-download each segment then combine everything with FFmpeg then discard the parts. There is a good chance it supports the site you want to download from natively, and you don't even need to do step #1.
If you find a site that is stubborn and you run into 403 errors... Telerik Fiddler to the rescue. It can catch and save anything transmitted (such as the video file) as it acts as a local proxy. Everything you see/hear can be downloaded, unless it's DRM content like Spotify.
Note: in the window, you can use youtube-dl.exe using "Command Prompt" or creating a batch file. i.e
Thanks
I posted this already at some other websites and though why not share it with guys/gals at stackoverflow.
Install the Video DownloadHelper extension on Firefox browser.
With DownloadHelper activated, navigate to the webpage containing the video that you want to download.
Once the video is streaming, click on the DownloadHelper icon. It will give you a list of all file formats available on the current video.
Scroll onto the file format that you wish to download
On the right hand side, you will see an arrow
Click on that arrow to get more information regarding the current video and the selected format
From the displayed window at the end of that arrow, scroll down and select "Details"
You now have all the details concerning the current video and the selected format. It is something like this.
Hit Details⊗
_needsAggregate
_needsCoapp
actions
bitrate
chunked
descrPrefix
durationFloat
extension
frameId
fromCache
group
hls
id
isPrivate
length
masterManifest
mediaManifest
originalId
referrer
size
status
title
topUrl
url
urlFilename
Now, look at the specifics of the referrer in that Hit Details. That's the url you want. Copy it and paste on your favorite downloader.
If the blob is instantiated with data from an F4M manifest (check the Network Tab in Chrome's Developer Tools), you can download the video file using the php script posted here: https://n1njahacks.wordpress.com/2015/01/29/how-to-save-hds-flash-streams-from-any-web-page/
By putting:
if ($manifest == '')
$manifest = $_GET['manifest'];
before:
if ($manifest)
you could even run it on a webserver, using requests with the query string: ?manifest=[manifest url].
Note that you'll probably want to use an FTP client to retrieve the downloaded video file and clean up after the script (it leaves all the downloaded video parts).
I have an MP4 file, where I've added chapters via ffmpeg.
But in the iOS 12 Podcasts app, from Apple, the chapters don't appear. This should happen, as noted by idownloadblog.com
In comparison, when using QuickLook on MacOS, the list of chapters can be seen by clicking the chapters button (in the bottom right hand side of the window).
And opening in QuickTime Player, while there isn't a list of chapters to view, you can use the "View > Next Chapter" menu item.
So I'm assuming this is a bug in iOS... but I'm wondering if there is another way to add chapters? or if I've made a mistake?
My current process is to create a "ffmetadata" file, as noted in the ffmpeg documentation:
;FFMETADATA1
title=Example
[CHAPTER]
TIMEBASE=1/1000
START=0
END=221913
title=Chapter 1
[CHAPTER]
TIMEBASE=1/1000
START=221913
END=1169241
title=Chapter 2
https://ffmpeg.org/ffmpeg-formats.html#Metadata-1
Then I've tried each of the following commands:
ffmpeg -i 2019-01-02.mp4 -i 2019-01-02.meta -map_metadata 1 -codec copy 2019-01-02-chapters.mp4
ffmpeg -i 2019-01-02.mp4 -i 2019-01-02.meta -map_metadata 1 2019-01-02-chapters.mp4
ffmpeg -i 2019-01-02.mp4 -i 2019-01-02.meta -map_metadata 1 2019-01-02-chapters.mp3
The first one is from the ffmpeg documentation, where -codec copy means the audio file is not re-encoded.
The second one takes longer, while it re-encodes the audio data.
The third one requires re-encoding to convert it into an MP3 file (which uses ID3 tags for the chapter data).
On a slightly unrelated note, the third party app "RSSRadio" does list the chapters, but the feature added in version 4 that allows you to "skip directly to the start of the next chapter", if the next chapter starts within the next 3 minutes, does not seem to work.
Simple answer, it needs to be an M4A file.
While MP4 and M4A files are pretty much the same, M4A is limited to audio format, whereas an MP4 can also contain video.
I'm using an offline terminal to create an .m3u8 file but I would like to play it using VLC. Every example so far has shown examples of using m3u8 online but this isn't an option for me. If you can play it offline using VLC how do you do so?
You have to download the playlist from m3u8 to watch it offline and to do that you have to either use youtube-dl or ffmpeg
i would recommend ffmpeg , and the command for downloading the m3u8 file goes like this :-
ffmpeg -i "here your m3u8 link" -c copy Output.mp4
This will download the highest quality video available in that m3u8 file.
To change quality you can use map command or simply download the m3u8 file and open it with notepad and there you will find links for other resolution.
m3u8 files are just playlists without actual contents, but just URL pointers. Those files are mostly being used these days for HLS.
While you can play the files while offline, you won't see anything unless you are also hosting the referenced Contents.
Sometimes it is useful to download a stream into our local machine.
Reasons could be
To make a manual modification to the manifest
For getting fast access to files of a server with poor networking.
If we try to use curl or wget to download the asset which is pointed by the URL for the stream, we end up downloading a small text file. It is surely not the video asset.
So how can we download the stream itself?
The actual script which does the download is given in the link at the bottom of my answer. But before we proceed to the how-to, let's first understand the steps for downloading a stream.
Without going into too much details, the URL pointing to the stream is typically named with the m3u8 extension. That file is called the manifest of the stream and is actually a text file containing, among other things, a list of pairs: a bitrate and a corresponding URL for the matching playlist file. Here is an excerpt from a manifest file:
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=380600,CODECS="avc1.4d00c,mp4a.40.2",RESOLUTION=320x180
http://f24hls-i.akamaihd.net/hls/live/221193-b/F24_EN_LO_HLS/master_250.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=655600,CODECS="avc1.77.30,mp4a.40.2",RESOLUTION=640x360
http://f24hls-i.akamaihd.net/hls/live/221193/F24_EN_LO_HLS/master_500.m3u8
A playlist file is another text file which tell the player which TS file is to be playing on each position of the playback head.
Here is the beginning of a typical playlist file:
#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:303165
#EXTINF:10.000,
20170216T114458/master_500/00151/master_500_01165.ts
#EXTINF:10.000,
20170216T114458/master_500/00151/master_500_01166.ts
So after downloading the playlist file for each of the bitrates, we can start downloading the TS files required to play the stream at each of the possible bitrates.
All this is done using a quite simple and self-explaining script which I was putting in GitHub: https://github.com/ishahak/HLS_Downloader
I hope it will be useful for others.
You can simply use ffmpeg. Like this:
fmpeg -i "http://somewhere.com/video.m3u8" -c copy -bsf:a aac_adtstoasc output.mp4
I've been asked by my client whether it is possible to download a video and stream it once a bit has downloaded, just like pocketcasts does. His reasoning is this will allow him to store his video files on a site such as godaddy and bypass the need to stream the file to the phone which normally requires a dedicated server.
Is this even possible? if so do you know anywhere I can look to find out how pocketcasts does it? At the moment my app just streams an mp4.
Thanks for looking,
Matt
Since you're targetting iOS, HLS (HTTP Live Streaming) is your friend: https://developer.apple.com/streaming/
Please see my answer here for how you can use it: Simultaneously downloading and playing a song that is pieced together from multiple URLs
It's very easy to run a long movie through the mediafilesegmenter tool from Apple (or FFMPEG) which spits out a number of small .ts files (MPEG 2 Transport Stream). Then you create a manifest (a .m3u8 file) which describes how these files fit together (which mediafilesegment will create for you too!). Then you just put the manifest file and the .ts files on a hosting provider (like GoDaddy) and you're all set.
For example, given a file called test.mp4, first turn it into a .ts file with ffmpeg:
ffmpeg -i test.mp4 -acodec copy -vcodec copy -bsf h264_mp4toannexb test.ts
Then turn it into a series of HLS segments with mediafilesegmenter (the same can be done using the ffmpeg segment muxer, but mediafilesegmenter seems to be more robust):
mediafilesegmenter -t 3 test.ts
The result is a bunch of 3 second clips (that's what -t 3 means) and an manifest file called prog_index.m3u8. The contents of that look like:
#EXTM3U
#EXT-X-TARGETDURATION:3
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:2.99520,
fileSequence0.ts
#EXTINF:2.99520,
fileSequence1.ts
#EXTINF:2.99520,
fileSequence2.ts
#EXTINF:2.99520,
fileSequence3.ts
...
#EXTINF:0.37440,
fileSequence75.ts
#EXT-X-ENDLIST
Simply putting all of the .ts files and the .m3u8 file on a web server and pointing your AVPlayer or MPMoviePlayerController in iOS at the URL for the .m3u8 will get you an excellent streaming performance.