I'm not sure if this is the right place to post this question,I googled a lot about this,but nothing turned up,. for a link of the form
http://www.youtube.com/watch?v=[video_id]
How do i get the link for the data stream?
The following bash script will retrieve youtube streaming url's. I know this is outdated, but maybe this will help someone.
#!/bin/bash
[ -z "$1" ] && printf "usage: `basename $0` <youtube_url>\n" && exit 1
ID="$(echo "$1" | grep -o "v=[^\&]*" | sed 's|v=||')"
URL="$(curl -s "http://www.youtube.com/get_video_info?&video_id=$ID" | sed -e 's|%253A|:|g' -e 's|%252F|/|g' -e 's|%253F|?|g' -e 's|%253D|=|g' -e 's|%2525|%|g' -e 's|%2526|\&|g' -e 's|\%26|\&|g' -e 's|%3D|=|g' -e 's|type=video/[^/]*&sig|signature|g' | grep -o "http://o-o---preferred[^:]*signature=[^\&]*" | head -1)"
[ -z "$URL" ] && printf "Nothing was found...\n" && exit 2
echo "$URL"
Here's a quick lesson in reverse-engineering the YouTube page to extract the stream data.
In the HTML you'll find a <script> tag which defines a variable "swfHTML" - it looks like this: "var swfHTML = (isIE) ? "...
The text in the quotes that follows that snippet is the HTML that displays the Flash object. Note, this text is a set of broken up strings that get concatenated so you'll need to clean it up (i.e. strip instances of '" + "' and and escaping backslashes in order to get the HTML string.)
Once clean you'll need to find the <param> tag with name="flashvars", the value of this tag is an &-delimited URL. Do a split on the & and you'll get your key-value pairs for all the data relating to this video.
The main key you're looking for is "fmt_url_map" and it's an URL Encoded string of Comma-Separated Values starting with "35|" or "34|" or other. (These are defined in another key, "fmt_list" to be files of resolution 854x480 for 35, 640x360 for 34, etc..)
each channel provides rss-data, wich is not updated immediatelly.
Here is a generator for Youtube RSS Files. You should be able to deduce the location of videofiles based on the RSS information. The flv files should be streamable but other formats are also provided.
EDIT:
http://www.referd.info/ is no longer available. It basically was a service where you provided the youtube link and it dereferenced it found all possible downloadsources for that video. I am sure those services are still out there... this one isnt anymore.
You Need Open Link Like This
http://www.youtube.com/get_video_info?&video_id=OjEG808MfF4
And Find Your Stream Your In Return Data
Related
I have a test.txt file with links for example:
google.com?test=
google.com?hello=
and this code
xargs -0 -n1 -a FUZZvul.txt -d '\n' -P 20 -I % curl -ks1L '%/?=DarkLotus' | grep -a 'DarkLotus'
When I type a specific word, such as DarkLotus, in the terminal, it checks the links in the file and it brings me the word which is reflected in the links i provided in the test file
There is no problem here, the problem is that I have many links, and when the result appears in the terminal, I do not know which site reflected the DarkLotus word.
How can i do it?
Try -n option. It shows the line number of file with the matched line.
Best Regards,
Haridas.
I'm not sure what you are up to there, but can you invert it? grep by default prints matching lines. The problem here is you are piping the input from the stdout of the previous commands into grep, and that can lack context at grep. Since you have a file to work with:
$ grep 'DarkLotus' FUZZvul.txt
If your intention is to also follow the link then it might be easier to write a bash script:
#!/bin/bash
for line in `grep 'DarkLotus FUZZvul.txt`
do
link=# extract link from line
echo ${link}
curl -ks1L ${link}
done
Then you could make your script accept user input:
#/bin/bash
word="${0}"
for line in `grep ${word} FUZZvul.txt`
...
and then
$ my_link_getter "DarkLotus"
https://google?somearg=DarkLotus
...
And then you could make the txt file a parameter.
etc.
I am trying to download videos of a channel which has subtitle. There are more than thousands files, but only a few has subtitle.
youtube-dl --all-subs -ciw -o "./tmp/%(playlist_index)s - %(title)s.%(ext)s" https://www.youtube.com/watch?v=eVW0Xz85qSA&list=PLElG6fwk_0UmBgC02jKJePx
If I can run a command after every url download, it can be good enough. In this case I will check existence of any subtitle file and decide to keep it or remove it.
maybe --exec is good, but it did not work for me as I expected.
This could be probably done in more elegant way.
But it works for me.
First extract urls that have subtitles (replace "playlist_url" with actual url of a playlist, of course)
youtube-dl --write-sub -ij playlist_url | jq -r ".subtitles" \
| grep -Eo "v=[^\&]+" | sort -u > urls.txt \
&& sed -i -e s7^7https:\/\/www\.youtube\.com\/watch\?7 urls.txt
and then download those files with a batch input
youtube-dl -cia urls.txt
* notice that proper playlist_url for channel adress you provided is "https://www.youtube.com/user/SonechkoProject/videos"
OK, this might be a silly question. I've got the test.json file:
{
"timestamp": 1234567890,
"report": "AgeReport"
}
What I want to do is to extract timestamp and report values and store them in some env variables:
export $(cat test.json | jq -r '#sh "TIMESTAMP=\(.timestamp) REPORT=\(.report)"')
and the result is:
echo $TIMESTAMP $REPORT
1234567890 'AgeReport'
The problem is that those single quotes break other commands.
How can I get rid of those single quotes?
NOTE: I'm gonna leave the accepted answer as is, but see #Inian's answer for a better solution.
Why make it convoluted with using eval and have a quoting mess? Rather simply emit the variables by joining them with NULL (\u0000) and read it back in the shell environment
{
IFS= read -r -d '' TIMESTAMP
IFS= read -r -d '' REPORT
} < <(jq -r '(.timestamp|tostring) + "\u0000" + .report + "\u0000"' test.json)
This makes your parsing more robust by making the fields joined by NULL delimiter, which can't be part of your string sequence.
From the jq man-page, the #sh command converts its input to be
escaped suitable for use in a command-line for a POSIX shell.
So, rather than attempting to splice the output of jq into the shell's export command which would require carefully removing some quoting, you can generate the entire commandline inside jq, and then execute it with eval:
eval "$(
cat test.json |\
jq -r '#sh "export TIMESTAMP=\(.timestamp) REPORT=\(.report)"'
)"
I want to download an AUDIO-ONLY playlist using youtube-dl. I've got the basics down. My version of youtube-dl is up to date, and the command I'm using is:
youtube-dl -x --extract-audio --yes-playlist --playlist-start 1 --playlist-end 18 \
https://www.youtube.com/watch?v=NRHoSXxTpgI&index=1&list=OLAK5uy_lowZyOuAZVs41vEtzV6e0KU8Iue1YQlzg
But it keeps getting stuck on
Deleting original file [filename] (pass -k to keep)
Github doesn't seem to be of any help: https://github.com/rg3/youtube-dl/issues/12570
Any ideas?
The ampersand (&) is a special character in most shells, advising the shell to run the command so far in the background. This should be plainly visible in the first lines after the command execution, which will likely say something like
[1] 17156
[2] 17157
These is your shell telling you the process ID of the new background processes.
You must escape ampersands with a backslash or quotes, like this:
youtube-dl -x --yes-playlist --playlist-start 1 --playlist-end 18 \
'https://www.youtube.com/watch?v=NRHoSXxTpgI&index=1&list=OLAK5uy_lowZyOuAZVs41vEtzV6e0KU8Iue1YQlzg'
--extract-audio is the same as -x, and thus can be deleted.
For more information, see the youtube-dl FAQ.
-x = extract audio only
-i = ignore errors (skip unavailable files)
use only the 'list=...' part, delete the other parameters from copied URL
default is first to last
youtube-dl -ix https://www.youtube.com/watch?list=...
for individual tracks add e.g. --playlist-items 4,7-9
see also: github/ytdl-org: output template examples
I am trying to use a shell script (well a "one liner") to find any common lines between around 50 files.
Edit: Note I am looking for a line (lines) that appears in all the files
So far i've tried grep grep -v -x -f file1.sp * which just matches that files contents across ALL the other files.
I've also tried grep -v -x -f file1.sp file2.sp | grep -v -x -f - file3.sp | grep -v -x -f - file4.sp | grep -v -x -f - file5.sp etc... but I believe that searches using the files to be searched as STD in not the pattern to match on.
Does anyone know how to do this with grep or another tool?
I don't mind if it takes a while to run, I've got to add a few lines of code to around 500 files and wanted to find a common line in each of them for it to insert 'after' (they were originally just c&p from one file so hopefully there are some common lines!)
Thanks for your time,
When I first read this I thought you were trying to find 'any common lines'. I took this as meaning "find duplicate lines". If this is the case, the following should suffice:
sort *.sp | uniq -d
Upon re-reading your question, it seems that you are actually trying to find lines that 'appear in all the files'. If this is the case, you will need to know the number of files in your directory:
find . -type f -name "*.sp" | wc -l
If this returns the number 50, you can then use awk like this:
WHINY_USERS=1 awk '{ array[$0]++ } END { for (i in array) if (array[i] == 50) print i }' *.sp
You can consolidate this process and write a one-liner like this:
WHINY_USERS=1 awk -v find=$(find . -type f -name "*.sp" | wc -l) '{ array[$0]++ } END { for (i in array) if (array[i] == find) print i }' *.sp
old, bash answer (O(n); opens 2 * n files)
From #mjgpy3 answer, you just have to make a for loop and use comm, like this:
#!/bin/bash
tmp1="/tmp/tmp1$RANDOM"
tmp2="/tmp/tmp2$RANDOM"
cp "$1" "$tmp1"
shift
for file in "$#"
do
comm -1 -2 "$tmp1" "$file" > "$tmp2"
mv "$tmp2" "$tmp1"
done
cat "$tmp1"
rm "$tmp1"
Save in a comm.sh, make it executable, and call
./comm.sh *.sp
assuming all your filenames end with .sp.
Updated answer, python, opens only each file once
Looking at the other answers, I wanted to give one that opens once each file without using any temporary file, and supports duplicated lines. Additionally, let's process the files in parallel.
Here you go (in python3):
#!/bin/env python
import argparse
import sys
import multiprocessing
import os
EOLS = {'native': os.linesep.encode('ascii'), 'unix': b'\n', 'windows': b'\r\n'}
def extract_set(filename):
with open(filename, 'rb') as f:
return set(line.rstrip(b'\r\n') for line in f)
def find_common_lines(filenames):
pool = multiprocessing.Pool()
line_sets = pool.map(extract_set, filenames)
return set.intersection(*line_sets)
if __name__ == '__main__':
# usage info and argument parsing
parser = argparse.ArgumentParser()
parser.add_argument("in_files", nargs='+',
help="find common lines in these files")
parser.add_argument('--out', type=argparse.FileType('wb'),
help="the output file (default stdout)")
parser.add_argument('--eol-style', choices=EOLS.keys(), default='native',
help="(default: native)")
args = parser.parse_args()
# actual stuff
common_lines = find_common_lines(args.in_files)
# write results to output
to_print = EOLS[args.eol_style].join(common_lines)
if args.out is None:
# find out stdout's encoding, utf-8 if absent
encoding = sys.stdout.encoding or 'utf-8'
sys.stdout.write(to_print.decode(encoding))
else:
args.out.write(to_print)
Save it into a find_common_lines.py, and call
python ./find_common_lines.py *.sp
More usage info with the --help option.
Combining this two answers (ans1 and ans2) I think you can get the result you are needing without sorting the files:
#!/bin/bash
ans="matching_lines"
for file1 in *
do
for file2 in *
do
if [ "$file1" != "$ans" ] && [ "$file2" != "$ans" ] && [ "$file1" != "$file2" ] ; then
echo "Comparing: $file1 $file2 ..." >> $ans
perl -ne 'print if ($seen{$_} .= #ARGV) =~ /10$/' $file1 $file2 >> $ans
fi
done
done
Simply save it, give it execution rights (chmod +x compareFiles.sh) and run it. It will take all the files present in the current working directory and will make an all-vs-all comparison leaving in the "matching_lines" file the result.
Things to be improved:
Skip directories
Avoid comparing all the files two times (file1 vs file2 and file2 vs file1).
Maybe add the line number next to the matching string
Hope this helps.
Best,
Alan Karpovsky
See this answer. I originally though a diff sounded like what you were asking for, but this answer seems much more appropriate.