I am trying to extract the audio from a HLS m3u8 live video stream into either FLAC, PCM, or OGG-OPUS. I've tried several options. They all create the file on my desktop but when I play the file, no audio is recorded.
Convert to FLAC
gst-launch-1.0 souphttpsrc location=[m3u8 URL] ! hlsdemux ! decodebin ! audioconvert ! flacenc ! filesink location=audio.flac
Convert to PCM
gst-launch-1.0 souphttpsrc location=[m3u8 URL] ! hlsdemux ! decodebin ! audioconvert ! audio/x-raw,format=S32BE,channels=1,rate=48000 ! filesink location=audio.pcm
Convert to OGG-OPUS
gst-launch-1.0 souphttpsrc location=[m3u8 URL] ! hlsdemux ! decodebin ! audioconvert ! opusenc ! filesink location=audio.ogg
Information about the m3u8 HLS stream from gst-discoverer-1.0
Properties:
Duration: 99:99:99.999999999
Seekable: no
Live: no
container: application/x-hls
container: MPEG-2 Transport Stream
audio: MPEG-4 AAC
Stream ID: 722f60699ac437d8b42b2325b9497eb8707874802bf34a0185ca68ebfd95dd38/src_0:1/00000101
Language: <unknown>
Channels: 2 (front-left, front-right)
Sample rate: 48000
Depth: 32
Bitrate: 0
Max bitrate: 0
video: H.264 (Main Profile)
Stream ID: 722f60699ac437d8b42b2325b9497eb8707874802bf34a0185ca68ebfd95dd38/src_0:1/00000100
Width: 768
Height: 432
Depth: 24
Frame rate: 30000/1001
Pixel aspect ratio: 1/1
Interlaced: false
Bitrate: 0
Max bitrate: 0
I figured this out and posted the solution on GitHub:
GStreamer Example on GitHub
This Node.js example will take a live m3u8 stream, use GStreamer to extract the audio, save it to a FLAC audio file, and send to AWS Transcribe all in real-time. (Some code copied from other examples on-line and combined into one working example.)
Related
I use some python code to decode H264 Video on NVIDIA JETSON NANO everything works fine.
I see some times that I get this warning:
[ WARN:0] global /home/ubuntu/opencv/modules/videoio/src/cap_gstreamer.cpp (961) open
OpenCV | GStreamer warning: Cannot query video position: status=1, value=253, duration=-1
I do not rely sure that this message is the problem but,
When I think to get this warning I get the performance of 7~7.5FPS, sometimes I don't get this warning, and my performance increase to 10.5FPS.
I would be happy to understand the problem I play some pcap video using colasoft player and capture the video in jetson using this python script.
class Video:
def __init__(self, urlName='udp://127.0.0.1:46002'):
print(f"Initialize Vieo, url: {urlName}")
self.pipeline = 'udpsrc port=46002 multicast-group=234.0.0.0 ! h264parse ! nvv4l2decoder ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink drop=1'
# self.cap = cv2.VideoCapture(urlName, cv2.CAP_FFMPEG)
self.cap = cv2.VideoCapture(self.pipeline, cv2.CAP_GSTREAMER)
if not self.cap.isOpened():
print('VideoCapture not opened')
exit(-1)
def readVideo(self):
"""
This Function read h.264 stream and calculate image info
:return: image, data_image
"""
try:
ret, frame = self.cap.read()
while frame == None:
ret, frame = self.cap.read()
except Exception as e:
pass
shape = frame.shape
data = {"image_width": shape[1], "image_height": shape[0], "channels": shape[2], "size_in_bytes": shape[0] * shape[1] * shape[2], "bits_per_pixel": shape[2] * 8} # data Video
return frame, data
I have a python script that receives jpeg encoded data using following pipeline and sends the jpeg data on a port.
rtspsrc location=rtsp://192.168.100.40:8554/ latency=0 retry=50 ! rtph265depay ! h265parse ! avdec_h265 ! videoscale ! videorate ! video/x-raw, framerate=10/1, width=1920, height=1080 ! jpegenc quality=85 ! image/jpeg ! appsink
At the receiver end I want to save the incoming data as a video, as described in this link
https://gstreamer.freedesktop.org/documentation/jpeg/jpegenc.html
gst-launch-1.0 videotestsrc num-buffers=50 ! video/x-raw, framerate='(fraction)'5/1 ! jpegenc ! avimux ! filesink location=mjpeg.avi
I have tried using opencv's VideoWriter with CAP_GSTREAMER
pipeline = f'appsrc ! avimux ! filesink location=recording.avi'
cap_write = cv2.VideoWriter(pipeline,cv2.CAP_GSTREAMER,0, 1, (1920,1080), False)
...
cap_write.write(jpgdata)
but it gives a runtime warning
[ WARN:0] global ../opencv/modules/videoio/src/cap_gstreamer.cpp (1948) writeFrame OpenCV | GStreamer warning: Error pushing buffer to GStreamer pipeline
If I modify the pipeline and use
pipeline = f'appsrc ! videoconvert ! videorate ! video/x-raw, framerate=1/1 ! filesink location=recording.avi'
The code does save the incoming frames as video but the saved video is too big with no bitrate and duration information in it.
ffmpeg -i recording.avi
...
[mjpeg # 0x560f4a408600] Format mjpeg detected only with low score of 25, misdetection possible!
Input #0, mjpeg, from 'recording.avi':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: mjpeg (Baseline), yuvj420p(pc, bt470bg/unknown/unknown), 1920x1080 [SAR 1:1 DAR 16:9], 25 tbr, 1200k tbn, 25 tbc
At least one output file must be specified
Need help!
Hello I am very new to gstreamer. As of now I am encoding frames using gstreamer pipeline in opencv in c++. But now I do not want dump each frame. I want to encode all frames and store it in a large buffer and whenever I want I will dump that buffer. so how can I do appsink in gstreamer opencv.
Below my code snippet where i am encoding each frame and dumping also
cv::VideoWriter out("appsrc ! videoconvert ! video/x-raw,width=1280,height=720 ! v4l2h264enc ! avimux ! filesink location=hellotest.avi",cv::CAP_GSTREAMER,0,30,cv::Size(1280,720),true);
out.write(Frame);
But I want appsink to store all encoded data into buffer I do not want to dump it or write it.
The short answer is no.
First of all you cannot write and read at the sametime with VideoWriter. What you can do is you can create two pipelines that communicate with each other using ipcpipeline elements.
For example
cv::VideoWriter out("appsrc ! videoconvert ! video/x-raw,width=1280,height=720 ! v4l2h264enc ! ipcpipelinesink",...);
cv::VideoCapture cap("ipcpipelinesrc ! ... ! appsink")
But this won't provide you with many alternatives because the data formats supported with VideoCapture is very limited, if you check the souce code.
// we support 11 types of data:
// video/x-raw, format=BGR -> 8bit, 3 channels
// video/x-raw, format=GRAY8 -> 8bit, 1 channel
// video/x-raw, format=UYVY -> 8bit, 2 channel
// video/x-raw, format=YUY2 -> 8bit, 2 channel
// video/x-raw, format=YVYU -> 8bit, 2 channel
// video/x-raw, format=NV12 -> 8bit, 1 channel (height is 1.5x larger than true height)
// video/x-raw, format=NV21 -> 8bit, 1 channel (height is 1.5x larger than true height)
// video/x-raw, format=YV12 -> 8bit, 1 channel (height is 1.5x larger than true height)
// video/x-raw, format=I420 -> 8bit, 1 channel (height is 1.5x larger than true height)
// video/x-bayer -> 8bit, 1 channel
// image/jpeg -> 8bit, mjpeg: buffer_size x 1 x 1
// bayer data is never decoded, the user is responsible for that
// everything is 8 bit, so we just test the caps for bit depth
I need to extract a MPEG-TS stream from a Wireshark capture. I have managed to do this but when I play it back using VLC the output is crappy, it's just a green window with some jitter on the top rows.
Here is how I did it:
Captured using ip.dest filter for the multicast stream.
Analyze -> Decode As -> UDP port (field), portnumber (value), MP2T (current)
Tools Dump MPEG TS Packets.
It does not play out correctly. Is there any other way of doing this
When I need to dump TS from a pcap file I do following:
If TS in plain UDP (column protocol shows MPEG TS for each packet) jump to step 3
If TS is packed in RTP, right click on any packet -> Decode as -> Choose RTP under field "Current"
Use tool MPEG Dump, Tools -> Dump MPEG TS Packets
I do not use MP2T packets decoding, it usually doesn't work.
If the TS is in plain UDP, it can happen that TS packets are shuffled and 4 bits long TS packet field which serves as a continuity counter is not long enough to correctly order TS packets. This can result in corrupted playback of dumped TS.
I've added two filtering options to the original pcap2mpeg.
You can find it on: https://github.com/bugre/pcap2mpegts
So you can:
filter by udp destination port
filter by mcast group IP and destination port
for the cases where the captured file has multiple TS on the same IP but on different ports, or on different mcast IP's.
you would run it as:
pcap2mpegts.pl -y -i 239.100.0.1 -p 2000 -l multi_ts_capture.pcap -o single-stream-output.ts
Not using Wireshark, you can use pcap2mpeg.pl. I tested it and it works well if there is a single MPEG
stream in the PCAP.
Here is the output of ffprobe on a mpeg file with 2 streams that was successfully extracted:
Input #0, mpegts, from 'test.mpeg':
Duration: 00:27:59.90, start: 4171.400000, bitrate: 8665 kb/s
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(progressive), 4096x2176 [SAR 1:1 DAR 32:17], 10 fps, 10 tbr, 90k tbn, 20 tbc
Stream #0:1[0x1001]: Data: bin_data ([6][0][0][0] / 0x0006)
First of all I'm a noob in both iOS and audio programming, so bear with me if I don't use the correct technical terms, but I'll do my best!
What we want to do:
In an iOS app we are developing, we want to be able to play sounds throughout 4 different outputs to have a mini surround system. That is, we want to have the Left and Right channels play through the Headphones, while the Center and Center surround play through an audio hardware connected to the lightning port. Since the audio files will be streamed/dowloaded from a remote server, using raw (PCM) audio files is not an option.
The problem:
Apple has, since iOS 6, made it possible to play an audio file using a multiroute configuration... and that is grate and exactly what we need... but, when ever we try to play a 4-channel audio file, AAC-encoded and encapsulated in an m4a (or CAF) file format, we get the following error:
ERROR: [0x19deee000] AVAudioFile.mm:86: AVAudioFileImpl: error 1718449215
(Which is the status code for "kAudioFileUnsupportedDataFormatError" )
We get the same error when we use the same audio encoded as lossless (ALAC) instead, but we don't get this error when playing the same audio befor encoding (PCM format).
We don't get the error neither when we use a stereo audio file, or a 5.1 audio file encoded, the same way as the 4-channels one, in both AAC and ALAC.
What we tried:
The encoding
The file was encoded using Apple's audio tools provided with Mac OS X: afconvert using this command:
afconvert -v -f 'm4af' -d "aac#44100" 4ch_master.caf 4ch_44100_AAC.m4a
and
afconvert -v -f 'caff' -d "alac#44100" 4ch_master.caf 4ch_44100_ALAC.caf
in the case of lossless encoding.
The audio format, as given by afinfo for the master (PCM) audio file:
File: 4ch_master.caf
File type ID: caff
Num Tracks: 1
----
Data format: 4 ch, 44100 Hz, 'lpcm' (0x0000000C) 16-bit little-endian signed integer
no channel layout.
estimated duration: 582.741338 sec
audio bytes: 205591144
audio packets: 25698893
bit rate: 2822400 bits per second
packet size upper bound: 8
maximum packet size: 8
audio data file offset: 4096
optimized
audio 25698893 valid frames + 0 priming + 0 remainder = 25698893
source bit depth: I16
The AAC-encoded format info:
File: 4ch_44100_AAC.m4a
File type ID: m4af
Num Tracks: 1
----
Data format: 4 ch, 44100 Hz, 'aac ' (0x00000000) 0 bits/channel, 0 bytes/packet, 1024 frames/packet, 0 bytes/frame
Channel layout: Quadraphonic
estimated duration: 582.741338 sec
audio bytes: 18338514
audio packets: 25099
bit rate: 251730 bits per second
packet size upper bound: 1039
maximum packet size: 1039
audio data file offset: 106496
optimized
audio 25698893 valid frames + 2112 priming + 371 remainder = 25701376
source bit depth: I16
format list:
[ 0] format: 4 ch, 44100 Hz, 'aac ' (0x00000000) 0 bits/channel, 0 bytes/packet, 1024 frames/packet, 0 bytes/frame
Channel layout: Quadraphonic
----
And for the lossless encoded audio file:
File: 4ch_44100_ALAC.caf
File type ID: caff
Num Tracks: 1
----
Data format: 4 ch, 44100 Hz, 'alac' (0x00000001) from 16-bit source, 4096 frames/packet
Channel layout: 4.0 (C L R Cs)
estimated duration: 582.741338 sec
audio bytes: 83333400
audio packets: 6275
bit rate: 1143862 bits per second
packet size upper bound: 16777
maximum packet size: 16777
audio data file offset: 20480
optimized
audio 25698893 valid frames + 0 priming + 3507 remainder = 25702400
source bit depth: I16
----
The code
In the code part, at the beginning, we followed the implementation presented at session 505 of WWDC12 using AVAudioPlayer API. At that level, multirouting didn't seemed to work reliably.. we didn't suspect that that might have been related to the audio format, so we moved on experimenting with AVAudioEngine API, presented at session 502 of WWDC14 and the sample code associated to it. We made the multirouting work for the master 4-channels audio file (after some adaptations), but then we hit the error mentioned above when calling scheduleFile, as shown in the code snippet below (Note: We are using Swift and all the necessary audio graph setup is done but not shown here):
var playerNode: AVAudioPlayerNode!
...
...
let audioFileToPlay = AVAudioFile(forReading: URLOfTheAudioFle)
playerNode.scheduleFile(audioFileToPlay, atTime: nil, completionHandler: nil)
Do someone have a hint on what could be wrong in the audio data format?
After contacting Apple Support, the answer was that this is not possible for the currently shipping system configurations:
"Thank you for contacting Apple Developer Technical Support (DTS). Our engineers have reviewed your request and have concluded that there is no supported way to achieve the desired functionality given the currently shipping system configurations."