Stream h264/aacplus media over udp - stream

Question

Does anyone have any experience in this area?
I've tried so far with ffmpeg (using libx264 and libaacplus) muxing into mpegts over udp, but that mpegts muxer is obviously broken (confirmed from several different sources).
I've also tried with vlc, but it can only encode AAC-LC and not HE-AAC v2 (aacplus).
Anyway, the problem I need to solve is to have several different geographic locations, covered with webcams and I need udp to be able to have incoming streams all the time and not to worry about networks ups and downs (udp will simply continue sending packets when the network is up again). So, did any of you have any experience using streaming live media using h264 with aacplus over udp protocol and if yes, can you please give me any links or directions how to accomplish it.
Thanks a lot in advance.

Answer 1

I'm developing a system which is kind of a DVR that must periodically record a h264 file from a video device and at the same time provide a local preview to allow for adjusting the video parameters and camera view. Although I am no expert at all in this field, I had a relative success streaming h264 over UDP, so I'll try to share what didn't and what did work out for me, which was all based on ffmpeg as a server (no audio in my case).
Initially I had set up my application to simultaneously record the video to a file and feed it to a .ffm file for ffserver to stream it up using RTP/UDP for the camera preview. The problem with that approach was exactly that when the feeding ffmpeg process stopped to change the video file, the preview would stop and never resume although the subsequent ffmpeg process would have already started feeding ffserver again. With RTP ffserver apparently complains the frames timestamp started back at 0 instead of continuing where they have stopped. Then I realized that if I were able to send the h264 packets over pure UDP I would get exactly the same effect you described, with preview resuming as soon as the next ffmpeg process would take over.
While trying to understand ffmpeg documentation I also tried using the mpegts format, but doing that I was getting MPEG2 video at the player on the other end (this shows a multicast address, but it also worked for a specific target):
$ ffmpeg -y -f video4linux2 -i /dev/video2 -vcodec libx264 -preset ultrafast /mnt/hd/video.mp4 -an -f mpegts udp://224.124.0.1:5000
ffmpeg version N-35860-g62adc60, Copyright (c) 2000-2011 the FFmpeg developers
built on Dec 16 2011 09:47:41 with gcc 4.5.3
configuration: --prefix=/usr --libdir=/usr/lib --shlibdir=/usr/lib --mandir=/usr/man --disable-debug --enable-shared --disable-static --enab
le-pthreads --enable-libtheora --enable-libvorbis --enable-gpl --enable-version3 --enable-postproc --enable-swscale --enable-avfilter --enable
-libx264 --enable-libvpx --enable-librtmp --disable-indev='v4l,dv1394'
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 46. 0 / 53. 46. 0
libavformat 53. 26. 0 / 53. 26. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[video4linux2,v4l2 # 0x8a96b00] Estimating duration from bitrate, this may be inaccurate
Input #0, video4linux2,v4l2, from '/dev/video2':
Duration: N/A, start: 1325538250.366878, bitrate: 27620 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, 27620 kb/s, 29.97 tbr, 1000k tbn, 29.97 tbc
[buffer # 0x8a9d8c0] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[buffer # 0x8a9c860] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[libx264 # 0x8a97780] using cpu capabilities: MMX2 Cache64
[libx264 # 0x8a97780] profile Constrained Baseline, level 1.3
[libx264 # 0x8a97780] 264 - core 120 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=0 ref=
1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11
fast_pskip=1 chroma_qp_offset=0 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=
0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
[mpegts # 0x8a98100] muxrate VBR, pcr every 2 pkts, sdt every 200, pat/pmt every 40 pkts
Output #0, mp4, to '/mnt/hd/video.mp4':
Metadata:
encoder : Lavf53.26.0
Stream #0:0: Video: h264 (![0][0][0] / 0x0021), yuv420p, 320x240, q=-1--1, 30k tbn, 29.97 tbc
Output #1, mpegts, to 'udp://224.124.0.1:5000':
Metadata:
encoder : Lavf53.26.0
Stream #1:0: Video: mpeg2video, yuv420p, 320x240, q=2-31, 200 kb/s, 90k tbn, 29.97 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo -> libx264)
Stream #0:0 -> #1:0 (rawvideo -> mpeg2video)
Press [q] to stop, [?] for help
On the client PC I was able to watch the video with ffplay, and it was indeed in MPEG2 format:
$ ffplay -f mpegts udp://224.124.0.1:5000
ffplay version N-35860-g62adc60, Copyright (c) 2003-2011 the FFmpeg developers
built on Dec 16 2011 09:47:41 with gcc 4.5.3
configuration: --prefix=/usr --libdir=/usr/lib --shlibdir=/usr/lib --mandir=/usr/man --disable-debug --enable-shared --disable-static --enable-pthreads --enable-libtheora --enable-libvorbis --enable-gpl --enable-version3 --enable-postproc --enable-swscale --enable-avfilter --enable-libx264 --enable-libvpx --enable-librtmp --disable-indev='v4l,dv1394'
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 46. 0 / 53. 46. 0
libavformat 53. 26. 0 / 53. 26. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[mpegts # 0x80f02e0] Unable to seek back to the start
[mpeg2video # 0x8111a00] mpeg_decode_postinit() failure
Last message repeated 6 times
[mpegts # 0x80f02e0] max_analyze_duration 5000000 reached at 5005000
[mpegts # 0x80f02e0] Estimating duration from bitrate, this may be inaccurate
Input #0, mpegts, from 'udp://224.124.0.1:5000':
Duration: N/A, start: 255.420433, bitrate: 104857 kb/s
Program 1
Metadata:
service_name : Service01
service_provider: FFmpeg
Stream #0:0[0x100]: Video: mpeg2video (Main) ([2][0][0][0] / 0x0002), yuv420p, 320x240 [SAR 1:1 DAR 4:3], 104857 kb/s, 30.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
VLC also worked to play the stream, but only after specifying the switch --demux ffmpeg (thanks to this):
vlc -vv --demux ffmpeg udp://#224.124.0.1:5000
Since I also wanted the preview to be in h264 but I already had it encoding the file, I tried to use the copy codec for the UDP streaming, but ffmpeg failed with a segfault (versions included for reference):
ffmpeg -y -f video4linux2 -i /dev/video2 -vcodec libx264 -preset ultrafast /mnt/hd/video.mp4 -an -vcodec cop
y -f h264 udp://224.124.0.1:5000
ffmpeg version N-35860-g62adc60, Copyright (c) 2000-2011 the FFmpeg developers
built on Dec 16 2011 09:47:41 with gcc 4.5.3
configuration: --prefix=/usr --libdir=/usr/lib --shlibdir=/usr/lib --mandir=/usr/man --disable-debug --enable-shared --disable-static --enab
le-pthreads --enable-libtheora --enable-libvorbis --enable-gpl --enable-version3 --enable-postproc --enable-swscale --enable-avfilter --enable
-libx264 --enable-libvpx --enable-librtmp --disable-indev='v4l,dv1394'
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 46. 0 / 53. 46. 0
libavformat 53. 26. 0 / 53. 26. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[video4linux2,v4l2 # 0x92c7b00] Estimating duration from bitrate, this may be inaccurate
Input #0, video4linux2,v4l2, from '/dev/video2':
Duration: N/A, start: 1325539132.411691, bitrate: 27620 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, 27620 kb/s, 29.97 tbr, 1000k tbn, 29.97 tbc
[buffer # 0x92ce860] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[libx264 # 0x92c8780] using cpu capabilities: MMX2 Cache64
[libx264 # 0x92c8780] profile Constrained Baseline, level 1.3
[libx264 # 0x92c8780] 264 - core 120 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=0 ref=
1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11
fast_pskip=1 chroma_qp_offset=0 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=
0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, mp4, to '/mnt/hd/video.mp4':
Metadata:
encoder : Lavf53.26.0
Stream #0:0: Video: h264 (![0][0][0] / 0x0021), yuv420p, 320x240, q=-1--1, 30k tbn, 29.97 tbc
Output #1, h264, to 'udp://224.124.0.1:5000':
Metadata:
encoder : Lavf53.26.0
Stream #1:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, q=2-31, 27620 kb/s, 90k tbn, 29.97 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo -> libx264)
Stream #0:0 -> #1:0 (copy)
Press [q] to stop, [?] for help
Segmentation fault
Although less than ideal, specifying the format h264 for the UDP streaming part led to a second concurrent h264 conversion, but it worked:
$ ffmpeg -y -f video4linux2 -i /dev/video2 -vcodec libx264 -preset ultrafast /mnt/hd/video.mp4 -an -f h264 -preset ultrafast udp://224.124.0.1:5000
ffmpeg version N-35860-g62adc60, Copyright (c) 2000-2011 the FFmpeg developers
built on Dec 16 2011 09:47:41 with gcc 4.5.3
configuration: --prefix=/usr --libdir=/usr/lib --shlibdir=/usr/lib --mandir=/usr/man --disable-debug --enable-shared --disable-static --enab
le-pthreads --enable-libtheora --enable-libvorbis --enable-gpl --enable-version3 --enable-postproc --enable-swscale --enable-avfilter --enable
-libx264 --enable-libvpx --enable-librtmp --disable-indev='v4l,dv1394'
libavutil 51. 32. 0 / 51. 32. 0
libavcodec 53. 46. 0 / 53. 46. 0
libavformat 53. 26. 0 / 53. 26. 0
libavdevice 53. 4. 0 / 53. 4. 0
libavfilter 2. 53. 0 / 2. 53. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 51. 2. 0 / 51. 2. 0
[video4linux2,v4l2 # 0x913ab00] Estimating duration from bitrate, this may be inaccurate
Input #0, video4linux2,v4l2, from '/dev/video2':
Duration: N/A, start: 1325539689.729735, bitrate: 27620 kb/s
Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 320x240, 27620 kb/s, 29.97 tbr, 1000k tbn, 29.97 tbc
[buffer # 0x9141840] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[buffer # 0x913e480] w:320 h:240 pixfmt:yuv420p tb:1/1000000 sar:0/1 sws_param:
[libx264 # 0x913b780] using cpu capabilities: MMX2 Cache64
[libx264 # 0x913b780] profile Constrained Baseline, level 1.3
[libx264 # 0x913b780] 264 - core 120 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=0 ref=
1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11
fast_pskip=1 chroma_qp_offset=0 threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=
0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
[libx264 # 0x913c820] using cpu capabilities: MMX2 Cache64
[libx264 # 0x913c820] profile Constrained Baseline, level 1.3
Output #0, mp4, to '/mnt/hd/video.mp4':
Metadata:
encoder : Lavf53.26.0
Stream #0:0: Video: h264 (![0][0][0] / 0x0021), yuv420p, 320x240, q=-1--1, 30k tbn, 29.97 tbc
Output #1, h264, to 'udp://224.124.0.1:5000':
Metadata:
encoder : Lavf53.26.0
Stream #1:0: Video: h264, yuv420p, 320x240, q=-1--1, 90k tbn, 29.97 tbc
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo -> libx264)
Stream #0:0 -> #1:0 (rawvideo -> libx264)
Press [q] to stop, [?] for help
On the client side ffplay repeatedly complained about some missing information, but after some seconds it finally showed up the video, which was ok but a bit choppy:
$ ffplay -f h264 udp://224.124.0.1:5000
[h264 # 0xa0be740] non-existing PPS referenced
[h264 # 0xa0be740] non-existing PPS 0 referenced
[h264 # 0xa0be740] decode_slice_header error
[h264 # 0xa0be740] no frame!
(...)
[h264 # 0xa0be740] non-existing PPS referenced
[h264 # 0xa0be740] non-existing PPS 0 referenced
[h264 # 0xa0be740] decode_slice_header error
[h264 # 0xa0be740] no frame!
[h264 # 0xa0e72e0] max_analyze_duration 5000000 reached at 5013967
[h264 # 0xa0e72e0] Estimating duration from bitrate, this may be inaccurate
Input #0, h264, from 'udp://224.124.0.1:5000':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: h264 (Constrained Baseline), yuv420p, 320x240, 47.27 fps, 29.97 tbr, 1200k tbn, 59.94 tbc
[h264 # 0xa0be740] Missing reference picture
[h264 # 0xa0be740] decode_slice_header error
[h264 # 0xa0be740] concealing 300 DC, 300 AC, 300 MV errors
[h264 # 0xa0be740] Missing reference picture 0KB sq= 0B f=0/0 0/0
[h264 # 0xa0be740] decode_slice_header error
[h264 # 0xa0be740] mmco: unref short failure
[h264 # 0xa0be740] concealing 300 DC, 300 AC, 300 MV errors
10.78 A-V: 0.000 fd= 0 aq= 0KB vq= 0KB sq= 0B f=0/0
I then tried VLC with the same ''--demux ffmpeg'' switch; it also complained about SPS/PPS (don't know what it is about yet) but in the end it played the video really smoothly:
$ vlc -v --demux ffmpeg udp/h264://#224.124.0.1:5000
VLC media player 1.1.12 The Luggage (revision exported)
Blocked: call to unsetenv("DBUS_ACTIVATION_ADDRESS")
Blocked: call to unsetenv("DBUS_ACTIVATION_BUS_TYPE")
[0x943346c] main libvlc: Running vlc with the default interface. Use 'cvlc' to use vlc without interface.
Blocked: call to setlocale(6, "")
Blocked: call to setlocale(6, "")
[0x94cde8c] qt4 interface error: Unable to load extensions module
[0x96720e4] h264 demux error: this doesn't look like a H264 ES stream, continuing anyway
[0x963989c] access_udp access warning: unimplemented query in control
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x967328c] packetizer_h264 demux packetizer warning: waiting for SPS/PPS
[0x94be644] main input error: Invalid PCR value in ES_OUT_SET_(GROUP_)PCR !
[0xb2a325f4] avcodec decoder warning: disabling direct rendering
[0x96dfac4] main video output warning: vlc_object_find_name(postproc) is not safe!
[0x94c2474] signals interface warning: signal 17 overridden (0xb6f31030)
[0x94c2474] signals interface warning: /usr/lib/qt/lib/libQtCore.so.4(?)[(nil)]
[0x96dfac4] main video output warning: late picture skipped (32703 > -4)
I'm still trying to figure out what is missing, but the results are pleasing already. After VLC is playing I can stop the feeding ffmpeg that playback pauses and resumes right away when I restart ffmpeg on the other end.
Hope this can be of any help, and please let me know if you have any additional info on using h264 as the chosen format in place of mpegts - I suspect this missing SPS/PPS info might have something to do with it.