I am receiving multiple x264 RTSP streams and I am decoding them using libVLC and I would like to use hardware acceleration for the task.
If I use the VLC player itself on Windows, I can choose "DirectX Video Acceleration (DXVA) 2.0" in Simple Preferences->Input/Codecs->Hardware-accelerated decoding and I can see a significant drop in CPU utilization when compared to disabling that option.
In the C++ code, I tried to add the option "--avcodec-hw=dxva2" to the arguments of libvlc_new() but no luck, hardware acceleration does not seem to be used (I would say the decoding is 50% slower than in the player with dxva2 on)
Found it. The option must be passed to libvlc_media_add_option(), not libvlc_new(). Note that the option changed with different versions of the library. With VLC 2.2.1 it works for me with:
libvlc_media_add_option(m, ":avcodec-hw=dxva2");
where dxva2 stands for the DirectX decoder. In UNIX systems the value of the parameter would be different (vda or vaapi I think)
Related
Currently, I am trying to use opencv to read a video from my Canon VB-H710F camera.
For this purpose I tried two different solutions:
SOLUTION 1: Read the stream from rtsp address
VideoCapture cam ("rtsp://root:camera#10.0.4.127/stream/profile1=u");
while(true)
cam >> frame;
In this case I am using opencv to directly read from a stream encoded with in H264 (profile1), however this yields the same problem reported here http://answers.opencv.org/question/34012/ip-camera-h264-error-while-decoding/
As suggested in the previous question, I tried to disable FFMPEG support in opencv installation, which solved the h264 decoding errors but raised other problem.
When accessing the stream with opencv, supported by gstreame, there is always a large delay associated.
With this solution I achieve 15 FPS but I have a delay of 5 seconds, which is not acceptable considering that I need a real time application.
SOLUTION 2: Read the frames from http address
while(true)
{
startTime=System.currentTimeMillis();
URL url = new URL("h t t p://[IP]/-wvhttp-01-/image.cgi");
URLConnection con = url.openConnection();
BufferedImage image = ImageIO.read(con.getInputStream());
showImage(image);
estimatedTime=System.currentTimeMillis()-startTime;
System.out.println(estimatedTime);
Thread.sleep(5);
}
This strategy simply grabs the frame from the url that the camera provides. The code is in Java but the results are the same in C++ with the curl library.
This solution avoids the delay of the first solution however it takes little more than 100 ms to grab each frame, which means that I can only achieve on average 10 FPS.
I would like to know how can I read the video using c++ or another library developed in c++ ?
I struggled with similar issues and think I have solved some of your problems using libVLC with OpenCV. FFMPEG seemed to have issues of not decoding H264 properly, plus the newer versions (2.4.11) seemed to have the TCP fix in there already for FFMPEG. Anyways, I use MS Visual Studio on Windows 7 and 8.1.
Details are given here: http://answers.opencv.org/question/65932
Personally, I suggest you to use ffmpeg to read rtsp streams from IP cameras, and then use openCV to read from decoded buffer from ffmpeg. ffmpeg has very good optimizations towards H.264 decoding, performance should not be a critical issue.
You can use ffmpeg binary to verify whether this can work correctly:
ffmpeg -i "rtsp://root:camera#10.0.4.127/stream/profile1=u" -vcodec copy -acodec none test.mp4
If test.mp4 can be played successfully, then it's definitely OK for you to integrate ffmpeg libs into your project.
Good luck!
You can process each frame using ffmpeg as well. you need to create your own filter as per your requirement. https://trac.ffmpeg.org/wiki/FilteringGuide
I am using the ExtAudioFile interface to decode audio, and would like to know whether it ever makes use of hardware-assisted audio decoding.
The docs for the kExtAudioFileProperty_CodecManufacturer property say:
Use this property in iOS to choose between a hardware or software encoder, by specifying kAppleHardwareAudioCodecManufacturer or kAppleSoftwareAudioCodecManufacturer.
This seems to indicate that ExtAudioFile can indeed make use of decoding hardware.
Elsewhere, in the Audio Format Services docs, I find:
Hardware-based codecs can be used only when playing or recording using Audio Queue Services or using interfaces, such as AV Foundation, which use Audio Queue Services. In particular, you cannot use hardware-based audio codecs with OpenAL or when using the I/O audio unit.
... which is not completely clear; if ExtAudioFile uses Audio Queue Services in its implementation, then maybe it can make use of hardware, but we don't actually know how it is implemented.
I tried to test at runtime whether the hardware was being used, but this itself proved difficult. One method, given in the Audio Format Services reference, is to use AudioFormatGetProperty to test the kAudioFormatProperty_HardwareCodecCapabilities property. But the sample code does not work, always returning kAudioFormatUnsupportedPropertyError. (After searching online, I found a few other questions from people who have this problem, but no reports of ever using it successfully.)
So... I'm wondering if anyone knows of any way to test for certain whether the hardware decoder is currently active (in which case I could test for myself whether ExtAudioFile is using it). Or alternatively if anyone has any definitive knowledge of whether ExtAudioFile does use hardware (not based solely on the vague mentions in the Apple docs).
Specifying kAppleHardwareAudioCodecManufacturer with ExtAudioFileSetProperty() appears to enable hardware decoding, because it fails with kAudioConverterErr_HardwareInUse when there is already one audio file open with that codec set, and with kAudioQueueErr_InvalidCodecAccess when the audio category is set to one that does not (according to the documentation) enable hardware decoding.
However, after profiling with Instruments, I found that performance was slightly worse with hardware decoding enabled, which I still can't explain...
I'm trying to build an iOS app that plays video files and does some interesting things using MTAudioProcessingTap. I need it to be able to play all sorts of formats, including some that are not supported by Apple. I'm thinking of branching out from VLC, but I can't figure out if it uses Core Audio/Video at any point or if it's running something else completely.
If it's not, is there a library I can use to take care of the 203572964 codecs being used out there?
Thanks.
Preliminary note: I'm the developer of VLC for iOS so the following may be biased.
MobileVLCKit for iOS includes 2 different audio output modules. One of them is a high level module based on AudioQueue which is fairly incomplex but a bit slow. The other is based on AudioUnit, the low level framework of CoreAudio, quite a bit more complex, but way faster. Depending on your current experience, either module would be a good way to start.
Regarding the one library supporting all codecs thing: basically there are two forks of the same library: libav and FFmpeg. VLC supports either flavor and abstracts the complexity and the ever-changing APIs (which are a real pain if you intend to keep maintaining your app across multiple releases of those libraries). Additionally, we include a quite well performing OpenGL ES 2 video output module which is using shaders to do chroma conversation. All you need to do is embedding a UIView. MobileVLCKit handles the rest.
Speaking of MobileVLCKit: this is a thin ObjC layer on top of libvlc simplifying the use of this library in third party applications by abstracting most commonly used features.
As implicitly mentioned by HalR, libvlc does not use hardware accelerated decoding on iOS yet. We are working with the libav developers on a generic approach, but we are not quite there yet. Thus, we have to do all the decoding on the CPU, which leads to the heating but allows us to play virtually anything instead of H264/MP4 using the default, accelerated API.
If you can't figure out how its playing the video, at its lower level, that perhaps is a sign that you should keep working with it instead of trying to outdo it. Video processing is pretty difficult and often unsupported formats are unsupported due to patent issues. I really haven't seen anything better than VLC that is publicly available.
VLC 2.1.x appears to use AudioToolbox and AVFoundation.
One other issue, though, is that when I was doing work with VLC, I was stunned how it turned my iPod Touch into a miniature iron, because it was working so hard to process the video. Manually processing video is very processor intensive and really is a drain. So your way or VLC could still have some additional issues.
I am importing a source code for stereo visions. The next code of the author works. It takes two cameras sources. I have two different cameras currently and i receive images. Both works. It crashes at capture2. interesting part is that if i change the orders of the webcams(Unplugging them and invert the orders) the first camera it will be the second one. We it doesn't work? I tested also with Windows XP sp3 and Windows 7 X64. The same problem.
//---------Starting WebCam----------
capture1= cvCaptureFromCAM(1);
assert(capture1!=NULL); cvWaitKey(100);
capture2= cvCaptureFromCAM(2);
assert(capture2!=NULL);
Also If i use -1 for paramters the just give me the first one(all the time).
Or any method to capture two camers using function cvCaptureFrom
Firstly the cameras are generally numbered from 0 - is this just the problem?
Secondly, directshow and multiple USB webcams is notoriously bad in windows. Sometimes it will work with two identical camera, sometimes only if they are different.
You can also try a delay between initialising the cameras, sometimes one will lock the capture stream until it is sending data, preventing the other being detected.
Often the drivers assume they are the only camera and make incorrect calls to lock up the entire capture graph. This isn't helped by it being extremely complicated to write correct drivers+fdirectshow filters in Windows
some mother board can not work with some usb 2.0 cameras. one usb 2.0 camera take 40-60% of usb controller. solution is connect second usb 2.0 camera from pci2usb controller
Get 2 PS3 Eyes, around EUR 10 each, and the free codelaboratories.com SDK, this gets you support up to 2 cameras using C, C#, Java, and AS3 incl. examples etc. You also get FIXED frame rates up 75 fps # 640*480. Their free driver only version 5.1.1.0177 provides decent DirectShow component, but for a single camera only.
COmment for the rest: Multi-cam DirectShow drivers should be a default for any manufacturer, not providing this is a direct failure to implement THE VERY BASIC PORPUSE AND FEATURE OF USB as an interface. It is also VERY EASY to implement, compared to implementing the driver itself for a particular sensor / chipset.
Alternatives that are confirmed to work in identical pairs (via DirectShow):
Microsoft Lifecam HD Cinema (use general UVC driver if you can, less limited fps)
Logitech Webcam Pro 9000 (not to be confused with QuickCam Pro 9000, which DOES NOT work)
Creative VF0220
Creative VF0330
Canyon WCAMN-1N
If you're serious about your work, get a pair of machine vision cameras to get PERFORMANCE. Cheapest on the market, with german engineering quality, CCD, CMOS, mono, colour, GigE (ethernet), USB, FireWire, excellent range of dedicated drivers:
http://www.theimagingsource.com
I was planning to use the vlc library to decode an H.264 based RTSP stream and extract each frame from it (convert vlc picture to IplImage). I have done a bit of exploration of the vlc code and concluded that there is a function called libvlc_video_take_snapshot which does a similar thing. However the captured frame in this case is saved on the hard disk which I wish to avoid due to the real time nature of my application. What would be the best way to do this? Would it be possible without modifying the vlc source (I want to avoid recompilation if possible). I have heard of vmem etc but could not really figure out what it does and how to use it.
The picture_t structure is internal to the library, how can we get an access to the same.
Awaiting your response.
P.S. Earlier I tried doing this using FFMPEG, however the ffmpeg library has a lot of issues while decoding an H.264 based RTSP stream on windows and hence I had to switch to VLC.
Regards,
Saurabh Gandhi