working on face tracking system and have problem with OpenCV.
If I instantiate the Capture and call QueryFrame() i get 640x480 image resolution. Everything is silky smooth.
_grabber = new Capture();
_grabber.QueryFrame();
However, if i try to increase the resolution lets say to 800x600 the fps decreases drastically.
_grabber = new Capture();
_grabber.SetCaptureProperty(CAP_PROP.CV_CAP_PROP_FRAME_HEIGHT, LiveFeedSize.Height);
_grabber.SetCaptureProperty(CAP_PROP.CV_CAP_PROP_FRAME_WIDTH, LiveFeedSize.Width);
_grabber.QueryFrame();
I am setting the width/height before every QueryFrame().
Anyone with ideas how to increase the frames per seconds? Thanks
so after long night I find the solution to the problem. The camera itself is able to work with full resolution, however, the problem was in this line of code
MCvAvgComp[][] facesDetected = _gray.DetectHaarCascade(_face, 1.2, 10, HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(150, 150));
where the last size parameter was small (27*27) so it takes significant amount of time to calculate. The solution was to increase the value to 150 which is basically minimal face detected size
Related
I'm using captureImage of ARFrame in ARKit to generate signatures for human faces and compare the similarities. However, due to the poor resolution of the captureImage, the quality of the comparison is negatively affected compared to the Vision framework.
capturedDepthData shows that the resolution of the capturedImage in ARKit is only 640x480.
First, I've tried improving the video format to the highest resolution:
let configuration = ARFaceTrackingConfiguration()
if let videoFormat = ARFaceTrackingConfiguration.supportedVideoFormats.sorted(by: { ($0.imageResolution.width * $0.imageResolution.height) < ($1.imageResolution.width * $1.imageResolution.height) }).last {
configuration.videoFormat = videoFormat
}
But, this does not seem to improve the resolution of the capturedImage.
Second, I tried using captureHighResolutionFrame and as well as change the video format:
if let videoFormat = ARFaceTrackingConfiguration.recommendedVideoFormatForHighResolutionFrameCapturing {
configuration.videoFormat = videoFormat
}
However, according to the documentation:
The system delivers a high-resolution frame out-of-band, which means that it doesn't affect the other frames that the session receives at a regular interval
which seems to toggle back and forth between the regular capturedImage and the high resolution images, instead of replacing the regular images due to the asynchronous nature of capturing the high resolution images. This is problematic because the size differences require displayTransform and CGAffineTransform to be used differently for scaling the images for each cases.
On top of that, the capturing of the images with this method creates the shutter sound at 60 times per second.
I did not try this specific model but I hope it's doing good work since people are using it, and the transformer-based models achieved excellent scores in NLP.
Here is the link for one of them, specialized in enhancing image resolution
model. Hope this work well for your case.
I am trying to read a high resolution video using OpenCV VideoCapture and it seems to be extremely slow. I read somewhere changing the buffer sizes might help, but I tried setting all kinds of buffer sizes and its still slow.
Any help here on what settings can help improve reading high resolution videos for java opencv is really appreciated
I am using VideoCapture to read a video from disk. I am using Mac OSX. Here is a snippet of my code:
while(camera.read(frame))
{
BufferedImage bufferedImage = MatToBufferedImage(frame);
BufferedImage scaledImage = (BufferedImage)getScaledImage(bufferedImage, frameWidth, frameHeight);
ImageIcon icon = new ImageIcon(scaledImage);
publish(icon);
}
I am using swingworker and doing this in the background thread.
I am not explicitly setting any openCV properties. Should I be setting any explicit properties is something I am not sure of.
Here is what I observe: My video starts off well and then around 50th frame or so, i see some lag and then again at around 120th frame and it almost completely stops at frame number 190ish.
Have you tried resizing the individual frames?
while(camera.read(frame))
{
resize(frame,frame,Size(640,360));
imshow("frame",frame);
}
I run into an optimization problem. I have a to display a lot of image a the same time on the screen (up to 150). These image are displayed like if it was a sphere, that can roll.
So far it run 60 fps on a new device, but there is some micro lags and I don't know what is the cause. The profiler tell me 59-60 fps but the problem is that the missing fps is visible and fell like a micro-slutter.
If I reduce the number of imageViews drastically, this problem disappear.
It doesn't seems to be gpu-limited, because I use something like 30% at most of the gpu, but the cpu never goes to more than 50% neither.
What could cause that/how can I detect the cause of this micro lag ? (the image are inited with contentOfFile)
What is the best solution to improve performance knowing that at least half of the image are not on the screen (other side of the sphere) ?
-Recycle UIImageViews
-Set not showed imageView to hidden (it's what i'm doing right now)
-Use rasterization layer (But since the size is changing everytime, it will re-render, right ?)
-Another magic solution that i'm not aware of.
If you want to try the app :
https://itunes.apple.com/us/app/oolala-instant-hangout-app/id972713282?mt=8
1) The way those shapes are created is also important. The most performant way to do that (that's I'm aware of) would be to pre-render images with alpha channel.
2) There is also a "classic" jpeg decompression problem
3) We used CALayer directly to render ~300 images in a UIScrollView on iPhone 4S screen without dropped frames. Might give it a shot.
_layer = [CALayer new];
_layer.contents = (id)_image.CGImage;
_layer.opaque = _opaque;
_layer.contentsScale = [UIScreen mainScreen].scale;
It was a long time ago and was used on iOS 5. You should probably disregard this suggestion unless you have a fast way to test it.
To clarify, I know that a texture atlas improves performance when using multiple distinct images. But I'm interested in how things are done when you are not doing this.
I tried doing some frame-by-frame animation manually in custom OpenGL where each frame I bind a new texture and draw it on the same point sprite. It works, but it is very slow compared to the UIImageView ability to abstract the same. I load all the textures up front, but the rebinding is done each frame. By comparison, UIImageView accepts the individual images, not a texture atlas, so I'd imagine it is doing similarly.
These are 76 images loaded individually, not as a texture atlas, and each is about 200px square. In OpenGL, I suspect the bottleneck is the requirement to rebind a texture at every frame. But how is UIImageView doing this as I'd expect a similar bottleneck?? Is UIImageView somehow creating an atlas behind the scenes so no rebinding of textures is necessary? Since UIKit ultimately has OpenGL running beneath it, I'm curious how this must be working.
If there is a more efficient means to animate multiple textures, rather than swapping out different bound textures each frame in OpenGL, I'd like to know, as it might hint at what Apple is doing in their framework.
If I did in fact get a new frame for each of 60 frames in a second, then it would take about 1.25 seconds to animate through my 76 frames. Indeed I get that with UIImageView, but the OpenGL is taking about 3 - 4 seconds.
I would say your bottleneck is somewhere else. The openGL is more then capable doing an animation the way you are doing. Since all the textures are loaded and you just bind another one each frame there is no loading time or anything else. Consider for a comparison I have an application that can in runtime generate or delete textures and can at some point have a great amount of textures loaded on the GPU, I have to bind all those textures every frame (not 1 every frame), using all from depth buffer, stencil, multiple FBOs, heavy user input, about 5 threads bottlenecked into 1 to process all the GL code and I have no trouble with the FPS at all.
Since you are working with the iOS I suggest you run some profilers to see what code is responsible for the overhead. And if for some reason your time profiler will tell you that the line with glBindTexture is taking too long I would still say that the problem is somewhere else.
So to answer your question, it is normal and great that UIImageView does its work so smoothly and there should be no problem achieving same performance with openGL. THOUGH, there are a few things to consider at this point. How can you say that image view does not skip images, you might be setting a pointer to a different image 60 times per second but the image view might just ask itself 30 times per second to redraw and when it does just uses a current image assigned to it. On the other hand with your GL code you are forcing the application to do the redraw 60FPS regardless to if it is capable of doing so.
Taking all into consideration, there is a thing called display link that apple developers created for you. I believe it is meant for exactly what you want to do. The display link will tell you how much time has elapsed between frames and by that key you should ask yourself what texture to bind rather then trying to force them all in a time frame that might be too short.
And another thing, I have seen that if you try to present render buffer at 100 FPS on most iOS devices (might be all), you will only get 60 FPS as the method to present render buffer will pause your thread if it has been called in less then 1/60s. That being said it is rather impossible do display anything at all at 60 FPS on iOS devices and everything running 30+ FPS is considered good.
"not as a texture atlas" is the sentence that is a red flag for me.
USing a texture atlas is a good thing....the texture is loaded into memory once and then you just move the rectangle position to play the animation. It's fast because its already all in memory. Any operation which involves constantly loading and reloading new image frames is going to be slower than that.
You'd have to post source code to get any more exact an answer than that.
I am trying to process the camera frames returned by the callback didOutputSampleBuffer, and I want a high frame rate to capture sudden changes in the image (like a flash light going off). The rate at which the callback is called seems to be independent of the frame rate set for the connection. Even if I set frame rate to say 60 via videoOut.minFrameDuration = CMTimeMake(1, 60);, the interval between two consecutive didOutputSampleBuffer seems to be around 60 - 80mS (which is a frame rate of around 17 - 12fps. Why is that so? Is it possible to increase it?
Looks like I have found the answer to the question. Basically the frame rate is affected by the preset you use to select the resolution of the image. So for 1MP resolution, you can get a faster frame rate compared to 8MP or and so on. So even if you try to set a minFrameDuration of 160, the hardware gives you what it can based on your resolution setting, which is probably fps of 15 or less.
So to increase the FPS, decrease the resolution along with increasing the minFrameDuration property.