Is it safe to call [EAGLContext presentRenderBuffer:] on a secondary thread? - ios

I have (multiple) UIViews with layers of type CAEAGLLayer, and am able to call [EAGLContext presentRenderBuffer:] on renderbuffers attached to these layers, on a secondary thread, without any kind of graphical glitches.
I would have expected to see at least some tearing, since other UI with which these UIViews are composited is updated on the main thread.
Does CAEAGLLayer (I have kEAGLDrawablePropertyRetainedBacking set to NO) do some double-buffering behind the scenes?
I just want to understand why it is that this works...
Example:
BView is a UIView subclass that owns a framebuffer with renderbuffer storage assigned to its OpenGLES layer, in a shared EAGLContext:
#implementation BView
-(id) initWithFrame:(CGRect)frame context:(EAGLContext*)context
{
self = [super initWithFrame:frame];
// Configure layer
CAEAGLLayer* eaglLayer = (CAEAGLLayer*)self.layer;
eaglLayer.opaque = YES;
eaglLayer.drawableProperties = #{ kEAGLDrawablePropertyRetainedBacking : [NSNumber numberWithBool:NO], kEAGLDrawablePropertyColorFormat : kEAGLColorFormatSRGBA8 };
// Create framebuffer with renderbuffer attached to layer
[EAGLContext setCurrentContext:context];
glGenFramebuffers( 1, &FrameBuffer );
glBindFramebuffer( GL_FRAMEBUFFER, FrameBuffer );
glGenRenderbuffers( 1, &RenderBuffer );
glBindRenderbuffer( GL_RENDERBUFFER, RenderBuffer );
[context renderbufferStorage:GL_RENDERBUFFER fromDrawable:(id<EAGLDrawable>)self.layer];
glFramebufferRenderbuffer( GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, RenderBuffer );
return self;
}
+(Class) layerClass
{
return [CAEAGLLayer class];
}`
A UIViewController adds a BView instance on the main thread at init time:
BView* view = [[BView alloc] initWithFrame:(CGRect){ 0.0, 0.0, 75.0, 75.0 } context:Context];
[self.view addSubView:view];
On a secondary thread, render to the framebuffer in the BView and present it; in this case it's in a callback from a video AVCaptureDevice, called regularly:
-(void) captureOutput:(AVCaptureOutput*)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection*)connection
{
[EAGLContext setCurrentContext:bPipe->Context.GlContext];
// Render into framebuffer ...
// Present renderbuffer
glBindRenderbuffer( GL_RENDERBUFFER, BViewsRenderBuffer );
[Context presentRenderbuffer:GL_RENDERBUFFER];
}

It used to not work. There used to be several issues with updating the view if the buffer was presented on any but the main thread. It seems this has been working for some time now but it is on your own risk to implement it as you do. Later versions may have issues with it as well as some older probably still do (not that you need to support some old OS versions anyway).
Apple was always a bit closed as to how things work internally but we may guess quite a few things. Since iOS seems to be the only platform that uses your main buffer as a FBO (frame buffer object) I would expect the main frame buffer is inaccessible for development and your main FBO is actually redrawn to the main frame buffer when you present the render buffer. The last time I checked the method to present the render buffer will block your current thread and seems to be limited to the screen refresh rate (60FPS in most cases) which implies there is still some locking mechanism. Some additional test should be done but I would expect there is some sort of a pool of buffers which need to be redrawn to the main buffer where in the pool only one unique buffer id can be present at the time or the calling thread is blocked. This would result in the first call to the present render buffer would not be blocked at all but each sequential would be if the previous buffer has not yet been redrawn.
If this is true then yes, a double buffering is mandatory at some point since you may immediately continue drawing to your buffer. Since the render buffer has the same id over the frames it may not be swapped (for what I know) but it could be redrawn/copied to another buffer (most likely a texture) which can be done on the fly at any given time. In this procedure then when you first present the buffer you will copy the buffer to the texture which will be locked. When the screen refreshes the texture will be collected and unlocked. So if this texture is locked your presentation call will block the thread, otherwise it will continue smoothly. It is hard to say this is double buffering then. It has 2 buffers but it still works with a locking mechanism.
I do hope you may then understand why it works. It is pretty much the same procedure you would use when loading large data structures on the separate shared context which runs on a separate thread.
Still most of this is just guessing unfortunately.

Related

GLKView with stencil buffer's glDrawElements results in black screen

I have an iOS app that uses GLKViewController and I set up the render buffer as follows:
inside
#interface RootViewController : GLKViewController<UIKeyInput>
- viewDidLoad {
[super viewDidLoad];
_context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
GLKView* view = (GLKView*)self.view;
view.context = _context;
view.drawableColorFormat = GLKViewDrawableColorFormatRGBA8888;
view.drawableDepthFormat = GLKViewDrawableDepthFormat24;
view.drawableStencilFormat = GLKViewDrawableStencilFormat8;
view.drawableMultisample = GLKViewDrawableMultisampleNone;
self.preferredFramesPerSecond = 60;
[EAGLContext setCurrentContext:_context];
}
However, when I call draw later:
glDrawElements(getGlPrimitiveType(ePrimType), numIndis, GL_UNSIGNED_SHORT, startIndis);
It results in black screen and upon Capture GPU Frame, this error shows up:
Your app rendered with STENCIL_TEST enabled into a framebuffer without an attached stencil buffer.
Is there anything that I missed?
I remembered having the same problem before due to depth testing, and I fix it with the view.drawableDepthFormat = GLKViewDrawableDepthFormat24; on viewDidLoad I am not sure about stencil testing, Apple's documentation is either very minimal or very general with theories all around (i.e: pretty much useless).
I found the culprit,
I've lost the original FBO-ID already setup by GLKView when I do render to texture:
uint m_nFboId;
glGenFramebuffers(1, &m_nFboId);
glBindFramebuffer(GL_FRAMEBUFFER, m_nFboId);
then, when I try to reset back to the original FBO-ID:
GLint defaultFBO;
glGetIntegerv(GL_FRAMEBUFFER_BINDING, &defaultFBO);
here, the defaultFBO is the value of m_nFboId generated before.. So the solution is to either back it up before the operation, or call back [GLKView bindDrawable];

CIImage and CIDetector use with AVCaptureOutput memory leak

I'm using a CIContext, CIDetector, and CIImage to detect rectangles in a vImage_Buffer derived from samples in captureOutput:didOutputSampleBuffer:fromConnection:. It seems that either the detector or the CIImage is retaining memory and it cannot be released.
Here is the code in question - skipping over this code shows memory held constant, otherwise in increases until crashing the app:
// ...rotatedBuf and format managed outside scope
// Use a CIDetector to find any potential subslices to process
CVPixelBufferRef cvBuffer;
vImageCVImageFormatRef cvFormat = vImageCVImageFormat_CreateWithCVPixelBuffer(pixelBuffer);
CVPixelBufferCreate(kCFAllocatorSystemDefault, rotatedBuf.width, rotatedBuf.height, kCVPixelFormatType_32BGRA, NULL, &cvBuffer);
CVPixelBufferLockBaseAddress(cvBuffer, kCVPixelBufferLock_ReadOnly);
err = vImageBuffer_CopyToCVPixelBuffer(&rotatedBuf, &format, cvBuffer, cvFormat, NULL, kvImageNoFlags);
CVPixelBufferUnlockBaseAddress(cvBuffer, kCVPixelBufferLock_ReadOnly);
if (![self vImageDidError:err]) {
CIImage *ciImage = [CIImage imageWithCVPixelBuffer:cvBuffer];
NSArray *feats = [self.ciDetector featuresInImage:ciImage options:nil];
if (feats && [feats count]) {
for (CIFeature *feat in feats) {
// The frame is currently in image space, so we must convert it to a unitless space like the other rects.
CGRect frame = feat.bounds;
CGRect clip = CGRectMake(frame.origin.x / rotatedBuf.width, frame.origin.y / rotatedBuf.height,
frame.size.width / rotatedBuf.width, frame.size.height / rotatedBuf.height);
rects = [rects arrayByAddingObject:[NSValue valueWithCGRect:clip]];
}
}
}
CVPixelBufferRelease(cvBuffer);
vImageCVImageFormat_Release(cvFormat);
Other answers seem to suggest wrapping in an autorelease pool or create a new CIDector each frame, but neither affect the memory use.
CIDetector isn't releasing memory
CIDetector won't release memory - swift
Edit: switching the dispatch queue to one other than dispatch_main_queue seemed to have cleared the memory issue and keeps the UI responsive.
I figured out a different solution - in my case I was running all my processing on the main dispatch queue. What fixed the situation was creating a new queue to run the processing in. I realized this may be the case when the majority of my CPU time was spent on the call to featuresInImage:options:. It doesn't explain what caused the memory issue, but now that I'm running in a separate queue, memory is nice and constant.

iOS AIR Native Extension With OpenGLES Causes App To Freeze

I created a Air Native Extension for iOS that opens a GLKViewController with a GLKView inside to render some 3D content, this all works fine. When the ViewController is dismissed, the AIR App has stopped rendering. Interaction still works (so the app is not frozen), but the rendering has stopped on the last frame before the native extension's view controller opened.
This is code from my view controller's viewDidLoad method (this is all the subclass does):
GLKView* view = (GLKView*)self.view;
if( view.context == nil )
{
EAGLContext *context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];
if (!context)
{
NSLog(#"Failed to create ES context");
return;
}
view.context = context; //Removing this fixes flash!
}
When I comment the view.context = context line then Flash will continue rendering fine (but obviously, I no longer have a context and can't render).
I assume Flash is losing it's EAGLContext when the GLKView sets the current context and is not resetting it. Is there a way I can fix or avoid this?
I have tried to save the current EAGLContext before opening the view controller, then to reset it when the view controller is closed, but that did not work.
To fix this I had to save Flash's EAGLContext before setting my own, then restore Flash's one after I had finished drawing/setting up the context. This let Flash continue to draw without knowing that my view controller was also drawing. I ended up doing this using the old EAGLView and a custom view controller as it wasn't clear where the GLKView was setting the context.
Of course, it would have been better for Flash to set the context itself before it tried to draw a frame, like Apple suggest!

A faster way to update interface using background threads

I'm building a spectrograph and would like to know how I can improve the performance of my UIView-based code. I know that I cannot update user interface for iPhone/iPad from a background thread, so I'm doing most of my processing using GCD. The issue that I'm running into is that my interface still updates way too slowly.
With the code below, I'm trying to take 32 stacked 4x4 pixel UIViews and change their background color (see the green squares on the attached image). The operation produces visible lag for other user interface.
Is there a way I can "prepare" these colors from some kind of background thread and then ask the main thread to refresh the interface all at once?
//create a color intensity map used to color pixels
static dispatch_once_t onceToken;
dispatch_once(&onceToken, ^{
colorMap = [[NSMutableDictionary alloc] initWithCapacity:128];
for(int i = 0; i<128; i ++)
{
[colorMap setObject:[UIColor colorWithHue:0.2 saturation:1 brightness:i/128.0 alpha:1] forKey:[NSNumber numberWithInt:i]];
}
});
-(void)updateLayoutFromMainThread:(id)sender
{
for(UIView* tempView in self.markerViews)
{
tempView.backgroundColor =[colorMap objectForKey:[NSNumber numberWithInt:arc4random()%128]];
}
}
//called from background, would do heavy processing and fourier transforms
-(void)updateLayout
{
//update the interface from the main thread
[self performSelectorOnMainThread:#selector(updateLayoutFromMainThread:) withObject:nil waitUntilDone:NO];
}
I ended up pre-calculating a dictionary of 256 colors and then asking the dictionary for the color based on the value that the circle is trying to display. Trying to allocate colors on the fly was the bottleneck.
, Yes, a couple of points.
While you shouldn't process UIView on the main thread, you can instantiate views on a background thread before using them. Not sure if that will help you at all. However beyond instantiating a view on a background thread, UIView's are really just a meta-data wrapper for CALayer objects and are optimised for flexibility rather than performance.
Your best bet is to draw to a layer object or an image object on a background thread (which is a slower process because drawing uses the CPU as well as the GPU), pass the layer object or image to the main thread, then draw the pre-rendered image to your view's layer (much faster because a simple call is made to get the Graphics Processor to blit the image to the UIView's backing store directly).
see this answer:
Render to bitmap then blit to screen
The code:
- (void)drawRect:(CGRect)rect {
CGContextRef context = UIGraphicsGetCurrentContext();
CGContextDrawImage(context, rect, image);
}
executes far faster than if you were to execute other drawing operations, such as drawing bezier curves, in the same method.

AV Foundation: AVCaptureVideoPreviewLayer and frame duration

I am using the AV Foundation to process frames from the video camera (iPhone 4s, iOS 6.1.2). I am setting up AVCaptureSession, AVCaptureDeviceInput, AVCaptureVideoDataOutput per the AV Foundation programming guide. Everything works as expected and I am able to recieve frames in the captureOutput:didOutputSampleBuffer:fromConnection: delegate.
I also have a preview layer set like this:
AVCaptureVideoPreviewLayer *videoPreviewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:_captureSession];
[videoPreviewLayer setFrame:self.view.bounds];
videoPreviewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
[self.view.layer insertSublayer:videoPreviewLayer atIndex:0];
Thing is, I don't need 30 frames per second in my frame handling and I am not able to process them so fast anyway. So I am using this code to limit the frame duration:
// videoOutput is AVCaptureVideoDataOutput set earlier
AVCaptureConnection *conn = [videoOutput connectionWithMediaType:AVMediaTypeVideo];
[conn setVideoMinFrameDuration:CMTimeMake(1, 10)];
[conn setVideoMaxFrameDuration:CMTimeMake(1, 2)];
This works fine and limits the frames recieved by the captureOutput delegate.
However, this also limits the frames per second on the preview layer and preview video becomes very unresponsive.
I understand from the documentation that the frame duration is set independently on the connection and the preview layer has indeed a different AVCaptureConnection. Checking the mix/max frame durations on [videoPreviewLayer connection] shows that it's indeed set to the defaults (1/30 and 1/24) and different than the durations set on the connection of the AVCaptureVideoDataOutput.
So, is it possible to limit the frame duration only on the frame capturing output and still see a 1/24-1/30 frame duration on the preview video? How?
Thanks.
While you're correct that there are two AVCaptureConnections, that doesn't mean they can have independently set the minimum and maximum frame durations. This is because they are sharing the same physical hardware.
If connection #1 is activating the rolling shutter at a rate of (say) five frames/sec with a frame duration of 1/5 sec, there is no way that connection #2 can simultaneously activate the shutter 30 times/sec with a frame duration of 1/30 sec.
To get the effect you want would require two cameras!
The only way to get close to what you want is to follow an approach along the lines of that outlined by Kaelin Colclasure in the answer of 22 March.
You do have options of being a little more sophisticated within that approach, however. For example, you can use a counter to decide which frames to drop, rather than making the thread sleep. You can make that counter respond to the actual frame-rate that's coming through (which you can get from the metadata that comes in to the captureOutput:didOutputSampleBuffer:fromConnection: delegate along with the image data, or which you can calculate yourself by manually timing the frames). You can even do a very reasonable imitation of a longer exposure by compositing frames rather than dropping them—just as a number of "slow shutter" apps in the App Store do (leaving aside details—such as differing rolling shutter artefacts—there's not really that much difference between one frame scanned at 1/5 sec and five frames each scanned at 1/25 sec and then glued together).
Yes, it's a bit of work, but you are trying to make one video camera behave like two, in real time—and that's never going to be easy.
Think of it this way:
You ask the capture device to limit frame duration, so you get better exposure.
Fine.
You want to preview at higher frame rate.
If you were to preview at higher rate, then the capture device (the camera) would NOT have enough time to expose the frame so you get better exposure at the captured frames.
It is like asking to see different frames in preview than the ones captured.
I think that, if it was possible, it would also be a negative user experience.
I had the same issue for my Cocoa (Mac OS X) application. Here's how I solved it:
First, make sure to process the captured frames on a separate dispatch queue. Also make sure any frames you're not ready to process are discarded; this is the default, but I set the flag below anyway just to document that I'm depending on it.
videoQueue = dispatch_queue_create("com.ohmware.LabCam.videoQueue", DISPATCH_QUEUE_SERIAL);
videoOutput = [[AVCaptureVideoDataOutput alloc] init];
[videoOutput setAlwaysDiscardsLateVideoFrames:YES];
[videoOutput setSampleBufferDelegate:self
queue:videoQueue];
[session addOutput:videoOutput];
Then when processing the frames in the delegate, you can simply have the thread sleep for the desired time interval. Frames that the delegate is not awake to handle are quietly discarded. I implement the optional method for counting dropped frames below just as a sanity check; my application never logs dropping any frames using this technique.
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didDropSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection;
{
OSAtomicAdd64(1, &videoSampleBufferDropCount);
}
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
fromConnection:(AVCaptureConnection *)connection;
{
int64_t savedSampleBufferDropCount = videoSampleBufferDropCount;
if (savedSampleBufferDropCount && OSAtomicCompareAndSwap64(savedSampleBufferDropCount, 0, &videoSampleBufferDropCount)) {
NSLog(#"Dropped %lld video sample buffers!!!", savedSampleBufferDropCount);
}
// NSLog(#"%s", __func__);
#autoreleasepool {
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CIImage * cameraImage = [CIImage imageWithCVImageBuffer:imageBuffer];
CIImage * faceImage = [self faceImage:cameraImage];
dispatch_sync(dispatch_get_main_queue(), ^ {
[_imageView setCIImage:faceImage];
});
}
[NSThread sleepForTimeInterval:0.5]; // Only want ~2 frames/sec.
}
Hope this helps.

Resources