I just stumbled over a quite tricky issue.
In context of a openGL app for iOS I tried to call glReadPixels.
Therefore, a global buffer variable was created/allocated once at the beginning.
I tried to use the glReadPixel-Function on that buffer, but it did not succeed. I did not get any new picture, just crap.
so my question: Why do I need to use a free() on my allocated buffer space, when I want to use the location of that memory a lot of times before I finally free it?
See for example:
int bytes = width*height*3; //Color space is RGB
if(buffer == null)
buffer = (GLubyte *)malloc(bytes);
glReadPixels(0, 0, width, height, GL_RGB, GL_UNSIGNED_BYTE, buffer);
free(buffer);
EDIT: I replaced free(bytes); with free(buffer);
Related
I have received a CMSampleBufferRef from a system API that contains CVPixelBufferRefs that are not RGBA (linear pixels). The buffer contains planar pixels (such as 420f aka kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange aka yCbCr aka YUV).
I would like to modify do some manipulation of this video data before sending it off to VideoToolkit to be encoded to h264 (drawing some text, overlaying a logo, rotating the image, etc), but I'd like for it to be efficient and real-time. Buuuut planar image data looks suuuper messy to work with -- there's the chroma plane and the luma plane and they're different sizes and... Working with this on a byte level seems like a lot of work.
I could probably use a CGContextRef and just paint right on top of the pixels, but from what I can gather it only supports RGBA pixels. Any advice on how I can do this with as little data copying as possible, yet as few lines of code as possible?
CGBitmapContextRef can only paint into something like 32ARGB, correct. This means that you will want to create ARGB (or RGBA) buffers, and then find a way to very quickly transfer YUV pixels onto this ARGB surface. This recipe includes using CoreImage, a home-made CVPixelBufferRef through a pool, a CGBitmapContextRef referencing your home made pixel buffer, and then recreating a CMSampleBufferRef resembling your input buffer, but referencing your output pixels. In other words,
Fetch the incoming pixels into a CIImage.
Create a CVPixelBufferPool with the pixel format and output dimensions you are creating. You don't want to create CVPixelBuffers without a pool in real time: you will run out of memory if your producer is too fast; you'll fragment your RAM as you won't be reusing buffers; and it's a waste of cycles.
Create a CIContext with the default constructor that you'll share between buffers. It contains no external state, but documentation says that recreating it on every frame is very expensive.
On incoming frame, create a new pixel buffer. Make sure to use an allocation threshold so you don't get runaway RAM usage.
Lock the pixel buffer
Create a bitmap context referencing the bytes in the pixel buffer
Use CIContext to render the planar image data into the linear buffer
Perform your app-specific drawing in the CGContext!
Unlock the pixel buffer
Fetch the timing info of the original sample buffer
Create a CMVideoFormatDescriptionRef by asking the pixel buffer for its exact format
Create a sample buffer for the pixel buffer. Done!
Here's a sample implementation, where I have chosen 32ARGB as the image format to work with, as that's something that both CGBitmapContext and CoreVideo enjoys working with on iOS:
{
CGPixelBufferPoolRef *_pool;
CGSize _poolBufferDimensions;
}
- (void)_processSampleBuffer:(CMSampleBufferRef)inputBuffer
{
// 1. Input data
CVPixelBufferRef inputPixels = CMSampleBufferGetImageBuffer(inputBuffer);
CIImage *inputImage = [CIImage imageWithCVPixelBuffer:inputPixels];
// 2. Create a new pool if the old pool doesn't have the right format.
CGSize bufferDimensions = {CVPixelBufferGetWidth(inputPixels), CVPixelBufferGetHeight(inputPixels)};
if(!_pool || !CGSizeEqualToSize(bufferDimensions, _poolBufferDimensions)) {
if(_pool) {
CFRelease(_pool);
}
OSStatus ok0 = CVPixelBufferPoolCreate(NULL,
NULL, // pool attrs
(__bridge CFDictionaryRef)(#{
(id)kCVPixelBufferPixelFormatTypeKey: #(kCVPixelFormatType_32ARGB),
(id)kCVPixelBufferWidthKey: #(bufferDimensions.width),
(id)kCVPixelBufferHeightKey: #(bufferDimensions.height),
}), // buffer attrs
&_pool
);
_poolBufferDimensions = bufferDimensions;
assert(ok0 == noErr);
}
// 4. Create pixel buffer
CVPixelBufferRef outputPixels;
OSStatus ok1 = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(NULL,
_pool,
(__bridge CFDictionaryRef)#{
// Opt to fail buffer creation in case of slow buffer consumption
// rather than to exhaust all memory.
(__bridge id)kCVPixelBufferPoolAllocationThresholdKey: #20
}, // aux attributes
&outputPixels
);
if(ok1 == kCVReturnWouldExceedAllocationThreshold) {
// Dropping frame because consumer is too slow
return;
}
assert(ok1 == noErr);
// 5, 6. Graphics context to draw in
CGColorSpaceRef deviceColors = CGColorSpaceCreateDeviceRGB();
OSStatus ok2 = CVPixelBufferLockBaseAddress(outputPixels, 0);
assert(ok2 == noErr);
CGContextRef cg = CGBitmapContextCreate(
CVPixelBufferGetBaseAddress(outputPixels), // bytes
CVPixelBufferGetWidth(inputPixels), CVPixelBufferGetHeight(inputPixels), // dimensions
8, // bits per component
CVPixelBufferGetBytesPerRow(outputPixels), // bytes per row
deviceColors, // color space
kCGImageAlphaPremultipliedFirst // bitmap info
);
CFRelease(deviceColors);
assert(cg != NULL);
// 7
[_imageContext render:inputImage toCVPixelBuffer:outputPixels];
// 8. DRAW
CGContextSetRGBFillColor(cg, 0.5, 0, 0, 1);
CGContextSetTextDrawingMode(cg, kCGTextFill);
NSAttributedString *text = [[NSAttributedString alloc] initWithString:#"Hello world" attributes:NULL];
CTLineRef line = CTLineCreateWithAttributedString((__bridge CFAttributedStringRef)text);
CTLineDraw(line, cg);
CFRelease(line);
// 9. Unlock and stop drawing
CFRelease(cg);
CVPixelBufferUnlockBaseAddress(outputPixels, 0);
// 10. Timings
CMSampleTimingInfo timingInfo;
OSStatus ok4 = CMSampleBufferGetSampleTimingInfo(inputBuffer, 0, &timingInfo);
assert(ok4 == noErr);
// 11. VIdeo format
CMVideoFormatDescriptionRef videoFormat;
OSStatus ok5 = CMVideoFormatDescriptionCreateForImageBuffer(NULL, outputPixels, &videoFormat);
assert(ok5 == noErr);
// 12. Output sample buffer
CMSampleBufferRef outputBuffer;
OSStatus ok3 = CMSampleBufferCreateForImageBuffer(NULL, // allocator
outputPixels, // image buffer
YES, // data ready
NULL, // make ready callback
NULL, // make ready refcon
videoFormat,
&timingInfo, // timing info
&outputBuffer // out
);
assert(ok3 == noErr);
[_consumer consumeSampleBuffer:outputBuffer];
CFRelease(outputPixels);
CFRelease(videoFormat);
CFRelease(outputBuffer);
}
Using openGL-ES3, running on an iPhone5s (hardware, not in the simulator) in Xcode 7.3 I receive an "invalid framebuffer operation" when doing a glClear.
The texture in question is a "final" texture for my GBuffer, much like in this tutorial http://ogldev.atspace.co.uk/www/tutorial37/tutorial37.html.
Key difference being that I'm requesting an sRGB texture and that I use GL_COLOR_ATTACHMENT3 (instead of 4), due to ES3 limitations.
glTexImage2D(GL_TEXTURE_2D, 0, GL_SRGB8, WindowWidth, WindowHeight, 0, GL_RGB, GL_UNSIGNED_BYTE, NULL);
// glTexParameteri ...
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT3, GL_TEXTURE_2D, m_finalTexture, 0);
GLenum Status = glCheckFramebufferStatus(GL_FRAMEBUFFER); // No errors here
Now when I try to clear it, I get an "invalid framebuffer operation"
glBindFramebuffer(GL_DRAW_FRAMEBUFFER, m_fbo);
// Element at index**[i]** needs to match GL_COLOR_ATTACHMENT**i** on GL-ES3!
GLenum drawbuf[4] = { GL_NONE, GL_NONE, GL_NONE, GL_COLOR_ATTACHMENT3 };
glDrawBuffers(sizeof(drawbuf)/sizeof(drawbuf[0]), drawbuf);
GLCheckError(); // no errors
glClear(GL_COLOR_BUFFER_BIT);
GLCheckError(); // => glGetError 506 GL_INVALID_FRAMEBUFFER_OPERATION
Now if instead I initialise the texture like this (so without sRGB), OpenGL doesn't give an error on the clear:
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB8, WindowWidth, WindowHeight, 0, GL_RGB, GL_UNSIGNED_BYTE, NULL);
Now as I understood it, sRGB is supported on OpenGL ES3... so why does glClear fail?
Any ideas anyone?
GL_SRGB8 is not a color-renderable format in ES 3.0. In the spec document:
In the "Required Texture Format" section starting on page 128, SRGB8 is listed under "Texture-only color formats".
In table 3.13, starting on page 130, SRGB8 does not have a checkmark in the "Color-renderable" column.
This also matches the EXT_srgb extension specification, under "Issues":
Do we require SRGB8_EXT be supported for RenderbufferStorage?
No. Some hardware would need to pad this out to RGBA and instead of adding that unknown for application developers we will simply not support that format in this extension.
glCheckFramebufferStatus() should return GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT in this case. If it's not doing that, that looks like a bug in the OpenGL implementation.
The closest alternative that is color-renderable is GL_SRGB8_ALPHA8.
try this
#define GL_COLOR_BUFFER_BIT 0x00004000
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
I'm using luaglut to do some graphics in lua. And I am struggling with this function glReadPixels, particularly with its last input argument GLvoid *pixels.
void glReadPixels (GLint x, GLint y, GLsizei width, GLsizei height, GLenum format, GLenum type, GLvoid *pixels);
pixels is a pointer type so in lua it is of type lightuserdata. I managed to get a lightuserdata type variable let's say img in lua, according to this post; however, after I get the frame I wanna grab into img by calling:
glReadPixels(0, 0, 250, 250, GL_RGB, GL_UNSIGNED_BYTE, img)
I could do nothing with img. I tried creating a same structure in lua using ffi and coverting this img to a torch.Tensor type, but it is too slow since I have to assign the values pixel by pixel.
So I am asking here if there is better ways to use this glReadPixels function to get img than this troublesome approach that I took? Both table and torch.Tensor types of img are OK. Thank you in advance!
I'm looking at optimizing a routine that fetches the pixel data from a CGImage. The way this currently is done (very inefficiently) is to create a new CGContext, draw the CGImage into the context, and then get the data from the context.
I have the following optimized routine to handle this:
CGImageRef imageRef = image.CGImage;
uint8_t *pixelData = NULL;
CGDataProviderRef imageDataProvider = CGImageGetDataProvider(imageRef);
CFDataRef imageData = CGDataProviderCopyData(imageDataProvider);
pixelData = (uint8_t *)malloc(CFDataGetLength(imageData));
CFDataGetBytes(imageData, CFRangeMake(0, CFDataGetLength(imageData)), pixelData);
CFRelease(imageData);
This almost works. After viewing and comparing the hex dump of the pixel data obtained through both methods, I found that in the above case, there are 8 bytes of 0's every 6360 bytes. Otherwise, the data is identical. e.g.
And here is the comparison with the unoptimized version:
After the 8 bytes of 0's, the correct pixel data continues. Anyone know why this is happening?
UPDATE:
Here is the routine I am optimizing (the snipped code is just getting size info, and other non-important things; the relevant bit being the pixel data returned):
CGContextRef context = NULL;
CGImageRef imageRef = image.CGImage;
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst;
// ... SNIP ...
context = CGBitmapContextCreate(...);
CGColorSpaceRelease(colorSpace);
// ... SNIP ...
CGContextDrawImage(context, rect, imageRef);
uint8_t *pixelData = (uint8_t *)CGBitmapContextGetData(context);
CGContextRelease(context);
Obviously this is an excessive amount of work just to get the underlying pixel data. Creating a context, then drawing into it. The first routine is between 5 - 10 times as fast. But as I pointed out, the pixel data returned by both routines are almost identical, except for the insertion of 8 zero-byte values every 6360 bytes in the optimized code (highlighted in the images).
Otherwise, everything else is the same -- color values, byte order, etc.
The bitmap data has padding at the end of each row of pixels, to round the number of bytes per row up to a larger value. (In this case, a multiple of 16 bytes.)
This padding is added to make it faster to process and draw the image.
You should use CGImageGetBytesPerRow() to find out how many bytes each row takes. Don't assume that it's the same as CGImageGetWidth() * CGImageGetBitsPerPixel() / 8; the bytes per row may be larger.
Keep in mind that the data behind an arbitrary CGImage may not be in the format that you expect. You cannot assume that all images are 32-bit-per-pixel ARGB with no padding. You should either use the CG functions to figure out what format the data might be, or redraw the image into a bitmap context that's in the exact format you expect. The latter is typically much easier -- let CG do the conversions for you.
(You don't show what parameters you're passing to CGBitmapContextCreate. Are you calculating an exact bytesPerRow or are you passing in 0? If you pass in 0, CG may add padding for you, and you may find that drawing into the context is faster.)
I have an EXC_BAD_ACCESS at the last line of this code (this code is fired several times per second), but I cannot figure out what is the problem:
[EAGLContext setCurrentContext:_context];
glActiveTexture(GL_TEXTURE0);
glPixelStorei(GL_PACK_ALIGNMENT, 1);
glBindTexture(GL_TEXTURE_2D, _backgroundTexture);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, _outputFrame.cols, _outputFrame.rows, 0, GL_BGRA, GL_UNSIGNED_BYTE, _outputFrame.data);
When debugging I make sure that the texture is created (the id is > 0), output frame has a valid pointer to the data and is a 4 channel matrix. I am inside the drawRect method of a GLKViewController. I think I should not have to bind the framebuffer as it is one of the things that are automated here. It doesn't crash at the first frame, but a few dozens frames later.
Can anybody spot the problem?
UPDATE:
It seems it's because of a race condition on _outputFrame, it's being updated while being read by glTexImage2D. I will try to lock it for read, then report back.
That was the solution indeed (see UPDATE), I fixed it with NSLock. Firstly I swapped the instance variable _outputFrame with a temporary one that gets updated from another thread and used the lock to update the instance variable:
[_frameLock lock];
_outputFrame = temp;
[_frameLock unlock];
Then used the lock when I wanted to read from the instance variable:
glActiveTexture(GL_TEXTURE0);
glPixelStorei(GL_PACK_ALIGNMENT, 1);
glBindTexture(GL_TEXTURE_2D, _backgroundTexture);
[_frameLock lock];
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, _outputFrame.cols, _outputFrame.rows, 0, GL_BGRA, GL_UNSIGNED_BYTE, _outputFrame.data);
[_frameLock unlock];
I just figured out some problem like this after several days.
1. better avoid rendering in multi-thread
2. better render in GLKView with base affect, and don't manually manage framebuffer& render buffer by yourself
3. base effect render raw pixel data like this
My solution:
glTexImage2D(...);
self.baseEffect.texture2d0.envMode = GLKTextureEnvModeReplace;
self.baseEffect.texture2d0.target = GLKTextureTarget2D;
self.baseEffect.texture2d0.name = texture;
self.baseEffect.texture2d0.enabled = YES;
self.baseEffect.useConstantColor = YES;