ios9 - Issues with cropping CVImageBuffer - ios

I am facing few issues related to cropping with iOS9 SDK.
I have the following code to resize a image (converting from 4:3 to 16:9 by cropping in middle). This used to work fine till iOS8 SDK. With iOS 9, the bottom area is blank.
(CMSampleBufferRef)resizeImage:(CMSampleBufferRef) sampleBuffer {
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer,0);
int target_width = CVPixelBufferGetWidth(imageBuffer);
int target_height = CVPixelBufferGetHeight(imageBuffer);
int height = CVPixelBufferGetHeight(imageBuffer);
int width = CVPixelBufferGetWidth(imageBuffer);
int x=0, y=0;
// Convert 16:9 to 4:3
if (((target_width*3)/target_height) == 4)
{
target_height = ((target_width*9)/16);
target_height = ((target_height + 15) / 16) * 16;
y = (height - target_height)/2;
}
else
if ((target_width == 352) && (target_height == 288))
{
target_height = ((target_width*9)/16);
target_height = ((target_height + 15) / 16) * 16;
y = (height - target_height)/2;
}
else
if (((target_height*3)/target_width) == 4)
{
target_width = ((target_height*9)/16);
target_width = ((target_width + 15) / 16) * 16;
x = ((width - target_width)/2);
}
else
if ((target_width == 288) && (target_height == 352))
{
target_width = ((target_height*9)/16);
target_width = ((target_width + 15) / 16) * 16;
x = ((width - target_width)/2);
}
CGRect cropRect;
NSLog(#"resizeImage x %d, y %d, target_width %d, target_height %d", x, y, target_width, target_height );
cropRect = CGRectMake(x, y, target_width, target_height);
CFDictionaryRef empty; // empty value for attr value.
CFMutableDictionaryRef attrs;
empty = CFDictionaryCreate(kCFAllocatorDefault, // our empty IOSurface properties dictionary
NULL,
NULL,
0,
&kCFTypeDictionaryKeyCallBacks,
&kCFTypeDictionaryValueCallBacks);
attrs = CFDictionaryCreateMutable(kCFAllocatorDefault,
1,
&kCFTypeDictionaryKeyCallBacks,
&kCFTypeDictionaryValueCallBacks);
CFDictionarySetValue(attrs,
kCVPixelBufferIOSurfacePropertiesKey,
empty);
OSStatus status;
CIImage *ciImage = [CIImage imageWithCVPixelBuffer:imageBuffer]; //options: [NSDictionary dictionaryWithObjectsAndKeys:[NSNull null], kCIImageColorSpace, nil]];
CVPixelBufferRef pixelBuffer;
status = CVPixelBufferCreate(kCFAllocatorSystemDefault, target_width, target_height, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange, attrs, &pixelBuffer);
if (status != 0)
{
NSLog(#"CVPixelBufferCreate error %d", (int)status);
}
[ciContext render:ciImage toCVPixelBuffer:pixelBuffer bounds:cropRect colorSpace:nil];
CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );
CVPixelBufferUnlockBaseAddress( imageBuffer,0);
CMSampleTimingInfo sampleTime = {
.duration = CMSampleBufferGetDuration(sampleBuffer),
.presentationTimeStamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer),
.decodeTimeStamp = CMSampleBufferGetDecodeTimeStamp(sampleBuffer)
};
CMVideoFormatDescriptionRef videoInfo = NULL;
status = CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, pixelBuffer, &videoInfo);
if (status != 0)
{
NSLog(#"CMVideoFormatDescriptionCreateForImageBuffer error %d", (int)status);
}
CMSampleBufferRef oBuf;
status = CMSampleBufferCreateForImageBuffer(kCFAllocatorDefault, pixelBuffer, true, NULL, NULL, videoInfo, &sampleTime, &oBuf);
if (status != 0)
{
NSLog(#"CMSampleBufferCreateForImageBuffer error %d", (int)status);
}
CFRelease(pixelBuffer);
ciImage = nil;
pixelBuffer = nil;
return oBuf;
}
}
Any ideas or suggestions regarding this? I tried changing the crop rectangle but with no effect.
Thanks

Are you aware that the doc comment of the function [CIContext toCVPixelBuffer: bounds: colorSpace:] says about iOS8- and iOS9+? (I could've not found any online resource to link, though.)
/* Render 'image' to the given CVPixelBufferRef.
* The 'bounds' parameter has the following behavior:
* In OS X and iOS 9 and later: The 'image' is rendered into 'buffer' so that
* point (0,0) of 'image' aligns to the lower left corner of 'buffer'.
* The 'bounds' acts like a clip rect to limit what region of 'buffer' is modified.
* In iOS 8 and earlier: The 'bounds' parameter acts to specify the region of 'image' to render.
* This region (regarless of its origin) is rendered at upper-left corner of 'buffer'.
*/
Taking it into account I solved my problem, which looks the same as yours.

Related

Memory leak from Objective-C code in iOS application

My code is eating memory. I added this function and it seems to the cause of all the problems as when I dont call it then I don't run out.
It's a function in Objective-C to crop an image. How do I release the memory that was used in the auction so that at the end of the function everything is cleaned up before exiting.
-(void) crop: (CVImageBufferRef)sampleBuffer
{
int cropX0, cropY0, cropHeight, cropWidth, outWidth, outHeight;
cropHeight = 720;
cropWidth = 1280;
cropX0 = 0;
cropY0 = 0;
outWidth = 1280;
outHeight = 720;
CVPixelBufferLockBaseAddress(sampleBuffer,0);
void *baseAddress = CVPixelBufferGetBaseAddress(sampleBuffer);
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(sampleBuffer);
vImage_Buffer inBuff;
inBuff.height = cropHeight;
inBuff.width = cropWidth;
inBuff.rowBytes = bytesPerRow;
int startpos = cropY0*bytesPerRow+4*cropX0;
inBuff.data = baseAddress+startpos;
unsigned char *outImg= (unsigned char*)malloc(4*outWidth*outHeight);
vImage_Buffer outBuff = {outImg, outHeight, outWidth, 4*outWidth};
vImage_Error err = vImageScale_ARGB8888(&inBuff, &outBuff, NULL, 0);
if (err != kvImageNoError)
{
NSLog(#" error %ld", err);
}
else
{
NSLog(#"Success");
}
CVPixelBufferRef pixelBuffer = NULL;
OSStatus result = CVPixelBufferCreateWithBytes(kCFAllocatorDefault,
inBuff.width,
inBuff.height,
kCVPixelFormatType_32BGRA,
outImg,
bytesPerRow,
NULL,
NULL,
NULL,
&pixelBuffer);
CVPixelBufferUnlockBaseAddress(sampleBuffer,0);
}
free(outImg);
at the end missing since you are not freeing the memory allocated.
It is a good practice in embedded programming and also here since you have const size pixel dimensions to use a const matrix that you can declare at the top of the function and initialized to zero.

CVPixelBufferCreate does not care Planar format

I try to rotate CoreVideo '420f' image without transfer to RGBA.
The incoming CMSampleBuffer Y-plane bytesPerRow is width + 32.
That means Y-plane row size is 8bit * width + sizeof(CVPlanarComponentInfo).
But if I call CVPixelBufferCreate(,,,'420f',,) , BytesPerRow == width.
CVPixelBufferCreate() does not care about planar format and did not add 32bytes.
I tried
vImage_Buffer myYBuffer = {buf, height, width, bytePerRow};
But there is no parameter for bitsPerPixel. I cannot use for UVBuffer.
I tried
vImageBuffer_Init(buf, height, width, bitPerPixel, flag);
But there is no parameter for bytesPerRow.
I like to know how to create vImageBuffer or CVPixelBuffer with '420f' planar format.
This is under construction code for rotation
NS_INLINE void dumpData(NSString* tag, unsigned char* p, size_t w) {
NSMutableString* str = [tag mutableCopy];
for(int i=0;i<w+100;++i) {
[str appendString:[NSString stringWithFormat:#"%02x ", *(p + i)]];
}
NSLog(#"%#", str);
}
- (CVPixelBufferRef) RotateBuffer:(CMSampleBufferRef)sampleBuffer withConstant:(uint8_t)rotationConstant
{
vImage_Error err = kvImageNoError;
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
size_t outHeight = width;
size_t outWidth = height;
assert(CVPixelBufferGetPixelFormatType(imageBuffer) == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange);
assert(CVPixelBufferGetPlaneCount(imageBuffer) == 2);
NSLog(#"YBuffer %ld %ld %ld", CVPixelBufferGetWidthOfPlane(imageBuffer, 0), CVPixelBufferGetHeightOfPlane(imageBuffer, 0),
CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0)); // BytesPerRow = width + 32
dumpData(#"Base=", CVPixelBufferGetBaseAddress(imageBuffer), width);
dumpData(#"Plane0=", CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0), width);
CVPixelBufferRef rotatedBuffer = NULL;
CVReturn ret = CVPixelBufferCreate(kCFAllocatorDefault, outWidth, outHeight, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange, NULL, &rotatedBuffer);
NSLog(#"CVPixelBufferCreate err=%d", ret);
CVPixelBufferLockBaseAddress(rotatedBuffer, 0);
NSLog(#"CVPixelBufferCreate init %ld %ld %ld p=%p", CVPixelBufferGetWidthOfPlane(rotatedBuffer, 0), CVPixelBufferGetHeightOfPlane(rotatedBuffer, 0),
CVPixelBufferGetBytesPerRowOfPlane(rotatedBuffer, 0), CVPixelBufferGetBaseAddressOfPlane(rotatedBuffer, 0));
// BytesPerRow = width ??? should be width + 32
// rotate Y plane
vImage_Buffer originalYBuffer = { CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0), CVPixelBufferGetHeightOfPlane(imageBuffer, 0),
CVPixelBufferGetWidthOfPlane(imageBuffer, 0), CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 0) };
vImage_Buffer rotatedYBuffer = { CVPixelBufferGetBaseAddressOfPlane(rotatedBuffer, 0), CVPixelBufferGetHeightOfPlane(rotatedBuffer, 0),
CVPixelBufferGetWidthOfPlane(rotatedBuffer, 0), CVPixelBufferGetBytesPerRowOfPlane(rotatedBuffer, 0) };
err = vImageRotate90_Planar8(&originalYBuffer, &rotatedYBuffer, 1, 0.0, kvImageNoFlags);
NSLog(#"rotatedYBuffer rotated %ld %ld %ld p=%p", rotatedYBuffer.width, rotatedYBuffer.height, rotatedYBuffer.rowBytes, rotatedYBuffer.data);
NSLog(#"RotateY err=%ld", err);
dumpData(#"Rotated Plane0=", rotatedYBuffer.data, outWidth);
// rotate UV plane
vImage_Buffer originalUVBuffer = { CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 1), CVPixelBufferGetHeightOfPlane(imageBuffer, 1),
CVPixelBufferGetWidthOfPlane(imageBuffer, 1), CVPixelBufferGetBytesPerRowOfPlane(imageBuffer, 1) };
vImage_Buffer rotatedUVBuffer = { CVPixelBufferGetBaseAddressOfPlane(rotatedBuffer, 1), CVPixelBufferGetHeightOfPlane(rotatedBuffer, 1),
CVPixelBufferGetWidthOfPlane(rotatedBuffer, 1), CVPixelBufferGetBytesPerRowOfPlane(rotatedBuffer, 1) };
err = vImageRotate90_Planar16U(&originalUVBuffer, &rotatedUVBuffer, 1, 0.0, kvImageNoFlags);
NSLog(#"RotateUV err=%ld", err);
dumpData(#"Rotated Plane1=", rotatedUVBuffer.data, outWidth);
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
CVPixelBufferUnlockBaseAddress(rotatedBuffer, 0);
return rotatedBuffer;
}
I found vImageBuffer BytesPerRow extra 32 byte is optional. Some Apple API add 32 byte on each row, some API does not add.
Actually questioned code works fine. CVPixelBufferCreate() creates buffer without extra 32 byte. vImageRotate90_Planar8() supports both formats, with 32 byte and without 32 byte.

How to convert opencv cv::Mat to CVPixelBuffer

I'm an undergraduate student and I'm doing some HumanSeg iPhone app using CoreML. Since my model needs resizing and black padding on the original video frames, I can't rely on Vision (which only provides resizing but no black padding) and have to do the converting myself.
I have CVPixelBuffer frames and I have converted it into cv::Mat using the following codes:
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
int bufferWidth = (int) CVPixelBufferGetWidth(pixelBuffer);
int bufferHeight = (int) CVPixelBufferGetHeight(pixelBuffer);
int bytePerRow = (int) CVPixelBufferGetBytesPerRow(pixelBuffer);
unsigned char *pixel = (unsigned char *) CVPixelBufferGetBaseAddress(pixelBuffer);
Mat image = Mat(bufferHeight, bufferWidth, CV_8UC4, pixel, bytePerRow);
CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
/*I'll do my resizing and padding here*/
// How can I implement this function?
convertToCVPixelBuffer(image);
But now, after I've done my preprocessing works, I have to convert the cv::Mat back to a CVPixelBuffer to feed it to the CoreML model. How can I achieve this? (Or can Vision achieve black padding using some special techniques?)
Any help will be appreciated.
Please see below the code... Checking whether width and height is divisible by 64 is necessary or else we get weird results due to BytesPerRow mismatch with cv::Mat and CVPixelBuffer
CVPixelBufferRef getImageBufferFromMat(cv::Mat matimg) {
cv::cvtColor(matimg, matimg, CV_BGR2BGRA);
/* Very much required see https://stackoverflow.com/questions/66434552/objective-c-cvmat-to-cvpixelbuffer
height & width has to be multiple of 64 for better caching
*/
int widthReminder = matimg.cols % 64, heightReminder = matimg.rows % 64;
if (widthReminder != 0 || heightReminder != 0) {
cv::resize(matimg, matimg, cv::Size(matimg.cols + (64 - widthReminder), matimg.rows + (64 - heightReminder)));
}
NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:
[NSNumber numberWithBool: YES], kCVPixelBufferCGImageCompatibilityKey,
[NSNumber numberWithBool: YES], kCVPixelBufferCGBitmapContextCompatibilityKey,
[NSNumber numberWithInt: matimg.cols], kCVPixelBufferWidthKey,
[NSNumber numberWithInt: matimg.rows], kCVPixelBufferHeightKey,
[NSNumber numberWithInt: matimg.step[0]], kCVPixelBufferBytesPerRowAlignmentKey,
nil];
CVPixelBufferRef imageBuffer;
CVReturn status = CVPixelBufferCreate(kCFAllocatorMalloc, matimg.cols, matimg.rows, kCVPixelFormatType_32BGRA, (CFDictionaryRef) CFBridgingRetain(options), &imageBuffer) ;
NSParameterAssert(status == kCVReturnSuccess && imageBuffer != NULL);
CVPixelBufferLockBaseAddress(imageBuffer, 0);
void *base = CVPixelBufferGetBaseAddress(imageBuffer);
memcpy(base, matimg.data, matimg.total() * matimg.elemSize());
CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
return imageBuffer;
}
First, convert mat to UIImage (or any other class from iOS APIs), check this question. Then, convert resulting image to CVPixelBuffer like this.
For people who will be using the new OpenCV Swift Wrapper, here is #Abhinava 's code translated to Swift
func matToCVPixelBuffer(mat: Mat)-> CVPixelBuffer? {
let matrix = Mat()
Imgproc.cvtColor(src: mat, dst: matrix, code: ColorConversionCodes.COLOR_BGR2BGRA)
let widthRemainder = matrix.cols() % 64
let heightRemainder = matrix.rows() % 64
if widthRemainder != 0 || heightRemainder != 0 {
Imgproc.resize(src: matrix, dst: matrix, dsize: Size(width: matrix.cols() + (64 - widthRemainder), height: matrix.rows() + (64 - heightRemainder)))
}
let attributes = [
kCVPixelBufferMetalCompatibilityKey: kCFBooleanTrue!,
kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue!,
kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue!,
kCVPixelBufferWidthKey: matrix.cols(),
kCVPixelBufferHeightKey: matrix.rows(),
kCVPixelBufferBytesPerRowAlignmentKey: matrix.step1(0)
] as CFDictionary
var pixelBuffer: CVPixelBuffer?
let status = CVPixelBufferCreate(
kCFAllocatorDefault, Int(matrix.cols()),
Int(matrix.rows()),
kCVPixelFormatType_32BGRA,
attributes,
&pixelBuffer)
guard let pixelBuffer = pixelBuffer, (status == kCVReturnSuccess) else {
return nil
}
CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
let base = CVPixelBufferGetBaseAddress(pixelBuffer)
memcpy(base, matrix.dataPointer(), matrix.total()*matrix.elemSize())
CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
return pixelBuffer
}

What's the most efficient way to check of CVPixelBufferRef has white pixels?

What's the most efficient way to check of CVPixelBufferRef has white pixels?
Below is the code that I'm currently using, but it's not returning an white pixels, when a photo clearly has white pixels.
const int kBytesPerPixel = 4;
CVPixelBufferLockBaseAddress( pixelBuffer, 0 );
int bufferWidth = (int)CVPixelBufferGetWidth( pixelBuffer );
int bufferHeight = (int)CVPixelBufferGetHeight( pixelBuffer );
size_t bytesPerRow = CVPixelBufferGetBytesPerRow( pixelBuffer );
uint8_t *baseAddress = CVPixelBufferGetBaseAddress( pixelBuffer );
int count = 0;
BOOL hasWhitePixels = NO;
for ( int row = 0; row < bufferHeight; row++ ){
uint8_t *pixel = baseAddress + row * bytesPerRow;
for ( int column = 0; column < bufferWidth; column++ ){
if (pixel[0] == 255 && pixel[1] == 255 && pixel[2] == 255) {
count++;
hasWhitePixels = YES;
NSLog(#"HAS WHITE PIXELS");
break;
}
pixel += kBytesPerPixel;
}
}
CVPixelBufferUnlockBaseAddress( pixelBuffer, 0 );

How to check if a uiimage is blank? (empty, transparent)

which is the best way to check whether a UIImage is blank?
I have this painting editor which returns a UIImage; I don't want to save this image if there's nothing on it.
Try this code:
BOOL isImageFlag=[self checkIfImage:image];
And checkIfImage method:
- (BOOL) checkIfImage:(UIImage *)someImage {
CGImageRef image = someImage.CGImage;
size_t width = CGImageGetWidth(image);
size_t height = CGImageGetHeight(image);
GLubyte * imageData = malloc(width * height * 4);
int bytesPerPixel = 4;
int bytesPerRow = bytesPerPixel * width;
int bitsPerComponent = 8;
CGContextRef imageContext =
CGBitmapContextCreate(
imageData, width, height, bitsPerComponent, bytesPerRow, CGImageGetColorSpace(image),
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big
);
CGContextSetBlendMode(imageContext, kCGBlendModeCopy);
CGContextDrawImage(imageContext, CGRectMake(0, 0, width, height), image);
CGContextRelease(imageContext);
int byteIndex = 0;
BOOL imageExist = NO;
for ( ; byteIndex < width*height*4; byteIndex += 4) {
CGFloat red = ((GLubyte *)imageData)[byteIndex]/255.0f;
CGFloat green = ((GLubyte *)imageData)[byteIndex + 1]/255.0f;
CGFloat blue = ((GLubyte *)imageData)[byteIndex + 2]/255.0f;
CGFloat alpha = ((GLubyte *)imageData)[byteIndex + 3]/255.0f;
if( red != 1 || green != 1 || blue != 1 || alpha != 1 ){
imageExist = YES;
break;
}
}
free(imageData);
return imageExist;
}
You will have to add OpenGLES framework and import this in the .m file:
#import <OpenGLES/ES1/gl.h>
One idea would be to call UIImagePNGRepresentation to get an NSData object then compare it with a pre-defined 'empty' version - ie: call:
- (BOOL)isEqualToData:(NSData *)otherData
to test?
Not tried this on large data; might want to check performance, if your image data is quite large, otherwise if it's small it is probably just like calling memcmp() in C.
Something along these lines:
Create a 1 px square CGContext
Draw the image so it fills the context
Test the one pixel of the context to see if it contains any data. If it's completely transparent, consider the picture blank
Others may be able to add more details to this answer.
Here's a solution in Swift that does not require any additional frameworks.
Thanks to answers in a related question here:
Get Pixel Data of ImageView from coordinates of touch screen on xcode?
func imageIsEmpty(_ image: UIImage) -> Bool {
guard let cgImage = image.cgImage,
let dataProvider = cgImage.dataProvider else
{
return true
}
let pixelData = dataProvider.data
let data: UnsafePointer<UInt8> = CFDataGetBytePtr(pixelData)
let imageWidth = Int(image.size.width)
let imageHeight = Int(image.size.height)
for x in 0..<imageWidth {
for y in 0..<imageHeight {
let pixelIndex = ((imageWidth * y) + x) * 4
let r = data[pixelIndex]
let g = data[pixelIndex + 1]
let b = data[pixelIndex + 2]
let a = data[pixelIndex + 3]
if a != 0 {
if r != 0 || g != 0 || b != 0 {
return false
}
}
}
}
return true
}
I'm not at my Mac, so I can't test this (and there are probably compile errors). But one method might be:
//The pixel format depends on what sort of image you're expecting. If it's RGBA, this should work
typedef struct
{
uint8_t red;
uint8_t green;
uint8_t blue;
uint8_t alpha;
} MyPixel_T;
UIImage *myImage = [self doTheThingToGetTheImage];
CGImageRef myCGImage = [myImage CGImage];
//Get a bitmap context for the image
CGBitmapContextRef *bitmapContext =
CGBitmapContextFreate(NULL, CGImageGetWidth(myCGImage), CGImageGetHeight(myCGImage),
CGImageGetBitsPerComponent(myCGImage), CGImageGetBytesPerRow(myCGImage),
CGImageGetColorSpace(myCGImage), CGImageGetBitmapInfo(myCGImage));
//Draw the image into the context
CGContextDrawImage(bitmapContext, CGRectMake(0, 0, CGImageGetWidth(myCGImage), CGImageGetHeight(myCGImage)), myCGImage);
//Get pixel data for the image
MyPixel_T *pixels = CGBitmapContextGetData(bitmapContext);
size_t pixelCount = CGImageGetWidth(myCGImage) * CGImageGetHeight(myCGImage);
for(size_t i = 0; i < pixelCount; i++)
{
MyPixel_T p = pixels[i];
//Your definition of what's blank may differ from mine
if(p.red > 0 && p.green > 0 && p.blue > 0 && p.alpha > 0)
return NO;
}
return YES;
I just encountered the same problem. Solved it by checking the dimensions:
Swift example:
let image = UIImage()
let height = image.size.height
let width = image.size.height
if (height > 0 && width > 0) {
// We have an image
} else {
// ...and we don't
}

Resources