How to scale/resize CVPixelBufferRef in objective C, iOS - ios

Question

I am trying to resize an image from a CVPixelBufferRef to 299x299.
Ideally is would also crop the image. The original pixelbuffer is 640x320, the goal is to scale/crop to 299x299 without loosing aspect ratio (crop to center).
I found code to resize a UIImage in objective c, but none to resize a CVPixelBufferRef. I have found various very complicated examples of object C many different image types, but none specifically for resizing a CVPixelBufferRef.
What is the easiest/best way to do this, please include the exact code.
... I tried the answer from selton, but this did not work as the resulting type in the scaled buffer is not correct (goes into assert code),
OSType sourcePixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer);
int doReverseChannels;
if (kCVPixelFormatType_32ARGB == sourcePixelFormat) {
doReverseChannels = 1;
} else if (kCVPixelFormatType_32BGRA == sourcePixelFormat) {
doReverseChannels = 0;
} else {
assert(false); // Unknown source format
}

Answer 1

Using CoreMLHelpers as an inspiration. We can create a C function that does what you need. Based on your pixel format requirements, I think this solution will be the most efficient option. I used an AVCaputureVideoDataOutput for testing.
I hope this helps!
AVCaptureVideoDataOutputSampleBufferDelegate implementation. The majority of the work here is creating a centered-cropping rectangle. Making use of AVMakeRectWithAspectRatioInsideRect is key (it does exactly what you want).
- (void)captureOutput:(AVCaptureOutput *)output didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection; {
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
if (pixelBuffer == NULL) { return; }
size_t height = CVPixelBufferGetHeight(pixelBuffer);
size_t width = CVPixelBufferGetWidth(pixelBuffer);
CGRect videoRect = CGRectMake(0, 0, width, height);
CGSize scaledSize = CGSizeMake(299, 299);
// Create a rectangle that meets the output size's aspect ratio, centered in the original video frame
CGRect centerCroppingRect = AVMakeRectWithAspectRatioInsideRect(scaledSize, videoRect);
CVPixelBufferRef croppedAndScaled = createCroppedPixelBuffer(pixelBuffer, centerCroppingRect, scaledSize);
// Do other things here
// For example
CIImage *image = [CIImage imageWithCVImageBuffer:croppedAndScaled];
// End example
CVPixelBufferRelease(croppedAndScaled);
}
Method 1: Data manipulation and Accelerate
The basic premise of this function is that it first crops to the specified rectangle then scales to the final desired size. The cropping is achieved by simply ignoring the data outside the rectangle. Scaling is achieved through Accelerate's vImageScale_ARGB8888 function. Again, thanks to CoreMLHelpers for the insight.
void assertCropAndScaleValid(CVPixelBufferRef pixelBuffer, CGRect cropRect, CGSize scaleSize) {
CGFloat originalWidth = (CGFloat)CVPixelBufferGetWidth(pixelBuffer);
CGFloat originalHeight = (CGFloat)CVPixelBufferGetHeight(pixelBuffer);
assert(CGRectContainsRect(CGRectMake(0, 0, originalWidth, originalHeight), cropRect));
assert(scaleSize.width > 0 && scaleSize.height > 0);
}
void pixelBufferReleaseCallBack(void *releaseRefCon, const void *baseAddress) {
if (baseAddress != NULL) {
free((void *)baseAddress);
}
}
// Returns a CVPixelBufferRef with +1 retain count
CVPixelBufferRef createCroppedPixelBuffer(CVPixelBufferRef sourcePixelBuffer, CGRect croppingRect, CGSize scaledSize) {
OSType inputPixelFormat = CVPixelBufferGetPixelFormatType(sourcePixelBuffer);
assert(inputPixelFormat == kCVPixelFormatType_32BGRA
|| inputPixelFormat == kCVPixelFormatType_32ABGR
|| inputPixelFormat == kCVPixelFormatType_32ARGB
|| inputPixelFormat == kCVPixelFormatType_32RGBA);
assertCropAndScaleValid(sourcePixelBuffer, croppingRect, scaledSize);
if (CVPixelBufferLockBaseAddress(sourcePixelBuffer, kCVPixelBufferLock_ReadOnly) != kCVReturnSuccess) {
NSLog(#"Could not lock base address");
return nil;
}
void *sourceData = CVPixelBufferGetBaseAddress(sourcePixelBuffer);
if (sourceData == NULL) {
NSLog(#"Error: could not get pixel buffer base address");
CVPixelBufferUnlockBaseAddress(sourcePixelBuffer, kCVPixelBufferLock_ReadOnly);
return nil;
}
size_t sourceBytesPerRow = CVPixelBufferGetBytesPerRow(sourcePixelBuffer);
size_t offset = CGRectGetMinY(croppingRect) * sourceBytesPerRow + CGRectGetMinX(croppingRect) * 4;
vImage_Buffer croppedvImageBuffer = {
.data = ((char *)sourceData) + offset,
.height = (vImagePixelCount)CGRectGetHeight(croppingRect),
.width = (vImagePixelCount)CGRectGetWidth(croppingRect),
.rowBytes = sourceBytesPerRow
};
size_t scaledBytesPerRow = scaledSize.width * 4;
void *scaledData = malloc(scaledSize.height * scaledBytesPerRow);
if (scaledData == NULL) {
NSLog(#"Error: out of memory");
CVPixelBufferUnlockBaseAddress(sourcePixelBuffer, kCVPixelBufferLock_ReadOnly);
return nil;
}
vImage_Buffer scaledvImageBuffer = {
.data = scaledData,
.height = (vImagePixelCount)scaledSize.height,
.width = (vImagePixelCount)scaledSize.width,
.rowBytes = scaledBytesPerRow
};
/* The ARGB8888, ARGB16U, ARGB16S and ARGBFFFF functions work equally well on
* other channel orderings of 4-channel images, such as RGBA or BGRA.*/
vImage_Error error = vImageScale_ARGB8888(&croppedvImageBuffer, &scaledvImageBuffer, nil, 0);
CVPixelBufferUnlockBaseAddress(sourcePixelBuffer, kCVPixelBufferLock_ReadOnly);
if (error != kvImageNoError) {
NSLog(#"Error: %ld", error);
free(scaledData);
return nil;
}
OSType pixelFormat = CVPixelBufferGetPixelFormatType(sourcePixelBuffer);
CVPixelBufferRef outputPixelBuffer = NULL;
CVReturn status = CVPixelBufferCreateWithBytes(nil, scaledSize.width, scaledSize.height, pixelFormat, scaledData, scaledBytesPerRow, pixelBufferReleaseCallBack, nil, nil, &outputPixelBuffer);
if (status != kCVReturnSuccess) {
NSLog(#"Error: could not create new pixel buffer");
free(scaledData);
return nil;
}
return outputPixelBuffer;
}
Method 2: CoreImage
This method is much simpler to read, and has the benefit of being pretty agnostic to the pixel buffer format you pass in, which is a plus for certain use cases. Granted, you're limited to which formats CoreImage supports.
CVPixelBufferRef createCroppedPixelBufferCoreImage(CVPixelBufferRef pixelBuffer,
CGRect cropRect,
CGSize scaleSize,
CIContext *context) {
assertCropAndScaleValid(pixelBuffer, cropRect, scaleSize);
CIImage *image = [CIImage imageWithCVImageBuffer:pixelBuffer];
image = [image imageByCroppingToRect:cropRect];
CGFloat scaleX = scaleSize.width / CGRectGetWidth(image.extent);
CGFloat scaleY = scaleSize.height / CGRectGetHeight(image.extent);
image = [image imageByApplyingTransform:CGAffineTransformMakeScale(scaleX, scaleY)];
// Due to the way [CIContext:render:toCVPixelBuffer] works, we need to translate the image so the cropped section is at the origin
image = [image imageByApplyingTransform:CGAffineTransformMakeTranslation(-image.extent.origin.x, -image.extent.origin.y)];
CVPixelBufferRef output = NULL;
CVPixelBufferCreate(nil,
CGRectGetWidth(image.extent),
CGRectGetHeight(image.extent),
CVPixelBufferGetPixelFormatType(pixelBuffer),
nil,
&output);
if (output != NULL) {
[context render:image toCVPixelBuffer:output];
}
return output;
}
Creating the CIContext can be done at the call site or it can be created and stored on a property. For information about options, see the documentation.
// Create a CIContext using default settings, this will
// typically use the GPU and Metal by default if supported
if (self.context == nil) {
self.context = [CIContext context];
}

Answer 2

func assertCropAndScaleValid(_ pixelBuffer: CVPixelBuffer, _ cropRect: CGRect, _ scaleSize: CGSize) {
let originalWidth: CGFloat = CGFloat(CVPixelBufferGetWidth(pixelBuffer))
let originalHeight: CGFloat = CGFloat(CVPixelBufferGetHeight(pixelBuffer))
assert(CGRect(x: 0, y: 0, width: originalWidth, height: originalHeight).contains(cropRect))
assert(scaleSize.width > 0 && scaleSize.height > 0)
}
func createCroppedPixelBufferCoreImage(pixelBuffer: CVPixelBuffer,
cropRect: CGRect,
scaleSize: CGSize,
context: inout CIContext
) -> CVPixelBuffer {
assertCropAndScaleValid(pixelBuffer, cropRect, scaleSize)
var image = CIImage(cvImageBuffer: pixelBuffer)
image = image.cropped(to: cropRect)
let scaleX = scaleSize.width / image.extent.width
let scaleY = scaleSize.height / image.extent.height
image = image.transformed(by: CGAffineTransform(scaleX: scaleX, y: scaleY))
image = image.transformed(by: CGAffineTransform(translationX: -image.extent.origin.x, y: -image.extent.origin.y))
var output: CVPixelBuffer? = nil
CVPixelBufferCreate(nil, Int(image.extent.width), Int(image.extent.height), CVPixelBufferGetPixelFormatType(pixelBuffer), nil, &output)
if output != nil {
context.render(image, to: output!)
} else {
fatalError("Error")
}
return output!
}
Swift version of #allenh 's answer

Answer 3

Step 1
Convert the CVPixelBuffer to UIImage by starting with [CIImage imageWithCVPixelBuffer: then converting that CIImage to CGImage then that CGImage to UIImage using the standard methods.
CIImage *ciimage = [CIImage imageWithCVPixelBuffer:pixelBuffer];
CIContext *context = [CIContext contextWithOptions:nil];
CGImageRef cgimage = [context
createCGImage:ciimage
fromRect:CGRectMake(0, 0,
CVPixelBufferGetWidth(pixelBuffer),
CVPixelBufferGetHeight(pixelBuffer))];
UIImage *uiimage = [UIImage imageWithCGImage:cgimage];
CGImageRelease(cgimage);
Step 2
Scale the image to desired size/cropping by placing it in a UIImageView
UIImageView *imageView = [[UIImageView alloc] initWithFrame:/*CGRect with new dimensions*/];
imageView.contentMode = /*UIViewContentMode with desired scaling/clipping style*/;
imageView.image = uiimage;
Step 3
Snapshot the CALayer of said imageView with something like this:
#define snapshotOfView(__view) (\
(^UIImage *(void) {\
CGRect __rect = [__view bounds];\
UIGraphicsBeginImageContextWithOptions(__rect.size, /*(BOOL)Opaque*/, /*(float)scaleResolution*/);\
CGContextRef __context = UIGraphicsGetCurrentContext();\
[__view.layer renderInContext:__context];\
UIImage *__image = UIGraphicsGetImageFromCurrentImageContext();\
UIGraphicsEndImageContext();\
return __image;\
})()\
)
In use:
uiimage = snapshotOfView(imageView);
Step 4
Convert said UIImage-snapshot image (cropped/scaled) back into a CVPixelBuffer using a method like this: https://stackoverflow.com/a/34990820/2057171
That is,
- (CVPixelBufferRef) pixelBufferFromCGImage: (CGImageRef) image
{
NSDictionary *options = #{
(NSString*)kCVPixelBufferCGImageCompatibilityKey : #YES,
(NSString*)kCVPixelBufferCGBitmapContextCompatibilityKey : #YES,
};
CVPixelBufferRef pxbuffer = NULL;
CVReturn status = CVPixelBufferCreate(kCFAllocatorDefault, CGImageGetWidth(image),
CGImageGetHeight(image), kCVPixelFormatType_32ARGB, (__bridge CFDictionaryRef) options,
&pxbuffer);
if (status!=kCVReturnSuccess) {
NSLog(#"Operation failed");
}
NSParameterAssert(status == kCVReturnSuccess && pxbuffer != NULL);
CVPixelBufferLockBaseAddress(pxbuffer, 0);
void *pxdata = CVPixelBufferGetBaseAddress(pxbuffer);
CGColorSpaceRef rgbColorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pxdata, CGImageGetWidth(image),
CGImageGetHeight(image), 8, 4*CGImageGetWidth(image), rgbColorSpace,
kCGImageAlphaNoneSkipFirst);
NSParameterAssert(context);
CGContextConcatCTM(context, CGAffineTransformMakeRotation(0));
CGAffineTransform flipVertical = CGAffineTransformMake( 1, 0, 0, -1, 0, CGImageGetHeight(image) );
CGContextConcatCTM(context, flipVertical);
CGAffineTransform flipHorizontal = CGAffineTransformMake( -1.0, 0.0, 0.0, 1.0, CGImageGetWidth(image), 0.0 );
CGContextConcatCTM(context, flipHorizontal);
CGContextDrawImage(context, CGRectMake(0, 0, CGImageGetWidth(image),
CGImageGetHeight(image)), image);
CGColorSpaceRelease(rgbColorSpace);
CGContextRelease(context);
CVPixelBufferUnlockBaseAddress(pxbuffer, 0);
return pxbuffer;
}
In use:
pixelBuffer = [self pixelBufferFromCGImage:uiimage];

Answer 4

You can consider using CIImage:
CIImage *image = [CIImage imageWithCVPixelBuffer:pxbuffer];
CIImage *scaledImage = [image imageByApplyingTransform:(CGAffineTransformMakeScale(0.1, 0.1))];
CVPixelBufferRef scaledBuf = [scaledImage pixelBuffer];
You should change the scale to fit your dest size.