Get grayscale image from CMSampleBufferRef - ios

I'm developing an application to detect text from an image. I use AVCaptureSession for getting sample buffer images. Also I use Tesseract lib for recognition. But it gets poor quality detection. I use following code to get image from CMSampleBufferRef. Can anyone edit that code to get grayscale image as output. I heard grayscale image input will be better for recognition in Tesseract.
// Create a UIImage from sample buffer data
- (UIImage *)imageFromSampleBuffer:(CMSampleBufferRef)sampleBuffer
{
// Get a CMSampleBuffer's Core Video image buffer for the media data
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
// Lock the base address of the pixel buffer
CVPixelBufferLockBaseAddress(imageBuffer, 0);
// Get the number of bytes per row for the pixel buffer
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
// Get the number of bytes per row for the pixel buffer
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
// Get the pixel buffer width and height
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
// Create a device-dependent RGB color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Create a bitmap graphics context with the sample buffer data
CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8,
bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
// Create a Quartz image from the pixel data in the bitmap graphics context
CGImageRef quartzImage = CGBitmapContextCreateImage(context);
// Unlock the pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
// Free up the context and color space
CGContextRelease(context);
CGColorSpaceRelease(colorSpace);
// Create an image object from the Quartz image
UIImage *image = [UIImage imageWithCGImage:quartzImage];
// Release the Quartz image
CGImageRelease(quartzImage);
return (image);
}

Related

What is the proper colorspace conversion for this image on iOS?

How do we fix our code below to properly color the image you see below from our incoming sampleBuffer?
We are attempting to convert an incoming sampleBuffer image to a UIImage but the result is the inverted off-color image you see below.
You can see our attempt to use the kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange option in the code - but the results were the same.
The incoming image we are trying to show has all the right colors as evidenced by the fact that if we render the image into a GLKView, all the colors are there.
Could this be a YUV420 conversion issue?
Here is the conversion code we are using:
- (CGImageRef) imageFromSampleBuffer:(CMSampleBufferRef) sampleBuffer // Create a CGImageRef from sample buffer data
{
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
CVPixelBufferLockBaseAddress(imageBuffer,0); // Lock the image buffer
uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0); // Get information of the image
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
CGContextRef newContext = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange | kCGImageAlphaPremultipliedFirst);
CGImageRef newImage = CGBitmapContextCreateImage(newContext);
CGContextRelease(newContext);
CGColorSpaceRelease(colorSpace);
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
/* CVBufferRelease(imageBuffer); */ // do not call this!
return newImage;
}
Here is the setup code we use for the incoming CVPixelBuffer:
// Now create the CVPixelBuffer to which we will render
CVReturn retVal = CVPixelBufferCreate(kCFAllocatorDefault,
self.screenWidth,
self.screenHeight,
kCVPixelFormatType_32BGRA,
attrs,
&_outputRenderTarget);
Any suggestions for what to try to restore and display all the proper colors?
There is no problem with the code shown apart from you need to swap back your commented/uncommented lines.
I would take a step back and look at what is writing the data to your incoming buffer. For example I use a AVAssetReaderVideoCompositionOutput object to generate the sample buffers. The initialisation of the video composition object has video settings that match the ones you use for creating your incoming cvpixelbuffer plus a cgimage and bitmap compatibility key.
It turns out we were inverting all the colors with a 1-r, 1-g, 1-b.
Once we removed the inversion the colors appeared normal again.

iOS drawing pixels in a UIView

I have an 2D array of RGB values (or any such data container) that I need to write to UIView that is currently displayed to the user. An example would be — while using the capture output from the camera, I run some algorithms to identify objects and then highlight them using custom defined RGB pixels.
What is the best way to do this as this whole thing is done in real-time every 10 frames per second for example?
Use the method below to create a UIImage from your 2D array. You can then display this image using a UIImageView.
-(UIImage *)imageFromArray:(void *)array width:(unsigned int)width height:(unsigned int)height {
/*
Assuming pixel color values are 8 bit unsigned
You need to create an array that is in the format BGRA (blue,green,red,alpha).
You can achieve this by implementing a for-loop that sets the values at each index.
I have not included a for-loop in this example because it depends on how the values are stored in your input 2D array.
You can set the alpha value to 255.
*/
unsigned char pixelData[width * height * 4];
// This is where the for-loop would be
void *baseAddress = &pixelData;
size_t bytesPerRow = width * 4;
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
CGImageRef cgImage = CGBitmapContextCreateImage(context);
UIImage *image = [UIImage imageWithCGImage:cgImage];
CGImageRelease(cgImage);
CGColorSpaceRelease(colorSpace);
CGContextRelease(context);
return image;
}

Convert CMSampleBufferRef to UIImage with YUV color space?

I'm working with AVCaptureVideoDataOutput and want to convert CMSampleBufferRef to UIImage. Many answers are the same, like this UIImage created from CMSampleBufferRef not displayed in UIImageView? and AVCaptureSession with multiple previews
It works fine if I set the VideoDataOutput color space to BGRA (credited to this answer CGBitmapContextCreateImage error)
NSString* key = (NSString*)kCVPixelBufferPixelFormatTypeKey;
NSNumber* value = [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA];
NSDictionary* videoSettings = [NSDictionary dictionaryWithObject:value forKey:key];
[dataOutput setVideoSettings:videoSettings];
Without the above videoSettings, I will receive the following errors
CGBitmapContextCreate: invalid data bytes/row: should be at least 2560 for 8 integer bits/component, 3 components, kCGImageAlphaPremultipliedFirst.
<Error>: CGBitmapContextCreateImage: invalid context 0x0
Working with BGRA is not a good choice, since there is conversion overhead from YUV (default AVCaptureSession color space) to BGRA, as stated by Brad and Codo in How to get the Y component from CMSampleBuffer resulted from the AVCaptureSession?
So is there a way to convert CMSampleBufferRef to UIImage and working with YUV color space ?
After doing a lots of research and read apple documentations and wikipedis. I figured out the answer and it is works perfectly for me. So for the future readers Im sharing the code to convert CMSampleBufferRef to UIImage when video pixel type is set as kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
// Create a UIImage from sample buffer data
// Works only if pixel format is kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
-(UIImage *) imageFromSamplePlanerPixelBuffer:(CMSampleBufferRef) sampleBuffer{
#autoreleasepool {
// Get a CMSampleBuffer's Core Video image buffer for the media data
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
// Lock the base address of the pixel buffer
CVPixelBufferLockBaseAddress(imageBuffer, 0);
// Get the number of bytes per row for the plane pixel buffer
void *baseAddress = CVPixelBufferGetBaseAddressOfPlane(imageBuffer, 0);
// Get the number of bytes per row for the plane pixel buffer
size_t bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(imageBuffer,0);
// Get the pixel buffer width and height
size_t width = CVPixelBufferGetWidth(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);
// Create a device-dependent gray color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceGray();
// Create a bitmap graphics context with the sample buffer data
CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8,
bytesPerRow, colorSpace, kCGImageAlphaNone);
// Create a Quartz image from the pixel data in the bitmap graphics context
CGImageRef quartzImage = CGBitmapContextCreateImage(context);
// Unlock the pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
// Free up the context and color space
CGContextRelease(context);
CGColorSpaceRelease(colorSpace);
// Create an image object from the Quartz image
UIImage *image = [UIImage imageWithCGImage:quartzImage];
// Release the Quartz image
CGImageRelease(quartzImage);
return (image);
}
}
// it works for me.
var image: CGImage?
VTCreateCGImageFromCVPixelBuffer(pixelBuffer, options: nil, imageOut: &image)
DispatchQueue.main.async {
self.imageView.image = UIImage(cgImage: image!)
}

Images being rotated when converted from Matrix

I am working on a app and making use of the opencv library.
The problem I am having happens only to certain images (usually if made with the phone's camera) and I pinpointed as being just a conversion problem. When I convert a (problematic) Image to a cv::Mat object and then back it just rotates 90 degrees.
Here is the call that causes the problem:
cv::Mat tmpMat = [sentImage CVMat];
UIImage * tmpImage = [[UIImage alloc] initWithCVMat:tmpMat];
[imageHolder setImage: tmpImage];
And here are the functions that do the conversion from image to matrix and vice-versa.
-(cv::Mat)CVMat
{
CGColorSpaceRef colorSpace = CGImageGetColorSpace(self.CGImage);
CGFloat cols = self.size.width;
CGFloat rows = self.size.height;
cv::Mat cvMat(rows, cols, CV_8UC4); // 8 bits per component, 4 channels
CGContextRef contextRef = CGBitmapContextCreate(cvMat.data, // Pointer to backing data
cols, // Width of bitmap
rows, // Height of bitmap
8, // Bits per component
cvMat.step[0], // Bytes per row
colorSpace, // Colorspace
kCGImageAlphaNoneSkipLast |
kCGBitmapByteOrderDefault); // Bitmap info flags
CGContextDrawImage(contextRef, CGRectMake(0, 0, cols, rows), self.CGImage);
CGContextRelease(contextRef);
return cvMat;
}
- (id)initWithCVMat:(const cv::Mat&)cvMat
{
NSData *data = [NSData dataWithBytes:cvMat.data length:cvMat.elemSize() * cvMat.total()];
CGColorSpaceRef colorSpace;
if (cvMat.elemSize() == 1)
{
colorSpace = CGColorSpaceCreateDeviceGray();
}
else
{
colorSpace = CGColorSpaceCreateDeviceRGB();
}
CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)data);
CGImageRef imageRef = CGImageCreate(cvMat.cols, // Width
cvMat.rows, // Height
8, // Bits per component
8 * cvMat.elemSize(), // Bits per pixel
cvMat.step[0], // Bytes per row
colorSpace, // Colorspace
kCGImageAlphaNone | kCGBitmapByteOrderDefault, // Bitmap info flags
provider, // CGDataProviderRef
NULL, // Decode
false, // Should interpolate
kCGRenderingIntentDefault); // Intent
self = [self initWithCGImage:imageRef];
CGImageRelease(imageRef);
CGDataProviderRelease(provider);
CGColorSpaceRelease(colorSpace);
return self;
}
Now I am using a "Aspect Fill" property in my imageHolder (a UIImageView) and tried changing it without success. I also tried seeing if it was a problem of a matrix being transposed on the conversion and tried to change without success and it also would not be logical since it does not turn every picture.
I do not understand why it works with some pictures but other not (all photos taken with the phone's camera don't work).
If anyone can shed a light on the matter I would appreciate.
Images from the camera that are taken with different orientations (Portrait / Landscape) are saved in the same resolution (same number of rows and columns) by the iPhone camera. The difference is that the JPEG contains a flag (to be precise, the Exif.Image.Orientation flag) to tell the displaying software how the image needs to be rotated to be displayed correctly.
My guess is that OpenCV looses that information (that is stored in the UIImage in the imageOrientation property) when converting, so when the image is converted back to UIImage this piece of information is set to default (UIImageOrientationUp), explaining why certain images appear to be rotated.
I was having the same issue. This solves the problem of the image rotating when converting from UIImage to cvMat. Add the method at the bottom, call it after you dismiss the picker controller. It is the 'second answer' located here: Rotating a CGImage
Also, there are two methods in the ios.h for UIImage to cvMat and vice versa, that you can just include. highgui/ios.h. Then add the rotation method and you are good to go.

Converting Images from camera buffer iOS. Capture still image using AVFoundation

I'm using this well known sample code from Apple to convert camera buffer still images into UIImages.
-(UIImage*) getUIImageFromBuffer:(CMSampleBufferRef) imageSampleBuffer{
// Get a CMSampleBuffer's Core Video image buffer for the media data
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(imageSampleBuffer);
if (imageBuffer==NULL) {
NSLog(#"No buffer");
}
// Lock the base address of the pixel buffer
if((CVPixelBufferLockBaseAddress(imageBuffer, 0))==kCVReturnSuccess){
NSLog(#"Buffer locked successfully");
}
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
// Get the number of bytes per row for the pixel buffer
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
NSLog(#"bytes per row %zu",bytesPerRow );
// Get the pixel buffer width and height
size_t width = CVPixelBufferGetWidth(imageBuffer);
NSLog(#"width %zu",width);
size_t height = CVPixelBufferGetHeight(imageBuffer);
NSLog(#"height %zu",height);
// Create a device-dependent RGB color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Create a bitmap graphics context with the sample buffer data
CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8,
bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
// Create a Quartz image from the pixel data in the bitmap graphics context
CGImageRef quartzImage = CGBitmapContextCreateImage(context);
// Free up the context and color space
CGContextRelease(context);
CGColorSpaceRelease(colorSpace);
// Create an image object from the Quartz image
UIImage *image= [UIImage imageWithCGImage:quartzImage scale:SCALE_IMAGE_RATIO orientation:UIImageOrientationRight];
// Release the Quartz image
CGImageRelease(quartzImage);
// Unlock the pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
return (image );}
The problem is that usually the image that you obtain is 90° rotated. Using the method +imageWithCGImage:scale:orientation I'm able to rotate it, but before getting into this method I was trying to rotate and scale the image using the CTM function, before passing it to a UIImage. the problem was that CTM transformation didn't affect the image.
I'm asking myself why... is that because I'm locking the buffer? or because the context is created with the image inside, so the changes will affect only the further mod?
Thank you
The answer is that it affects only further modifications, and it has nothing to deal with the buffer locking.
As you can read from this answer mod to the context are applied by time Vertical flip of CGContext

Resources