Converting UIImage to cv::Mat - ios

I have a UIImage which is a pic captured from an iPhone Camera now I want the UIImage to be converted to cv::Mat (OpenCV). I am using the following lines of code to accomplish this :
-(cv::Mat)CVMat
{
CGColorSpaceRef colorSpace = CGImageGetColorSpace(self.CGImage);
CGFloat cols = self.size.width;
CGFloat rows = self.size.height;
cv::Mat cvMat(rows, cols, CV_8UC4); // 8 bits per component, 4 channels
CGContextRef contextRef = CGBitmapContextCreate(cvMat.data, // Pointer to backing data
cols, // Width of bitmap
rows, // Height of bitmap
8, // Bits per component
cvMat.step[0], // Bytes per row
colorSpace, // Colorspace
kCGImageAlphaNoneSkipLast |
kCGBitmapByteOrderDefault); // Bitmap info flags
CGContextDrawImage(contextRef, CGRectMake(0, 0, cols, rows), self.CGImage);
CGContextRelease(contextRef);
return cvMat;
}
This Code works fine for the UIImage in Landscape mode but when I use the same code with Images taken from Portrait mode the images gets rotated by 90 degrees towards the right.
I am a newbie in iOS and Objective C and hence I am not able to figure out what is wrong.
Can somebody tell me what is wrong with the code or am I missing out something.

This will be because the UIImage is not actually portrait. All photos taken with the iPhone camera are landscape in their raw bitmap state, eg 3264 wide x 2488 high. A "portrait" photo is displayed as such by the orientation EXIF flag set in the image, which is honoured, for example, by the photo library app which swivels images according to this flag and the viewing orientation of the camera.
The flag also affects how UIImage reports its width and height properties, transposing them from their bitmap values for images flagged as portrait.
cv::Mat doesn't bother with any of that. This means that (i) when translating to cv::Mat a portrait image will have its size.width and size.height values transposed, and (ii) when translating back from cv::Mat you will have lost the orientation flag.
The simplest way to handle this when going from UIImage to cv::Mat is to swap width and height values if the image is flagged as portrait:
if (self.imageOrientation == UIImageOrientationLeft
|| self.imageOrientation == UIImageOrientationRight) {
cols = self.size.height;
rows = self.size.width;
}
When translating back from cv::Mat to UIImage, you will want to reinstate the orientation flag. Assuming your cv::Mat -> UIImage code contains this:
self = [self initWithCGImage:imageRef];
you can use this method instead, and reset the orientation as per the original.
self = [self initWithCGImage:imageRef scale:1 orientation:orientation];

You should consider using native OpenCV functions to convert forth and back :
#import <opencv2/imgcodecs/ios.h>
...
UIImage* MatToUIImage(const cv::Mat& image);
void UIImageToMat(const UIImage* image,
cv::Mat& m, bool alphaExist = false);
Note: if your UIImage comes from the camera, you should 'normalize' it ( iOS UIImagePickerController result image orientation after upload) before converting to cv::Mat since OpenCV does not take into account Exif data. If you don't do that the result should be misoriented.

I was trying to convert a portrait UIImage to Mat and back, but I had problems with both approaches of this post.
The native approach with MatToUIImage and UIImageToMat resulted in a rotated image.
The custom approach with CGContext ended up in a strange color bug (not only RGB / BGR swap). Red was red, blue was pink, green was yellow.
I ended up using the native OpenCV with a rotation:
#import <opencv2/imgcodecs/ios.h>
- (cv::Mat)convertUIImageToCVMat:(UIImage *)image
{
cv::Mat imageMat;
UIImageToMat(image, imageMat);
cv::rotate(imageMat, imageMat, cv::ROTATE_90_CLOCKWISE);
return imageMat;
}
- (UIImage *)convertCVMatToUIImage:(cv::Mat)cvMat
{
UIImage *image = MatToUIImage(cvMat);
return image;
}

Related

iOS drawing pixels in a UIView

I have an 2D array of RGB values (or any such data container) that I need to write to UIView that is currently displayed to the user. An example would be — while using the capture output from the camera, I run some algorithms to identify objects and then highlight them using custom defined RGB pixels.
What is the best way to do this as this whole thing is done in real-time every 10 frames per second for example?
Use the method below to create a UIImage from your 2D array. You can then display this image using a UIImageView.
-(UIImage *)imageFromArray:(void *)array width:(unsigned int)width height:(unsigned int)height {
/*
Assuming pixel color values are 8 bit unsigned
You need to create an array that is in the format BGRA (blue,green,red,alpha).
You can achieve this by implementing a for-loop that sets the values at each index.
I have not included a for-loop in this example because it depends on how the values are stored in your input 2D array.
You can set the alpha value to 255.
*/
unsigned char pixelData[width * height * 4];
// This is where the for-loop would be
void *baseAddress = &pixelData;
size_t bytesPerRow = width * 4;
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8, bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
CGImageRef cgImage = CGBitmapContextCreateImage(context);
UIImage *image = [UIImage imageWithCGImage:cgImage];
CGImageRelease(cgImage);
CGColorSpaceRelease(colorSpace);
CGContextRelease(context);
return image;
}

Images being rotated when converted from Matrix

I am working on a app and making use of the opencv library.
The problem I am having happens only to certain images (usually if made with the phone's camera) and I pinpointed as being just a conversion problem. When I convert a (problematic) Image to a cv::Mat object and then back it just rotates 90 degrees.
Here is the call that causes the problem:
cv::Mat tmpMat = [sentImage CVMat];
UIImage * tmpImage = [[UIImage alloc] initWithCVMat:tmpMat];
[imageHolder setImage: tmpImage];
And here are the functions that do the conversion from image to matrix and vice-versa.
-(cv::Mat)CVMat
{
CGColorSpaceRef colorSpace = CGImageGetColorSpace(self.CGImage);
CGFloat cols = self.size.width;
CGFloat rows = self.size.height;
cv::Mat cvMat(rows, cols, CV_8UC4); // 8 bits per component, 4 channels
CGContextRef contextRef = CGBitmapContextCreate(cvMat.data, // Pointer to backing data
cols, // Width of bitmap
rows, // Height of bitmap
8, // Bits per component
cvMat.step[0], // Bytes per row
colorSpace, // Colorspace
kCGImageAlphaNoneSkipLast |
kCGBitmapByteOrderDefault); // Bitmap info flags
CGContextDrawImage(contextRef, CGRectMake(0, 0, cols, rows), self.CGImage);
CGContextRelease(contextRef);
return cvMat;
}
- (id)initWithCVMat:(const cv::Mat&)cvMat
{
NSData *data = [NSData dataWithBytes:cvMat.data length:cvMat.elemSize() * cvMat.total()];
CGColorSpaceRef colorSpace;
if (cvMat.elemSize() == 1)
{
colorSpace = CGColorSpaceCreateDeviceGray();
}
else
{
colorSpace = CGColorSpaceCreateDeviceRGB();
}
CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)data);
CGImageRef imageRef = CGImageCreate(cvMat.cols, // Width
cvMat.rows, // Height
8, // Bits per component
8 * cvMat.elemSize(), // Bits per pixel
cvMat.step[0], // Bytes per row
colorSpace, // Colorspace
kCGImageAlphaNone | kCGBitmapByteOrderDefault, // Bitmap info flags
provider, // CGDataProviderRef
NULL, // Decode
false, // Should interpolate
kCGRenderingIntentDefault); // Intent
self = [self initWithCGImage:imageRef];
CGImageRelease(imageRef);
CGDataProviderRelease(provider);
CGColorSpaceRelease(colorSpace);
return self;
}
Now I am using a "Aspect Fill" property in my imageHolder (a UIImageView) and tried changing it without success. I also tried seeing if it was a problem of a matrix being transposed on the conversion and tried to change without success and it also would not be logical since it does not turn every picture.
I do not understand why it works with some pictures but other not (all photos taken with the phone's camera don't work).
If anyone can shed a light on the matter I would appreciate.
Images from the camera that are taken with different orientations (Portrait / Landscape) are saved in the same resolution (same number of rows and columns) by the iPhone camera. The difference is that the JPEG contains a flag (to be precise, the Exif.Image.Orientation flag) to tell the displaying software how the image needs to be rotated to be displayed correctly.
My guess is that OpenCV looses that information (that is stored in the UIImage in the imageOrientation property) when converting, so when the image is converted back to UIImage this piece of information is set to default (UIImageOrientationUp), explaining why certain images appear to be rotated.
I was having the same issue. This solves the problem of the image rotating when converting from UIImage to cvMat. Add the method at the bottom, call it after you dismiss the picker controller. It is the 'second answer' located here: Rotating a CGImage
Also, there are two methods in the ios.h for UIImage to cvMat and vice versa, that you can just include. highgui/ios.h. Then add the rotation method and you are good to go.

iOS Performance Tuning: fastest way to get pixel color for large images

There are a number of questions/answers regarding how to get the pixel color of an image for a given point. However, all of these answers are really slow (100-500ms) for large images (even as small as 1000 x 1300, for example).
Most of the code samples out there draw to an image context. All of them take time when the actual draw takes place:
CGContextDrawImage(context, CGRectMake(0.0f, 0.0f, (CGFloat)width, (CGFloat)height), cgImage)
Examining this in Instruments reveals that the draw is being done by copying the data from the source image:
I have even tried a different means of getting at the data, hoping that getting to the bytes themselves would actually prove much more efficient.
NSInteger pointX = trunc(point.x);
NSInteger pointY = trunc(point.y);
CGImageRef cgImage = CGImageCreateWithImageInRect(self.CGImage,
CGRectMake(pointX * self.scale,
pointY * self.scale,
1.0f,
1.0f));
CGDataProviderRef provider = CGImageGetDataProvider(cgImage);
CFDataRef data = CGDataProviderCopyData(provider);
CGImageRelease(cgImage);
UInt8* buffer = (UInt8*)CFDataGetBytePtr(data);
CGFloat red = (float)buffer[0] / 255.0f;
CGFloat green = (float)buffer[1] / 255.0f;
CGFloat blue = (float)buffer[2] / 255.0f;
CGFloat alpha = (float)buffer[3] / 255.0f;
CFRelease(data);
UIColor *pixelColor = [UIColor colorWithRed:red green:green blue:blue alpha:alpha];
return pixelColor;
This method takes it's time on the data copy:
CFDataRef data = CGDataProviderCopyData(provider);
It would appear that it too is reading the data from disk, instead of the CGImage instance I am creating:
Now, this method, in some informal testing does perform better, but it is still not as fast I want it to be. Does anyone know of an even faster way of getting the underlying pixel data???
If it's possible for you to draw this image to the screen via OpenGL ES, you can get extremely fast random access to the underlying pixels in iOS 5.0 via the texture caches introduced in that version. They allow for direct memory access to the underlying BGRA pixel data stored in an OpenGL ES texture (where your image would be residing), and you could pick out any pixel from that texture almost instantaneously.
I use this to read back the raw pixel data of even large (2048x2048) images, and the read times are at worst in the range of 10-20 ms to pull down all of those pixels. Again, random access to a single pixel there takes almost no time, because you're just reading from a location in a byte array.
Of course, this means that you'll have to parse and upload your particular image to OpenGL ES, which will involve the same reading from disk and interactions with Core Graphics (if going through a UIImage) that you'd see if you tried to read pixel data from a random PNG on disk, but it sounds like you just need to render once and sample from it multiple times. If so, OpenGL ES and the texture caches on iOS 5.0 would be the absolute fastest way to read back this pixel data for something also displayed onscreen.
I encapsulate these processes in the GPUImagePicture (image upload) and GPUImageRawData (fast raw data access) classes within my open source GPUImage framework, if you want to see how something like that might work.
I have yet to find a way to get access to the drawn (in frame buffer) pixels. The fastest method I've measured is:
Indicate you want the image to be cached by specifying kCGImageSourceShouldCache when creating it.
(optional) Precache the image by forcing it to render.
Draw the image a 1x1 bitmap context.
The cost of this method is the cached bitmap, which may have a lifetime as long as the CGImage it is associated with. The code ends up looking something like this:
Create image w/ ShouldCache flag
NSDictionary *options = #{ (id)kCGImageSourceShouldCache: #(YES) };
CGImageSourceRef imageSource = CGImageSourceCreateWithData((__bridge CFDataRef)imageData, NULL);
CGImageRef cgimage = CGImageSourceCreateImageAtIndex(imageSource, 0, (__bridge CFDictionaryRef)options);
UIImage *image = [UIImage imageWithCGImage:cgimage];
CGImageRelease(cgimage);
Precache image
UIGraphicsBeginImageContext(CGSizeMake(1, 1));
[image drawAtPoint:CGPointZero];
UIGraphicsEndImageContext();
Draw image to a 1x1 bitmap context
unsigned char pixelData[] = { 0, 0, 0, 0 };
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef context = CGBitmapContextCreate(pixelData, 1, 1, 8, 4, colorSpace, kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGImageRef cgimage = image.CGImage;
int imageWidth = CGImageGetWidth(cgimage);
int imageHeight = CGImageGetHeight(cgimage);
CGContextDrawImage(context, CGRectMake(-testPoint.x, testPoint.y - imageHeight, imageWidth, imageHeight), cgimage);
CGColorSpaceRelease(colorSpace);
CGContextRelease(context);
pixelData has the R, G, B, and A values of the pixel at testPoint.
A CGImage context is possibly nearly empty and contains no actual pixel data until you try to read the first pixel or draw it, so trying to speed up getting pixels from an image might not get you anywhere. There's nothing to get yet.
Are you trying to read pixels from a PNG file? You could try going directly after the file and mmap'ing it and decoding the PNG format yourself. It will still take awhile to pull the data from storage.
- (BOOL)isWallPixel: (UIImage *)image: (int) x :(int) y {
CFDataRef pixelData = CGDataProviderCopyData(CGImageGetDataProvider(image.CGImage));
const UInt8* data = CFDataGetBytePtr(pixelData);
int pixelInfo = ((image.size.width * y) + x ) * 4; // The image is png
//UInt8 red = data[pixelInfo]; // If you need this info, enable it
//UInt8 green = data[(pixelInfo + 1)]; // If you need this info, enable it
//UInt8 blue = data[pixelInfo + 2]; // If you need this info, enable it
UInt8 alpha = data[pixelInfo + 3]; // I need only this info for my maze game
CFRelease(pixelData);
//UIColor* color = [UIColor colorWithRed:red/255.0f green:green/255.0f blue:blue/255.0f alpha:alpha/255.0f]; // The pixel color info
if (alpha) return YES;
else return NO;
}

Can you load only a smaller rectangular portion of a larger on-disk image into memory?

On iOS and most mobile devices there is a restriction on the size of the image that you can load, due to memory contraints. Is it possible to have a large image on disk (say 5,000 pixels by 5,000 pixels) but only read a smaller rectangle within that image (say 100x100) into memory for display?
In other words, do you need to load the entire image into memory if you just want to see a small subsection of it? If it's possible to load just the smaller portion, how can we do this?
This way, one could save a lot of space like spritesheets do for repetitive content. It would be important to note that the overall goal is to minimize the file size so the large image should be compressed with jpeg or png or some other kind of compression. I suspect video formats are like this because you never load an entire video into the memory.
Although I have not utilized the techniques, you might find the following Apple Sample useful:
LargeImageDownsizing Sample
You could do something with mapped NSData like this:
UIImage *pixelDataForRect(NSString *fileName, const CGRect pixelRect)
{
// get the pixels from that image
uint32_t width = pixelRect.size.width;
uint32_t height = pixelRect.size.height;
// create the context
UIGraphicsBeginImageContext(CGSizeMake(width, height));
CGContextRef bitMapContext = UIGraphicsGetCurrentContext();
CGAffineTransform flipVertical = CGAffineTransformMake(1, 0, 0, -1, 0, height);
CGContextConcatCTM(bitMapContext, flipVertical);
// render the image (assume PNG compression)
CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef) [NSData dataWithContentsOfMappedFile:fileName]);
CGImageRef image = CGImageCreateWithPNGDataProvider(provider, NULL, YES, kCGRenderingIntentDefault);
CGDataProviderRelease(provider);
uint32_t imageWidth = CGImageGetWidth(image);
uint32_t imageHeight = CGImageGetHeight(image);
CGRect drawRect = CGRectMake(-pixelRect.origin.x, -((imageHeight - pixelRect.origin.y) - height), imageWidth, imageHeight);
CGContextDrawImage(bitMapContext, drawRect, image);
CGImageRelease(image);
UIImage *retImage = UIGraphicsGetImageFromCurrentImageContext();
UIGraphicsEndImageContext();
return retImage;
}
Your best bet is using UIScrollView with CATiledLayer.
Check out the "Designing Apps with Scroll Views presentation from WWDC 2010 for a description of how to do this:
https://developer.apple.com/videos/wwdc/2010/
The idea is to take your large image and chop it down into tiles, and then use a UIScrollView to provide your user with a scrollable view of the image, only loading those sections of the image that are necessary based on the position of the scrollview. This is accomplished using CATiledLayer.

Converting Images from camera buffer iOS. Capture still image using AVFoundation

I'm using this well known sample code from Apple to convert camera buffer still images into UIImages.
-(UIImage*) getUIImageFromBuffer:(CMSampleBufferRef) imageSampleBuffer{
// Get a CMSampleBuffer's Core Video image buffer for the media data
CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(imageSampleBuffer);
if (imageBuffer==NULL) {
NSLog(#"No buffer");
}
// Lock the base address of the pixel buffer
if((CVPixelBufferLockBaseAddress(imageBuffer, 0))==kCVReturnSuccess){
NSLog(#"Buffer locked successfully");
}
void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
// Get the number of bytes per row for the pixel buffer
size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
NSLog(#"bytes per row %zu",bytesPerRow );
// Get the pixel buffer width and height
size_t width = CVPixelBufferGetWidth(imageBuffer);
NSLog(#"width %zu",width);
size_t height = CVPixelBufferGetHeight(imageBuffer);
NSLog(#"height %zu",height);
// Create a device-dependent RGB color space
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Create a bitmap graphics context with the sample buffer data
CGContextRef context = CGBitmapContextCreate(baseAddress, width, height, 8,
bytesPerRow, colorSpace, kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst);
// Create a Quartz image from the pixel data in the bitmap graphics context
CGImageRef quartzImage = CGBitmapContextCreateImage(context);
// Free up the context and color space
CGContextRelease(context);
CGColorSpaceRelease(colorSpace);
// Create an image object from the Quartz image
UIImage *image= [UIImage imageWithCGImage:quartzImage scale:SCALE_IMAGE_RATIO orientation:UIImageOrientationRight];
// Release the Quartz image
CGImageRelease(quartzImage);
// Unlock the pixel buffer
CVPixelBufferUnlockBaseAddress(imageBuffer,0);
return (image );}
The problem is that usually the image that you obtain is 90° rotated. Using the method +imageWithCGImage:scale:orientation I'm able to rotate it, but before getting into this method I was trying to rotate and scale the image using the CTM function, before passing it to a UIImage. the problem was that CTM transformation didn't affect the image.
I'm asking myself why... is that because I'm locking the buffer? or because the context is created with the image inside, so the changes will affect only the further mod?
Thank you
The answer is that it affects only further modifications, and it has nothing to deal with the buffer locking.
As you can read from this answer mod to the context are applied by time Vertical flip of CGContext

Resources