I have used vImageHistogramCalculation in my current application for calculating the histogram of image and I am getting EXC_BAD_ACCESS in some cases.I went through the cases as follows-
- (void)histogramForImage:(UIImage *)image {
vImage_Buffer inBuffer;
CGImageRef img = image.CGImage;
CGDataProviderRef inProvider = CGImageGetDataProvider(img);
CFDataRef inBitmapData = CGDataProviderCopyData(inProvider);
inBuffer.width = CGImageGetWidth(img);
inBuffer.height = CGImageGetHeight(img);
inBuffer.rowBytes = CGImageGetBytesPerRow(img);
inBuffer.data = (void*)CFDataGetBytePtr(inBitmapData);
vImagePixelCount histogram[4][8] = {{0}};
vImagePixelCount *histogramPointers[4] = { &histogram[0][0], &histogram[1][0], &histogram[2][0], &histogram[3][0] };
vImage_Error error = vImageHistogramCalculation_ARGBFFFF(&inBuffer, histogramPointers, 8, 0, 255, kvImageNoFlags);
if (error) {
NSLog(#"error %ld", error);
}
CGDataProviderRelease(inProvider);
}
I used the image from iPhone Camera Roll in PNG form and manually put in bundle in the above code and it works fine.
I used the same code for the image from iPhone Camera Roll in JPG format then I am getting EXC_BAD_ACCESS error.
I tried to get image from Camera Roll using Photos Framework and then passed to the same code then also I am getting EXC_BAD_ACCESS error.
What I actually need is to find the histogram of all the images of iPhone Camera Roll.So I am unable to find out why the above code is working fine for one image format and failing for other.Is there any other reason for the crash?
EDIT 1- what I came to its not about image format its working fine for some JPG images too but still its crashing in some cases.How should I figure that out?
Reference:
https://developer.apple.com/library/mac/documentation/Performance/Reference/vImage_histogram/#//apple_ref/c/func/vImageHistogramCalculation_ARGBFFFF
Compute the histogram of an image using vImageHistogramCalculation
vImageHistogramCalculation_ARGBFFFF is for four-channel floating-point data. Chances are extremely high that the data you are getting out of the data provider is RGB or RGBA 8-bit integer data. Check with the CGImageRef for the storage format of the image data.
If you want a specific data format out of a CGImageRef, you can call vImageBuffer_InitWithCGImage().
Related
I am currently using tesseract to scan receipts. The quality wasn't good so I read this article on how to improve it: https://github.com/tesseract-ocr/tesseract/wiki/ImproveQuality#noise-removal. I implemented resizing, deskewing(aligning), and gaussian blur. But none of them seem to have a positive effect on the accuracy of the OCR except the deskewing. Here is my code for resizing and gaussian blur. Am I doing anything wrong? If not, what else can I do to help?
Code:
+(UIImage *) prepareImage: (UIImage *)image{
//converts UIImage to Mat format
Mat im = cvMatWithImage(image);
//grayscale image
Mat gray;
cvtColor(im, gray, CV_BGR2GRAY);
//deskews text
//did not provide code because I know it works
Mat preprocessed = preprocess2(gray);
double skew = hough_transform(preprocessed, im);
Mat rotated = rot(im,skew* CV_PI/180);
//resize image
Mat scaledImage = scaleImage(rotated, 2);
//Guassian Blur
GaussianBlur(scaledImage, scaledImage, cv::Size(1, 1), 0, 0);
return UIImageFromCVMat(scaledImage);
}
// Organization -> Resizing
Mat scaleImage(Mat mat, double factor){
Mat resizedMat;
double width = mat.cols;
double height = mat.rows;
double aspectRatio = width/height;
resize(mat, resizedMat, cv::Size(width*factor*aspectRatio, height*factor*aspectRatio));
return resizedMat;
}
Receipt:
If you read the Tesseract documentation you will see that tesseract engine works best with texts in a single line in a square. Passing it the whole receipt image reduces the engine's accuracy. What you need to do is use the new iOS framework CITextFeature to detect texts in your receipt into multiple blocks of images. Then only you can pass those images to tesseract for processing.
I've been struggling to figure out what the problem is with my code. I'm creating a planar CVPixelBufferRef to write to an AVAssetWriter. This pixel buffer is created manually through some other process (i.e., I'm not getting these samples from the camera or anything like that). On the iOS Simulator, it has no problem appending the samples and creating a valid output movie.
But on the device, it immediately fails at the first sample and provides less than useless error information:
AVAssetWriterError: Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSUnderlyingError=0x12fd2c670 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}, NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed}
I'm very new to pixel formats, and I wouldn't be surprised if I've somehow created invalid pixel buffers, but the fact that it works just fine on the Simulator (i.e., OS X) leaves me confused.
Here's my code:
const int pixelBufferWidth = img->get_width();
const int pixelBufferHeight = img->get_height();
size_t planeWidths[3];
size_t planeHeights[3];
size_t planeBytesPerRow[3];
void* planeBaseAddresses[3];
for (int c=0;c<3;c++) {
int stride;
const uint8_t* p = de265_get_image_plane(img, c, &stride);
int width = de265_get_image_width(img,c);
int height = de265_get_image_height(img, c);
planeWidths[c] = width;
planeHeights[c] = height;
planeBytesPerRow[c] = stride;
planeBaseAddresses[c] = const_cast<uint8_t*>(p);
}
void* descriptor = calloc(1, sizeof(CVPlanarPixelBufferInfo_YCbCrPlanar));
CVPixelBufferRef pixelBufferRef;
CVReturn result = CVPixelBufferCreateWithPlanarBytes(NULL,
pixelBufferWidth,
pixelBufferHeight,
kCVPixelFormatType_420YpCbCr8Planar,
NULL,
0,
3,
planeBaseAddresses,
planeWidths,
planeHeights,
planeBytesPerRow,
&pixelBufferReleaseCallback,
NULL,
NULL,
&pixelBufferRef);
CMFormatDescriptionRef formatDescription = NULL;
CMVideoFormatDescriptionCreateForImageBuffer(NULL, pixelBufferRef, &formatDescription);
if (assetWriter == nil) {
// ... create output file path in Caches directory
assetWriter = [AVAssetWriter assetWriterWithURL:fileOutputURL fileType:AVFileTypeMPEG4 error:nil];
NSDictionary *videoSettings = #{AVVideoCodecKey : AVVideoCodecH264,
AVVideoWidthKey : #(pixelBufferWidth),
AVVideoHeightKey : #(pixelBufferHeight),
AVVideoCompressionPropertiesKey : #{AVVideoMaxKeyFrameIntervalKey : #1}};
assetWriterInput = [AVAssetWriterInput assetWriterInputWithMediaType:AVMediaTypeVideo outputSettings:videoSettings sourceFormatHint:formatDescription];
[assetWriter addInput:assetWriterInput];
NSDictionary *pixelBufferAttributes = #{(id)kCVPixelBufferPixelFormatTypeKey : #(kCVPixelFormatType_420YpCbCr8Planar),
(id)kCVPixelBufferWidthKey : #(pixelBufferWidth),
(id)kCVPixelBufferHeightKey : #(pixelBufferHeight)};
pixelBufferAdaptor = [AVAssetWriterInputPixelBufferAdaptor assetWriterInputPixelBufferAdaptorWithAssetWriterInput:assetWriterInput sourcePixelBufferAttributes:pixelBufferAttributes];
[assetWriter startWriting];
[assetWriter startSessionAtSourceTime:kCMTimeZero];
}
samplePresentationTime = CMTimeMake(frameIndex++, framesPerSecond);
BOOL success = [pixelBufferAdaptor appendPixelBuffer:pixelBufferRef withPresentationTime:samplePresentationTime];
success is always NO, and the error from the asset writer is what I pasted above.
I also tried creating the sample buffers manually instead of using AVAssetWriterInputPixelBufferAdaptor just to eliminate that as a possible problem, but the results are the same.
Again, this does work on the Simulator, so I know my pixel buffers do contain the right data.
Also, I verified that I can write to the file destination. I tried creating a dummy file at that location, and it succeeded.
I would like to avoid converting my buffer to RGB since I shouldn't have to. I have Y'CbCr buffers to begin with, and I want to just encode them into an H.264 video, which supports Y'CbCr.
The source that is creating these buffers states the following:
The image is currently always 3-channel YCbCr, with 4:2:0 chroma.
I confirmed that it always enters its loop logic that deals with 8-bit YUV channels.
What am I doing wrong?
So, I can't confirm this officially, but it appears that AVAssetWriter doesn't like 3-plane pixel formats (i.e., kCVPixelFormatType_420YpCbCr8Planar) on iOS. On OS X, it appears to work with pretty much anything. When I converted my 3-plane buffers to a bi-planar pixel buffer format, this worked on iOS. This is unsurprising since the camera natively captures in kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange pixel format, so AV Foundation would likely also work with that format.
Still, it'd be nice if I didn't have to do this explicit conversion step myself, though vImageConvert_PlanarToChunky8 helps to interleave the Cb and Cr planes into a single plane.
I'm trying to use openCV 3 on iOS to produce an HDR image from multiple exposures that will eventually be output as an EXR file. I noticed I was getting garbled output when I tried to create an HDR image. Thinking it was a mistake in trying to create a camera response, I started from scratch and adapted the HDR imaging tutorial material on the openCV to iOS but it produces similar results. The following C++ code returns a garbled image:
cv::Mat mergeToHDR (vector<Mat>& images, vector<float>& times)
{
imgs = images;
Mat response;
//Ptr<CalibrateDebevec> calibrate = createCalibrateDebevec();
//calibrate->process(images, response, times);
Ptr<CalibrateRobertson> calibrate = createCalibrateRobertson();
calibrate->process(images, response, times);
// create HDR
Mat hdr;
Ptr<MergeDebevec> merge_debevec = createMergeDebevec();
merge_debevec->process(images, hdr, times, response);
// create LDR
Mat ldr;
Ptr<TonemapDurand> tonemap = createTonemapDurand(2.2f);
tonemap->process(hdr, ldr);
// create fusion
Mat fusion;
Ptr<MergeMertens> merge_mertens = createMergeMertens();
merge_mertens->process(images, fusion);
/*
Uncomment what kind of tonemapped image or hdr to return
Returning one of the images in the array produces ungarbled output
so we know the problem is unlikely with the openCV to UIImage conversion
*/
//give back one of the images from the image array
//return images[0];
//give back one of the hdr images
return fusion * 255;
//return ldr * 255;
//return hdr
}
This is what the image looks like:
Bad image output
I've analysed the image, tried various colour space conversions, but the data appears to be junk.
The openCV framework is the latest compiled 3.0.0 version from the openCV.org website. The RC and alpha produce the same results, and the current version won't build (for iOS or OSX). I was thinking my next steps would be to try and get the framework to compile from scratch, or to get the example working under another platform to see if the issue is platform specific or with the openCV HDR functions themselves. But before I do that I thought I would throw the issue up on stack overflow to see if anyone had come across the same issue or if I am missing something blindingly obvious.
I have uploaded the example xcode project to here:
https://github.com/artandmath/openCVHDRSwiftExample
Getting openCV to work with swift was with the help from user foundry on Github
Thanks foundry for pointing me in the right direction. The UIImage+OpenCV class extension is expecting 8-bits per colour channel, however the HDR functions are spitting out 32-bits per channel (which is actually what I want). Converting the image matrix back to 8-bits per channel for display purposes before converting it to a UIImage fixes the issue.
Here is the resulting image:
The expected result!
Here is the fixed function:
cv::Mat mergeToHDR (vector<Mat>& images, vector<float>& times)
{
imgs = images;
Mat response;
//Ptr<CalibrateDebevec> calibrate = createCalibrateDebevec();
//calibrate->process(images, response, times);
Ptr<CalibrateRobertson> calibrate = createCalibrateRobertson();
calibrate->process(images, response, times);
// create HDR
Mat hdr;
Ptr<MergeDebevec> merge_debevec = createMergeDebevec();
merge_debevec->process(images, hdr, times, response);
// create LDR
Mat ldr;
Ptr<TonemapDurand> tonemap = createTonemapDurand(2.2f);
tonemap->process(hdr, ldr);
// create fusion
Mat fusion;
Ptr<MergeMertens> merge_mertens = createMergeMertens();
merge_mertens->process(images, fusion);
/*
Uncomment what kind of tonemapped image or hdr to return
Convert back to 8-bits per channel because that is what
the UIImage+OpenCV class extension is expecting
*/
// tone mapped
/*
Mat ldr8bit;
ldr = ldr * 255;
ldr.convertTo(ldr8bit, CV_8U);
return ldr8bit;
*/
// fusion
Mat fusion8bit;
fusion = fusion * 255;
fusion.convertTo(fusion8bit, CV_8U);
return fusion8bit;
// hdr
/*
Mat hdr8bit;
hdr = hdr * 255;
hdr.convertTo(hdr8bit, CV_8U);
return hdr8bit;
*/
}
Alternatively here is a fix for the initWithCVMat method in the OpenCV+UIImage class extension based on one of the iOS tutorials in the iOS section on opencv.org:
http://docs.opencv.org/2.4/doc/tutorials/ios/image_manipulation/image_manipulation.html#opencviosimagemanipulation
When creating a new CGImageRef with floating point data, it needs to be explicitly told that it expects floating point data, and the byte order of the image data from openCV needs to be reversed. Now iOS/Quartz has the float data! It's a bit of a hacky fix, because the method still only deals with 8 bit or 32 bits per channel or alphas and doesn't take into account every kind of image that could be passed from Mat to UIImage.
- (id)initWithCVMat:(const cv::Mat&)cvMat
{
NSData *data = [NSData dataWithBytes:cvMat.data length:cvMat.elemSize() * cvMat.total()];
CGColorSpaceRef colorSpace;
size_t elemSize = cvMat.elemSize();
size_t elemSize1 = cvMat.elemSize1();
size_t channelCount = elemSize/elemSize1;
size_t bitsPerChannel = 8 * elemSize1;
size_t bitsPerPixel = bitsPerChannel * channelCount;
if (channelCount == 1) {
colorSpace = CGColorSpaceCreateDeviceGray();
} else {
colorSpace = CGColorSpaceCreateDeviceRGB();
}
// Tell CGIImageRef different bitmap info if handed 32-bit
uint32_t bitmapInfo = kCGImageAlphaNone | kCGBitmapByteOrderDefault;
if (bitsPerChannel == 32 ){
bitmapInfo = kCGImageAlphaNoneSkipLast | kCGBitmapFloatComponents | kCGBitmapByteOrder32Little;
}
CGDataProviderRef provider = CGDataProviderCreateWithCFData((__bridge CFDataRef)data);
// Creating CGImage from cv::Mat
CGImageRef imageRef = CGImageCreate(cvMat.cols, //width
cvMat.rows, //height
bitsPerChannel, //bits per component
bitsPerPixel, //bits per pixel
cvMat.step[0], //bytesPerRow
colorSpace, //colorspace
bitmapInfo, // bitmap info
provider, //CGDataProviderRef
NULL, //decode
false, //should interpolate
kCGRenderingIntentDefault //intent
);
// Getting UIImage from CGImage
self = [self initWithCGImage:imageRef];
CGImageRelease(imageRef);
CGDataProviderRelease(provider);
CGColorSpaceRelease(colorSpace);
return self;
}
From iOS6, Apple has given the provision to use native YUV to CIImage through this call
initWithCVPixelBuffer:options:
In the core Image Programming guide, they have mentioned about this feature
Take advantage of the support for YUV image in iOS 6.0 and later.
Camera pixel buffers are natively YUV but most image processing
algorithms expect RBGA data. There is a cost to converting between the
two. Core Image supports reading YUB from CVPixelBuffer objects and
applying the appropriate color transform.
options = #{ (id)kCVPixelBufferPixelFormatTypeKey :
#(kCVPixelFormatType_420YpCvCr88iPlanarFullRange) };
But, I am unable to use it properly. I have a raw YUV data. So, this is what i did
void *YUV[3] = {data[0], data[1], data[2]};
size_t planeWidth[3] = {width, width/2, width/2};
size_t planeHeight[3] = {height, height/2, height/2};
size_t planeBytesPerRow[3] = {stride, stride/2, stride/2};
CVPixelBufferRef pixelBuffer = NULL;
CVReturn ret = CVPixelBufferCreateWithPlanarBytes(kCFAllocatorDefault,
width,
height,
kCVPixelFormatType_420YpCbCr8PlanarFullRange,
nil,
width*height*1.5,
3,
YUV,
planeWidth,
planeHeight,
planeBytesPerRow,
nil,
nil, nil, &pixelBuffer);
NSDict *opt = #{ (id)kCVPixelBufferPixelFormatTypeKey :
#(kCVPixelFormatType_420YpCbCr8PlanarFullRange) };
CIImage *image = [[CIImage alloc] initWithCVPixelBuffer:pixelBuffer options:opt];
I am getting nil for image. Anyy idea what I am missing.
EDIT:
I added lock and unlock base address before call. Also, I dumped the data of pixelbuffer to ensure pixellbuffer propely hold the data. It looks like something wrong with the init call only. Still CIImage object is returning nil.
CVPixelBufferLockBaseAddress(pixelBuffer, 0);
CIImage *image = [[CIImage alloc] initWithCVPixelBuffer:pixelBuffer options:opt];
CVPixelBufferUnlockBaseAddress(pixelBuffer,0);
There should be error message in console: initWithCVPixelBuffer failed because the CVPixelBufferRef is not IOSurface backed. See Apple's Technical Q&A QA1781 for how to create an IOSurface-backed CVPixelBuffer.
Calling CVPixelBufferCreateWithBytes() or CVPixelBufferCreateWithPlanarBytes() will result in CVPixelBuffers that are not IOSurface-backed...
...To do that, you must specify kCVPixelBufferIOSurfacePropertiesKey in the pixelBufferAttributes dictionary when creating the pixel buffer using CVPixelBufferCreate().
NSDictionary *pixelBufferAttributes = [NSDictionary dictionaryWithObjectsAndKeys:
[NSDictionary dictionary], (id)kCVPixelBufferIOSurfacePropertiesKey,
nil];
// you may add other keys as appropriate, e.g. kCVPixelBufferPixelFormatTypeKey, kCVPixelBufferWidthKey, kCVPixelBufferHeightKey, etc.
CVPixelBufferRef pixelBuffer;
CVPixelBufferCreate(... (CFDictionaryRef)pixelBufferAttributes, &pixelBuffer);
Alternatively, you can make IOSurface-backed CVPixelBuffers using CVPixelBufferPoolCreatePixelBuffer() from an existing pixel buffer pool, if the pixelBufferAttributes dictionary provided to CVPixelBufferPoolCreate() includes kCVPixelBufferIOSurfacePropertiesKey.
I am working on a similar problem and kept finding that same quote from Apple without any further information on how to work in a YUV color space. I came upon the following:
By default, Core Image assumes that processing nodes are 128 bits-per-pixel, linear light, premultiplied RGBA floating-point values that use the GenericRGB color space. You can specify a different working color space by providing a Quartz 2D CGColorSpace object. Note that the working color space must be RGB-based. If you have YUV data as input (or other data that is not RGB-based), you can use ColorSync functions to convert to the working color space. (See Quartz 2D Programming Guide for information on creating and using CGColorspace objects.)
With 8-bit YUV 4:2:2 sources, Core Image can process 240 HD layers per gigabyte. Eight-bit YUV is the native color format for video source such as DV, MPEG, uncompressed D1, and JPEG. You need to convert YUV color spaces to an RGB color space for Core Image.
I note that there are no YUV color spaces, only Gray and RGB; and their calibrated cousins. I'm not sure how to convert the color space yet, but will certainly report here if I find out.
I'm working on a photo app for iPhone/iPod.
I'd like to get the raw data from a large image in an iPhone app and perform some pixel manipulation on it and write it back to the disk/gallery.
So far I've been converting the UIImage obtained from image picker to unsigned char pointers using the following technique:
CGImageRef imageBuff = [imageBuffer CGImage];//imageBuffer is an UIImage *
CFDataRef pixelData = CGDataProviderCopyData(CGImageGetDataProvider(imageBuff));
unsigned char *input_image = (unsigned char *)CFDataGetBytePtr(pixelData);
//height & width represents the dimensions of the input image
unsigned char *resultant = (unsigned char *)malloc(height*4*width);
for (int i=0; i<height;i++)
{
for (int j=0; j<4*width; j+=4)
{
resultant[i*4*width+4*(j/4)+0] = input_image[i*4*width+4*(j/4)];
resultant[i*4*width+4*(j/4)+1] = input_image[i*4*width+4*(j/4)+1];
resultant[i*4*width+4*(j/4)+2] = input_image[i*4*width+4*(j/4)+2];
resultant[i*4*width+4*(j/4)+3] = 255;
}
}
CFRelease(pixelData);
I'm doing all operations on resultant and writing it back to disk in the original resolution using:
NSData* data = UIImagePNGRepresentation(image);
[data writeToFile:path atomically:YES];
I'd like to know:
is the transformation actually lossless?
if there's a 20-22 MP image at hand... is it wise to do this operation in a background thread? (chances of crashing etc... I'd like to know the best practice for doing this).
is there a better method for implementing this (getting the pixel data is a necessity here)?
Yes the method is lossless but i am not sure about 20-22 MP images. I think the Iphone is not at all a suitable choice if i want to edit that big images!
I have been successful in capturing and editing image upto 22 MP using this technique.
Tested this on an iPhone 4s and it worked fine. However, some of the effects I'm using required Core Image filters. It seems like CIFilters do not support more than 16 MP images. the filters will return a blank image if used on an image >16MP.
I'd still like people to comment on lossless large image editing strategies in iOS.