I am fighting with an internal caching (about 90 MB for 15 mp image ) in CGContextDrawImage/CGDataProviderCopyData functions.
Here is the stack-trace in profiler:
In all cases, IOSurface is created as a "cache", and isn't cleaned after #autoreleasepool is drained. This leaves a very few chances for an app to survive.
Caching doesn't depend on image size: I tried to render 512x512, as well as 4500x512 and 4500x2500 (full-size) image chunks.
I use #autoreleasepool, CFGetRetainCount returns 1 for all CG-objects before cleaning them.
The code which manipulates the data:
+ (void)render11:(CIImage*)ciImage fromRect:(CGRect)roi toBitmap:(unsigned char*)bitmap {
#autoreleasepool
{
int w = CGRectGetWidth(roi), h = CGRectGetHeight(roi);
CIContext* ciContext = [CIContext contextWithOptions:nil];
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGContextRef cgContext = CGBitmapContextCreate(bitmap, w, h,
8, w*4, colorSpace,
kCGImageAlphaPremultipliedLast | kCGBitmapByteOrder32Big);
CGImageRef cgImage = [ciContext createCGImage:ciImage
fromRect:roi
format:kCIFormatRGBA8
colorSpace:colorSpace
deferred:YES];
CGContextDrawImage(cgContext, CGRectMake(0, 0, w, h), cgImage);
assert( CFGetRetainCount(cgImage) == 1 );
CGColorSpaceRelease(colorSpace);
CGContextRelease(cgContext);
CGImageRelease(cgImage);
}
}
What I know about IOSurface: it's from the previously private framework IOSurface.
CIContext has a function render: ... toIOSurface:.
I've created my IOSurfaceRef and passed it to this function, and the internal implementation still creates its own surface, and doesn't clean it.
So, do you know (or assume):
1. Are there other ways to read CGImage's data buffer except
CGContextDrawImage/CGDataProviderCopyData ?
2. Is there a way to disable caching at render?
3. Why does the caching happen?
4. Can I use some lower-level (while non-private) API to manually clean up system memory?
Any suggestions are welcome.
To answer your second question,
Is there a way to disable caching at render?
setting the environment variable CI_SURFACE_CACHE_CAPACITY to 0 will more-or-less disable the CIContext surface cache. Moreover, you can specify a custom (approximate) cache limit by setting that variable to a given value in bytes. For example, setting CI_SURFACE_CACHE_CAPACITY to 2147483648 specifies a 2 GiB surface cache limit.
Note it appears that all of a process's CIContext instances share a single surface cache. It does not appear to be possible to use separate caches per CIContext.
If you just need to manipulate CIImage data, may consider to use CIImageProcessorKernel to put data into CPU or GPU calculation without extracting them.
I notice that
[ciContext
render:image toBitmap:bitmap rowBytes: w*4 bounds:image.extent format:kCIFormatRGBA8 colorSpace:colorSpace];
There is no such 90M cache. Maybe it's what you want.
Related
I'm looking at optimizing a routine that fetches the pixel data from a CGImage. The way this currently is done (very inefficiently) is to create a new CGContext, draw the CGImage into the context, and then get the data from the context.
I have the following optimized routine to handle this:
CGImageRef imageRef = image.CGImage;
uint8_t *pixelData = NULL;
CGDataProviderRef imageDataProvider = CGImageGetDataProvider(imageRef);
CFDataRef imageData = CGDataProviderCopyData(imageDataProvider);
pixelData = (uint8_t *)malloc(CFDataGetLength(imageData));
CFDataGetBytes(imageData, CFRangeMake(0, CFDataGetLength(imageData)), pixelData);
CFRelease(imageData);
This almost works. After viewing and comparing the hex dump of the pixel data obtained through both methods, I found that in the above case, there are 8 bytes of 0's every 6360 bytes. Otherwise, the data is identical. e.g.
And here is the comparison with the unoptimized version:
After the 8 bytes of 0's, the correct pixel data continues. Anyone know why this is happening?
UPDATE:
Here is the routine I am optimizing (the snipped code is just getting size info, and other non-important things; the relevant bit being the pixel data returned):
CGContextRef context = NULL;
CGImageRef imageRef = image.CGImage;
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
CGBitmapInfo bitmapInfo = kCGBitmapByteOrder32Little | kCGImageAlphaPremultipliedFirst;
// ... SNIP ...
context = CGBitmapContextCreate(...);
CGColorSpaceRelease(colorSpace);
// ... SNIP ...
CGContextDrawImage(context, rect, imageRef);
uint8_t *pixelData = (uint8_t *)CGBitmapContextGetData(context);
CGContextRelease(context);
Obviously this is an excessive amount of work just to get the underlying pixel data. Creating a context, then drawing into it. The first routine is between 5 - 10 times as fast. But as I pointed out, the pixel data returned by both routines are almost identical, except for the insertion of 8 zero-byte values every 6360 bytes in the optimized code (highlighted in the images).
Otherwise, everything else is the same -- color values, byte order, etc.
The bitmap data has padding at the end of each row of pixels, to round the number of bytes per row up to a larger value. (In this case, a multiple of 16 bytes.)
This padding is added to make it faster to process and draw the image.
You should use CGImageGetBytesPerRow() to find out how many bytes each row takes. Don't assume that it's the same as CGImageGetWidth() * CGImageGetBitsPerPixel() / 8; the bytes per row may be larger.
Keep in mind that the data behind an arbitrary CGImage may not be in the format that you expect. You cannot assume that all images are 32-bit-per-pixel ARGB with no padding. You should either use the CG functions to figure out what format the data might be, or redraw the image into a bitmap context that's in the exact format you expect. The latter is typically much easier -- let CG do the conversions for you.
(You don't show what parameters you're passing to CGBitmapContextCreate. Are you calculating an exact bytesPerRow or are you passing in 0? If you pass in 0, CG may add padding for you, and you may find that drawing into the context is faster.)
I'm currently working on a project that must generate a collage of a 9000x6000 pixels resolution, generated from 15 photos. The problem that I'm facing is that when I finish drawing I'm getting an empty image (those 15 images are not being drawn in the context).
This problem is only present on devices with 512MB of RAM like iPhone 4/4S or iPad 2 and I think that this is a problem caused by the system because it cannot allocate enough memory for this app. When I run this line: UIGraphicsBeginImageContextWithOptions(outputSize, opaque, 1.0f); the app's memory usage raises by 216MB and the total memory usage gets to ~240MB RAM.
The thing that I cannot understand is why on Earth the images that I'm trying to draw within the for loop are not being rendered always on the currentContext? I emphasized the word always, because only once in 30 tests the images were rendered (without changing any line of code).
Question nr. 2: If this is a problem caused by the system because it cannot allocate enough memory, is there any other way to generate this image, like a CGContextRef backed by a file output stream, so that it won't keep the image in the memory?
This is the code:
CGSize outputSize = CGSizeMake(9000, 6000);
BOOL opaque = YES;
UIGraphicsBeginImageContextWithOptions(outputSize, opaque, 1.0f);
CGContextRef currentContext = UIGraphicsGetCurrentContext();
CGContextSetFillColorWithColor(currentContext, [UIColor blackColor].CGColor);
CGContextFillRect(currentContext, CGRectMake(0, 0, outputSize.width, outputSize.height));
for (NSUInteger i = 0; i < strongSelf.images.count; i++)
{
#autoreleasepool
{
AGAutoCollageImageData *imageData = (AGAutoCollageImageData *)strongSelf.layout.images[i];
CGRect destinationRect = CGRectMake(floorf(imageData.destinationRectangle.origin.x * scaleXRatio),
floorf(imageData.destinationRectangle.origin.y * scaleYRatio),
floorf(imageData.destinationRectangle.size.width * scaleXRatio),
floorf(imageData.destinationRectangle.size.height * scaleYRatio));
CGRect sourceRect = imageData.sourceRectangle;
// Draw clipped image
CGImageRef clippedImageRef = CGImageCreateWithImageInRect(((ALAsset *)strongSelf.images[i]).defaultRepresentation.fullScreenImage, sourceRect);
CGContextDrawImage(currentContext, drawRect, clippedImageRef);
CGImageRelease(clippedImageRef);
}
}
// Pull the image from our context
strongSelf.result = UIGraphicsGetImageFromCurrentImageContext();
// Pop the context
UIGraphicsEndImageContext();
P.S: The console doesn't show anything but 'memory warnings', which are expected to see.
Sound like a cool project.
Tactic: try also releasing imageData at the end of every loop (explicitly, after releasing the clippedImageRef)
Strategic:
If you do need to support such "low" RAM requirements with such "high" input, maybe you should consider 2 alternative options:
Compress (obviously): even minimal, naked to the eye, JPEG compression can go a long way.
Split: never "really" merge the image. Have an arrayed datastructure which represents a BigImage. make utilities for the presentation logic.
I'm currently using a UIGraphicsBeginImageContext(resultingImageSize); to create an image.
But when I call this function, I don't know exactly the width of resultingImageSize.
Indeed, I developed some kind of video processing which consume lots of memory, and I cannot process first then draw after: I must draw during the video process.
If I set, for example UIGraphicsBeginImageContext(CGSizeMake(300, 400));, the drawn part over 400 is lost.
So is there a solution to set a variable size of CGContext, or resize a CGContext with very few memory consume?
I found a solution by creating a new larger Context each time it must be resized. Here's the magic function:
void MPResizeContextWithNewSize(CGContextRef *c, CGSize s) {
size_t bitsPerComponents = CGBitmapContextGetBitsPerComponent(*c);
size_t numberOfComponents = CGBitmapContextGetBitsPerPixel(*c) / bitsPerComponents;
CGContextRef newContext = CGBitmapContextCreate(NULL, s.width, s.height, bitsPerComponents, sizeof(UInt8)*s.width*numberOfComponents,
CGBitmapContextGetColorSpace(*c), CGBitmapContextGetBitmapInfo(*c));
// Copying context content
CGImageRef im = CGBitmapContextCreateImage(*c);
CGContextDrawImage(newContext, CGRectMake(0, 0, CGBitmapContextGetWidth(*c), CGBitmapContextGetHeight(*c)), im);
CGImageRelease(im);
CGContextRelease(*c);
*c = newContext;
}
I wonder if it could be optimized, for example with memcpy, as suggested here. I tried but it makes my code crash.
I have a simple drawing issue. I have prepared a 2 dimensional array which has an animated wave motion. The array is updated every 1/10th of a second (this can be changed by the user). After the array is updated I want to display it as a 2 dimensional image with each array value as a pixel with color range from 0 to 255.
Any pointers on how to do this most efficiently...
Appreciate any help on this...
KAS
If it's just a greyscale then the following (coded as I type, probably worth checking for errors) should work:
CGDataProviderRef dataProvider =
CGDataProviderCreateWithData(NULL, pointerToYourData, width*height, NULL);
CGColorSpaceRef colourSpace = CGColorSpaceCreateDeviceGray();
CGImageRef inputImage = CGImageCreate( width, height,
8, 8, width,
colourSpace,
kCGBitmapByteOrderDefault,
dataProvider,
NULL, NO,
kCGRenderingIntentDefault);
CGDataProviderRelease(dataProvider);
CGColorSpaceRelease(colourSpace);
UIImage *image = [UIImage imageWithCGImage:inputImage];
CGImageRelease(inputImage);
someImageView.image = image;
That'd be for a one-shot display, assuming you didn't want to write a custom UIView subclass (which is worth the effort only if performance is a problem, probably).
My understanding from the docs is that the data provider can be created just once for the lifetime of your C buffer. I don't think that's true of the image, but if you created a CGBitmapContext to wrap your buffer rather than a provider and an image, that would safely persist and you could use CGBitmapContextCreateImage to get a CGImageRef to be moving on with. It's probably worth benchmarking both ways around if it's an issue.
EDIT: so the alternative way around would be:
// get a context from your C buffer; this is now something
// CoreGraphics could draw to...
CGColorSpaceRef colourSpace = CGColorSpaceCreateDeviceGray();
CGContextRef context =
CGBitmapContextCreate(pointerToYourData,
width, height,
8, width,
colourSpace,
kCGBitmapByteOrderDefault);
CGColorSpaceRelease(colourSpace);
// get an image of the context, which is something
// CoreGraphics can draw from...
CGImageRef image = CGBitmapContextCreateImage(context);
/* wrap in a UIImage, push to a UIImageView, as before, remember
to clean up 'image' */
CoreGraphics copies things about very lazily, so neither of these solutions should be as costly as the multiple steps imply.
Does anyone know how to tell core image to process a CIImage through a CIFilter using the CPU instead of the GPU? I need to process some very large images and I get strange results using the GPU. I don't care how long it takes to CPU will be fine.
kCIContextUseSoftwareRenderer is key here:
+ (CIContext*)coreContextFor:(NSGraphicsContext *)context forceSoftware:(BOOL)forceSoftware
{
//CGColorSpaceRef colorSpace = CGColorSpaceCreateWithName(kCGColorSpaceGenericRGB);
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
NSDictionary *contextOptions = [NSDictionary dictionaryWithObjectsAndKeys:
(id)colorSpace, kCIContextWorkingColorSpace,
(id)colorSpace, kCIContextOutputColorSpace,
[NSNumber numberWithBool:forceSoftware], kCIContextUseSoftwareRenderer,
nil];
CIContext* result = [CIContext contextWithCGContext:(CGContext *)[context graphicsPort] options:contextOptions];
CGColorSpaceRelease(colorSpace);
return result;
}
Rendering in software more (CPU) solves some issues, but... Performance penalty is so strong on modern machines that I can't say it is solution. I use CoreImage in my apps and I always render with GPU on-screen while forced CPU is used for saving only. I have noticed that CPU rendering is a bit more accurate on my test hardwares, and saving filtered image is a long process, I can sacrifice the speed here.
Here is a Swift version :
func coreContextFor(context : CGContext,forceSoftware force : Bool) -> CIContext {
let colorSpace = CGColorSpaceCreateDeviceRGB()
let options : [String:AnyObject] = [
kCIContextWorkingColorSpace: colorSpace!,
kCIContextOutputColorSpace : colorSpace!,
kCIContextUseSoftwareRenderer : NSNumber(bool: force)
]
return CIContext(CGContext: context, options: options)
}
Not an answer, but you always can re-write CIkernel as function which operates on pixel RGB value, and after that apply that function in a loop on unsigned char [] data of image and convert result to CIImage.