Per the Getting the Best Performance Page,
Use Core Graphics or Image I/O functions to crop or downsample, such as the functions CGImageCreateWithImageInRect or CGImageSourceCreateThumbnailAtIndex.
However, I'm wondering how true this is if you're working solely in Core Image for image processing. If I have an image that needs to be downsampled and then filtered, along with other things, wouldn't it be less efficient to convert to CGImage, downsample, then convert back to CIImage for other uses?
I'm wondering if it would simply be better to work in the Core Image framework if downsampling is apart of the image processing algorithm you're performing. Certainly if the above is faster I'd like to give it a try, but I'm not sure there's any other way to downsample something as fast as possible. No, unfortunately CILanczosScaleTransform is horribly slow, I wish Core Image had a faster way in built to scale images besides this.
I'm using the code below, which I found here:
http://flexmonkey.blogspot.com/2014/12/scaling-resizing-and-orienting-images.html
extension UIImage {
public func resizeToBoundingSquare(_ boundingSquareSideLength : CGFloat) -> UIImage {
let imgScale = self.size.width > self.size.height ? boundingSquareSideLength / self.size.width : boundingSquareSideLength / self.size.height
let newWidth = self.size.width * imgScale
let newHeight = self.size.height * imgScale
let newSize = CGSize(width: newWidth, height: newHeight)
UIGraphicsBeginImageContext(newSize)
self.draw(in: CGRect(x: 0, y: 0, width: newWidth, height: newHeight))
let resizedImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext();
return resizedImage!
}
}
After downsizing things and/or making various pixel sizes consistent, I then use CI filters, both custom and chaining. I'm not seeing any performance or memory issues.
I think that I have seen fastest performance by using a CGAffineTransform(scale X, y) on the CIImage itself. I tried the CGImagesource thumbnail method and the overhead way outstripped any benefits, perhaps due to having to do more source conversions. Note that I am starting with a CVSampleBuffer so my chain is like:
CVSampleBuffer -> CIImage -> downsample with CGAffineTransform -> CIFiltering using a filter chain... input directly to VNRequest. I am able to get 60Hz direct-from-camera realtime processing using this method, although I would like to optimize it further which is why I am searching for an even faster option!
Related
I'm testing a rotation of an image on my Swift project using unit tests.
I'm getting different results to pngData() when I test with iOS 13.7 then when I test with iOS 11.2.
Also strange and I think it's related and also my real problem is that -
On iOS 13.7 comparing 2 images - static image and rotated image - return that they are the same data size.
On iOS 11.2 - my static image changed its data size by X AMOUNT and my rotated image changed its data size by Y AMOUNT and now they have different data sizes and my test fails.
The rotate func -
func cld_rotate(_ degree: Float) -> UIImage? {
var newSize = CGRect(origin: CGPoint.zero, size: self.size).applying(CGAffineTransform(rotationAngle: cld_radians(from: Double(degree)))).size
// Trim off the extremely small float value to prevent core graphics from rounding it up
newSize.width = floor(newSize.width)
newSize.height = floor(newSize.height)
UIGraphicsBeginImageContextWithOptions(newSize, false, self.scale)
let context = UIGraphicsGetCurrentContext()!
// Move origin to middle
context.translateBy(x: newSize.width/2, y: newSize.height/2)
// Rotate around middle
context.rotate(by: cld_radians(from: Double(degree)))
// Draw the image at its center
draw(in: CGRect(x: -self.size.width/2, y: -self.size.height/2, width: self.size.width, height: self.size.height))
let newImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
return newImage
}
Any idea about why pngData() is not the same used in 2 different iOS? And why the rotated image change its data size by a different amount than the static one?
PNG images are compressed. Rotating the image will likely require that it be re-compressed, which may yield different output. I would expect the filesize to change slightly, since compression algorithms can yield different file sizes based on different input data. (I originally stated that PNG images used lossy compression, but I was mistaken.)
To me the question is how does iOS 13.7 preserve the file size on rotation. I wonder if it is able to recognize a 90 degree rotation and transform the compressed image data somehow, where iOS 11.2 isn't able to do that? (My guess is that the image compression/decompression algorithm got smarter between iOS 11.2 and iOS 13.7, and now it's able to recognize a 90 degree rotation and use an algorithm on the data without having to decompresss and re-compress the image.)
I'm not sure what you are saying about a static image. Are you saying that you open the PNG image into a UIImage and then export it back to a PNG without transforming it?
I'm currently implementing some sort of coloring book and I'm curious about the best way to change pixels in UIImage. Here is my code:
self.context = CGContext(data: nil, width: image.width, height: image.height, bitsPerComponent: 8, bytesPerRow: image.width * 4, space: colorSpace, bitmapInfo: CGBitmapInfo.byteOrder32Little.rawValue | CGImageAlphaInfo.premultipliedFirst.rawValue)!
self.context?.draw(image.cgImage, in: CGRect(x: 0, y: 0, width: CGFloat(image.width), height: CGFloat(image.height)))
let ptr = context.data
self.pixelBuffer = ptr!.bindMemory(to: UInt32.self, capacity: image.width * image.height)
And change pixels using this function:
#inline (__always) func fill(matrixPosition: MatrixPosition, color: UInt32) {
pixelsBuffer?[self.linearIndex(for: matrixPosition)] = color
}
The problem is that every time when I change pixels I have to invoke makeImage on context to generate new image and it takes a lot of time:
func generateImage() -> UIImage {
let cgImage = context.makeImage()!
let uiimage = UIImage(cgImage: cgImage)
return uiimage
}
Does my approach is correct? What are better and faster ways to implement it? Thanks.
Manipulating individual pixels and then copying the entire memory buffer to a CGContext and then creating a UIImage with that context is going to end up being inefficient, as you are discovering.
You can continue to improve and optimize a CoreGraphics canvas approach by being more efficient about what part of your offscreen is copied onto screen. You can detect the pixels that have changed and only copy the minimum bounding rectangle of those pixels onto screen. This approach may be good enough for your use case where you are only filling in areas with colors.
Instead of copying the entire offscreen, copy just the changed area:
self.context?.draw(image.cgImage, in: CGRect(x: diffX, y: diffY, width: diffWidth, height: diffHeight))
It is up to you to determine the changed rectangle and when to update the screen.
Here is an example of a painting app that uses CoreGraphics, CoreImage and CADisplayLink. The code is a bit old, but the concepts are still valid and will serve as a good starting point. You can see how the changes are accumulated and drawn to the screen using a CADisplayLink.
If you want to introduce various types of ink and paint effects, a CoreGraphics approach is going to be more challenging. You will want to look at Apple's Metal API. A good tutorial is here.
I'm working with a CIImage, and while I understand it's not a linear image, it does hold some data.
My question is whether or not a CIImage's extent property returns pixels or points? According to the documentation, which says very little, it's working space coordinates. Does this mean there's no way to get the pixels / points from a CIImage and I must convert to a UIImage to use the .size property to get the points?
I have a UIImage with a certain size, and when I create a CIImage using the UIImage, the extent is shown in points. But if I run a CIImage through a CIFilter that scales it, I sometimes get the extent returned in pixel values.
I'll answer the best I can.
If your source is a UIImage, its size will be the same as the extent. But please, this isn't a UIImageView (which the size is in points). And we're just talking about the source image.
Running something through a CIFilter means you are manipulating things. If all you are doing is manipulating color, its size/extent shouldn't change (the same as creating your own CIColorKernel - it works pixel-by-pixel).
But, depending on the CIFilter, you may well be changing the size/extent. Certain filters create a mask, or tile. These may actually have an extent that is infinite! Others (blurs are a great example) sample surrounding pixels so their extent actually increases because they sample "pixels" beyond the source image's size. (Custom-wise these are a CIWarpKernel.)
Yes, quite a bit. Taking this to a bottom line:
What is the filter doing? Does it need to simply check a pixel's RGB and do something? Then the UIImage size should be the output CIImage extent.
Does the filter produce something that depends on the pixel's surrounding pixels? Then the output CIImage extent is slightly larger. How much may depend on the filter.
There are filters that produce something with no regard to an input. Most of these may have no true extent, as they can be infinite.
Points are what UIKit and CoreGraphics always work with. Pixels? At some point CoreImage does, but it's low-level to a point (unless you want to write your own kernel) you shouldn't care. Extents can usually - but keep in mind the above - be equated to a UIImage size.
EDIT
Many images (particularly RAW ones) can have so large a size as to affect performance. I have an extension for UIImage that resizes an image to a specific rectangle to help maintain consistent CI performance.
extension UIImage {
public func resizeToBoundingSquare(_ boundingSquareSideLength : CGFloat) -> UIImage {
let imgScale = self.size.width > self.size.height ? boundingSquareSideLength / self.size.width : boundingSquareSideLength / self.size.height
let newWidth = self.size.width * imgScale
let newHeight = self.size.height * imgScale
let newSize = CGSize(width: newWidth, height: newHeight)
UIGraphicsBeginImageContext(newSize)
self.draw(in: CGRect(x: 0, y: 0, width: newWidth, height: newHeight))
let resizedImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext();
return resizedImage!
}
}
Usage:
image = image.resizeToBoundingSquare(640)
In this example, an image size of 3200x2000 would be reduced to 640x400. Or an image size or 320x200 would be enlarged to 640x400. I do this to an image before rendering it and before creating a CIImage to use in a CIFilter.
I suggest you think of them as points. There is no scale and no screen (a CIImage is not something that is drawn), so there are no pixels.
A UIImage backed by a CGImage is the basis for drawing, and in addition to the CGImage it has a scale; together with the screen resolution, that gives us our translation from points to pixels.
When cropping a CGImage in Swift 3 (using the .cropping method), the original CGImage is referenced by the cropped version - both according to the documentation, and according to what the Allocations instruments shows me.
I am placing the cropped CGImage objects on an undo stack, so having the original versions retained 'costs' me about 21mb of memory per undo element.
Since there is no obvious way to 'compact' a cropped CGImage and have it made independent from the original, I have currently done something similar to the following (without all the force unwrapping):
let croppedImage = original.cropping(to: rect)!
let data = UIImagePNGRepresentation(UIImage(cgImage: croppedImage))!
let compactedCroppedImage = UIImage(data: data)!.cgImage!
This works perfectly, and now each undo snapshot takes up only the amount of memory that it is supposed to.
My question is: Is there a better / faster way to achieve this?
Your code involves a PNG compression and decompression. This can be avoided. Just create an offscreen bitmap of the target size, draw the original image into it and use it as an image:
UIGraphicsBeginImageContext(rect.size)
let targetRect = CGRect(x: -rect.origin.x, y: -rect.origin.y, width: original.size.width, height: original.size.height)
original.draw(in: targetRect)
let croppedImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
Note: The result is slightly different if you don't have integral coordinates.
Currently I'm working on downloading all the image's provided within a user's selected folder. So this process consists of:
Requesting all the thumbnails of the images
Requesting all the original images
Take the original and create a retina compressed version to display
The reason we need to keep the original is because that's the file we will be printing on anything from 8x10 picture frames to 40x40 canvas wraps, so having the original is important. The only part that's causing the crash is taking the original and creating the compressed version. I ended up using this:
autoreleasepool({
self.compressed = self.saveImageWithReturn(image:self.original!.scaledImage(newSize: 2048), type: .Compressed)
})
scaling the image by calling:
func scaledImage(newSize newHeight : CGFloat) -> UIImage {
let scale = newHeight / size.height
let newWidth = size.width * scale
UIGraphicsBeginImageContext(CGSizeMake(newWidth, newHeight))
drawInRect(CGRectMake(0, 0, newWidth, newHeight))
let newImage = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
return newImage
}
which saves the image to the device documents by using this:
private func saveImageWithReturn(image img: UIImage, type: PhotoType) -> UIImage? {
guard let path = ASSET_PATH.URLByAppendingPathComponent(type.rawValue).path,
let imageData = UIImageJPEGRepresentation(img, type.compression())
else { return nil }
imageData.writeToFile(path, atomically: true)
return UIImage(data: imageData)
}
The autoreleasepool actually fixes the problem of it crashing, but it's operating on the main thread basically freezing all user interaction. Then I tried
dispatch_async(dispatch_get_global_queue(QOS_CLASS_USER_INITIATED, 0), {
autoreleasepool({
self.compressed = self.saveImageWithReturn(image: self.original!.scaledImage(newSize: 2048), type: .Compressed)
})
})
and it results in it not properly releasing memory quick enough and it crashes. The reason I believe this is happening because it's not processing the scaledImage(newSize: 2048) quick enough causing the multiple requests to stack and all try to process this and having multiple instances all trying to hold onto an original image will result in memory warnings or a crash from it. So far I know it works perfectly on the iPad Air 2, but the iPad Generation 4 seems to process it slow.
Not sure if this is the best way of doing things, or if I should be finding another way to scale and compress the original file. Any help would be really appreciated.