MTKView frequently displaying scrambled MTLTextures - metal

I am working on an MTKView-backed paint program which can replay painting history via an array of MTLTextures that store keyframes. I am having an issue in which sometimes the content of these MTLTextures is scrambled.
As an example, say I want to store a section of the drawing below as a keyframe:
During playback, sometimes the drawing will display exactly as intended, but sometimes, it will display like this:
Note the distorted portion of the picture. (The undistorted portion constitutes a static background image that's not part of the keyframe in question)
I describe the way I Create individual MTLTextures from the MTKView's currentDrawable below. Because of color depth issues I won't go into, the process may seem a little round-about.
I first get a CGImage of the subsection of the screen that constitutes a keyframe.
I use that CGImage to create an MTLTexture tied to the MTKView's device.
I store that MTLTexture into a MTLTextureStructure that stores the MTLTexture and the keyframe's bounding-box (which I'll need later)
Lastly, I store in an array of MTLTextureStructures (keyframeMetalArray). During playback, when I hit a keyframe, I get it from this keyframeMetalArray.
The associated code is outlined below.
let keyframeCGImage = weakSelf!.canvasMetalViewPainting.mtlTextureToCGImage(bbox: keyframeBbox, copyMode: copyTextureMode.textureKeyframe) // convert from MetalTexture to CGImage
let keyframeMTLTexture = weakSelf!.canvasMetalViewPainting.CGImageToMTLTexture(cgImage: keyframeCGImage)
let keyframeMTLTextureStruc = mtlTextureStructure(texture: keyframeMTLTexture, bbox: keyframeBbox, strokeType: brushTypeMode.brush)
weakSelf!.keyframeMetalArray.append(keyframeMTLTextureStruc)
Without providing specifics about how each conversion is happening, I wonder if, from an architecture design point, I'm overlooking something that is corrupting my data stored in the keyframeMetalArray. It may be unwise to try to store these MTLTextures in volatile arrays, but I don't know that for a fact. I just figured using MTLTextures would be the quickest way to update content.
By the way, when I swap out arrays of keyframes to arrays of UIImage.pngData, I have no display issues, but it's a lot slower. On the plus side, it tells me that the initial capture from currentDrawable to keyframeCGImage is working just fine.
Any thoughts would be appreciated.
p.s. adding a bit of detail based on the feedback:
mtlTextureToCGImage:
func mtlTextureToCGImage(bbox: CGRect, copyMode: copyTextureMode) -> CGImage {
let kciOptions = [convertFromCIContextOption(CIContextOption.outputPremultiplied): true,
convertFromCIContextOption(CIContextOption.useSoftwareRenderer): false] as [String : Any]
let bboxStrokeScaledFlippedY = CGRect(x: (bbox.origin.x * self.viewContentScaleFactor), y: ((self.viewBounds.height - bbox.origin.y - bbox.height) * self.viewContentScaleFactor), width: (bbox.width * self.viewContentScaleFactor), height: (bbox.height * self.viewContentScaleFactor))
let strokeCIImage = CIImage(mtlTexture: metalDrawableTextureKeyframe,
options: convertToOptionalCIImageOptionDictionary(kciOptions))!.oriented(CGImagePropertyOrientation.downMirrored)
let imageCropCG = cicontext.createCGImage(strokeCIImage, from: bboxStrokeScaledFlippedY, format: CIFormat.RGBA8, colorSpace: colorSpaceGenericRGBLinear)
cicontext.clearCaches()
return imageCropCG!
} // end of func mtlTextureToCGImage(bbox: CGRect)
CGImageToMTLTexture:
func CGImageToMTLTexture (cgImage: CGImage) -> MTLTexture {
// Note that we forego the more direct method of creating stampTexture:
//let stampTexture = try! MTKTextureLoader(device: self.device!).newTexture(cgImage: strokeUIImage.cgImage!, options: nil)
// because MTKTextureLoader seems to be doing additional processing which messes with the resulting texture/colorspace
let width = Int(cgImage.width)
let height = Int(cgImage.height)
let bytesPerPixel = 4
let rowBytes = width * bytesPerPixel
//
let texDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm,
width: width,
height: height,
mipmapped: false)
texDescriptor.usage = MTLTextureUsage(rawValue: MTLTextureUsage.shaderRead.rawValue)
texDescriptor.storageMode = .shared
guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else {
return brushTextureSquare // return SOMETHING
}
let dstData: CFData = (cgImage.dataProvider!.data)!
let pixelData = CFDataGetBytePtr(dstData)
let region = MTLRegionMake2D(0, 0, width, height)
print ("[MetalViewPainting]: w= \(width) | h= \(height) region = \(region.size)")
stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))
return stampTexture
} // end of func CGImageToMTLTexture (cgImage: CGImage)

The type of distortion looks like a bytes-per-row alignment issue between CGImage and MTLTexture. You're probably only seeing this issue when your image is a certain size that falls outside of the bytes-per-row alignment requirement of your MTLDevice. If you really need to store the texture as a CGImage, ensure that you are using the bytesPerRow value of the CGImage when copying back to the texture.

Related

How to convert a numerical data array into RAW image data in Swift?

I have a data array of Int16 or Int32 numerical values that are the raw image data from a 11MP camera chip with an RGGB pixel layout (CFA). The data are exported by the camera driver as FITS data, which is basically a vector or long string of bytes or 16bit/pixel data in my case.
I like to convert these data into a raw image format in Swift in order to use the powerful debayering and demosaicing features and algorithms in iOS/Swift. I do not intend to demosaic myself, since iOS has a great library for this already (see WWDC2016 keynote on Raw Processing with Core Image).
I need to make iOS “believe” my data are actual raw image data.
I tried using CreatePixelBufferWithBytes in Swift and then CIImage from pixelbuffer but to no avail. The CIImage.cgimage is not an RGB color image.
Is there a simple way to create a raw or DNG image in Swift from raw numerical data?
Here is what I tried with the CVPixelBuffer approach, but I do not get any color image out of this:
imgRawData is a [Int32] or [Float32] array with width*height number of elements.
var pixelBuffer: CVPixelBuffer?
let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue ]
CVPixelBufferCreateWithBytes(kCFAllocatorDefault, width, height, kCVPixelFormatType_14Bayer_RGGB, &imgRawData, 2*width, nil, nil, attrs as CFDictionary, &pixelBuffer)
let dummyImg = UIImage(systemName: "star.fill")?.cgImage
let ciiraw = CIImage(cvPixelBuffer: pixelBuffer!)
let cif = CIFilter.lanczosScaleTransform()
cif.scale = 0.25
cif.inputImage = ciiraw
let cii = cif.outputImage
let context: CIContext = CIContext.init(options: nil)
guard let cgi = context.createCGImage(cii!, from: cii!.extent) else { return dummyImg! }
Quickview of Xcode shows me only black&white or grayscale images. So does the SwiftUI View of the CGImage...
You can use CGContext and pass your raw values in as bitmapinfo, see init:
init?(data: UnsafeMutableRawPointer?, width: Int, height: Int, bitsPerComponent: Int, bytesPerRow: Int, space: CGColorSpace, bitmapInfo: UInt32)
And for space parameter, which takes CGColorSpace you would use CGColorSpaceCreateDeviceRGB().
You will then use your image with a code similar to this one:
let imageRef = CGContext.makeImage(context!)
let imageRep = NSBitmapImageRep(cgImage: imageRef()!)
Play around with it for a bit, I think you will find what you are looking for.

Generate Laplacian image by Apple-Metal MPSImageLaplacian

I am trying to generate Laplacian image out of rgb CGImage by using metal laplacian.
The current code used:
if let croppedImage = self.cropImage2(image: UIImage(ciImage: image), rect: rect)?.cgImage {
let commandBuffer = self.commandQueue.makeCommandBuffer()!
let laplacian = MPSImageLaplacian(device: self.device)
let textureLoader = MTKTextureLoader(device: self.device)
let options: [MTKTextureLoader.Option : Any]? = nil
let srcTex = try! textureLoader.newTexture(cgImage: croppedImage, options: options)
let desc = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: srcTex.pixelFormat, width: srcTex.width, height: srcTex.height, mipmapped: false)
let lapTex = self.device.makeTexture(descriptor: desc)
laplacian.encode(commandBuffer: commandBuffer, sourceTexture: srcTex, destinationTexture: lapTex!)
let output = CIImage(mtlTexture: lapTex!, options: [:])?.cgImage
print("output: \(output?.width)")
print("")
}
I suspect the problem is in makeTexture:
let lapTex = self.device.makeTexture(descriptor: desc)
the width and height of the lapTex in debugger are invalid although the desc and srcTex contains valid data including width and height.
Looks like order or initialisation is wrong but couldn't find what.
Does anyone has an idea what is wrong?
Thanks
There are a few things wrong here.
First, as mentioned in my comment, the command buffer isn't being committed, so the kernel work is never being performed.
Second, you need to wait for the work to complete before attempting to read back the results. (On macOS you'd additionally need to use a blit command encoder to ensure that the contents of the texture are copied back to CPU-accessible memory.)
Third, it's important to create the destination texture with the appropriate usage flags. The default of .shaderRead is insufficient in this case, since the MPS kernel writes to the texture. Therefore, you should explicitly set the usage property on the texture descriptor (to either [.shaderRead, .shaderWrite] or .shaderWrite, depending on how you go on to use the texture).
Fourth, it may be the case that the pixel format of your source texture isn't a writable format, so unless you're absolutely certain it is, consider setting the destination pixel format to a known-writable format (like .rgba8unorm) instead of assuming the destination should match the source. This also helps later when creating CGImages.
Finally, there is no guarantee that the cgImage property of a CIImage is non-nil when it wasn't created from a CGImage. Calling the property doesn't (necessarily) create a new backing CGImage. So, you need to explicitly create a CGImage somehow.
One way of doing this would be to create a Metal device-backed CIContext and use its createCGImage(_:from:) method. Although this might work, it seems redundant if the intent is simply to create a CGImage from a MTLTexture (for display purposes, let's say).
Instead, consider using the getBytes(_:bytesPerRow:from:mipmapLevel:) method to get the bytes from the texture and load them into a CG bitmap context. It's then trivial to create a CGImage from the context.
Here's a function that computes the Laplacian of an image and returns the resulting image:
func laplacian(_ image: CGImage) -> CGImage? {
let commandBuffer = self.commandQueue.makeCommandBuffer()!
let laplacian = MPSImageLaplacian(device: self.device)
let textureLoader = MTKTextureLoader(device: self.device)
let options: [MTKTextureLoader.Option : Any]? = nil
let srcTex = try! textureLoader.newTexture(cgImage: image, options: options)
let desc = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: srcTex.pixelFormat,
width: srcTex.width,
height: srcTex.height,
mipmapped: false)
desc.pixelFormat = .rgba8Unorm
desc.usage = [.shaderRead, .shaderWrite]
let lapTex = self.device.makeTexture(descriptor: desc)!
laplacian.encode(commandBuffer: commandBuffer, sourceTexture: srcTex, destinationTexture: lapTex)
#if os(macOS)
let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder()!
blitCommandEncoder.synchronize(resource: lapTex)
blitCommandEncoder.endEncoding()
#endif
commandBuffer.commit()
commandBuffer.waitUntilCompleted()
// Note: You may want to use a different color space depending
// on what you're doing with the image
let colorSpace = CGColorSpaceCreateDeviceRGB()
// Note: We skip the last component (A) since the Laplacian of the alpha
// channel of an opaque image is 0 everywhere, and that interacts oddly
// when we treat the result as an RGBA image.
let bitmapInfo = CGImageAlphaInfo.noneSkipLast.rawValue
let bytesPerRow = lapTex.width * 4
let bitmapContext = CGContext(data: nil,
width: lapTex.width,
height: lapTex.height,
bitsPerComponent: 8,
bytesPerRow: bytesPerRow,
space: colorSpace,
bitmapInfo: bitmapInfo)!
lapTex.getBytes(bitmapContext.data!,
bytesPerRow: bytesPerRow,
from: MTLRegionMake2D(0, 0, lapTex.width, lapTex.height),
mipmapLevel: 0)
return bitmapContext.makeImage()
}

Cropping CIImage

I have a class that takes an UIImage, initializes a CIImage with it like so:
workingImage = CIImage.init(image: baseImage!)
Then the image is used to cut out 9 neighbouring squares in a 3x3 pattern out of it - in a loop:
for x in 0..<3
{
for y in 0..<3
{
croppingRect = CGRect(x: CGFloat(Double(x) * sideLength + startPointX),
y: CGFloat(Double(y) * sideLength + startPointY),
width: CGFloat(sideLength),
height: CGFloat(sideLength))
let tmpImg = (workingImage?.cropping(to: croppingRect))!
}
}
Those tmpImgs are inserted into a table and later used, but thats besides the point.
This code works on IOS 9, and on IOS 10 simulators, but not on an actual IOS 10 device. The images produced are either all empty, or one of them is like a half of what its supposed to be, with the rest being, again, empty.
Is this not how its supposed to be done in IOS 10?
The heart of the matter is that passing through CIImage is not the way to crop a UIImage. For one thing, coming back from CIImage to UIImage is a complicated business. For another, the whole round-trip is unnecessary.
How To Crop
To crop an image, make an image graphics context of the desired cropped size and call draw(at:) on the UIImage to draw it at the desired point relative to the graphics context, so that the desired portion of the image falls into the context. Now extract the resulting new image and close the context.
To demonstrate, I'll crop to one of the thirds you are trying to crop to, namely the lower right third:
let sz = baseImage.size
UIGraphicsBeginImageContextWithOptions(
CGSize(width:sz.width/3.0, height:sz.height/3.0),
false, 0)
baseImage.draw(at:CGPoint(x: -sz.width/3.0*2.0, y: -sz.height/3.0*2.0))
let tmpImg = UIGraphicsGetImageFromCurrentImageContext()
UIGraphicsEndImageContext()
Original image (baseImage):
Cropped image (tmpImg):
The other sections are completely parallel.
Core Image's coordinate system mismatches with UIKit, so the rect needs to be mirrored.
So in your specific case, you want:
var ciRect = croppingRect
ciRect.origin.y = workingImage!.extent.height - ciRect.origin.y - ciRect.height
let tmpImg = workingImage!.cropped(to: ciRect)
This definitely works for iOS 10+.
In a more general case, we would make a UIImage extension that covers both possible coordinate systems, and that's way faster than draw(at:):
extension UIImage {
/// Return a new image cropped to a rectangle.
/// - parameter rect:
/// The rectangle to crop.
open func cropped(to rect: CGRect) -> UIImage {
// a UIImage is either initialized using a CGImage, a CIImage, or nothing
if let cgImage = self.cgImage {
// CGImage.cropping(to:) is magnitudes faster than UIImage.draw(at:)
if let cgCroppedImage = cgImage.cropping(to: rect) {
return UIImage(cgImage: cgCroppedImage)
} else {
return UIImage()
}
}
if let ciImage = self.ciImage {
// Core Image's coordinate system mismatch with UIKit, so rect needs to be mirrored.
var ciRect = rect
ciRect.origin.y = ciImage.extent.height - ciRect.origin.y - ciRect.height
let ciCroppedImage = ciImage.cropped(to: ciRect)
return UIImage(ciImage: ciCroppedImage)
}
return self
}
}
I've made a pod for it, so the source code is at https://github.com/Coeur/ImageEffects/blob/master/SwiftImageEffects/ImageEffects%2Bextensions.swift

How to drawInRect asynchronously

I'm trying to load a huge image (talking about 131.072x131.072 pixels) tiled up nicely into 512x512 tiles of 256x256 pixels from a bunch of URLs.
Once my function returns the Image I want to draw it in a Rect on the proper position.
Since this process takes a while, I want to run the whole thing asynchronously.
Below is what I've tried so far:
override func drawRect(rect: CGRect) {
let firstColumn = Int(CGRectGetMinX(rect) / sideLength)
let lastColumn = Int(CGRectGetMaxX(rect) / sideLength)
let firstRow = Int(CGRectGetMinY(rect) / sideLength)
let lastRow = Int(CGRectGetMaxY(rect) / sideLength)
let qos = Int(QOS_CLASS_USER_INITIATED.rawValue)
dispatch_async(dispatch_get_global_queue(qos, 0)) { () -> Void in
for row in firstRow...lastRow {
for column in firstColumn...lastColumn {
let url = NSURL(string: "https://someURL/\(row)/\(column).jpg")
let tile = UIImage(data: NSData(contentsOfURL: url!)!)!
let x = self.sideLength * CGFloat(column)
let y = self.sideLength * CGFloat(row)
let point = CGPoint(x: x, y: y)
let size = CGSize(width: self.sideLength, height: self.sideLength)
var tileRect = CGRect(origin: point, size: size)
tileRect = CGRectIntersection(self.bounds, tileRect)
dispatch_async(dispatch_get_main_queue()) {
tile.drawInRect(tileRect)
}
}
}
}
}
And I'm getting this error:
<Error>: CGContextRestoreGState: invalid context 0x0. Backtrace:
<-[UIImage drawInRect:]+66>
<_TFFFC6H1Z1DB15MyClass8drawRectFS0_FVSC6CGRectT_U_FT_T_U_FT_T_+122>
<_TTRXFo__dT__XFdCb__dT__+39>
<_dispatch_call_block_and_release+12>
<_dispatch_client_callout+8>
<_dispatch_main_queue_callback_4CF+1738>
<__CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__+9>
<__CFRunLoopRun+2073>
<CFRunLoopRunSpecific+488>
<GSEventRunModal+161>
<UIApplicationMain+171>
<main+109>
Can anybody give me a hint on how to retrieve and draw the image asynchronously?
Don't use drawRect. There's no reason to use drawRect in this scenario. Simply use views, or layers, or SpriteKit, or OpenGL ES (there are probably more possible choices). In the first two cases, you'll probably have to add/remove bits and pieces based on the part of the view which is visible on screen, but using standard views/layers will get you much better performance. Apple strongly recommends against using drawRect.
If you do use drawRect, certainly don't load data while in there. Apple clearly states that you should be drawing, not doing anything else while in there. And you certainly don't want to start asynchronous tasks while in there, this will just lead to a catastrophe. Load the data beforehand, store it somewhere, and just do the drawing while in drawRect. If you load data as the user moves around, do the loading as the user moves, not when you draw. You'll probably need to invalidate rects when the image has actually been loaded so that drawRect is then called. But again, don`t use drawRect. Just add/remove views/layers.
Also I recommend not using NSData(contentsOfURL:). Use an NSURLSession dataTask* with the appropriate completion handler. This way, all of your loads will happen simultaneously (up to the set limits), not one after the other.

Calculating size (in bytes) of an image in memory

I am writing a small app in Swift to resize an image. I would like to calculate the size of the resized image (in bytes/KB). How do I do that?
Here is the piece of code I am working on:
var assetRepresentation : ALAssetRepresentation = asset.defaultRepresentation()
self.originalImageSize = assetRepresentation.size()
selectedImageSize = self.originalImageSize
// now scale the image
let image = selectedImage
let hasAlpha = false
let scale: CGFloat = 0.0 // Automatically use scale factor of main screen
UIGraphicsBeginImageContextWithOptions(sizeChange, !hasAlpha, scale)
image.drawInRect(CGRect(origin: CGPointZero, size: sizeChange))
let scaledImage = UIGraphicsGetImageFromCurrentImageContext()
self.backgroundImage.image = scaledImage
Since scaledImage is not yet saved, how do I go about calculating its size?
Since you're looking to display the size of the file to your user, NSByteCountFormatter is a good solution. It takes NSData, and can output a String representing the size of the data in a human readable format (like 1 KB, 2 MB, etc).
Since you're dealing with a UIImage though, you'll have to convert the UIImage to NSData to use this, which for example, can be done using UIImagePNGRepresentation() or UIImageJPEGRepresentation(), which returns NSData representative of the image in the specified format. A usage example could look something like this:
let data = UIImagePNGRepresentation(scaledImage)
let formatted = NSByteCountFormatter.stringFromByteCount(
Int64(data.length),
countStyle: NSByteCountFormatterCountStyle.File
)
println(formatted)
Edit: If as suggested by your title, you're looking to show this information with a specific unit of measurement (bytes), this can also be achieved with NSByteCountFormatter. You just have to create an instance of the class and set its allowedUnits property.
let data = UIImagePNGRepresentation(scaledImage)
let formatter = NSByteCountFormatter()
formatter.allowedUnits = NSByteCountFormatterUnits.UseBytes
formatter.countStyle = NSByteCountFormatterCountStyle.File
let formatted = formatter.stringFromByteCount(Int64(data.length))
println(formatted)
I used this to create my image:
var imageBuffer: UnsafeMutablePointer<UInt8> = nil
let ctx = CGBitmapContextCreate(imageBuffer, UInt(width), UInt(height), UInt(8), bitmapBytesPerRow, colorSpace, bitmapInfo)
imageBuffer is allocated automatically (see according documentation).

Resources