I'm trying to get CIImage uncompress data.
For now the only way I found to get compressed data is using CIContext as follow:
let ciContext = CIContext()
let ciImage = CIImage(color: .red).cropped(to: .init(x: 0, y: 0, width: 192, height: 192))
guard let ciImageData = ciContext.jpegRepresentation(of: ciImage, colorSpace: CGColorSpace(name: CGColorSpace.sRGB)!, options: [:]) else {
fatalError()
}
print(ciImageData.count) // Prints 1331
Is it possible to get (as efficiently as possible) the uncompressed CIImage data?
As you can see, ciContext.jpegRepresentation is compressing the image data as JPEG and gives you a Data object that can be written as-is as a JPEG file to disk (including image metadata).
You need to use a different CIContext API for rendering directly into (uncompressed) bitmap data:
let rowBytes = 4 * Int(ciImage.extent.width) // 4 channels (RGBA) of 8-bit data
let dataSize = rowBytes * Int(ciImage.extent.height)
var data = Data(count: dataSize)
data.withUnsafeMutableBytes { data in
ciContext.render(ciImage, toBitmap: data, rowBytes: rowBytes, bounds: ciImage.extent, format: .RGBA8, colorSpace: CGColorSpace(name: CGColorSpace.sRGB)!)
}
Alternatively, you can create a CVPixelBuffer with the correct size and format and render into that with CIContext.render(_ image: CIImage, to buffer: CVPixelBuffer). I think Core ML has direct support for CVPixelBuffer inputs, so this might be the better option.
Related
I am using the following code to create the ARGB8888 image. Is this the correct Image format for ARBG8888 or should I use a different format.
This link is used to create the below Image FOrmat
guard let imageConversionToCGImage = img.cgImage,
let imageFormat = vImage_CGImageFormat(
bitsPerComponent: 8,
bitsPerPixel: 32,
colorSpace: CGColorSpaceCreateDeviceRGB(),
bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.first.rawValue),
renderingIntent: .defaultIntent),
//Source buffer
let sourceBuffer = try? vImage_Buffer(cgImage: imageConversionToCGImage, format: imageFormat, flags: .noFlags),
//ARGBImage
let argb8888CGImage = try? sourceBuffer.createCGImage(format: imageFormat) else { return } ```
and I am getting the image data using the DataProvider.CFData for getting the data . Is this the right format or am I doing it wrong. What should I do to make it so I get the ARGB data.
When you pass a populated vImage_CGImageFormat to that initializer, vImage will attempt to convert the source image to the specified format. For example, you could pass a grayscale image format such as:
let imageFormat = vImage_CGImageFormat(
bitsPerComponent: 8,
bitsPerPixel: 8,
colorSpace: CGColorSpaceCreateDeviceGray(),
bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.none.rawValue),
renderingIntent: .defaultIntent)
And the returned buffer will contain a grayscale representation of the source image regardless of the source image's format.
If you want to populate a buffer based on the format of the source image, pass an empty vImage_CGImageFormat and use the vImageBuffer_InitWithCGImage function:
var imageFormat = vImage_CGImageFormat()
var sourceBuffer = vImage_Buffer()
vImageBuffer_InitWithCGImage(&sourceBuffer,
&imageFormat,
nil,
cgImage,
vImage_Flags(kvImageNoFlags))
On return, imageFormat contains the color space and bit depth of the source image, and sourceBuffer contains the image itself.
Apple have just released the vImage.PixelBuffer that provides a Swift friendly API to vImage. Take a look at init(cgImage:cgImageFormat:pixelFormat:).
Finally, init(data:width:height:byteCountPerRow:pixelFormat:) shows an example of creating a vImage pixel buffer from a Core Graphics image's underlying data.
I have a data array of Int16 or Int32 numerical values that are the raw image data from a 11MP camera chip with an RGGB pixel layout (CFA). The data are exported by the camera driver as FITS data, which is basically a vector or long string of bytes or 16bit/pixel data in my case.
I like to convert these data into a raw image format in Swift in order to use the powerful debayering and demosaicing features and algorithms in iOS/Swift. I do not intend to demosaic myself, since iOS has a great library for this already (see WWDC2016 keynote on Raw Processing with Core Image).
I need to make iOS “believe” my data are actual raw image data.
I tried using CreatePixelBufferWithBytes in Swift and then CIImage from pixelbuffer but to no avail. The CIImage.cgimage is not an RGB color image.
Is there a simple way to create a raw or DNG image in Swift from raw numerical data?
Here is what I tried with the CVPixelBuffer approach, but I do not get any color image out of this:
imgRawData is a [Int32] or [Float32] array with width*height number of elements.
var pixelBuffer: CVPixelBuffer?
let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue,
kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue ]
CVPixelBufferCreateWithBytes(kCFAllocatorDefault, width, height, kCVPixelFormatType_14Bayer_RGGB, &imgRawData, 2*width, nil, nil, attrs as CFDictionary, &pixelBuffer)
let dummyImg = UIImage(systemName: "star.fill")?.cgImage
let ciiraw = CIImage(cvPixelBuffer: pixelBuffer!)
let cif = CIFilter.lanczosScaleTransform()
cif.scale = 0.25
cif.inputImage = ciiraw
let cii = cif.outputImage
let context: CIContext = CIContext.init(options: nil)
guard let cgi = context.createCGImage(cii!, from: cii!.extent) else { return dummyImg! }
Quickview of Xcode shows me only black&white or grayscale images. So does the SwiftUI View of the CGImage...
You can use CGContext and pass your raw values in as bitmapinfo, see init:
init?(data: UnsafeMutableRawPointer?, width: Int, height: Int, bitsPerComponent: Int, bytesPerRow: Int, space: CGColorSpace, bitmapInfo: UInt32)
And for space parameter, which takes CGColorSpace you would use CGColorSpaceCreateDeviceRGB().
You will then use your image with a code similar to this one:
let imageRef = CGContext.makeImage(context!)
let imageRep = NSBitmapImageRep(cgImage: imageRef()!)
Play around with it for a bit, I think you will find what you are looking for.
I am trying to generate Laplacian image out of rgb CGImage by using metal laplacian.
The current code used:
if let croppedImage = self.cropImage2(image: UIImage(ciImage: image), rect: rect)?.cgImage {
let commandBuffer = self.commandQueue.makeCommandBuffer()!
let laplacian = MPSImageLaplacian(device: self.device)
let textureLoader = MTKTextureLoader(device: self.device)
let options: [MTKTextureLoader.Option : Any]? = nil
let srcTex = try! textureLoader.newTexture(cgImage: croppedImage, options: options)
let desc = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: srcTex.pixelFormat, width: srcTex.width, height: srcTex.height, mipmapped: false)
let lapTex = self.device.makeTexture(descriptor: desc)
laplacian.encode(commandBuffer: commandBuffer, sourceTexture: srcTex, destinationTexture: lapTex!)
let output = CIImage(mtlTexture: lapTex!, options: [:])?.cgImage
print("output: \(output?.width)")
print("")
}
I suspect the problem is in makeTexture:
let lapTex = self.device.makeTexture(descriptor: desc)
the width and height of the lapTex in debugger are invalid although the desc and srcTex contains valid data including width and height.
Looks like order or initialisation is wrong but couldn't find what.
Does anyone has an idea what is wrong?
Thanks
There are a few things wrong here.
First, as mentioned in my comment, the command buffer isn't being committed, so the kernel work is never being performed.
Second, you need to wait for the work to complete before attempting to read back the results. (On macOS you'd additionally need to use a blit command encoder to ensure that the contents of the texture are copied back to CPU-accessible memory.)
Third, it's important to create the destination texture with the appropriate usage flags. The default of .shaderRead is insufficient in this case, since the MPS kernel writes to the texture. Therefore, you should explicitly set the usage property on the texture descriptor (to either [.shaderRead, .shaderWrite] or .shaderWrite, depending on how you go on to use the texture).
Fourth, it may be the case that the pixel format of your source texture isn't a writable format, so unless you're absolutely certain it is, consider setting the destination pixel format to a known-writable format (like .rgba8unorm) instead of assuming the destination should match the source. This also helps later when creating CGImages.
Finally, there is no guarantee that the cgImage property of a CIImage is non-nil when it wasn't created from a CGImage. Calling the property doesn't (necessarily) create a new backing CGImage. So, you need to explicitly create a CGImage somehow.
One way of doing this would be to create a Metal device-backed CIContext and use its createCGImage(_:from:) method. Although this might work, it seems redundant if the intent is simply to create a CGImage from a MTLTexture (for display purposes, let's say).
Instead, consider using the getBytes(_:bytesPerRow:from:mipmapLevel:) method to get the bytes from the texture and load them into a CG bitmap context. It's then trivial to create a CGImage from the context.
Here's a function that computes the Laplacian of an image and returns the resulting image:
func laplacian(_ image: CGImage) -> CGImage? {
let commandBuffer = self.commandQueue.makeCommandBuffer()!
let laplacian = MPSImageLaplacian(device: self.device)
let textureLoader = MTKTextureLoader(device: self.device)
let options: [MTKTextureLoader.Option : Any]? = nil
let srcTex = try! textureLoader.newTexture(cgImage: image, options: options)
let desc = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: srcTex.pixelFormat,
width: srcTex.width,
height: srcTex.height,
mipmapped: false)
desc.pixelFormat = .rgba8Unorm
desc.usage = [.shaderRead, .shaderWrite]
let lapTex = self.device.makeTexture(descriptor: desc)!
laplacian.encode(commandBuffer: commandBuffer, sourceTexture: srcTex, destinationTexture: lapTex)
#if os(macOS)
let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder()!
blitCommandEncoder.synchronize(resource: lapTex)
blitCommandEncoder.endEncoding()
#endif
commandBuffer.commit()
commandBuffer.waitUntilCompleted()
// Note: You may want to use a different color space depending
// on what you're doing with the image
let colorSpace = CGColorSpaceCreateDeviceRGB()
// Note: We skip the last component (A) since the Laplacian of the alpha
// channel of an opaque image is 0 everywhere, and that interacts oddly
// when we treat the result as an RGBA image.
let bitmapInfo = CGImageAlphaInfo.noneSkipLast.rawValue
let bytesPerRow = lapTex.width * 4
let bitmapContext = CGContext(data: nil,
width: lapTex.width,
height: lapTex.height,
bitsPerComponent: 8,
bytesPerRow: bytesPerRow,
space: colorSpace,
bitmapInfo: bitmapInfo)!
lapTex.getBytes(bitmapContext.data!,
bytesPerRow: bytesPerRow,
from: MTLRegionMake2D(0, 0, lapTex.width, lapTex.height),
mipmapLevel: 0)
return bitmapContext.makeImage()
}
I'm trying to load a large image into a MTLTexture and it works with 4000x6000 images. But when I try with 6000x8000 it can't.
func setTexture(device: MTLDevice, imageName: String) -> MTLTexture? {
let textureLoader = MTKTextureLoader(device: device)
var texture: MTLTexture? = nil
// In iOS 10 the origin was changed.
let textureLoaderOptions: [MTKTextureLoader.Option: Any]
if #available(iOS 10.0, *) {
let origin = MTKTextureLoader.Origin.bottomLeft.rawValue
textureLoaderOptions = [MTKTextureLoader.Option.origin : origin]
} else {
textureLoaderOptions = [:]
}
if let textureURL = Bundle.main.url(forResource: imageName, withExtension: nil, subdirectory: "Images") {
do {
texture = try textureLoader.newTexture(URL: textureURL, options: textureLoaderOptions)
} catch {
print("Texture not created.")
}
}
return texture
}
Pretty basic code. I'm running it in an iPad Pro with A9 chip, GPU family 3. It should handle textures this large. Should I manually tile it somehow if it doesn't accept this size? In that case, what's the best approach: using MTLRegionMake to copy bytes, slicing in Core Image or a Core Graphics context...
I appreciate any help
Following your helpful comments I decided to load it manually drawing to a CGContext and copying to a MTLTexture. I'm adding the solution code below. The context shouldn't be created each time a texture is created, it's better to put it outside the function and keep reusing it.
// Grab the CGImage, w = width, h = height...
let context = CGContext(data: nil, width: w, height: h, bitsPerComponent: bpc, bytesPerRow: (bpp / 8) * w, space: colorSpace!, bitmapInfo: bitmapInfo.rawValue)
let flip = CGAffineTransform(a: 1, b: 0, c: 0, d: -1, tx: 0, ty: CGFloat(h))
context?.concatenate(flip)
context?.draw(cgImage, in: CGRect(x: 0, y: 0, width: CGFloat(w), height: CGFloat(h)))
let textureDescriptor = MTLTextureDescriptor()
textureDescriptor.pixelFormat = .rgba8Unorm
textureDescriptor.width = w
textureDescriptor.height = h
guard let data = context?.data else {print("No data in context."); return nil}
let texture = device.makeTexture(descriptor: textureDescriptor)
texture?.replace(region: MTLRegionMake2D(0, 0, w, h), mipmapLevel: 0, withBytes: data, bytesPerRow: 4 * w)
return texture
I had this issue before, a texture would load on one device and not on another. I think it is a bug with the texture loader.
You can load in a texture manually using CGImage and a CGContext, draw the image into the context. Create a MTLTexture buffer, then copy the bytes from the CGContext into the texture using a MTLRegion.
It's not fool proof, you have to make sure to use the correct pixel format for the metal buffer or you'll get strange results, so either you code for one specific format of image you're importing, or do a lot of checking. Apples' Basic Texturing example shows how you can change the color order before writing the bytes to the texture using MTLRegion.
I am trying to use a MTKTextureLoader to load a CGImage as a texture. Here is the original image
However after I convert that CGImage into a MTLTexture and that texture back to a CGImage it looks horrible, like this:
Here is sorta what is going on in code.
The image is loaded in as a CGImage (I have checked and that image does appear to have the full visual quality)
I have a function view() that allows me to view a NSImage by using it in a CALayer like so:
func view() {
.....
imageView!.layer = CALayer()
imageView!.layer!.contentsGravity = kCAGravityResizeAspectFill
imageView!.layer!.contents = img
imageView!.wantsLayer = true
So I did the following
let cg = CoolImage()
let ns = NSImage(cgImage: cg, size: Size(width: cg.width, height: cg.height))
view(image: ns)
And checked sure enough it had the full visual fidelity.
So then I loaded the cg image into a MTLTexture like so
let textureLoader = MTKTextureLoader(device: metalState.sharedDevice!)
let options = [
MTKTextureLoader.Option.textureUsage: NSNumber(value: MTLTextureUsage.shaderRead.rawValue | MTLTextureUsage.shaderWrite.rawValue | MTLTextureUsage.renderTarget.rawValue),
MTKTextureLoader.Option.SRGB: false
]
return ensure(try textureLoader.newTexture(cgImage: cg, options: options))
I then converted the MTLTexture back to a UIImage like so:
let texture = self
let width = texture.width
let height = texture.height
let bytesPerRow = width * 4
let data = UnsafeMutableRawPointer.allocate(bytes: bytesPerRow * height, alignedTo: 4)
defer {
data.deallocate(bytes: bytesPerRow * height, alignedTo: 4)
}
let region = MTLRegionMake2D(0, 0, width, height)
texture.getBytes(data, bytesPerRow: bytesPerRow, from: region, mipmapLevel: 0)
var buffer = vImage_Buffer(data: data, height: UInt(height), width: UInt(width), rowBytes: bytesPerRow)
var map: [UInt8] = [0, 1, 2, 3]
if (pixelFormat == .bgra8Unorm) {
map = [2, 1, 0, 3]
}
vImagePermuteChannels_ARGB8888(&buffer, &buffer, map, 0)
guard let colorSpace = CGColorSpace(name: CGColorSpace.genericRGBLinear) else { return nil }
guard let context = CGContext(data: data, width: width, height: height, bitsPerComponent: 8, bytesPerRow: bytesPerRow, space: colorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipLast.rawValue) else { return nil }
guard let cgImage = context.makeImage() else { return nil }
return NSImage(cgImage: cgImage, size: Size(width: width, height: height))
And viewed it.
The resulting image was quite saturated and I believe it was because of the CGImage to MTLTexture conversion which I have been fairly successful with in the past.
Please note that this texture was never rendered only converted.
You are probably wondering why I am using all of these conversions and that is a great point. My actual pipeline does not work anything like this HOWEVER it does require each of these conversion components to be working smoothly. This is not my actual use case just something to show the problem.
The problem here isn't the conversion from CGImage to MTLTexture. The problem is that you're assuming that the color space of the source image is linear. More likely than not, the image data is actually sRGB-encoded, so by creating a bitmap context with a generic linear color space, you're incorrectly telling CG that it should gamma-encode the image data before display, which leads to the desaturation you're seeing.
You can fix this by using the native color space of the original CGImage, or by otherwise accounting for the fact that your image data is sRGB-encoded.