Present a pixel buffer to a MTKView - ios

Question

Here is my problem : I want to display a pixel buffer that I calculated to a MTKView. I searched for MTLTexture, MTLBuffer and other Metal objects, but I can't find any way to just present a pixel buffer.
Every tutorial I saw are about presenting 3D objects with vertex and fragments shaders.
I think the buffer has to be presented within the drawInMTKView function (maybe with the MTLRenderCommandEncoder), but again, I can't find any information about this.
I hope I'm not asking an obvious question.
Thanks

Answer 1

Welcome!
I recommend you use Core Image for rendering the content of the pixel buffer into the view. This requires the least manual Metal setup.
Setup the MTKView and some required objects as follows (assuming you have a view controller and a storyboard setup):
import UIKit
import CoreImage
class PreviewViewController: UIViewController {
#IBOutlet weak var metalView: MTKView!
var device: MTLDevice!
var commandQueue: MTLCommandQueue!
var ciContext: CIContext!
var pixelBuffer: CVPixelBuffer?
override func viewDidLoad() {
super.viewDidLoad()
self.device = MTLCreateSystemDefaultDevice()
self.commandQueue = self.device.makeCommandQueue()
self.metalView.delegate = self
self.metalView.device = self.device
// this allows us to render into the view's drawable
self.metalView.framebufferOnly = false
self.ciContext = CIContext(mtlDevice: self.device)
}
}
In the delegate method you use Core Image to transform the pixel buffer to fit the contents of the view (this is a bonus, adapt it to your use case) and render it using the CIContext:
extension PreviewViewController: MTKViewDelegate {
func draw(in view: MTKView) {
guard let pixelBuffer = self.pixelBuffer,
let commandBuffer = self.commandQueue.makeCommandBuffer() else { return }
// turn the pixel buffer into a CIImage so we can use Core Image for rendering into the view
let image = CIImage(cvPixelBuffer: pixelBuffer)
// bonus: transform the image to aspect-fit the view's bounds
let drawableSize = view.drawableSize
let scaleX = drawableSize.width / image.extent.width
let scaleY = drawableSize.height / image.extent.height
let scale = min(scaleX, scaleY)
let scaledImage = image.transformed(by: CGAffineTransform(scaleX: scale, y: scale))
// center in the view
let originX = max(drawableSize.width - scaledImage.extent.size.width, 0) / 2
let originY = max(drawableSize.height - scaledImage.extent.size.height, 0) / 2
let centeredImage = scaledImage.transformed(by: CGAffineTransform(translationX: originX, y: originY))
// Create a render destination that allows to lazily fetch the target texture
// which allows the encoder to process all CI commands _before_ the texture is actually available.
// This gives a nice speed boost because the CPU doesn't need to wait for the GPU to finish
// before starting to encode the next frame.
// Also note that we don't pass a command buffer here, because according to Apple:
// "Rendering to a CIRenderDestination initialized with a commandBuffer requires encoding all
// the commands to render an image into the specified buffer. This may impact system responsiveness
// and may result in higher memory usage if the image requires many passes to render."
let destination = CIRenderDestination(width: Int(drawableSize.width),
height: Int(drawableSize.height),
pixelFormat: view.colorPixelFormat,
commandBuffer: nil,
mtlTextureProvider: { () -> MTLTexture in
return currentDrawable.texture
})
// render into the view's drawable
let _ = try! self.ciContext.startTask(toRender: centeredImage, to: destination)
// present the drawable
commandBuffer.present(currentDrawable)
commandBuffer.commit()
}
}
There is a slightly simpler way for rendering into the drawable texture instead of using CIRenderDestination, but this is recommended if you want to achieve high frame rates (see comment).

Answer 2

I think I found a solution : https://developer.apple.com/documentation/metal/creating_and_sampling_textures?language=objc.
In this exemple, they show how to render an image to a Metal view, using just a few vertices and a fragment shader to render the texture to a 2D square.
I'll go from there. Not sure if there isn't a better (simpler ?) way to do that. But I guess that's how Metal wants us to do this.