Problems (wrong scale, aspect ratio) while rendering MTKTexture to MTKView after applying compute shaders - ios

Question

I am trying to process video frames from the camera using compute metal shaders and display it to the user. The problem is with displaying the modified frames. The output contains stacked copies of the processed frame with some of them clipped and they don't fill the screen completely.
P.S I am new to both iOS and metal
So far, I have identified variables that control this:
1. Number of Thread groups launched
2. MTKView's drawable size
3. sampling id in the metal shader
I have played around with these with no good result.
Below are the code and my output
The function that sets up the MTKView
func initMetalView() {
metalView = MTKView(frame: view.frame, device: metalDevice)
metalView.delegate = self
metalView.framebufferOnly = false
metalView.colorPixelFormat = .bgra8Unorm
metalView.autoResizeDrawable = false
metalView.drawableSize = CGSize(width: 1920, height: 1080)
metalView.layer.transform = CATransform3DMakeRotation(CGFloat(Float.pi),0.0,1.0,0.0)
view.insertSubview(metalView, at: 0)
}
The AVCaptureVideoDataOutputSampleBufferDelegate used to convert CMSampleBuffer to MTLTexture
extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
// sample buffer -> image buffer -> CoreVideo metal texture -> MTL texture
guard let cvImageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
else { fatalError("can't get image buffer") }
var textureCache: CVMetalTextureCache?
guard CVMetalTextureCacheCreate(kCFAllocatorDefault, nil, metalDevice, nil, &textureCache) == kCVReturnSuccess else { fatalError("cant create texture cache") }
let width = CVPixelBufferGetWidth(cvImageBuffer)
let height = CVPixelBufferGetHeight(cvImageBuffer)
var imageTexture: CVMetalTexture?
let result = CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache!, cvImageBuffer, nil, MTLPixelFormat.bgra8Unorm, width, height, 0, &imageTexture)
guard let unwrappedImageTexture = imageTexture,
result == kCVReturnSuccess
else { fatalError("failed to create texture from image") }
inputTexture = CVMetalTextureGetTexture(unwrappedImageTexture)
}
}
The MTKViewDelegate used to apply the shader on inputTexture and display the outputTexture to metalView
extension ViewController: MTKViewDelegate {
func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
}
func draw(in view: MTKView) {
guard let inputTexture = inputTexture,
let commandQueue = commandQueue,
let commandBuffer = commandQueue.makeCommandBuffer(),
let encoder = commandBuffer.makeComputeCommandEncoder(),
let pipelineState = pipelineState
else { return }
encoder.setComputePipelineState(pipelineState)
encoder.setTextures([metalView.currentDrawable!.texture, inputTexture], range: 0..<2)
encoder.dispatchThreadgroups(MTLSizeMake(inputTexture.width/16, inputTexture.height/16, 1), threadsPerThreadgroup: threadsPerBlock)
// inputTexture w:1920, h:1080
encoder.endEncoding()
commandBuffer.present(metalView.currentDrawable!)
commandBuffer.commit()
}
}
The metal compute shader
#include <metal_stdlib>
using namespace metal;
kernel void blacky (texture2d<float, access::write> outTexture [[texture(0)]],
texture2d<float, access::read> inTexture [[texture(1)]],
uint2 id [[thread_position_in_grid]]) {
uint2 flipped_id = uint2(id.y, id.x);
float3 val = inTexture.read(flipped_id).rgb;
float g = (val.r + val.g + val.b)/3.0;
float4 out = float4(g, g, g, 1);
outTexture.write(out.rgba, id);
}
You can see the current output here: https://i.imgur.com/hVDox3U