Rendering MTLTexture on MTKView is not keeping aspect ratio - ios

Question

I have a texture that's 1080x1920 pixels. And I'm trying to render it on a MTKView that isn't the same aspect ratio. (i.e iPad/iPhone X full screen).
This is how I'm rendering the texture for the MTKView:
private func render(_ texture: MTLTexture, withCommandBuffer commandBuffer: MTLCommandBuffer, device: MTLDevice) {
guard let currentRenderPassDescriptor = metalView?.currentRenderPassDescriptor,
let currentDrawable = metalView?.currentDrawable,
let renderPipelineState = renderPipelineState,
let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: currentRenderPassDescriptor) else {
semaphore.signal()
return
}
encoder.pushDebugGroup("RenderFrame")
encoder.setRenderPipelineState(renderPipelineState)
encoder.setFragmentTexture(texture, index: 0)
encoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4, instanceCount: 1)
encoder.popDebugGroup()
encoder.endEncoding()
// Called after the command buffer is scheduled
commandBuffer.addScheduledHandler { [weak self] _ in
guard let strongSelf = self else {
return
}
strongSelf.didRender(texture: texture)
strongSelf.semaphore.signal()
}
commandBuffer.present(currentDrawable)
commandBuffer.commit()
}
I want the texture to be rendered like .scaleAspectFill on a UIView and I'm trying to learn Metal so I'm not sure where I should be looking for this (the .metal file, the pipeline, the view itself, the encoder, etc.)
Thanks!
Edit: Here is the shader code:
#include <metal_stdlib> using namespace metal;
typedef struct {
float4 renderedCoordinate [[position]];
float2 textureCoordinate; } TextureMappingVertex;
vertex TextureMappingVertex mapTexture(unsigned int vertex_id [[ vertex_id ]]) {
float4x4 renderedCoordinates = float4x4(float4( -1.0, -1.0, 0.0, 1.0 ),
float4( 1.0, -1.0, 0.0, 1.0 ),
float4( -1.0, 1.0, 0.0, 1.0 ),
float4( 1.0, 1.0, 0.0, 1.0 ));
float4x2 textureCoordinates = float4x2(float2( 0.0, 1.0 ),
float2( 1.0, 1.0 ),
float2( 0.0, 0.0 ),
float2( 1.0, 0.0 ));
TextureMappingVertex outVertex;
outVertex.renderedCoordinate = renderedCoordinates[vertex_id];
outVertex.textureCoordinate = textureCoordinates[vertex_id];
return outVertex; }
fragment half4 displayTexture(TextureMappingVertex mappingVertex [[ stage_in ]],texture2d<float, access::sample> texture [[ texture(0) ]]) {
constexpr sampler s(address::clamp_to_edge, filter::linear);
return half4(texture.sample(s, mappingVertex.textureCoordinate));
}

Answer 1

A few general things to start with when dealing with Metal textures or Metal in general:
You should take into account the difference between points and pixels, refer to the documentation here. The frame property of a UIView subclass (as MTKView is one) always gives you the width and the height of the view in points.
The mapping from points to actual pixels is controlled through the contentScaleFactor option. The MTKView automatically selects a texture with a fitting aspect ratio that matches the actual pixels of your device. For example, the underlying texture of a MTKView on the iPhone X would have a resolution of 2436 x 1125 (the actual display size in pixels). This is documented here: "The MTKView class automatically supports native screen scale. By default, the size of the view’s current drawable is always guaranteed to match the size of the view itself."
As documented here, the .scaleAspectFill option "scale[s] the content to fill the size of the view. Some portion of the content may be clipped to fill the view’s bounds". You want to simulate this behavior.
Rendering with Metal is nothing more than "drawing" to the resolve texture, which is automatically set by the MTKView. However, you still have full control and could do it on your own by manually creating textures and setting them in your renderPassDescriptor. But you don't need to care about this right now. The single thing you should care about is what, where and which part of the 1080x1920 pixels texture in your resolve texture you want to render in your resolve texture (which might have a different aspect ratio). We want to fully fill ("scaleAspectFill") the resolve texture, so we leave the renderedCoordinates in your fragment shader as they are. The are defining a rectangle over the whole resolve texture, which means the fragment shader is called for every single pixel in the resolve texture. Following, we will simply change the texture coordinates.
Let's define the aspect ratio as ratio = width / height, the resolve texture as r_tex and the texture you want to render as tex.
So assuming your resolve texture does not have the same aspect ratio, there are two possible scenarios:
The aspect ratio of your texture that you want to render is larger than the aspect ratio of your resolve texture (the texture Metal renders to), that means the texture you want to render has a larger width than the resolve texture. In this case we leave the y values of the coordinate as they are. The x values of texture coordinates will be changed:
x_left = 0 + ((tex.width - r_tex.width) / 2.0)
x_right = tex_width - ((tex.width - r_tex_width) / 2.0)
These values must be normalized because the texture samples needs coordinates in the range from 0 to 1:
x_left = x_left / tex.width
x_right = x_right / tex.width
We have our new texture coordinates:
topLeft = float2(x_left,0)
topRight = float2(x_right,0)
bottomLeft = float2(x_left,1)
bottomRight = float2(x_right,1)
This will have the effect that nothing of the top or the bottom of your texture will be cut off, but some outer parts at the left and right side will be clipped, i.e. not visible.
The aspect ratio of your texture that you want to render is smaller than the aspect ratio of your resolve texture. The procedure is the same as with first scenario, but this time we will change the y coordinates
This should render your texture so that the resolve texture is completely filled and the aspect ratio of your texture is maintained on the x-axis. Maintaining the y-axis will work similarly. Additionally you have to check which side of the texture is larger/smaller and incorporate this in your calculation. This will clip parts of your texture as it would be when using scaleAspectFill. Be aware that the above solution is untested. But I hope it is helpful. Be sure to visit Metal Best Practices documentation from time to time, it's very helpful to get the basic concepts right. Have fun with Metal!

Answer 2

So your vertex shader pretty directly dictates that the source texture be stretched to the dimensions of the viewport. You are rendering a quad that fills the viewport, because its coordinates are at the extremes ([-1, 1]) of the Normalized Device Coordinate system in the horizontal and vertical directions.
And you are mapping the source texture corner-to-corner over that same range. That's because you specify the extremes of texture coordinate space ([0, 1]) for the texture coordinates.
There are various approaches to achieve what you want. You could pass the vertex coordinates in to the shader via a buffer, instead of hard-coding them. That way, you can compute the appropriate values in app code. You'd compute the desired destination coordinates in the render target, expressed in NDC. So, conceptually, something like left_ndc = (left_pixel / target_width) * 2 - 1, etc.
Alternatively, and probably easier, you can leave the shader as-is and change the viewport for the draw operation to target only the appropriate portion of the render target.