Metal vertex shader draw points of a Texture - ios

I want to execute Metal (or OpenGLES 3.0) shader that draws Points primitive with blending. To do that, I need to pass all the pixel coordinates of the texture to Vertex shader as vertices which computes the position of the vertex to be passed to fragment shader. The fragment shader simply outputs the color for the point with blending enabled. My problem is if there is an efficient was to pass coordinates of vertices to the vertex shader, since there would be too many vertices for 1920x1080 image, and that needs to be done 30 times in a second? Like we do in a compute shader by using dispatchThreadgroups command, except that compute shader can not draw a geometry with blending enabled.
EDIT: This is what I did -
let vertexFunctionRed = library!.makeFunction(name: "vertexShaderHistogramBlenderRed")
let fragmentFunctionAccumulator = library!.makeFunction(name: "fragmentShaderHistogramAccumulator")
let renderPipelineDescriptorRed = MTLRenderPipelineDescriptor()
renderPipelineDescriptorRed.vertexFunction = vertexFunctionRed
renderPipelineDescriptorRed.fragmentFunction = fragmentFunctionAccumulator
renderPipelineDescriptorRed.colorAttachments[0].pixelFormat = .bgra8Unorm
renderPipelineDescriptorRed.colorAttachments[0].isBlendingEnabled = true
renderPipelineDescriptorRed.colorAttachments[0].rgbBlendOperation = .add
renderPipelineDescriptorRed.colorAttachments[0].sourceRGBBlendFactor = .one
renderPipelineDescriptorRed.colorAttachments[0].destinationRGBBlendFactor = .one
do {
histogramPipelineRed = try device.makeRenderPipelineState(descriptor: renderPipelineDescriptorRed)
} catch {
print("Unable to compile render pipeline state Histogram Red!")
return
}
Drawing code:
let commandBuffer = commandQueue?.makeCommandBuffer()
let renderEncoder = commandBuffer?.makeRenderCommandEncoder(descriptor: renderPassDescriptor!)
renderEncoder?.setRenderPipelineState(histogramPipelineRed!)
renderEncoder?.setVertexTexture(metalTexture, index: 0)
renderEncoder?.drawPrimitives(type: .point, vertexStart: 0, vertexCount: 1, instanceCount: metalTexture!.width*metalTexture!.height)
renderEncoder?.drawPrimitives(type: .point, vertexStart: 0, vertexCount: metalTexture!.width*metalTexture!.height, instanceCount: 1)
and Shaders:
vertex MappedVertex vertexShaderHistogramBlenderRed (texture2d<float, access::sample> inputTexture [[ texture(0) ]],
unsigned int vertexId [[vertex_id]])
{
MappedVertex out;
constexpr sampler s(s_address::clamp_to_edge, t_address::clamp_to_edge, min_filter::linear, mag_filter::linear, coord::pixel);
ushort width = inputTexture.get_width();
ushort height = inputTexture.get_height();
float X = (vertexId % width)/(1.0*width);
float Y = (vertexId/width)/(1.0*height);
int red = inputTexture.sample(s, float2(X,Y)).r;
out.position = float4(-1.0 + (red * 0.0078125), 0.0, 0.0, 1.0);
out.pointSize = 1.0;
out.colorFactor = half3(1.0, 0.0, 0.0);
return out;
}
fragment half4 fragmentShaderHistogramAccumulator ( MappedVertex in [[ stage_in ]]
)
{
half3 colorFactor = in.colorFactor;
return half4(colorFactor*(1.0/256.0), 1.0);
}

Maybe you can draw a single point instanced 1920x1080 times. Something like:
vertex float4 my_func(texture2d<float, access::read> image [[texture(0)]],
constant uint &width [[buffer(0)]],
uint instance_id [[instance_id]])
{
// decompose the instance ID to a position
uint2 pos = uint2(instance_id % width, instance_id / width);
return float4(image.read(pos).r * 255, 0, 0, 0);
}

Related

Metal shader 3D texture lookup

I created a 3D texture from a LUT file on iOS as follows:
let dim = 33
let textureDescriptor = MTLTextureDescriptor()
textureDescriptor.textureType = .type3D
textureDescriptor.pixelFormat = .rgba32Float
textureDescriptor.width = dim
textureDescriptor.height = dim
textureDescriptor.depth = dim
textureDescriptor.usage = .shaderRead
let texture = device.makeTexture(descriptor: textureDescriptor)
texture!.replace(region: MTLRegionMake3D(0, 0, 0, dim, dim, dim),
mipmapLevel:0,
slice:0,
withBytes:values!,
bytesPerRow:dim * MemoryLayout<Float>.size * 4,
bytesPerImage:dim * dim * MemoryLayout<Float>.size * 4)
and then I try to use this LUT in fragment shader as follows:
fragment half4 fragmentShader ( MappedVertex in [[ stage_in ]],
texture2d<float, access::sample> inputTexture [[ texture(0) ]],
texture3d<float, access::sample> lutTexture [[ texture(1) ]]
)
{
constexpr sampler s(s_address::clamp_to_edge, t_address::clamp_to_edge, min_filter::linear, mag_filter::linear);
float3 rgb = inputTexture.sample(s, in.textureCoordinate).rgb;
float3 lookupColor = lutTexture.sample(s, rgb).rgb;
return half4(half3(lookupColor), 1.h);
}
I am afraid that I'm not getting the correct results. Is everything in the code perfect? Am I sampling the 3d texture correctly?

Execution of the command buffer was aborted due to an error during execution

I've been stuck on this for a while; this isn't a particularly expensive shader (at least, based on my very limited experience with Metal), yet I still get this message for the first few frames:
Execution of the command buffer was aborted due to an error during execution. Caused GPU Hang Error (IOAF code 3)
And then I get this one all subsequent frames:
Execution of the command buffer was aborted due to an error during execution. Ignored (for causing prior/excessive GPU errors) (IOAF code 4)
This is the vertex shader, which doesn't really do anything:
vertex VertexOut viewportProgram(uint vertexID [[ vertex_id ]],
constant float2 *positions [[ buffer(0) ]],
constant float2 *texcoords [[ buffer(1) ]]) {
VertexOut out;
out.position.xy = positions[vertexID];
out.position.z = 0.0; // Only 2D; no depth
out.position.w = 1.0; // Only 2D; no perspective divide
out.texcoord = texcoords[vertexID];
return out;
}
Here's my fragment shader, which converts colors from YUV to RGB:
fragment float4 colorConvertProgram(VertexOut in [[stage_in]],
texture2d<float, access::sample> yTexture [[ texture(0) ]],
texture2d<float, access::sample> uvTexture [[ texture(1) ]],
sampler textureSampler [[ sampler(0) ]]) {
float3 colorOffset = float3(-(16.0/255.0), -0.5, -0.5);
float3x3 colorMatrix = float3x3(float3(1.164, 0.000, 1.596),
float3(1.164, -0.392, -0.813),
float3(1.164, 2.017, 0.000));
float3 yuv = float3(yTexture.sample(textureSampler, in.texcoord).r,
uvTexture.sample(textureSampler, in.texcoord).rg);
float3 rgb = (yuv + colorOffset) * colorMatrix;
return float4(rgb, 1.0);
}
And this is my Swift code that puts it all together:
let samplerDescriptor = MTLSamplerDescriptor()
samplerDescriptor.minFilter = .linear
samplerDescriptor.mipFilter = .linear
samplerDescriptor.sAddressMode = .clampToZero
samplerDescriptor.tAddressMode = .clampToZero
let sampler = device.makeSamplerState(descriptor: samplerDescriptor)
renderEncoder.setViewport(MTLViewport(originX: 0.0, originY: 0.0,
width: Double(bounds.width),
height: Double(bounds.height),
znear: -1.0, zfar: 1.0))
renderEncoder.setRenderPipelineState(renderPipelineState)
renderEncoder.setVertexBuffer(vertexPositionBuffer, offset: 0, index: 0)
renderEncoder.setVertexBuffer(vertexTexcoordBuffer, offset: 0, index: 1)
renderEncoder.setFragmentTexture(yTexture, index: 0)
renderEncoder.setFragmentTexture(uvTexture, index: 1)
renderEncoder.setFragmentSamplerState(sampler, index: 0)
renderEncoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4)
renderEncoder.endEncoding()
commandBuffer.present(drawable)
commandBuffer.commit()

Update texture from vector directx9

I' m trying to render two textures one for RGB and another on for the alpha channel, I blend them together with a shader.
The alpha channel texture doesn't overlap properly to the RGB one. It seems to be stretched.
The alpha channel texture changes at every frame and I need to fill starting from an array of uint8_t by the following fuction:
D3DLOCKED_RECT locked_rect;
HRESULT hr = alpha_tex->LockRect(0, &locked_rect, nullptr, 0);
if (!FAILED(hr)) {
ret_code = 0;
BYTE *p_dst = (BYTE *)locked_rect.pBits;
for (uint y = 0; y < height; y++) {
memcpy(p_dst, alpha_array, width);
alpha_array += width;
p_dst += locked_rect.Pitch;
}
alpha_tex->UnlockRect(0);
}
where the alpha_array is a uint8_t array containing the alpha values.
To render the texture i use the following function:
hwctx->d3d9device->Clear(0, 0, D3DCLEAR_TARGET | D3DCLEAR_ZBUFFER, 0xffeeeeee, 1.0f, 0);
hwctx->d3d9device->BeginScene();
ctx->mFX->SetTechnique(ctx->mhTech);
ctx->texRGB->GetSurfaceLevel(0, &ctx->surfRGB);
hwctx->d3d9device->StretchRect((IDirect3DSurface9*)s->vdrFrame->data[3], NULL, ctx->surfRGB, NULL, D3DTEXF_LINEAR);
ctx->mFX->SetTexture(ctx->mhTexRGB, ctx->texRGB);
ctx->mFX->SetTexture(ctx->mhTexAlpha, ctx->texAlpha);
// Enable alpha blending.
hwctx->d3d9device->SetRenderState(D3DRS_ALPHABLENDENABLE, true);
hwctx->d3d9device->SetRenderState(D3DRS_SRCBLEND, D3DBLEND_SRCALPHA);
hwctx->d3d9device->SetRenderState(D3DRS_DESTBLEND, D3DBLEND_INVSRCALPHA);
UINT numPasses = 0;
ctx->mFX->Begin(&numPasses, 0);
for (UINT i = 0; i < numPasses; ++i){
ctx->mFX->BeginPass(i);
hwctx->d3d9device->DrawPrimitive(D3DPT_TRIANGLEFAN, 0, 2);
ctx->mFX->EndPass();
}
ctx->mFX->End();
hwctx->d3d9device->EndScene();
hwctx->d3d9device->Present(0, 0, 0, 0);
// Disable alpha blending.
hwctx->d3d9device->SetRenderState(D3DRS_ALPHABLENDENABLE, false);
I combine the textures by vertex/pixel shader:
uniform extern texture gTexRGB;
uniform extern texture gTexAlpha;
sampler TexRGB = sampler_state{
Texture = <gTexRGB>;
AddressU = WRAP;
AddressV = WRAP;
};
sampler TexAlpha = sampler_state{
Texture = <gTexAlpha>;
AddressU = WRAP;
AddressV = WRAP;
};
struct OutputVS{
float4 posH : POSITION0;
float2 tex0 : TEXCOORD0;
};
OutputVS TextureBlendingVS(float2 tex0: TEXCOORD0){
// Zero out our output.
OutputVS outVS = (OutputVS)0;
// Pass on texture coordinates to be interpolated in rasterization.
outVS.tex0 = tex0;
// Done--return the output.
return outVS;
}
float4 TextureBlendingPS(float2 tex0 : TEXCOORD0) : COLOR{
float3 rgb = tex2D(TexRGB, tex0).rgb;
float alpha = tex2D(TexAlpha, tex0).a;
return float4(rgb, alpha);
}
technique DirLightTexTech{
pass P0 {
// Specify the vertex and pixel shader associated with this pass.
vertexShader = compile vs_2_0 TextureBlendingVS();
pixelShader = compile ps_2_0 TextureBlendingPS();
}
}
The size of the textures is the same but during the rendering something goes wrong.
Please help me. :)

Programmatically writing a texture and reading it in fragment shader

I've written a Compute shader that outputs to a Texture. The coordinate system of the output texture is in pixels. I then have a basic vertex and fragment shader that should simply sample the value and respond with what I thought would be in normalised coordinates. However, I thought this mapping between my programmatically drawn texture and the vertices of my rendering surface would match up.
The Compute Function
Can be summarized as
texture.write(color, uint2(x, y));
where x and y are integer pixel locations.
The Vertex Data
// position.x, position.y, texCoords.x, texCoords.y
let vertexData = [Float](arrayLiteral:
-1, 1, 0, 0,
-1, -1, 0, 1,
1, -1, 1, 1,
1, -1, 1, 1,
1, 1, 1, 0,
-1, 1, 0, 0)
The Metal Shader
typedef struct {
packed_float2 position;
packed_float2 texCoords;
} VertexIn;
typedef struct {
float4 position [[ position ]];
float2 texCoords;
} FragmentVertex;
vertex FragmentVertex simple_vertex(device VertexIn *vertexArray [[ buffer(0) ]],
uint vertexIndex [[ vertex_id ]])
{
VertexIn in = vertexArray[vertexIndex];
FragmentVertex out;
out.position = float4(in.position, 0.f, 1.f);
out.texCoords = in.texCoords;
return out;
}
fragment float4 simple_fragment(FragmentVertex in [[ stage_in ]],
texture2d<uint, access::sample> inputTexture [[ texture(0) ]],
sampler linearSampler [[ sampler(0) ]])
{
const uint2 imageSizeInPixels = uint2(360, 230);
float imageSizeInPixelsWidth = imageSizeInPixels.x;
float imageSizeInPixelsHeight = imageSizeInPixels.y;
float2 coords = float2(in.position.x / 360.f, in.position.y / 230.f);
float color = inputTexture.sample(linearSampler, in.texCoords).x / 255.f;
return float4(float3(color), 1.f);
}
The Sampler
let samplerDescriptor = MTLSamplerDescriptor()
samplerDescriptor.normalizedCoordinates = true
samplerDescriptor.minFilter = .linear
samplerDescriptor.magFilter = .linear
samplerDescriptor.sAddressMode = .clampToZero
samplerDescriptor.rAddressMode = .clampToZero
self.samplerState = self.metalDevice?.makeSamplerState(descriptor: samplerDescriptor)
In this experiment the only value that seems to work is coords, based upon the normalized in.position value. in.texCoords seems to always be zero. Shouldn't the texcoords and position received by the vertex and fragment shader be values be in the range of values defined in the vertex data?
My Vertex Buffer was right, but wrong
In the process of converting Obj-C code to Swift I failed to copy the vertex completely.
The Correct Copy
let byteCount = vertexData.count * MemoryLayout<Float>.size
let vertexBuffer = self.metalDevice?.makeBuffer(bytes: vertexData, length: byteCount, options: options)
The Source of my Woes
let vertexBuffer = self.metalDevice?.makeBuffer(bytes: vertexData, length: vertexData.count, options: options)
The Complete Vertex Buffer Creation
// Vertex data for a full-screen quad. The first two numbers in each row represent
// the x, y position of the point in normalized coordinates. The second two numbers
// represent the texture coordinates for the corresponding position.
let vertexData = [Float](arrayLiteral:
-1, 1, 0, 0,
-1, -1, 0, 1,
1, -1, 1, 1,
1, -1, 1, 1,
1, 1, 1, 0,
-1, 1, 0, 0)
// Create a buffer to hold the static vertex data
let options = MTLResourceOptions().union(.storageModeShared)
let byteCount = vertexData.count * MemoryLayout<Float>.size
let vertexBuffer = self.metalDevice?.makeBuffer(bytes: vertexData, length: byteCount, options: options)
vertexBuffer?.label = "Image Quad Vertices"
self.vertexBuffer = vertexBuffer

Indexed drawing with metal

I am trying to load a model (form .OBJ) and draw it to the screen on iOS with MetalKit. The problem is that instead of my model, I get some random polygons...
Here is the code that is tend to load the model(The code is based on a tutorial from raywenderlich.com:
let allocator = MTKMeshBufferAllocator(device: device)
let vertexDescriptor = MDLVertexDescriptor()
let vertexLayout = MDLVertexBufferLayout()
vertexLayout.stride = sizeof(Vertex)
vertexDescriptor.layouts = [vertexLayout]
vertexDescriptor.attributes = [MDLVertexAttribute(name: MDLVertexAttributePosition, format: MDLVertexFormat.Float3, offset: 0, bufferIndex: 0),
MDLVertexAttribute(name: MDLVertexAttributeColor, format: MDLVertexFormat.Float4, offset: sizeof(float3), bufferIndex: 0),
MDLVertexAttribute(name: MDLVertexAttributeTextureCoordinate, format: MDLVertexFormat.Float2, offset: sizeof(float3)+sizeof(float4), bufferIndex: 0),
MDLVertexAttribute(name: MDLVertexAttributeNormal, format: MDLVertexFormat.Float3, offset: sizeof(float3)+sizeof(float4)+sizeof(float2), bufferIndex: 0)]
var error: NSError?
let asset = MDLAsset(URL: path, vertexDescriptor: vertexDescriptor, bufferAllocator: allocator, preserveTopology: true, error: &error)
if error != nil{
print(error)
return nil
}
let model = asset.objectAtIndex(0) as! MDLMesh
let mesh = try MTKMesh(mesh: model, device: device)
And here is my drawing method:
func render(commandQueue: MTLCommandQueue, pipelineState: MTLRenderPipelineState,drawable: CAMetalDrawable,projectionMatrix: float4x4,modelViewMatrix: float4x4, clearColor: MTLClearColor){
dispatch_semaphore_wait(bufferProvider.availibleResourcesSemaphore, DISPATCH_TIME_FOREVER)
let renderPassDescriptor = MTLRenderPassDescriptor()
renderPassDescriptor.colorAttachments[0].texture = drawable.texture
renderPassDescriptor.colorAttachments[0].loadAction = .Clear
renderPassDescriptor.colorAttachments[0].clearColor = clearColor
renderPassDescriptor.colorAttachments[0].storeAction = .Store
let commandBuffer = commandQueue.commandBuffer()
commandBuffer.addCompletedHandler { (buffer) in
dispatch_semaphore_signal(self.bufferProvider.availibleResourcesSemaphore)
}
let renderEncoder = commandBuffer.renderCommandEncoderWithDescriptor(renderPassDescriptor)
renderEncoder.setCullMode(MTLCullMode.None)
renderEncoder.setRenderPipelineState(pipelineState)
renderEncoder.setVertexBuffer(vertexBuffer, offset: 0, atIndex: 0)
renderEncoder.setFragmentTexture(texture, atIndex: 0)
if let samplerState = samplerState{
renderEncoder.setFragmentSamplerState(samplerState, atIndex: 0)
}
var nodeModelMatrix = self.modelMatrix()
nodeModelMatrix.multiplyLeft(modelViewMatrix)
uniformBuffer = bufferProvider.nextUniformsBuffer(projectionMatrix, modelViewMatrix: nodeModelMatrix, light: light)
renderEncoder.setVertexBuffer(self.uniformBuffer, offset: 0, atIndex: 1)
renderEncoder.setFragmentBuffer(uniformBuffer, offset: 0, atIndex: 1)
if indexBuffer != nil{
renderEncoder.drawIndexedPrimitives(.Triangle, indexCount: self.indexCount, indexType: self.indexType, indexBuffer: self.indexBuffer!, indexBufferOffset: 0)
}else{
renderEncoder.drawPrimitives(.Triangle, vertexStart: 0, vertexCount: vertexCount, instanceCount: vertexCount/3)
}
renderEncoder.endEncoding()
commandBuffer.presentDrawable(drawable)
commandBuffer.commit()
}
Here is my vertex shader:
struct VertexIn{
packed_float3 position;
packed_float4 color;
packed_float2 texCoord;
packed_float3 normal;
};
struct VertexOut{
float4 position [[position]];
float3 fragmentPosition;
float4 color;
float2 texCoord;
float3 normal;
};
struct Light{
packed_float3 color;
float ambientIntensity;
packed_float3 direction;
float diffuseIntensity;
float shininess;
float specularIntensity;
};
struct Uniforms{
float4x4 modelMatrix;
float4x4 projectionMatrix;
Light light;
};
vertex VertexOut basic_vertex(
const device VertexIn* vertex_array [[ buffer(0) ]],
const device Uniforms& uniforms [[ buffer(1) ]],
unsigned int vid [[ vertex_id ]]) {
float4x4 mv_Matrix = uniforms.modelMatrix;
float4x4 proj_Matrix = uniforms.projectionMatrix;
VertexIn VertexIn = vertex_array[vid];
VertexOut VertexOut;
VertexOut.position = proj_Matrix * mv_Matrix * float4(VertexIn.position,1);
VertexOut.fragmentPosition = (mv_Matrix * float4(VertexIn.position,1)).xyz;
VertexOut.color = VertexIn.color;
VertexOut.texCoord = VertexIn.texCoord;
VertexOut.normal = (mv_Matrix * float4(VertexIn.normal, 0.0)).xyz;
return VertexOut;
}
And here is how it looks like:
link
Actually I have an other class that is completely written by me to load models. It works fine, the problem is that it is not using indexing so f I try to load models that are more complex than a low-poly sphere, the GPU crashes... Anyways I tried to modify it to use indexing and I got the same result.. than I added hardcoded indices for testing and I got a really weird result. When I had 3 indices it drew a triangle, when I added 3 more, it drew the same triangle and after 3 more vertices it drew 2 triangles...
Edit:
Here is my Vertex structure:
struct Vertex:Equatable{
var x,y,z: Float
var r,g,b,a: Float
var s,t: Float
var nX,nY,nZ:Float
func floatBuffer()->[Float]{
return [x,y,z,r,g,b,a,s,t,nX,nY,nZ]
}
}
I see a couple of potential issues here.
1) Your vertex descriptor does not map exactly to your Vertex struct. The position variables (x, y, z) occupy 12 bytes, so the color variables start at an offset of 12 bytes. This matches the packed_float3 position field in your shader's VertexIn struct, but in the vertex descriptor you provide to Model I/O, you use sizeof(Float3), which is 16, as the offset of the color attribute. Because you're packing the position field, you should use sizeof(Float) * 3 for this value instead, and likewise in the subsequent offsets. I suspect this is the main cause of your problems.
More generally, it's a good idea to use strideof rather than sizeof to account for alignment, though--by chance--it wouldn't make a difference here.
2) Model I/O is allowed to use a single MTLBuffer to store both vertices and indices, so you should use the offset member of each MTKMeshBuffer when setting the vertex buffer or specifying the index buffer in each draw call, rather than assuming the offsets to be 0.

Resources