fragment half4 fragmen_shader_test(WaterColorCloudOut params[[stage_in]],
texture2d<float , access::sample>cloud1 [[texture(0)]],
texture2d<half, access::sample> cloud2 [[texture(1)]],
texture2d<half, access::sample> cloud3 [[texture(2)]]
)
{
constexpr sampler defaultSampler;
float4 color1;
if(params.index == 0){
color1= float4(cloud1.sample(defaultSampler, float2(params.textureCoordinates))) * params.color ;
}
else if(params.index == 1){
color1= float4(cloud2.sample(defaultSampler, float2(params.textureCoordinates))) * params.color ;
} else{
color1= float4(cloud3.sample(defaultSampler, float2(params.textureCoordinates))) * params.color ;
}
return half4(color1);
}
Here I am using three textures because of If-else condition the performance drops with time. I feel if i send a texture array to the shader there is no need to perform if else statement. In CPU I have three MTLTexture. How can I bind the three texture to an array and pass it shader.
In CPU side I created three Textures and Created a MTLTexture array
var textureArray:[MTLTexture] = []
Then I append the Texture to that array.
In MTLRenderCommandEncoder
let myRange: CountableRange = 0..<2
commandEncoder.setFragmentTextures(textureArray, range: myRange)
In Shader
texture2d_array<float , access::sample> texture [[ texture(0) ]]
While sampling in Shader
float4 color = texture.sample(defaultSampler, float2(params.textureCoordinates),0) * float4(1,0,0,1.0);
I am doing like this currently ButI couldn't get the correct Texture output
You don't have to change the app code. Since you are using consecutive texture indexes, you can just replace the three separate parameters with a single texture array parameter bound to index 0 and the bindings will connect the elements of the array with the textures you've provided from the app code.
In other words, if you declare a texture array with size 3 and bound to texture index 0, the elements of the array are taken from texture index 0, 1, and 2.
Update for your edited question:
You are confusing an array of textures (also called a texture array) and an array texture. The terminology is confusingly subtle.
An array texture is a single texture with multiple 2D planes in it. All of the planes have the same size and pixel format. The number of planes in the texture does not have to be known at shader compile time.
An array of textures is an array of independent textures, which may have different sizes or pixel formats. The length of the array is a compile-time constant.
You used texture2d_array<...>. That's an array texture.
For a texture array, or array of textures, you should use array<texture2d<float, access::sample>, 3> clouds [[texture(0)]].
To sample from a texture within the array, you first index into the array and then call a texture function on that element:
float4 color = clouds[params.index].sample(defaultSampler, float2(params.textureCoordinates)) * float4(1,0,0,1.0);
Related
I'm using Metal to render a scene with a z buffer and now need to integrate this z-buffer into SceneKit's rendering. However I can't figure out how to get SceneKit to use this depth better correctly and am not even 100% sure what format SceneKit expects it's z-buffers to be in
Base on this question, my understanding was that SceneKit uses a reverse logarithmic z-buffer in the range of 1 (near) to 0 (far). However I can't get this working and objects I draw with SceneKit don't properly respect the depth buffer: they are either always showing or always hidden
First, here's how the generate a z-buffer texture in a Metal render pass:
struct FragmentOut {
float4 color [[color(0)]];
float depth [[depth(any)]];
};
fragment FragmentOut metalRenderFragment(const InOut in [[ stage_in ]]) {
FragmentOut out;
out.depth = 0; // 0 is far with reverse z buffer
...
float cameraSpaceZ = ...; // Computed in shader
// There constants are taken from SceneKit's camera and inlined here
const float zNear = 0.0010000000474974513;
const float zFar = 1000.0;
float logDepth = log(z / zNear) / log(zFar / zNear);
out.depth = 1.0 - logDepth; // Reverse the depth for scenekit
return out;
}
Then to integrate the depth buffer into SceneKit, I render a full screen quad in scenekit with a SCNProgram that uses the depth texture generated in the previous step:
fragment FragmentOut sceneKitFullScreenQuadFragment(const InOut in [[ stage_in ]],
depth2d<float, access::sample> depthTexture [[texture(1)]])
{
constexpr sampler sampler(filter::linear);
const float depth = depthTexture.sample(sampler, in.uv);
return {
.color = float4(0),
.depth = depth,
};
}
So two questions:
What format does SceneKit use for its z-buffer? Is it a reversed logarithmic z-buffer?
What am I doing wrong in generating the z-buffer values for SceneKit?
SceneKit uses a reverse logarithmic Z-Buffer. This post and this post show you how to get a normalized linear mapping space [0...1]. You need the opposite formula.
Also, you can toggle the value from reverseZ to directZ this way:
let sceneView = self.view as! SCNView
sceneView.usesReverseZ = true // default
Andy Jazz's answer helped but I still found the links confusing. Here's what ultimately worked for me (although there are possibly other ways to do this):
When generating the depth map (this would be inside the the metal shader in my original example) pass in SceneKit's projection transform matrix and use this to transform the depth value:
// In a metal shader generating the depth map
// The z distance from the camera, e.g. if the object
// at the current position is 5 units away, this would be 5.
const float z = ...;
// The camera points along the -z axis, so transform the -z position
// with SceneKit's projection matrix (you can get this from SCNCamera)
const float4 depthPos = (sceneKitState.projectionTransform * float4(0, 0, -z, 1));
// Then do perspective division to get the final depth value
out.depth = depthPos.z / depthPos.w;
Then inside of the SceneKit shader, simply write out the depth, taking into account usesReverseZ:
// In a scenekit, full screen quad shader
const float depth = depthTexture.sample(sampler, in.uv);
return {
.color = float4(0),
.depth = 1.0 - depth,
};
❗️ The above assumes you are using sceneView.usesReverseZ = true (the default). If you are using usesReverseZ = false, simply do .depth = depth instead
I have an MTLTexture in RGBA8Unorm format, and a screen texture (in MTKView) in BGRA8Unorm format (reversed). In the Metal shader, when I sample from that texture using sample(), I get float4. When I write to texture in metal shader, I also write float4. It seems that when I am inside the shader code, float4 always represents the same order of components RGBA regardless of the original format the texture is in ([0] for red, [1] for green, [2] for blue, and [3] for alpha). Is my conclusion correct that the meaning of the components of the sampled/written float4 is always the same inside the shader, regardless of what the storage format of the texture is?
UPDATE: I use the following code to write to a texture with RGBA8Unnorm format:
kernel void
computeColourMap(constant Uniforms &uniforms [[buffer(0)]],
constant array<float, 120> &s [[buffer(1)]],
constant array<float, 120> &red [[buffer(2)]],
constant array<float, 120> &green [[buffer(3)]],
constant array<float, 120> &blue [[buffer(4)]],
texture2d<float, access::write> output [[texture(0)]],
uint2 id [[thread_position_in_grid]])
{
if (id.x >= output.get_width() || id.y >= output.get_height()) {
return;
}
uint i = id.x % 120;
float4 col (0, 0, 0, 1);
col.x += amps[i] * red[i];
col.y += amps[i] * green[i];
col.z += amps[i] * blue[i];
output.write(col, id);
}
I then use the following shaders for the rendering stage:
vertex VertexOut
vertexShader(const device VertexIn *vertexArray [[buffer(0)]],
unsigned int vid [[vertex_id]])
{
VertexIn vertex_in = vertexArray[vid];
VertexOut vertex_out;
vertex_out.position = vertex_in.position;
vertex_out.textureCoord = vertex_in.textureCoord;
return vertex_out;
}
fragment float4
fragmentShader(VertexOut interpolated [[stage_in]],
texture2d<float> colorTexture [[ texture(0) ]])
{
const float4 colorSample = colorTexture.sample(nearestSampler,
interpolated.textureCoord);
return colorSample;
}
where colourTexture passed into the fragment shader is the one I generated in RGBA8Unorm format, and in Swift I have:
let renderPipelineDescriptor = MTLRenderPipelineDescriptor()
renderPipelineDescriptor.vertexFunction = library.makeFunction(name: "vertexShader")!
renderPipelineDescriptor.fragmentFunction = library.makeFunction(name: "fragmentShader")!
renderPipelineDescriptor.colorAttachments[0].pixelFormat = colorPixelFormat
the colorPixelFormat of the MTKView is BGRA8Unorm (reversed relative to texture), which is not the same as my texture, but the colours on the screen come out correct.
UPDATE 2: one further pointer that within a shader the colour represented by float4 always has order of rgba is: float4 type actually has accessors called v.r, v.g, v.b, v.rgb, etc...
The vector always has 4 components, but the type of the components is not necessarily float. When you declare a texture, you specify the component type as a template argument (texture2d<float ...> in your code).
For example, from Metal Shading Language Specification v2.1, section 5.10.1:
The following member functions can be used to sample from a 1D
texture.
Tv sample(sampler s, float coord) const
Tv is a 4-component vector type based on the templated type used
to declare the texture type. If T is float, Tv is float4. If T is half,
Tv is half4. If T is int, Tv is int4. If T is uint, Tv is uint4. If T
is short, Tv is short4 and if T is ushort, Tv is ushort4.
The same Tv type is used in the declaration of write(). The functions for other texture types are documented in a similar manner.
And, yes, component .r always contains the red component (if present), etc. And [0] always corresponds to .r (or .x).
I have array with SCNVector3, one for each vertex.
var terrainArray = [SCNVector3]()
I need to provide this data per vertex in my fragment shader. Something like this:
struct TerrainVertexInput
{
float3 position [[attribute(SCNVertexSemanticPosition)]];
float4 color [[attribute(SCNVertexSemanticColor)]];
};
struct TerrainVertexOutput
{
float4 position [[position]];
float3 terrain;
float4 color;
};
vertex TerrainVertexOutput terrainVertex(TerrainVertexInput in [[stage_in]],
constant SCNSceneBuffer& scn_frame [[buffer(0)]],
constant MyNodeBuffer& scn_node [[buffer(1)]],
constant float3 terrain [[buffer(2)]])
{
TerrainVertexOutput v;
v.position = scn_node.modelViewProjectionTransform * float4(in.position, 1.0);
v.terrain = terrain;
v.color = in.color;
return v;
}
As I understand I need to create Data object with array data and provide it to program with setValue(_:forKey:) but I'm not sure if vertex function will get right element for vertex.
How to do this right?
You're on the right track, but you don't want to use SCNVector3 for data you're passing into a Metal shader. SceneKit's vector types have components of type CGFloat, the size of which is platform-dependent.
Instead, your data should use one of the simd vector types. In Swift and Metal, that means float3 or float4. Note that float3 actually occupies 16 bytes of space; there's a dummy element at the end for alignment purposes. If you want to pack your data tightly, using exactly 3 floats per vertex, you can type your buffer in Metal as packed_float3 and write 3 contiguous floats into your data buffer for each vertex. There is no three-element packed float vector type in Swift.
There are many ways to copy an array of SCNVector3 into a suitably-typed data buffer. Here's one:
// Allocate enough memory to store three floats per vertex, ensuring we free it later
let terrainBuffer = UnsafeMutableBufferPointer<Float>.allocate(capacity: terrainArray.count * 3)
defer {
terrainBuffer.deallocate()
}
// Copy each element of each vector into the buffer
terrainArray.enumerated().forEach { i, v in
terrainBuffer[i * 3 + 0] = Float(v.x)
terrainBuffer[i * 3 + 1] = Float(v.y)
terrainBuffer[i * 3 + 2] = Float(v.z)
}
// Copy the buffer data into a Data object, as expected by SceneKit
let terrainData = Data(buffer: terrainBuffer)
You can then use setValue(:forKey:) on your geometry or material:
material.setValue(terrainData, forKey: "terrain")
Rather than taking a single float3 as a parameter in your vertex function, instead take a pointer to packed_float3 and index into it according to the vertex ID:
vertex TerrainVertexOutput terrainVertex(TerrainVertexInput in [[stage_in]],
constant SCNSceneBuffer& scn_frame [[buffer(0)]],
constant MyNodeBuffer& scn_node [[buffer(1)]],
constant packed_float3 *terrain [[buffer(2)]],
uint vid [[vertex_id]]) {
// ...
v.terrain = terrain[vid];
// ...
}
This assumes an exact correspondence between vertices in your geometry and terrain data points. Rather than using the vertex ID directly, you can of course do whatever sort of fancy indexing you want to look up the terrain data for a given vertex.
Render an image in two steps.
It needs two pair vertex and fragment shader.
In the second step, the second fragment shader needs the frame buffer rendered by first pair shader.
How to get the frame buffer or the color in a special coordinate.
I have read the following answer.
iOS Metal Shader - Texture read and write access?
How to chain filters in Metal for iOS?
fragment float4 second_fragment(VertexOut vert [[stage_in]],
texture2d<float> tex [[texture(0)]],
float4 framebuffer [[color(0)]]
) {
float4 textureColor = tex.sample(basicSampler,vert.texCoor);
return textureColor;
}
[[color(0)]] qualifier only get the color in current coordinate. I need to know the color in any other coordinate.
I generate simple 2D grid with triangle strip representing water surface. First generated vertex has position [0,0] and the last one has [1,1]. For my water simulation I need to store current positions of vertices to a texture and then sample these values from the texture in the next frame to get the previous state of the water surface.
So, I created the texture in a size of vertices. For example if I will have a 10x10 vertices grid, I use a texture with 10x10 pixels (one pixel for one vertex data). And set this texture as a render target to render all vertex data into it.
According to this: MSDN Coordinate Systems, If I will use current positions of vertices in the grid (bottom-left at [0;0], top-right at [1;1]), rendered texture looks like this:
So I need to do some conversion to NDC. I convert it in a vertex shader like this:
[vertex.x * 2 - 1; vertex.y * 2 - 1]
Consider this 3x3 grid:
Now, grid is stretched to whole texture size. Texture coordinates are different from NDC and apparently I can use original coordinates of the grid (before conversion) to sample values from the texture and get previous values (positions) of vertices.
Here is a sample of my vertex/pixel shader code:
This vertex shader converts coordinates and sends it to pixel shader with SV_POSITION semantics (describes the pixel location).
struct VertexInput
{
float4 pos : POSITION;
float2 tex : TEXCOORD;
};
struct VertexOutput
{
float4 pos : SV_POSITION;
float2 tex : TEXCOORD;
};
// convertes coordinates from 0,0 origin to -1,-1, etc.
float2 toNDC(float2 px)
{
return float2(px.x * 2 - 1, px.y * 2 - 1);
}
VertexOutput main( VertexInput input )
{
VertexOutput output;
float2 ndc = toNDC(float2(input.pos.x, input.pos.z));
output.pos = float4(ndc, 1, 1);
output.tex = float2(input.pos.x, input.pos.z);
return output;
}
And here's the pixel shader saving values from vertex shader at defined pixel location (SV_POSITION).
struct PixelInput
{
float4 pos : SV_POSITION;
float2 tex : TEXCOORD;
};
float4 main(PixelInput input) : SV_TARGET
{
return float4(input.tex.x, input.tex.y, 0, 1);
}
And we're finally getting to my problem! I use graphics debugger in Visual Studio 2012 which allows me to look at the rendered texture and its values. I would expect that at the pixel location [0,1] (in texel coordinate system) should be value [0,0] (or [0,0,0,1] to be precise, for RGBA format) but it seems that value of final pixel is interpolated between 3 vertices and I have a wrong value for a given vertex.
Screenshot from VS graphics debugger:
Rendered 3x3 texture ([0;1] location in texel coordinate system):
Values from vertex and pixel shader:
How to render the exact value from vertex shader to texture for a given pixel?
I am pretty new to computer graphics and Direct3D 11, so please excuse my deficiencies.