I've got a YUV420 pixelbuffer in a UInt8 Array. I need to create a texture out of it in order to render it with OpenGL. In Android there is an easy way to decode my array to an RGB array for the texture. The code is the following:
BitmapFactory.Options bO = new BitmapFactory.Options();
bO.inJustDecodeBounds = false;
bO.inPreferredConfig = Bitmap.Config.RGB_565;
try {
myBitmap= BitmapFactory.decodeByteArray( yuvbuffer,
0,
yuvbuffer.length,
bO);
} catch (Throwable e) {
// ...
}
I need to decode the yuv buffer on my ios platform (Xcode 8.3.3, Swift 3.1) in order to put it into the following method as data:
void glTexImage2D( GLenum target,
GLint level,
GLint internalFormat,
GLsizei width,
GLsizei height,
GLint border,
GLenum format,
GLenum type,
const GLvoid * data);
How can I achieve this decoding?
ALTERNATIVE:
I've described the way I am decoding the YUV-buffer on Android. Maybe there is an other way to create a texture based on yuvpixels without decoding it like this. I've already tried the following method using the FragmentShader (Link), but it is not working for me. I'm getting a black screen or a green screen, but the image is never rendered. There are also some methods using two seperate buffers for Y and for UV - but on this I don't know how to split my YUV-buffer into Y and UV.
Do you have any new examples/samples for yuv-rendering which are not outdated and working?
If you need only to display that image/video, then you don't really need to convert it to rgb texture. You can bind all 3 planes (Y/Cb/Cr) as separate textures, and perform yuv->rgb conversion in fragment shader, with just a three dot products.
Related
Given an array of RBG pixels that updates every frame (e.g. 1024x1024), a ID3D11RenderTargetView, ID3D11Device and ID3D11DeviceContext, what's the easiest way to draw these pixels to the render view?
I've been working the angle of creating a vertex buffer for a square (two triangles), trying to make pixels be a proper texture, and figuring out how to make a shader reference the texture sampler. I've been following this tutorial https://learn.microsoft.com/en-us/windows/uwp/gaming/applying-textures-to-primitives .... But to be honest, I don't see how this tutorial has shaders that even reference the texture data (shaders defined on the proceeding tutorial, here).
I am a total DirectX novice, but I am writing a plugin for an application where I am given a directx11 device/view/context, and need to fill it with my pixel data. Many thanks!
IF you can make sure your staging resource matches the exact resolution and format of the render target you are given:
Create a staging resource
Map the staging resource, and copy your data into it.
Unmap the staging resource
UseGetResource on the RTV to get the resource
CopyResource from your staging to that resource.
Otherwise, IF you can count on Direct3D Hardware Feature level 10.0 or better, the easiest way would be:
Create a texture with USAGE_DYNAMIC.
Map it and copy your data into the texture.
Unmap the resource
Render the dynamic texture as a 'full-screen' quad using the 'big-triangle' self-generation trick in the vertex shader:
SamplerState PointSampler : register(s0);
Texture2D<float4> Texture : register(t0);
struct Interpolators
{
float4 Position : SV_Position;
float2 TexCoord : TEXCOORD0;
};
Interpolators main(uint vI : SV_VertexId)
{
Interpolators output;
// We use the 'big triangle' optimization so you only Draw 3 verticies instead of 4.
float2 texcoord = float2((vI << 1) & 2, vI & 2);
output.TexCoord = texcoord;
output.Position = float4(texcoord.x * 2 - 1, -texcoord.y * 2 + 1, 0, 1);
return output;
}
and a pixel shader of:
float4 main(Interpolators In) : SV_Target0
{
return Texture.Sample(PointSampler, In.TexCoord);
}
Then draw with:
ID3D11ShaderResourceView* textures[1] = { texture };
context->PSSetShaderResources(0, 1, textures);
// You need a sampler object.
context->PSSetSamplers(0, 1, &sampler);
// Depending on your desired result, you may need state objects here
context->OMSetBlendState(nullptr, nullptr, 0xffffffff);
context->OMSetDepthStencilState(nullptr, 0);
context->RSSetState(nullptr);
context->IASetInputLayout(nullptr);
contet->IASetPrimitiveTopology(D3D11_PRIMITIVE_TOPOLOGY_TRIANGLELIST);
Draw(3, 0);
For full source for the "Full Screen Quad" drawing, see GitHub.
I'm using luaglut to do some graphics in lua. And I am struggling with this function glReadPixels, particularly with its last input argument GLvoid *pixels.
void glReadPixels (GLint x, GLint y, GLsizei width, GLsizei height, GLenum format, GLenum type, GLvoid *pixels);
pixels is a pointer type so in lua it is of type lightuserdata. I managed to get a lightuserdata type variable let's say img in lua, according to this post; however, after I get the frame I wanna grab into img by calling:
glReadPixels(0, 0, 250, 250, GL_RGB, GL_UNSIGNED_BYTE, img)
I could do nothing with img. I tried creating a same structure in lua using ffi and coverting this img to a torch.Tensor type, but it is too slow since I have to assign the values pixel by pixel.
So I am asking here if there is better ways to use this glReadPixels function to get img than this troublesome approach that I took? Both table and torch.Tensor types of img are OK. Thank you in advance!
I'm using AVFoundation to get camera stream.
I'm using this code to get MTLTextures from:
- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
id<MTLTexture> texture = nil;
{
size_t width = CVPixelBufferGetWidth(pixelBuffer);
size_t height = CVPixelBufferGetHeight(pixelBuffer);
MTLPixelFormat pixelFormat = MTLPixelFormatBGRA8Unorm;
CVMetalTextureRef metalTextureRef = NULL;
CVReturn status = CVMetalTextureCacheCreateTextureFromImage(NULL, _textureCache, pixelBuffer, NULL, pixelFormat, width, height, 0, &metalTextureRef);
if(status == kCVReturnSuccess)
{
texture = CVMetalTextureGetTexture(metalTextureRef);
if (self.delegate){
[self.delegate textureUpdated:texture];
}
CFRelease(metalTextureRef);
}
}
}
It works fine, except for that generated MTLTexture object is not mipmaped (has only one mip level).
In this call:
CVMetalTextureCacheCreateTextureFromImage(NULL, _textureCache, pixelBuffer, NULL, pixelFormat, width, height, 0, &metalTextureRef);
There is a third parameter called "textureAtributes", I think it's possible to specify that I want mipmaped texture, but I haven't found any word in documentation what exactly goes there. Neither had I find a source code in which something else is passed instead of NULL.
In OpenGLES for iOS there is similar method, with same parameter, and also no words in documentation .
Just received an answer from Metal engineer here. Here's a quote:
No, it is not possible to generate a mipmapped texture from a
CVPixelBuffer directly.
CVPixelBuffer images typically have a linear/stride layout, as non-GPU hardware blocks might be interacting with those, and most GPU
hardware only supports mipmapping from tiled textures. You'll need to
issue a blit to copy from the linear MTLTexture to a private
MTLTexture of your own creation, then generate mipmaps.
As for textureAttributes, there is only one key supported: kCVMetalTextureCacheMaximumTextureAgeKey
There isn't a method to get a mipmapped texture directly, but you can generate one yourself easily enough.
First use your Metal device to create an empty Metal texture that is the same size and format as your existing texture, but has a full mipmap chain. See newTexture documentation
Use your MTLCommandBuffer object to create a blitEncoder object. See blitCommandEncoder documentation
Use the blitEncoder to copy from your camera texture to your empty texture. destinationLevel should be zero as you are only copying the top level mipmap. See copyFromTexture documentation
Finally use the blitEncoder to generate all the mip levels by calling generateMipmapsForTexture. See generateMipMapsForTexture documentation
At the end of this you have a metal texture from the camera with a full mip chain.
I've been asked to split the question below into multiple questions:
HLSL and Pix number of questions
This is asking the first question, can I in HLSL 3 run a pixel shader without a vertex shader. In HLSL 2 I notice you can but I can't seem to find a way in 3?
The shader will compile fine, I will then however get this error from Visual Studio when calling SpriteBatch Draw().
"Cannot mix shader model 3.0 with earlier shader models. If either the vertex shader or pixel shader is compiled as 3.0, they must both be."
I don't believe I've defined anything in the shader to use anything earlier then 3. So I'm left a bit confused. Any help would be appreciated.
The problem is that the built-in SpriteBatch shader is 2.0. If you specify a pixel shader only, SpriteBatch still uses its built-in vertex shader. Hence the version mismatch.
The solution, then, is to also specify a vertex shader yourself. Fortunately Microsoft provides the source to XNA's built-in shaders. All it involves is a matrix transformation. Here's the code, modified so you can use it directly:
float4x4 MatrixTransform;
void SpriteVertexShader(inout float4 color : COLOR0,
inout float2 texCoord : TEXCOORD0,
inout float4 position : SV_Position)
{
position = mul(position, MatrixTransform);
}
And then - because SpriteBatch won't set it for you - setting your effect's MatrixTransform correctly. It's a simple projection of "client" space (source from this blog post). Here's the code:
Matrix projection = Matrix.CreateOrthographicOffCenter(0,
GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height, 0, 0, 1);
Matrix halfPixelOffset = Matrix.CreateTranslation(-0.5f, -0.5f, 0);
effect.Parameters["MatrixTransform"].SetValue(halfPixelOffset * projection);
You can try the simple examples here. The greyscale shader is a very good example to understand how a minimal pixel shader works.
Basically, you create a Effect under your content project like this one:
sampler s0;
float4 PixelShaderFunction(float2 coords: TEXCOORD0) : COLOR0
{
// B/N
//float4 color = tex2D(s0, coords);
//color.gb = color.r;
// Transparent
float4 color = tex2D(s0, coords);
return color;
}
technique Technique1
{
pass Pass1
{
PixelShader = compile ps_2_0 PixelShaderFunction();
}
}
You also need to:
Create an Effect object and load its content.
ambienceEffect = Content.Load("Effects/Ambient");
Call your SpriteBatch.Begin() method passing the Effect object you want to use
spriteBatch.Begin( SpriteSortMode.FrontToBack,
BlendState.AlphaBlend,
null,
null,
null,
ambienceEffect,
camera2d.GetTransformation());
Inside the SpriteBatch.Begin() - SpriteBatch.End() block, you must call the Technique inside the Effect
ambienceEffect.CurrentTechnique.Passes[0].Apply();
What's the efficient way to render a bunch of layered textures? I have some semitransparent textured rectangles that I position randomly in 3D space and render them from back to front.
Currently I call d3dContext->PSSetShaderResources() to feed the pixel shader with a new texture before each call to d3dContext->DrawIndexed(). I have a feeling that I am copying the texture to the GPU memory before each draw. I might have 10-30 ARGB textures roughly 1024x1024 pixels each and they are associated across 100-200 rectangles that I render on screen. My FPS is OK at 100, but goes pretty bad around 200. I possibly have some inefficiencies elsewhere since this is my first semi-serious D3D code, but I strongly suspect this has to do with copying the textures back and forth. 30*1024*1024*4 is 120MB, which is a bit high for a Metro Style App that should target any Windows 8 device. So putting them all in there might be a stretch, but maybe I could at least cache a few somehow? Any ideas?
*EDIT - Some code snippets added
Constant Buffer
struct ModelViewProjectionConstantBuffer
{
DirectX::XMMATRIX model;
DirectX::XMMATRIX view;
DirectX::XMMATRIX projection;
float opacity;
float3 highlight;
float3 shadow;
float textureTransitionAmount;
};
The Render Method
void RectangleRenderer::Render()
{
// Clear background and depth stencil
const float backgroundColorRGBA[] = { 0.35f, 0.35f, 0.85f, 1.000f };
m_d3dContext->ClearRenderTargetView(
m_renderTargetView.Get(),
backgroundColorRGBA
);
m_d3dContext->ClearDepthStencilView(
m_depthStencilView.Get(),
D3D11_CLEAR_DEPTH,
1.0f,
0
);
// Don't draw anything else until all textures are loaded
if (!m_loadingComplete)
return;
m_d3dContext->OMSetRenderTargets(
1,
m_renderTargetView.GetAddressOf(),
m_depthStencilView.Get()
);
UINT stride = sizeof(BasicVertex);
UINT offset = 0;
// The vertext buffer only has 4 vertices of a rectangle
m_d3dContext->IASetVertexBuffers(
0,
1,
m_vertexBuffer.GetAddressOf(),
&stride,
&offset
);
// The index buffer only has 4 vertices
m_d3dContext->IASetIndexBuffer(
m_indexBuffer.Get(),
DXGI_FORMAT_R16_UINT,
0
);
m_d3dContext->IASetPrimitiveTopology(D3D11_PRIMITIVE_TOPOLOGY_TRIANGLELIST);
m_d3dContext->IASetInputLayout(m_inputLayout.Get());
FLOAT blendFactors[4] = { 0, };
m_d3dContext->OMSetBlendState(m_blendState.Get(), blendFactors, 0xffffffff);
m_d3dContext->VSSetShader(
m_vertexShader.Get(),
nullptr,
0
);
m_d3dContext->PSSetShader(
m_pixelShader.Get(),
nullptr,
0
);
m_d3dContext->PSSetSamplers(
0, // starting at the first sampler slot
1, // set one sampler binding
m_sampler.GetAddressOf()
);
// number of rectangles is in the 100-200 range
for (int i = 0; i < m_rectangles.size(); i++)
{
// start rendering from the farthest rectangle
int j = (i + m_farthestRectangle) % m_rectangles.size();
m_vsConstantBufferData.model = m_rectangles[j].transform;
m_vsConstantBufferData.opacity = m_rectangles[j].Opacity;
m_vsConstantBufferData.highlight = m_rectangles[j].Highlight;
m_vsConstantBufferData.shadow = m_rectangles[j].Shadow;
m_vsConstantBufferData.textureTransitionAmount = m_rectangles[j].textureTransitionAmount;
m_d3dContext->UpdateSubresource(
m_vsConstantBuffer.Get(),
0,
NULL,
&m_vsConstantBufferData,
0,
0
);
m_d3dContext->VSSetConstantBuffers(
0,
1,
m_vsConstantBuffer.GetAddressOf()
);
m_d3dContext->PSSetConstantBuffers(
0,
1,
m_vsConstantBuffer.GetAddressOf()
);
auto a = m_rectangles[j].textureId;
auto b = m_rectangles[j].targetTextureId;
auto srv1 = m_textures[m_rectangles[j].textureId].textureSRV.GetAddressOf();
auto srv2 = m_textures[m_rectangles[j].targetTextureId].textureSRV.GetAddressOf();
ID3D11ShaderResourceView* srvs[2];
srvs[0] = *srv1;
srvs[1] = *srv2;
m_d3dContext->PSSetShaderResources(
0, // starting at the first shader resource slot
2, // set one shader resource binding
srvs
);
m_d3dContext->DrawIndexed(
m_indexCount,
0,
0
);
}
}
Pixel Shader
cbuffer ModelViewProjectionConstantBuffer : register(b0)
{
matrix model;
matrix view;
matrix projection;
float opacity;
float3 highlight;
float3 shadow;
float textureTransitionAmount;
};
Texture2D baseTexture : register(t0);
Texture2D targetTexture : register(t1);
SamplerState simpleSampler : register(s0);
struct PixelShaderInput
{
float4 pos : SV_POSITION;
float3 norm : NORMAL;
float2 tex : TEXCOORD0;
};
float4 main(PixelShaderInput input) : SV_TARGET
{
float3 lightDirection = normalize(float3(0, 0, -1));
float4 baseTexelColor = baseTexture.Sample(simpleSampler, input.tex);
float4 targetTexelColor = targetTexture.Sample(simpleSampler, input.tex);
float4 texelColor = lerp(baseTexelColor, targetTexelColor, textureTransitionAmount);
float4 shadedColor;
shadedColor.rgb = lerp(shadow.rgb, highlight.rgb, texelColor.r);
shadedColor.a = texelColor.a * opacity;
return shadedColor;
}
As Jeremiah has suggested, you are not probably moving texture from CPU to GPU for each frame as you would have to create new texture for each frame or using "UpdateSubresource" or "Map/UnMap" methods.
I don't think that instancing is going to help for this specific case, as the number of polygons is extremely low (I would start to worry with several millions of polygons). It is more likely that your application is going to be bandwidth/fillrate limited, as your are performing lots of texture sampling/blending (It depends on tecture fillrate, pixel fillrate and the nunber of ROP on your GPU).
In order to achieve better performance, It is highly recommended to:
Make sure that all your textures have all mipmaps generated. If they
don't have any mipmaps, It will hurt a lot the cache of the GPU. (I also assume that you are using texture.Sample method in HLSL, and not texture.SampleLevel or variants)
Use Direct3D11 Block Compressed texture on the GPU, by using a tool
like texconv.exe or preferably the sample from "Windows DirectX 11
Texture Converter".
On a side note, you will probably get more attention for this kind of question on https://gamedev.stackexchange.com/.
I don't think you are doing any copying back and forth from GPU to system memory. You usually have to explicitly do that a call to Map(...), or by blitting to a texture you created in system memory.
One issue, is you are making a DrawIndexed(...) call for each texture. GPUs work most efficiently if you send it a bunch of work to do by batching. One way to accomplish this is to set n-amount of textures to PSSetShaderResources(i, ...), and do a DrawIndexedInstanced(...). Your shader code would then read each of the shader resources and draw them. I do this in my C++ DirectCanvas code here (SpriteInstanced.cpp). This can make for a lot of code, but the result is very efficient (I even do the matrix ops in the shader for more speed).
One other, maybe a lot easier way, is to give the DirectXTK spritebatch a shot.
I used it here in this project...only for a simple blit but it may be a good start to see the minor amount of setup needed to use the spritebatch.
Also, if possible, try to "atlas" your texture. For instance, try to fit as many "images" in a texture as possible and blit from them vs having a single texture for each.