My component is given a source ID3D11Texture2D buffer that, most of the time, I copy into a staging ID3D11Texture2D buffer using ID3D11DeviceContext::CopyResource. These two buffers are the same size.
Sometimes, I want to copy only a rectangular sub-region of the source buffer into the staging buffer and stretch it so the staging buffer is still the same size. The rectangular sub-region has the same aspect ratio as the source.
The staging buffer is never used for rendering. It's eventually saved to a media file.
Related
I need to grab the screen pixels into a texture to perform post processing.
Previously, i have been using BlitCommandEncoder to copy from texture to texture. Source texture being the MTLDrawable texture, onto my destination texture. They both have the same MTLPixelFormatBGRA8Unorm so everything works just fine.
However, now i need to use a frame buffer color attachment texture of MTLPixelFormatRGBA16Float for HDR rendering. So, when i am grabbing the screen pixels, i am actually grabbing from this color attachment texture instead of the Drawable texture. And i am getting this error:
[MTLDebugBlitCommandEncoder internalValidateCopyFromTexture:sourceSlice:sourceLevel:sourceOrigin:sourceSize:toTexture:destinationSlice:destinationLevel:destinationOrigin:options:]:447: failed assertion [sourceTexture pixelFormat](MTLPixelFormatRGBA16Float) must equal [destinationTexture pixelFormat](MTLPixelFormatBGRA8Unorm)
I don't think i need to change my destination texture to RGBA16Float format? Because that will take up double the memory. One full screen texture (color attachment) with that format should be enough for HDR to work right?
Is there other method to successfully perform this kind of copy?
On openGL there is no error when copying with glCopyTexImage2D
Metal automatically converts from source to destination format during rendering. So you could just do a no-op rendering pass to perform the conversion.
Alternatively, if you want to avoid boilerplate no-op rendering code, you can use the MPSImageConversion performance shader that's basically doing the same.
If I wanted to copy a low resolution texture onto a higher resolution texture and I used a Blit Command Encoder would it preform linear sampling on the texture while it stretched it?
Blit command encoders can't do that at all, let alone perform interpolation while doing so. All of the copy methods take only a single size parameter, which is the size in both the source and, implicitly, the destination of the region to copy. It can't resize.
To do what you want, you need to use a render command encoder and draw a quad which samples from the source texture and uses the destination texture as the render target (color attachment). At that point, you have control over sampling/interpolation via the fragment shader and the sampler object you use to sample from the source when determining the color of a fragment.
ARKit runs at 60 frames/sec, which equates to 16.6ms per frame.
My current code to convert the CVPixelBufferRef (kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format) to a cv::Mat (YCrCb) runs in 30ms, which causes ARKit to stall and everything to lag.
Does anyone have any ideas on how to to a quicker conversion or do I need to drop the frame rate?
There is a suggestion by Apple to use Metal, but I'm not sure how to do that.
Also I could just take the grayscale plane, which is the first channel, which runs in <1ms, but ideally I need the colour information as well.
In order to process an image in a pixel buffer using Metal, you need to do following.
Call CVMetalTextureCacheCreateTextureFromImage to create CVMetalTexture object on top of the pixel buffer.
Call CVMetalTextureGetTexture to create a MTLTexture object, which Metal code (GPU) can read and write.
Write some Metal code to convert the color format.
I have an open source project (https://github.com/snakajima/vs-metal), which processes pixel buffers (from camera, not ARKit) using Metal. Feel free to copy any code from this project.
I tried to convert Ycbcr to RGB, do image processing in RGB mat and convert it back to Ycbcr, it worked very slowly. I suggest only do that with a static image. For realtime processing, we should process directly in cv::Mat. ARFrame.capturedImage is Ycbcr buffer. So, the solution is
Sperate the buffer to 2 cv::Mat (yPlane and cbcrPlane). Keep in mind, we do not clone memory, we create 2 cv::Mat with base addresses is yPlane address and cbcrPlane address.
Do image process on yPlane and cbcrPlane, size(cbcrPlane) = size(yPlane) / 2.
You can check out my code here: https://gist.github.com/ttruongatl/bb6c69659c48bac67826be7368560216
I am trying to play a video using OpenGL ES 2.0 in iOS. I am not able to get a sample code or starting point of how to achieve this. Can anybody help me with this?
What you are looking for is getting a raw buffer for the video in real time. I believe you need to look into AVFoundation and somehow extract the CVPixelBufferRef. If I remember correctly you have a few ways; one is on demand at specific time, another for processing where you will get a fast iteration of the frames in a block, and the one you probably need is to receive the frames in real time. So with this you can extract a raw RGB buffer which needs to be pushed to the texture and then drawn to the render buffer.
I suggest you create a texture once (per video) and try making it as small as possible but ensure that the video frame will fit. You might need the POT (power of two) textures so to get the texture dimension from video width you need something like:
GLInt textureWidth = 1.0f;
while(textureWidth<videoWidth) textureWidth <<= 1; // Multiplies by 2
So the texture size is expected to be larger then the video. To push the data to the texture you then need to use texture subimage glTexSubImage2D. Which expects a pointer to your raw data and rectangle parameters where to save the data which are then (0, 0, sampleWidth, sampleHeight). Also then the texture coordinates must computed so they are not in range [0, 1] but rather for x: [0, sampleWidth/textureWidth].
So then you just need to put it all together:
Have a system to keep generating the video raw sample buffers
Generate a texture to fit video size
On new sample update the texture using glTexSubImage2D (watch out for threads)
After the data is loaded into the texture draw the texture as full screen rectangle (watch out for threads)
You might need to watch out for video orientation, transformation. So if possible do test your system with a few videos that have been recorded on the device in different orientations. I think there is now a support to receive the buffers already correctly oriented. But by default the sample at least used to be "wrong"; the portrait recorded video still had the samples in landscape but a transformation matrix or orientation was given with the asset.
I'm working on a volume rendering program using DirectX 11.
I render both to a window ( HWND ) and to a texture ( ID3D11Texture2D ).
While the rendering for the HWND always looks correct, my ID3D11Texture2D looks corrupt for render sizes smaller than 64x64:
I wonder whether there is a minimum size limit for textures in DirectX 11.
Unfortunately, I was only able to find information about the maximum texture size limit.
There is no minimum texture size; 1x1x1 is valid.
It looks to me like you've mapped the 3D texture and are extracting the data while ignoring the "RowPitch" returned. On textures that are sufficiently small (or of unusual dimensions) the address at which the next row of texels begins need not necessarily be contiguous after the previous row, but will instead begin "RowPitch" bytes after the last.
See D3D11_MAPPED_SUBRESOURCE