Doing readback from Direct3D textures and surfaces - directx

I need to figure out how to get the data from D3D textures and surfaces back to system memory. What's the fastest way to do such things and how?
Also if I only need one subrect, how can one read back only that portion without having to read back the entire thing to system memory?
In short I'm looking for concise descriptions of how to copy the following to system memory:
a texture
a subset of a texture
a surface
a subset of a surface
a D3DUSAGE_RENDERTARGET texture
a subset of a D3DUSAGE_RENDERTARGET texture
This is Direct3D 9, but answers about newer versions of D3D would be appreciated too.

The most involved part is reading from some surface that is in video memory ("default pool"). This is most often render targets.
Let's get the easy parts first:
reading from a texture is the same as reading from 0-level surface of that texture. See below.
the same for subset of a texture.
reading from a surface that is in non-default memory pool ("system" or "managed") is just locking it and reading bytes.
the same for subset of surface. Just lock relevant portion and read it.
So now we have left surfaces that are in video memory ("default pool"). This would be any surface/texture marked as render target, or any regular surface/texture that you have created in default pool, or the backbuffer itself. The complex part here is that you can't lock it.
Short answer is: GetRenderTargetData method on D3D device.
Longer answer (a rough outline of the code that will be below):
rt = get render target surface (this can be surface of the texture, or backbuffer, etc.)
if rt is multisampled (GetDesc, check D3DSURFACE_DESC.MultiSampleType), then: a) create another render target surface of same size, same format but without multisampling; b) StretchRect from rt into this new surface; c) rt = this new surface (i.e. proceed on this new surface).
off = create offscreen plain surface (CreateOffscreenPlainSurface, D3DPOOL_SYSTEMMEM pool)
device->GetRenderTargetData( rt, off )
now off contains render target data. LockRect(), read data, UnlockRect() on it.
cleanup
Even longer answer (paste from the codebase I'm working on) follows. This will not compile out of the box, because it uses some classes, functions, macros and utilities from the rest of codebase; but it should get you started. I also ommitted most of error checking (e.g. whether given width/height is out of bounds). I also omitted the part that reads actual pixels and possibly converts them into suitable destination format (that is quite easy, but can get long, depending on number of format conversions you want to support).
bool GfxDeviceD3D9::ReadbackImage( /* params */ )
{
HRESULT hr;
IDirect3DDevice9* dev = GetD3DDevice();
SurfacePointer renderTarget;
hr = dev->GetRenderTarget( 0, &renderTarget );
if( !renderTarget || FAILED(hr) )
return false;
D3DSURFACE_DESC rtDesc;
renderTarget->GetDesc( &rtDesc );
SurfacePointer resolvedSurface;
if( rtDesc.MultiSampleType != D3DMULTISAMPLE_NONE )
{
hr = dev->CreateRenderTarget( rtDesc.Width, rtDesc.Height, rtDesc.Format, D3DMULTISAMPLE_NONE, 0, FALSE, &resolvedSurface, NULL );
if( FAILED(hr) )
return false;
hr = dev->StretchRect( renderTarget, NULL, resolvedSurface, NULL, D3DTEXF_NONE );
if( FAILED(hr) )
return false;
renderTarget = resolvedSurface;
}
SurfacePointer offscreenSurface;
hr = dev->CreateOffscreenPlainSurface( rtDesc.Width, rtDesc.Height, rtDesc.Format, D3DPOOL_SYSTEMMEM, &offscreenSurface, NULL );
if( FAILED(hr) )
return false;
hr = dev->GetRenderTargetData( renderTarget, offscreenSurface );
bool ok = SUCCEEDED(hr);
if( ok )
{
// Here we have data in offscreenSurface.
D3DLOCKED_RECT lr;
RECT rect;
rect.left = 0;
rect.right = rtDesc.Width;
rect.top = 0;
rect.bottom = rtDesc.Height;
// Lock the surface to read pixels
hr = offscreenSurface->LockRect( &lr, &rect, D3DLOCK_READONLY );
if( SUCCEEDED(hr) )
{
// Pointer to data is lt.pBits, each row is
// lr.Pitch bytes apart (often it is the same as width*bpp, but
// can be larger if driver uses padding)
// Read the data here!
offscreenSurface->UnlockRect();
}
else
{
ok = false;
}
}
return ok;
}
SurfacePointer in the code above is a smart pointer to a COM object (it releases object on assignment or destructor). Simplifies error handling a lot. This is very similar to _comptr_t things in Visual C++.
The code above reads back whole surface. If you want to read just a part of it efficiently, then I believe fastest way is roughly:
create a default pool surface that is of the needed size.
StretchRect from part of original surface to that smaller one.
proceed as normal with the smaller one.
In fact this is quite similar to what code above does to handle multi-sampled surfaces. If you want to get just a part of a multi-sampled surface, you can do a multisample resolve and get part of it in one StretchRect, I think.
Edit: removed piece of code that does actual read of pixels and format conversions. Was not directly related to question, and the code was long.
Edit: updated to match edited question.

Related

What are drawBuffers?

The opengl docs as well as OpenGL ES docs state regarding clearBuffer[fiuv]:
If buffer is GL_COLOR, a particular draw buffer GL_DRAW_BUFFERi is specified by passing i as drawbuffer
Indeed, the WebGL spec defines values for DRAW_BUFFER0 .. DRAW_BUFFER15
But what are these things? I've heard of framebuffers, and renderbuffers... but what are drawbuffers? how do I create one? What's the relationship to framebuffers? How do I know which one I'm supposed to clear if I have some deferred rendering pipeline that uses multiple framebuffers?
note: this may seem like a duplicate of Regarding drawBuffer in glClearBufferiv but that question is old, tagged under OpenGL, and doesn't answer the same question exactly
Framebuffers have attachments. Those attachments are also referred to as "drawbuffers" when a framebuffer is the currently bound DRAW_FRAMEBUFFER as they are the buffers that will be drawn to.
There is a function gl.drawBuffers that let you choose which attachments will actually be written to
gl.drawBuffers([
gl.COLOR_ATTACHMENT0, // draw to the first attachment
gl.NONE, // don't draw to the 2nd attachment,
gl.COLOR_ATTACHMENT2, // draw to the 3rd attachment
]);
Note that these settings are part of the currently bound framebuffer's state. They are not global settings. (and there is a set of this state for the canvas itself when you call the function above with no framebuffer bound)
Similarly you can call gl.clearBufferXXX to clear a specific attachment of the currently bound framebuffer. For example clearing the 3rd attachment as in
const drawbuffer = 2
gl.clearBufferfv(gl.COLOR, drawBuffer, [1, 0, 0, 1]);
Then the 3rd color attachment on the currently bound DRAW_FRAMEBUFFER will be cleared. (0 being the first, 1 being the 2nd, 2 being the 3rd attachment)
The only use of the constants DRAW_BUFFER0 ... DRAW_BUFFER15 is to query the values set by gl.drawBuffers. Eg. gl.getParameter(gl.DRAW_BUFFER2) gets the drawbuffer setting for the currently bound framebuffer.

Direct2D: How to save content of ID2D1RenderTarget to an image file?

The question is very similar to this, but that one didn't get answered yet. My question is, I have a D2D DXGI RenderTarget from d2dfactory->CreateDxgiSurfaceRenderTarget(), and I want to save its content to an image file using WIC. I was just reading this and this, so it looks to me that I can not just create a ID2D1Bitmap on a WIC render target and use ID2D1Bitmap::CopyFromRenderTarget() to copy from the input render target I want to save, because they are using different resources. So here is what I came up with using ID2D1RenderTarget::CreateSharedBitmap():
HRESULT SaveRenderTargetToFile(
ID2D1RenderTarget* pRTSrc,
LPCWSTR uri
)
{
HRESULT hr = S_OK;
ComPtr<IWICBitmap> spWICBitmap;
ComPtr<ID2D1RenderTarget> spRT;
ComPtr<IWICBitmapEncoder> spEncoder;
ComPtr<IWICBitmapFrameEncode> spFrameEncode;
ComPtr<IWICStream> spStream;
//
// Create WIC bitmap to save and associated render target
//
UINT bitmapWidth = static_cast<UINT>(pRTSrc->GetSize().width + .5f);
UINT bitmapHeight = static_cast<UINT>(pRTSrc->GetSize().height + .5f);
HR(m_spWICFactory->CreateBitmap(
bitmapWidth,
bitmapHeight,
GUID_WICPixelFormat32bppPBGRA,
WICBitmapCacheOnLoad,
&spWICBitmap
));
D2D1_RENDER_TARGET_PROPERTIES prop = D2D1::RenderTargetProperties();
prop.pixelFormat = D2D1::PixelFormat(
DXGI_FORMAT_B8G8R8A8_UNORM,
D2D1_ALPHA_MODE_PREMULTIPLIED
);
prop.type = D2D1_RENDER_TARGET_TYPE_DEFAULT;
prop.usage = D2D1_RENDER_TARGET_USAGE_NONE;
HR(m_spD2D1Factory->CreateWicBitmapRenderTarget(
spWICBitmap,
prop,
&spRT
));
//
// Create a shared bitmap from this RenderTarget
//
ComPtr<ID2D1Bitmap> spBitmap;
D2D1_BITMAP_PROPERTIES bp = D2D1::BitmapProperties();
bp.pixelFormat = prop.pixelFormat;
HR(spRT->CreateSharedBitmap(
__uuidof(IWICBitmap),
static_cast<void*>(spWICBitmap.GetRawPointer()),
&bp,
&spBitmap
)); // <------------------------- This fails with E_INVALIDARG
//
// Copy the source RenderTarget to this bitmap
//
HR(spBitmap->CopyFromRenderTarget(nullptr, pRTSrc, nullptr));
//
// Draw this bitmap to the output render target
//
spRT->BeginDraw();
spRT->Clear(D2D1::ColorF(D2D1::ColorF::GreenYellow));
spRT->DrawBitmap(spBitmap);
HR(spRT->EndDraw());
//
// Save image to file
//
HR(m_spWICFactory->CreateStream(&spStream));
WICPixelFormatGUID format = GUID_WICPixelFormat32bppPBGRA;
HR(spStream->InitializeFromFilename(uri, GENERIC_WRITE));
HR(m_spWICFactory->CreateEncoder(GUID_ContainerFormatPng, nullptr, &spEncoder));
HR(spEncoder->Initialize(spStream, WICBitmapEncoderNoCache));
HR(spEncoder->CreateNewFrame(&spFrameEncode, nullptr));
HR(spFrameEncode->Initialize(nullptr));
HR(spFrameEncode->SetSize(bitmapWidth, bitmapHeight));
HR(spFrameEncode->SetPixelFormat(&format));
HR(spFrameEncode->WriteSource(spWICBitmap, nullptr));
HR(spFrameEncode->Commit());
HR(spEncoder->Commit());
HR(spStream->Commit(STGC_DEFAULT));
done:
return hr;
}
Anything wrong with this code? (I'm sure there's a lot :)) Somewhere on MSDN it says that WIC render target only supports software mode, while DXGI render target only supports hardware mode. Is this the reason why the above call to CreateSharedBitmap() fails? How should I save a DXGI surface content to an image file with D2D then?
With some limitations, you can use D3DX11SaveTextureToFile. Use QI on your surface to get the ID3D11Resource.
On the same page they are recommending DirectXTex library as a replacement, CaptureTexture then SaveToXXXFile (where XXX is WIC, DDS, or TGA). So that's another option.
Also, if your surface has been created as GDI compatible, you can use IDXGISurface1::GetDC. (Use QI on your IDXGISurface to get the IDXGISurface1). Saving DC to a file is left as an exercise to the reader.
Remember to use the Debug Layer for help with cryptic return codes like E_INVALIDARG.
You could try this (I haven't):
Make your old DXGISurface.
Make an auxiliary ID2D1DeviceContext render target.
Use ID2D1DeviceContext::CreateBitmapFromDxgiSurface to create an ID2D1Bitmap1 associated to the DXGI surface.
Draw on your DXGISurface. You should get the same on the ID2D1Bitmap1.
Use ID2D1Bitmap1::Map to get a memory pointer to the pixeldata.
Copy the pixeldata to file, or to a wicbitmap for encoding (jpeg, tiff, etc.)
Perhaps this:(succeed running)
D2DFactory->CreateHwndRenderTarget(D2D1::RenderTargetProperies(D2D1_RENDER_TARGET_TYPE_SOFTWARE,D2D1::Pixel Format(DXGI_FORMAT_B8G8R8A8_UNORM,D2D1_ALPHA_MODE_PREMULTIPLIED)), ……)
your RenderTarget should be set the static of SOFTWARE the same as WICRenderTarget.

Can I get a calibrated image and skeleton from the Official Kinect SDK at the same time?

Basically, I just want those three things.
Color, Depth, and Skeleton. But I also want the depth and color lined up.
I know that in the official examples, there's GreenScreen.cpp, which is an example of depth/color cut-outs, but no skeleton. There's also SkeletonBasics which has a skeleton, but no image.
I've tried learning from both of them and implementing them together, but I never get a callback for when the skeleton is ready (this: m_hNextSkeletonEvent).
SkeletonBasics.cpp uses this:
hEvents[0] = m_hNextSkeletonEvent;
// Check to see if we have either a message (by passing in QS_ALLEVENTS)
// Or a Kinect event (hEvents)
// Update() will check for Kinect events individually, in case more than one are signalled
DWORD dwEvent = MsgWaitForMultipleObjects(eventCount, hEvents, FALSE, INFINITE, QS_ALLINPUT);
// Check if this is an event we're waiting on and not a timeout or message
if (WAIT_OBJECT_0 == dwEvent)
{
Update();
}
And GreenScreen.cpp uses this:
hEvents[0] = m_hNextDepthFrameEvent;
hEvents[1] = m_hNextColorFrameEvent;
// Check to see if we have either a message (by passing in QS_ALLINPUT)
// Or a Kinect event (hEvents)
// Update() will check for Kinect events individually, in case more than one are signalled
DWORD dwEvent = MsgWaitForMultipleObjects(eventCount, hEvents, FALSE, INFINITE, QS_ALLINPUT);
// Check if this is an event we're waiting on and not a timeout or message
if (WAIT_OBJECT_0 == dwEvent || WAIT_OBJECT_0 + 1 == dwEvent)
{
Update();
}
But, like I said, combining them yields no
if ( WAIT_OBJECT_0 == WaitForSingleObject(m_hNextSkeletonEvent, 0) )
happening. (The KinectBridgeWithOpenCVBasics-D2D also does Skeleton, Color, and Depth, but still is unaligned (like this example I found in a search).
Do you know why the event is not firing? Or perhaps do you know of an example somewhere that has all of these abilities? I've searched high and low, and I've tried tons of different things messing with these examples, but I just cant seem to get it.
Thanks for your help.
EDIT:
At the end of KinectBridgeWithOpenCVBasics, I'm left with a m_colorMat and an m_depthMat. Even if at that point I could align the depth to color, I think that would work out just fine.

How do I set up DirectX 9 so that backface culling is off, z-buffering is on, and gouraud shading works, for triangle meshes without normals data?

I've been having difficulty identifying the correct parameters for the PresentParameters and DirectX device, so that there can be both vertex-level gouraud shading and the use of a z buffer. Some triangle meshes work fine, others have background triangles appearing in front of triangles which are closer-to-camera.
An example of this is found here: http://gallery.me.com/robert.perkins/100045/zBufferGone. The input data is a simple list of vertices in facets. The winding order of the vertices in each facet is nondeterministic (comes from various CAD software export functions) and there is no normals data.
The PresentParameters are being set up right now as follows. I realize this is C# instead of C++ but I think it's descriptive enough, and the parameters pass through to C++ code. This produces the image in the picture; the behavior is the same on the Reference device:
pParams = new PresentParameters()
{
BackBufferWidth = this.ClientSize.Width,
BackBufferHeight = this.ClientSize.Height,
AutoDepthStencilFormat = Format.D16,
EnableAutoDepthStencil = true,
SwapEffect = SwapEffect.Discard,
Windowed = true
};
_engineDX9 = new EngineDX9(this, SlimDX.Direct3D9.DeviceType.Hardware, SlimDX.Direct3D9.CreateFlags.SoftwareVertexProcessing, pParams);
_engineDX9.DefaultCamera.NearPlane = 0;
_engineDX9.DefaultCamera.FarPlane = 10;
_engineDX9.D3DDevice.SetRenderState(RenderState.Ambient, false);
_engineDX9.D3DDevice.SetRenderState(RenderState.ZEnable, ZBufferType.UseZBuffer);
_engineDX9.D3DDevice.SetRenderState(RenderState.ZWriteEnable, true);
_engineDX9.D3DDevice.SetRenderState(RenderState.ZFunc, Compare.Always);
_engineDX9.BackColor = Color.White;
_engineDX9.FillMode = FillMode.Solid;
_engineDX9.CullMode = Cull.None;
_engineDX9.DefaultCamera.AspectRatio = (float)this.Width / this.Height;
All of my other setup attempts, even on the reference device, return a COM error code ({"D3DERR_INVALIDCALL: Invalid call (-2005530516)"}). What are the correct setup parameters?
EDIT: The C++ class which interfaces with DirectX9 sets defaults like this:
PresentParameters::PresentParameters()
{
BackBufferWidth = 640;
BackBufferHeight = 480;
BackBufferFormat = Format::X8R8G8B8;
BackBufferCount = 1;
Multisample = MultisampleType::None;
MultisampleQuality = 0;
SwapEffect = SlimDX::Direct3D9::SwapEffect::Discard;
DeviceWindowHandle = IntPtr::Zero;
Windowed = true;
EnableAutoDepthStencil = true;
AutoDepthStencilFormat = Format::D24X8;
PresentFlags = SlimDX::Direct3D9::PresentFlags::None;
FullScreenRefreshRateInHertz = 0;
PresentationInterval = PresentInterval::Immediate;
}
Where does it return an invalid call?
Edit: I'm assuming in the new EngineDX9 call? Have you tried setting a device window handle in the present parameters?
Edit 2: Have you turned on the debug spew in the DirectX control panel to see whether it tells you what the error is?
Edit3: You have tried setting backbufferWidth and Height to 0? What is backbuffercount set to? Might also be worth trying "Format.D24S8" on the backbuffer? Its "possible" your graphics card doesn't support 16-bit (unlikely though). Have you checked in the caps that the mode you are trying to create is valid? I asssume, btw, that the CLR language you are using automagically sets the parameters you don't set to 0? I,personally, always prefer to be explicit in such cases ....
PS I'm guessing here because im a native C++ DX9 coder not a CLR SlimDX coder ...
Edit4: I'm sure its the lack of window handle ... I'm probably wrong but thats the only thing i can see REALLY wrong with your setup. A windowed DX9 device requires a window. Btw set width and height to 0 to just use the window you are setting the device too's size ...
Edit 5: I've really been heading down the wrong route here. There is nothing wrong with the creation of the device that produced your "incorrect" device. Do not mess with the present parameters they are fine. The main reason you'll have problems with your Z-Buffering is that you set the compare function to always. This means that, regardless of what the z-buffer contains, pas the pixel and write its z into the z-buffer overwriting whatever is there already. I'd wager therein lies your Z-buffering problem.

How to copy the pixel data from a texture into a bitmap (direct3d)? [duplicate]

I need to figure out how to get the data from D3D textures and surfaces back to system memory. What's the fastest way to do such things and how?
Also if I only need one subrect, how can one read back only that portion without having to read back the entire thing to system memory?
In short I'm looking for concise descriptions of how to copy the following to system memory:
a texture
a subset of a texture
a surface
a subset of a surface
a D3DUSAGE_RENDERTARGET texture
a subset of a D3DUSAGE_RENDERTARGET texture
This is Direct3D 9, but answers about newer versions of D3D would be appreciated too.
The most involved part is reading from some surface that is in video memory ("default pool"). This is most often render targets.
Let's get the easy parts first:
reading from a texture is the same as reading from 0-level surface of that texture. See below.
the same for subset of a texture.
reading from a surface that is in non-default memory pool ("system" or "managed") is just locking it and reading bytes.
the same for subset of surface. Just lock relevant portion and read it.
So now we have left surfaces that are in video memory ("default pool"). This would be any surface/texture marked as render target, or any regular surface/texture that you have created in default pool, or the backbuffer itself. The complex part here is that you can't lock it.
Short answer is: GetRenderTargetData method on D3D device.
Longer answer (a rough outline of the code that will be below):
rt = get render target surface (this can be surface of the texture, or backbuffer, etc.)
if rt is multisampled (GetDesc, check D3DSURFACE_DESC.MultiSampleType), then: a) create another render target surface of same size, same format but without multisampling; b) StretchRect from rt into this new surface; c) rt = this new surface (i.e. proceed on this new surface).
off = create offscreen plain surface (CreateOffscreenPlainSurface, D3DPOOL_SYSTEMMEM pool)
device->GetRenderTargetData( rt, off )
now off contains render target data. LockRect(), read data, UnlockRect() on it.
cleanup
Even longer answer (paste from the codebase I'm working on) follows. This will not compile out of the box, because it uses some classes, functions, macros and utilities from the rest of codebase; but it should get you started. I also ommitted most of error checking (e.g. whether given width/height is out of bounds). I also omitted the part that reads actual pixels and possibly converts them into suitable destination format (that is quite easy, but can get long, depending on number of format conversions you want to support).
bool GfxDeviceD3D9::ReadbackImage( /* params */ )
{
HRESULT hr;
IDirect3DDevice9* dev = GetD3DDevice();
SurfacePointer renderTarget;
hr = dev->GetRenderTarget( 0, &renderTarget );
if( !renderTarget || FAILED(hr) )
return false;
D3DSURFACE_DESC rtDesc;
renderTarget->GetDesc( &rtDesc );
SurfacePointer resolvedSurface;
if( rtDesc.MultiSampleType != D3DMULTISAMPLE_NONE )
{
hr = dev->CreateRenderTarget( rtDesc.Width, rtDesc.Height, rtDesc.Format, D3DMULTISAMPLE_NONE, 0, FALSE, &resolvedSurface, NULL );
if( FAILED(hr) )
return false;
hr = dev->StretchRect( renderTarget, NULL, resolvedSurface, NULL, D3DTEXF_NONE );
if( FAILED(hr) )
return false;
renderTarget = resolvedSurface;
}
SurfacePointer offscreenSurface;
hr = dev->CreateOffscreenPlainSurface( rtDesc.Width, rtDesc.Height, rtDesc.Format, D3DPOOL_SYSTEMMEM, &offscreenSurface, NULL );
if( FAILED(hr) )
return false;
hr = dev->GetRenderTargetData( renderTarget, offscreenSurface );
bool ok = SUCCEEDED(hr);
if( ok )
{
// Here we have data in offscreenSurface.
D3DLOCKED_RECT lr;
RECT rect;
rect.left = 0;
rect.right = rtDesc.Width;
rect.top = 0;
rect.bottom = rtDesc.Height;
// Lock the surface to read pixels
hr = offscreenSurface->LockRect( &lr, &rect, D3DLOCK_READONLY );
if( SUCCEEDED(hr) )
{
// Pointer to data is lt.pBits, each row is
// lr.Pitch bytes apart (often it is the same as width*bpp, but
// can be larger if driver uses padding)
// Read the data here!
offscreenSurface->UnlockRect();
}
else
{
ok = false;
}
}
return ok;
}
SurfacePointer in the code above is a smart pointer to a COM object (it releases object on assignment or destructor). Simplifies error handling a lot. This is very similar to _comptr_t things in Visual C++.
The code above reads back whole surface. If you want to read just a part of it efficiently, then I believe fastest way is roughly:
create a default pool surface that is of the needed size.
StretchRect from part of original surface to that smaller one.
proceed as normal with the smaller one.
In fact this is quite similar to what code above does to handle multi-sampled surfaces. If you want to get just a part of a multi-sampled surface, you can do a multisample resolve and get part of it in one StretchRect, I think.
Edit: removed piece of code that does actual read of pixels and format conversions. Was not directly related to question, and the code was long.
Edit: updated to match edited question.

Resources