Basically, I just want those three things.
Color, Depth, and Skeleton. But I also want the depth and color lined up.
I know that in the official examples, there's GreenScreen.cpp, which is an example of depth/color cut-outs, but no skeleton. There's also SkeletonBasics which has a skeleton, but no image.
I've tried learning from both of them and implementing them together, but I never get a callback for when the skeleton is ready (this: m_hNextSkeletonEvent).
SkeletonBasics.cpp uses this:
hEvents[0] = m_hNextSkeletonEvent;
// Check to see if we have either a message (by passing in QS_ALLEVENTS)
// Or a Kinect event (hEvents)
// Update() will check for Kinect events individually, in case more than one are signalled
DWORD dwEvent = MsgWaitForMultipleObjects(eventCount, hEvents, FALSE, INFINITE, QS_ALLINPUT);
// Check if this is an event we're waiting on and not a timeout or message
if (WAIT_OBJECT_0 == dwEvent)
{
Update();
}
And GreenScreen.cpp uses this:
hEvents[0] = m_hNextDepthFrameEvent;
hEvents[1] = m_hNextColorFrameEvent;
// Check to see if we have either a message (by passing in QS_ALLINPUT)
// Or a Kinect event (hEvents)
// Update() will check for Kinect events individually, in case more than one are signalled
DWORD dwEvent = MsgWaitForMultipleObjects(eventCount, hEvents, FALSE, INFINITE, QS_ALLINPUT);
// Check if this is an event we're waiting on and not a timeout or message
if (WAIT_OBJECT_0 == dwEvent || WAIT_OBJECT_0 + 1 == dwEvent)
{
Update();
}
But, like I said, combining them yields no
if ( WAIT_OBJECT_0 == WaitForSingleObject(m_hNextSkeletonEvent, 0) )
happening. (The KinectBridgeWithOpenCVBasics-D2D also does Skeleton, Color, and Depth, but still is unaligned (like this example I found in a search).
Do you know why the event is not firing? Or perhaps do you know of an example somewhere that has all of these abilities? I've searched high and low, and I've tried tons of different things messing with these examples, but I just cant seem to get it.
Thanks for your help.
EDIT:
At the end of KinectBridgeWithOpenCVBasics, I'm left with a m_colorMat and an m_depthMat. Even if at that point I could align the depth to color, I think that would work out just fine.
Related
I am learning about fluid dynamics (and Haxe) and have come across this awesome project and thought I would try to extend to it to help me learn. A demo of the original project in action can be seen here.
So far, I have created a side menu of items containing different shapes. When the user clicks on one of the shapes, then, clicks onto the canvas, the image selected should be imprinted onto the dye. The user will then move the mouse and explore the art etc.
To try and achieve this I did the following:
import js.html.webgl.RenderingContext;
function imageSelection(): Void{
document.querySelector('.myscrollbar1').addEventListener('click', function() {
// twilight image clicked
closeNav();
reset();
var image:js.html.ImageElement = cast document.querySelector('img[src="images/twilight.jpg"]');
gl.current_context.texSubImage2D(cast fluid.dyeRenderTarget.writeToTexture, 0, Math.round(mouse.x), Math.round(mouse.y), RenderingContext.RGB, RenderingContext.UNSIGNED_BYTE, image);
TWILIGHT = true;
});
After this call, inside the update function, I have the following:
override function update( dt:Float ){
time = haxe.Timer.stamp() - initTime;
performanceMonitor.recordFrameTime(dt);
//Smaller number creates a bigger ripple, was 0.016
dt = 0.090;//#!
//Physics
//interaction
updateDyeShader.isMouseDown.set(isMouseDown && lastMousePointKnown);
mouseForceShader.isMouseDown.set(isMouseDown && lastMousePointKnown);
//step physics
fluid.step(dt);
particles.flowVelocityField = fluid.velocityRenderTarget.readFromTexture;
if(renderParticlesEnabled){
particles.step(dt);
}
//Below handles the cycling of colours once the mouse is moved and then the image should be disrupted into the set dye colours.
}
However, although the project builds, I can't seem to get the image imprinted onto the canvas. I have checked the console log and I can see the following error:
WebGL: INVALID_ENUM: texSubImage2D: invalid texture target
Is it safe to assume that my cast for the first param is not allowed?
I have read that the texture target is the first parameter and INVALID_ENUM in particular means that one of the gl.XXX parameters are just flat out wrong for that particular function.
Looking through to the file writeToTexture is declared as so: public var writeToTexture (default, null):GLTexture;. WriteToTexture is a wrapper around a regular webgl handle.
I am using Haxe version 3.2.1 and using Snow to build the project. WriteToTexture is defined inside HaxeToolkit\haxe\lib\gltoolbox\git\gltoolbox\render
writeToTexture in gltoolbox is a GLTexture. With snow and snow_web, this is defined in snow.modules.opengl.GL as:
typedef GLTexture = js.html.webgl.Texture;
So we're simply dealing with a js.html.webgl.Texture here, or WebGLTexture in native JS.
Which means that yes, this is definitely not a valid value for texSubImage2D()'s target, which is specified to take one of the gl.TEXTURE_* constants.
A GLenum specifying the binding point (target) of the active texture.
From this description it's obvious that the parameter isn't actually for the texture itself - it merely gives some info on how the active texture should be used.
The question then becomes how the "active" texture can be set. bindTexture() can be used for this.
Given stage coordinates (x,y), I want to make my flash app behave just as if the user clicked at position (x,y). That is, something like
function simulateClick(x:Number, y:Number):void{
var e:MouseEvent = new MouseEvent(MouseEvent.CLICK, true, false, x, y)
stage.dispatchEvent(e);
}
I've found a bunch of pages talking about this kind of thing, and they all give solutions similar to the above. However, this isn't equivalent to the user clicking at (x,y). There are two problems:
The first is that e.stageX and e.stageY are both 0. I can't set them directly. The documentation says they are calculated when e.localX and e.localY are set, but this isn't happening when I set e.localX before dispatchEvent, nor in the event listener.
I could rewrite all of my event listeners with something like this:
var p:Point = e.target.localToGlobal(new Point(e.localX, e.localY));
Is this the only option?
The second problem is that my event listeners are registered with children of stage, not stage itself. So I need to find out what target to call dispatchEvent on. Clearly Flash is capable of determining what the target should be, ie which object owns the topmost pixel visible at position (x,y), because it does so when the user actually clicks. Is there an easy way to get at this information, or should I just write my own recursive function to do the same thing? I'm using DisplayObjectContainer.getObjectsUnderPoint at the moment, but it's not quite right.
I'm writing in FlashDevelop, if that makes any difference.
e.stageX/Y is populated correctly for me... also getObjectsUnderPoint() seems to work fine. I'm assuming that the x/y values passed to simulateClick are global coordinates?
edit: as pointed out in the comments, the mouse event must be dispatched on InteractiveObject instances... modified the code accordingly.
import flash.display.Bitmap;
import flash.display.BitmapData;
import flash.display.DisplayObject;
import flash.display.InteractiveObject;
import flash.display.Sprite;
import flash.events.MouseEvent;
import flash.geom.Point;
public function simulateClick(x:Number, y:Number):void
{
var objects:Array = stage.getObjectsUnderPoint(new Point(x, y));
var target:DisplayObject;
while(target = objects.pop())
{
if(target is InteractiveObject)
{
break;
}
}
if(target !== null)
{
var local:Point = target.globalToLocal(new Point(x, y));
var e:MouseEvent = new MouseEvent(MouseEvent.CLICK, true, false, local.x, local.y);
target.dispatchEvent(e);
}
}
public function addedToStage():void
{
var parent:Sprite = new Sprite();
stage.addChild(parent);
var child:Sprite = new Sprite();
child.name = 'child 1';
child.graphics.beginFill(0xff0000, 1);
child.graphics.drawRect(0, 0, 200, 200);
child.graphics.endFill();
var child2:Sprite = new Sprite();
child2.name = 'child 2';
child2.graphics.beginFill(0xff00ff, 1);
child2.graphics.drawRect(0, 0, 100, 100);
child2.graphics.endFill();
child2.x = 150;
child2.y = 150;
var bmpData:BitmapData = new BitmapData(80, 80, false, 0x00ff00);
var bmp:Bitmap = new Bitmap(bmpData);
bmp.name = 'bitmap';
child2.addChild(bmp);
parent.addChild(child);
parent.addChild(child2);
child2.addEventListener(MouseEvent.CLICK, function(e:MouseEvent):void
{
trace('target: ' + e.target.name);
trace('localX: ' + e.localX);
trace('localY: ' + e.localY);
trace('stageX: ' + e.stageX);
trace('stageY: ' + e.stageY);
});
simulateClick(190, 190);
}
Output:
target: child 2
localX: 40
localY: 40
stageX: 190
stageY: 190
For question 1: After you create the MouseEvent (assigning it a local x,y) you should be able to directly reference e.stageX and set it to what you want prior to dispatching the event. It's just a property of the MouseEvent instance.
For #2, currentTarget is always the thing that is topmost under the mouse, while target is the thing that is dispatching the event -- assuming the event is genuinely being dispatched by mouse interaction. In your case, you can set the target to be whatever object you have dispatching the event, and set the currentTarget arbitrarily. The question really is whether this is the most efficient way to deal with what's under the mouse right now; and the answer is, probably not. You'd be a lot better off using a MOUSE_OVER event to keep tabs on what the mouse is over right now, store that as a variable you can use when you want to call this, and don't try to iterate the whole display chain all the time (because Flash natively does that much faster than you can do it in a loop). If you put a mouseOver on the stage, and just check the currentTarget, you'll be getting whatever the topmost item is under the mouse on every frame where it changes.
You should be aware that (to prevent some obvious nasty scripts), certain actions cannot be triggered by mouse events that are generated dynamically by actionscript. These include opening a file reference and going fullscreen.
I have faced this issue too, gave me a bit of a headache.
In my situation I was creating the event, performing a bit of complex computations, but I couldn't retrieve global coordinates even though I had already set local coordinates.
Actually the solution was quite obvious in my case...
Global coordinates are populated only AFTER the event is dispatched, otherwise how can the event know how to translate local to global?
This is another pitfall, on top of not checking for the object used to dispatch event being an InteractiveObject.
I post this because someone else may face this issue due to both pitfalls. A quick answer easy to read.
I'm having a weird problem while developing my DirectShow application. I am using Delphi 6 with the DSPACK DirectShow component library. One of the IBaseFilter instances doesn't seem to recognize a pin that it owns when I try to find the pin in the filter using it's TPinInfo.achName property (_PinInfo). (Note, in this case it is the IBaseFilter created by the TSampleGrabber component that is exhibiting this weird behavior).
The sequence of events, encapsulated in the code sample below is this:
Find the first available input pin in the IBaseFilter instance. In the code below this is the pin passed to testPinInfo().
Execute QueryPinInfo() on the returned pin to get that information. The returned information shows the pin's achName as 'Input'.
Try to find a pin named 'Input' in the very same IBaseFilter instance using IBaseFilter.findPin().
Get NIL back indicating a pin could not be found with that name. This in my opinion is a really strange condition (error).
Does anyone know what kind of conditions could cause this scenario? I don't think it's a memory corruption problem because the data structures involved look fine when I inspect them in the debugger. Is it possible that some IBaseFilter implementations neglect to implement the FindPin() method properly?
Here's the code below:
procedure testPinInfo(intfInputPin: IPin);
var
intfTestPin: IPin;
pinInfo_input: TPinInfo;
begin
intfTestPin := nil;
// Get the pin information.
ZeroMemory(#pinInfo_input, SizeOf(pinInfo_input));
intfInputPin.QueryPinInfo(pinInfo_input);
// Now immediately turn around and try to find the pin in the filter that
// owns it, using the name found in pinInfo_input
pinInfo_input.pFilter.FindPin(pinInfo_input.achName, intfTestPin);
// >>> intfTestPin is NIL (unassigned). This is an error.
end;
Don't use FindPin, you always have better ways to do it. Look for unconnected pin of desired direction with the media type of interest. If you look for preview/capture pins specifically, you always have an option to use IKsPropertySet interface to unambiguously identify the pins you need.
I had a similar issue to this so I made my own version of FindPin :-
HRESULT GraphControl::FindPinByName(IBaseFilter* pFilter,LPCWSTR pName,IPin** ppPin)
{
HRESULT hr = E_FAIL;
IEnumPins* pEnum = NULL;
IPin* pPin = NULL;
DWORD pFetched = 0;
PIN_INFO pinInfo = {0};
// Create a pin enumerator
if(FAILED(pFilter->EnumPins(&pEnum)))
return E_FAIL;
// Get the first instance
hr = pEnum->Next(1,&pPin,&pFetched);
while( hr == S_OK )
{
pPin->QueryPinInfo(&pinInfo);
// Compare the names
if (wcscmp(pName,pinInfo.achName) == 0 )
{
// pin names match so use this one and exit
*ppPin = pPin;
break;
}
SAFE_RELEASE(pinInfo.pFilter);
SAFE_RELEASE(pPin);
hr = pEnum->Next(1,&pPin,&pFetched);
}
SAFE_RELEASE(pinInfo.pFilter);
SAFE_RELEASE(pEnum);
// if the pPin address is null we didnt find a pin with the wanted name
if(&*pPin == NULL)
hr = VFW_E_NOT_FOUND;
return hr;
}
For FindPin you need the corresponding Id, check QueryId(). For Input it's usually "In".
I need to figure out how to get the data from D3D textures and surfaces back to system memory. What's the fastest way to do such things and how?
Also if I only need one subrect, how can one read back only that portion without having to read back the entire thing to system memory?
In short I'm looking for concise descriptions of how to copy the following to system memory:
a texture
a subset of a texture
a surface
a subset of a surface
a D3DUSAGE_RENDERTARGET texture
a subset of a D3DUSAGE_RENDERTARGET texture
This is Direct3D 9, but answers about newer versions of D3D would be appreciated too.
The most involved part is reading from some surface that is in video memory ("default pool"). This is most often render targets.
Let's get the easy parts first:
reading from a texture is the same as reading from 0-level surface of that texture. See below.
the same for subset of a texture.
reading from a surface that is in non-default memory pool ("system" or "managed") is just locking it and reading bytes.
the same for subset of surface. Just lock relevant portion and read it.
So now we have left surfaces that are in video memory ("default pool"). This would be any surface/texture marked as render target, or any regular surface/texture that you have created in default pool, or the backbuffer itself. The complex part here is that you can't lock it.
Short answer is: GetRenderTargetData method on D3D device.
Longer answer (a rough outline of the code that will be below):
rt = get render target surface (this can be surface of the texture, or backbuffer, etc.)
if rt is multisampled (GetDesc, check D3DSURFACE_DESC.MultiSampleType), then: a) create another render target surface of same size, same format but without multisampling; b) StretchRect from rt into this new surface; c) rt = this new surface (i.e. proceed on this new surface).
off = create offscreen plain surface (CreateOffscreenPlainSurface, D3DPOOL_SYSTEMMEM pool)
device->GetRenderTargetData( rt, off )
now off contains render target data. LockRect(), read data, UnlockRect() on it.
cleanup
Even longer answer (paste from the codebase I'm working on) follows. This will not compile out of the box, because it uses some classes, functions, macros and utilities from the rest of codebase; but it should get you started. I also ommitted most of error checking (e.g. whether given width/height is out of bounds). I also omitted the part that reads actual pixels and possibly converts them into suitable destination format (that is quite easy, but can get long, depending on number of format conversions you want to support).
bool GfxDeviceD3D9::ReadbackImage( /* params */ )
{
HRESULT hr;
IDirect3DDevice9* dev = GetD3DDevice();
SurfacePointer renderTarget;
hr = dev->GetRenderTarget( 0, &renderTarget );
if( !renderTarget || FAILED(hr) )
return false;
D3DSURFACE_DESC rtDesc;
renderTarget->GetDesc( &rtDesc );
SurfacePointer resolvedSurface;
if( rtDesc.MultiSampleType != D3DMULTISAMPLE_NONE )
{
hr = dev->CreateRenderTarget( rtDesc.Width, rtDesc.Height, rtDesc.Format, D3DMULTISAMPLE_NONE, 0, FALSE, &resolvedSurface, NULL );
if( FAILED(hr) )
return false;
hr = dev->StretchRect( renderTarget, NULL, resolvedSurface, NULL, D3DTEXF_NONE );
if( FAILED(hr) )
return false;
renderTarget = resolvedSurface;
}
SurfacePointer offscreenSurface;
hr = dev->CreateOffscreenPlainSurface( rtDesc.Width, rtDesc.Height, rtDesc.Format, D3DPOOL_SYSTEMMEM, &offscreenSurface, NULL );
if( FAILED(hr) )
return false;
hr = dev->GetRenderTargetData( renderTarget, offscreenSurface );
bool ok = SUCCEEDED(hr);
if( ok )
{
// Here we have data in offscreenSurface.
D3DLOCKED_RECT lr;
RECT rect;
rect.left = 0;
rect.right = rtDesc.Width;
rect.top = 0;
rect.bottom = rtDesc.Height;
// Lock the surface to read pixels
hr = offscreenSurface->LockRect( &lr, &rect, D3DLOCK_READONLY );
if( SUCCEEDED(hr) )
{
// Pointer to data is lt.pBits, each row is
// lr.Pitch bytes apart (often it is the same as width*bpp, but
// can be larger if driver uses padding)
// Read the data here!
offscreenSurface->UnlockRect();
}
else
{
ok = false;
}
}
return ok;
}
SurfacePointer in the code above is a smart pointer to a COM object (it releases object on assignment or destructor). Simplifies error handling a lot. This is very similar to _comptr_t things in Visual C++.
The code above reads back whole surface. If you want to read just a part of it efficiently, then I believe fastest way is roughly:
create a default pool surface that is of the needed size.
StretchRect from part of original surface to that smaller one.
proceed as normal with the smaller one.
In fact this is quite similar to what code above does to handle multi-sampled surfaces. If you want to get just a part of a multi-sampled surface, you can do a multisample resolve and get part of it in one StretchRect, I think.
Edit: removed piece of code that does actual read of pixels and format conversions. Was not directly related to question, and the code was long.
Edit: updated to match edited question.
I need to figure out how to get the data from D3D textures and surfaces back to system memory. What's the fastest way to do such things and how?
Also if I only need one subrect, how can one read back only that portion without having to read back the entire thing to system memory?
In short I'm looking for concise descriptions of how to copy the following to system memory:
a texture
a subset of a texture
a surface
a subset of a surface
a D3DUSAGE_RENDERTARGET texture
a subset of a D3DUSAGE_RENDERTARGET texture
This is Direct3D 9, but answers about newer versions of D3D would be appreciated too.
The most involved part is reading from some surface that is in video memory ("default pool"). This is most often render targets.
Let's get the easy parts first:
reading from a texture is the same as reading from 0-level surface of that texture. See below.
the same for subset of a texture.
reading from a surface that is in non-default memory pool ("system" or "managed") is just locking it and reading bytes.
the same for subset of surface. Just lock relevant portion and read it.
So now we have left surfaces that are in video memory ("default pool"). This would be any surface/texture marked as render target, or any regular surface/texture that you have created in default pool, or the backbuffer itself. The complex part here is that you can't lock it.
Short answer is: GetRenderTargetData method on D3D device.
Longer answer (a rough outline of the code that will be below):
rt = get render target surface (this can be surface of the texture, or backbuffer, etc.)
if rt is multisampled (GetDesc, check D3DSURFACE_DESC.MultiSampleType), then: a) create another render target surface of same size, same format but without multisampling; b) StretchRect from rt into this new surface; c) rt = this new surface (i.e. proceed on this new surface).
off = create offscreen plain surface (CreateOffscreenPlainSurface, D3DPOOL_SYSTEMMEM pool)
device->GetRenderTargetData( rt, off )
now off contains render target data. LockRect(), read data, UnlockRect() on it.
cleanup
Even longer answer (paste from the codebase I'm working on) follows. This will not compile out of the box, because it uses some classes, functions, macros and utilities from the rest of codebase; but it should get you started. I also ommitted most of error checking (e.g. whether given width/height is out of bounds). I also omitted the part that reads actual pixels and possibly converts them into suitable destination format (that is quite easy, but can get long, depending on number of format conversions you want to support).
bool GfxDeviceD3D9::ReadbackImage( /* params */ )
{
HRESULT hr;
IDirect3DDevice9* dev = GetD3DDevice();
SurfacePointer renderTarget;
hr = dev->GetRenderTarget( 0, &renderTarget );
if( !renderTarget || FAILED(hr) )
return false;
D3DSURFACE_DESC rtDesc;
renderTarget->GetDesc( &rtDesc );
SurfacePointer resolvedSurface;
if( rtDesc.MultiSampleType != D3DMULTISAMPLE_NONE )
{
hr = dev->CreateRenderTarget( rtDesc.Width, rtDesc.Height, rtDesc.Format, D3DMULTISAMPLE_NONE, 0, FALSE, &resolvedSurface, NULL );
if( FAILED(hr) )
return false;
hr = dev->StretchRect( renderTarget, NULL, resolvedSurface, NULL, D3DTEXF_NONE );
if( FAILED(hr) )
return false;
renderTarget = resolvedSurface;
}
SurfacePointer offscreenSurface;
hr = dev->CreateOffscreenPlainSurface( rtDesc.Width, rtDesc.Height, rtDesc.Format, D3DPOOL_SYSTEMMEM, &offscreenSurface, NULL );
if( FAILED(hr) )
return false;
hr = dev->GetRenderTargetData( renderTarget, offscreenSurface );
bool ok = SUCCEEDED(hr);
if( ok )
{
// Here we have data in offscreenSurface.
D3DLOCKED_RECT lr;
RECT rect;
rect.left = 0;
rect.right = rtDesc.Width;
rect.top = 0;
rect.bottom = rtDesc.Height;
// Lock the surface to read pixels
hr = offscreenSurface->LockRect( &lr, &rect, D3DLOCK_READONLY );
if( SUCCEEDED(hr) )
{
// Pointer to data is lt.pBits, each row is
// lr.Pitch bytes apart (often it is the same as width*bpp, but
// can be larger if driver uses padding)
// Read the data here!
offscreenSurface->UnlockRect();
}
else
{
ok = false;
}
}
return ok;
}
SurfacePointer in the code above is a smart pointer to a COM object (it releases object on assignment or destructor). Simplifies error handling a lot. This is very similar to _comptr_t things in Visual C++.
The code above reads back whole surface. If you want to read just a part of it efficiently, then I believe fastest way is roughly:
create a default pool surface that is of the needed size.
StretchRect from part of original surface to that smaller one.
proceed as normal with the smaller one.
In fact this is quite similar to what code above does to handle multi-sampled surfaces. If you want to get just a part of a multi-sampled surface, you can do a multisample resolve and get part of it in one StretchRect, I think.
Edit: removed piece of code that does actual read of pixels and format conversions. Was not directly related to question, and the code was long.
Edit: updated to match edited question.