How to use a stencil buffer mask in ogre? - ogre3d

Question

How to use the stencil buffer to create a mask in OGRE?
That is, some objects should be rendered first into the stencil buffer and generate a mask (say 0 is background, 0xFF is foreground); then render the scene itself, using the stencil buffer as a mask so only pixels where it is 0xFF are rendered.
I guess I should use a RenderQueueListener, but I can't make it work. Here's what I'm doing right now:
void StencilOpQueueListener::renderQueueStarted(Ogre::uint8 queueGroupId, const Ogre::String& invocation, bool& skipThisInvocation) {
if (queueGroupId == Ogre::RENDER_QUEUE_1) {
Ogre::RenderSystem *rs = Ogre::Root::getSingleton().getRenderSystem();
rs->clearFrameBuffer(Ogre::FBT_STENCIL, Ogre::ColourValue::Black, 1.0, 0xFF);
rs->setStencilCheckEnabled(true);
rs->_setColourBufferWriteEnabled(false, false, false, false);
rs->setStencilBufferParams(
Ogre::CMPF_ALWAYS_PASS, // compare
0x1, // refvalue
0xFFFFFFFF, // mask
Ogre::SOP_REPLACE, Ogre::SOP_REPLACE, // stencil fail, depth fail
Ogre::SOP_REPLACE, // stencil pass + depth pass
false); // two-sided operation? no
}
}
void StencilOpQueueListener::renderQueueEnded(Ogre::uint8 queueGroupId, const Ogre::String& invocation, bool& repeatThisInvocation) {
if (queueGroupId == Ogre::RENDER_QUEUE_1) {
Ogre::RenderSystem *rs = Ogre::Root::getSingleton().getRenderSystem();
rs->setStencilCheckEnabled(false);
rs->setStencilBufferParams();
}
}
And I set the entities that should render to the stencil buffer are set with:
entity->setRenderQueueGroup(RENDER_QUEUE_1);
What am I doing wrong? Any examples of how this is done in Ogre? Thanks!
For reference, this is how I do it in pure OpenGL:
/* Enable stencil test and leave it enabled throughout */
glClearStencil(0xFF);
glClearColor(1, 1, 1, 1);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
glEnable(GL_STENCIL_TEST);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glDepthMask(GL_FALSE);
glStencilFunc(GL_ALWAYS, 0x1, 0xFFFFFFFF);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE);
// render into the stencil buffer. This should render only the selector objects
renderStencil();
// restore the rendering
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
glDepthMask(GL_TRUE);
// only render "inside" the silhouette -- we want the overlap
glStencilFunc(GL_EQUAL, 0x1, 0xFFFFFFFF);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
render(); // now we render objects against that mask

Answer 1

you should synthesize the second step as well, which you mentioned in
// only render "inside" the silhouette -- we want the overlap
glStencilFunc(GL_EQUAL, 0x1, 0xFFFFFFFF); glStencilOp(GL_KEEP,
GL_KEEP, GL_KEEP);
This means, you should apply the mask you set in RENDER_QUEUE_1 to the rendered scene, like you do in the case of gl calls. The main rendering takes place in Ogre's RENDER_QUEUE_MAIN by default, thus you should add this to your renderQueueStarted method:
if (queueGroupId == Ogre::RENDER_QUEUE_MAIN)
{
Ogre::RenderSystem * rs = Ogre::Root::getSingleton().getRenderSystem();
rs->_setColourBufferWriteEnabled(true, true, true, true);
rs->setStencilCheckEnabled(true);
rs->setStencilBufferParams(Ogre::CMPF_EQUAL,0x1,0xFFFFFFFF,
Ogre::SOP_KEEP,Ogre::SOP_KEEP,Ogre::SOP_KEEP,false);
}
also, switch off the stenciling in renderQueueEnded only after main rendering is finished:
if ( queueGroupId == Ogre::RENDER_QUEUE_MAIN )
{
Ogre::RenderSystem *rs = Ogre::Root::getSingleton().getRenderSystem();
rs->setStencilCheckEnabled(false);
rs->setStencilBufferParams();
}