8

I've been reading about the stencil buffer in OpenGL. The basic concept makes sense; a fragment is only drawn if it meets a certain condition after being bitwise ANDed with a value in the stencil buffer. But one thing I don't understand is how you actually write to the stencil buffer. Is there a function that I'm missing here?

P.S. When I say write, I mean specify the specific values in the stencil buffer.

paper man
  • 488
  • 5
  • 19
  • Read this: https://learnopengl.com/#!Advanced-OpenGL/Stencil-testing – Francis Cugler Jan 14 '18 at 03:09
  • Writing to stencil buffer is done by the same functions which write to RGB and/or depth. This is how I explained a friend: stencil test test is somehow like depth test. You can disable it at all. You can write only (without testing). You can enable testing (and overwrite in case). – Scheff's Cat Jan 14 '18 at 07:30
  • @Scheff how would I write to the stencil buffer without drawing anything in the color buffer? – paper man Jan 14 '18 at 13:10
  • @Scheff nevermind, I think I got it – paper man Jan 14 '18 at 13:12
  • I'm not sure whether you can write to stencil buffer only. I just had a look into code where we used stencil for vertical/horicontal interlaced stereo rendering. (The stencil buffer is initially written with a line/no line pattern to restrict the rendering to the appropriate even or odd rows/columns.) I believe it just renders RGB as well but you won't see it. It is overridden by the following rendering which passes the stencil test (and before the final `glSwap()` is called). – Scheff's Cat Jan 14 '18 at 15:39
  • @Scheff can’t you use glColorMask(false, false, false, false)? That should disable all writing to the color buffer – paper man Jan 14 '18 at 15:41
  • This might work. I was thinking on blending (i.e. ful transparency) instead. However, this might cost performance. Hence, I wouldn't care too much about temporary color values if they are overwritten. I guess the normal case is to combine stencil writing with rendering to mask pixels for following rendering commands. This is the way we recently used it: draw something with stencil writing, then draw the same as (thick) lines only with stencil test. The effect is like a thick border - I wanted this for highlighting of objects. I'm not fully happy with the result but impl. was quite easy. – Scheff's Cat Jan 14 '18 at 15:46

2 Answers2

14

Nobody answered this question and it's a valid question, so more than a year later, here's an answer to your question.

The stencil buffer is theoretically a buffer like the back buffer and the depth buffer. The three of them are written to at the same time (when enabled). You can enable/disable writing to them with specific calls:

  • glColorMask(red, green, blue, alpha) - for the back buffer
  • glDepthMask(t/f) - for the depth buffer
  • glStencilMask(value) - for the stencil buffer

For the depth and stencil buffer, you can specifically enable/disable further with:

  • glEnable/glDisable(GL_DEPTH_TEST)
  • glEnable/glDisable(GL_STENCIL_TEST)

Any triangle you render to the screen will write to all enabled buffers unless some operation functionality prevents it. For the stencil buffer, these can be set with several functions. Please look up the functionalities on the OpenGL reference pages, but here is a simple example of masking a part of the screen and then rendering only on that masked part of the screen, just to get you started.

glClearColor(0, 0, 0, 1);
glClearStencil(0);
glStencilMask(0xFF);
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE); // Do not draw any pixels on the back buffer
glEnable(GL_STENCIL_TEST); // Enables testing AND writing functionalities
glStencilFunc(GL_ALWAYS, 1, 0xFF); // Do not test the current value in the stencil buffer, always accept any value on there for drawing
glStencilMask(0xFF);
glStencilOp(GL_REPLACE, GL_REPLACE, GL_REPLACE); // Make every test succeed

// ... here you render the part of the scene you want masked, this may be a simple triangle or square, or for example a monitor on a computer in your spaceship ...

glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP); // Make sure you will no longer (over)write stencil values, even if any test succeeds
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); // Make sure we draw on the backbuffer again.

glStencilFunc(GL_EQUAL, 1, 0xFF); // Now we will only draw pixels where the corresponding stencil buffer value equals 1

// ... here you render your image on the computer screen (or whatever) that should be limited by the previous geometry ...

glDisable(GL_STENCIL_TEST);

Note that I omitted any depth code on purpose to make sure you see that it has nothing to do with stencilling. If you render 3D geometry, you might need to enable it. You may even need to NOT write a stencil value if a depth-test fails.

Note that it is important that when you render the masking geometry, you set the stencil func to GL_ALWAYS, because otherwise, the current value in the stencil buffer (which was cleared in the example) is tested against whatever was last used and your masking geometry may not even be drawn at all.

So there are no special functions to write to the stencil buffer. I'm not even sure if it can be written to like you can write data directly into the back buffer and the depth buffer video memory, but that's not the way it should be done anyway (because it's terribly slow). The stencil buffer is memory shared with the depth buffer, so it might be possible by changing the parameters of the write functions. I wouldn't count on it working on all video drivers though.

Good luck to anyone who needed this information!

scippie
  • 2,011
  • 1
  • 26
  • 42
  • wow it's been over a year and half since I asked this question! I was able to understand it eventually but I appreciate that you still wrote an answer for anyone else who comes across it :) – paper man Sep 16 '19 at 04:01
  • Thanks for accepting it :-). I know from experience that you are not the only one who struggled with this, and that's why I share the answer. – scippie Sep 16 '19 at 06:49
  • You mention that you can write to the stencil buffer as with a normal buffer, but I'm not seeing how you can bind the stencil buffer so that glBufferData will write to it. glBindBuffer wants a "name" (I don't know why on earth they call a number a name) created by glGenBuffers, which doesn't seem applicable to the stencil buffer. – Trevor Giddings Apr 06 '21 at 05:09
  • No, it doesn't work like that @TrevorGiddings, you must look at the stencil buffer as an extension to the currently selected render target (which can simply be the back buffer), whatever happens on that target, will happen on the stencil buffer if enabled like I've shown in my post. If you need to do this without actually changing that render target, you disable the color writes (like in the above post). You can't set the stencil buffer as a render target, that's not possible. You should look at it as if it is a carbon coated paper you connect to the render target. – scippie Apr 07 '21 at 07:18
  • @scippie if you can only write to it by rendering to it, then I don't understand "well no different than on any other buffers that is", and I don't see how "You could of course really send pixel data to the graphical memory" -- unless you count sending a texture that will later by used when rendering a full-screen quad. – Trevor Giddings Apr 08 '21 at 04:12
  • I've corrected my post. Thanks for pointing that out! – scippie Apr 09 '21 at 07:04
0

You can read, write and copy values (bytes,floats,dwords ...) to the frame and depth buffer but not to the stencil buffer.

All three buffers are locate at the GPU memory the problem are the stencil buffer used the remaining 8 bits from the 24 bit depth buffer !

This is why you must render something to set or overwrite values in the 8 bit stencil buffer (in real a region in the depth buffer)

DJ

DJLinux
  • 1