0

I am attempting to draw concave polygons using the stencil buffer trick outlined in the OpenGL redbook but I can't seem to get it working correctly.

Per the redbook, I clear the stencil buffer first prior to enabling the stencil test. Next, I then disable any writes to the color buffer and draw my triangles. The redbook noted that I can use any arbitrary point for the start of my triangle fan call, so I used (0, 0). See the code below for reference:

    glMatrixMode(GL_MODELVIEW);
    glPushAttrib( GL_CURRENT_BIT );
    glDisable(GL_DEPTH_TEST);
    glPushMatrix();
    glClear( GL_STENCIL_BUFFER_BIT );

    glEnable( GL_STENCIL_TEST );
    glColorMask( GL_FALSE,GL_FALSE,GL_FALSE,GL_FALSE );
    glStencilOp( GL_KEEP, GL_KEEP, GL_INVERT );
    glStencilFunc( GL_ALWAYS, 0x1, 0x1 );

    glBegin(GL_TRIANGLE_FAN);

    glVertex2d(0.0, 0.0);

    for(int i = 0; i < oglPoints.size(); i++)
    {
        glVertex2d(oglPoints[i].x, oglPoints[i].y);
    }

    glVertex2d(oglPoints[0].x, oglPoints[0].y);

    glEnd();

    glColor3d(1.0, 1.0, 0.0);
    glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE );
    glStencilFunc( GL_EQUAL, 0x1, 0x1 );

    glBegin(GL_TRIANGLE_FAN);

    for(int i = 0; i < oglPoints.size(); i++)
    {
        glVertex2d(oglPoints[i].x, oglPoints[i].y);
    }

    // glVertex2d(oglPoints[0].x, oglPoints[0].y);

    glEnd();

    glDisable( GL_STENCIL_TEST );
    glPopMatrix();
    glPopAttrib();

As can be seen from my attached images, I get mixed results. For some points it works, but for others it doesn't.

Correct concave polygon:

Correct concave polygon

Incorrect concave polygon 1:

Incorrect concave polygon 1

Incorrect concave polygon 2:

Incorrect concave polygon 2

genpfault
  • 51,148
  • 11
  • 85
  • 139
JDBones
  • 495
  • 1
  • 7
  • 18
  • Have you disabled backface culling? Why is your second pass the polygon again and not just a quad? – Nico Schertler Feb 28 '18 at 13:27
  • [This code](https://stackoverflow.com/a/15285096/44729) work for you? – genpfault Feb 28 '18 at 15:02
  • @NicoSchertler - Drawing a quad on the second pass just gets me, well, a quad. Am I doing something wrong with the stencil test? – JDBones Feb 28 '18 at 18:05
  • 1
    @Rabbid76 - How is this not an OpenGL issue? I want to fill my polygon using the stencil test trick that was mentioned in the redbook, but I seem to have an issue getting it to work properly. – JDBones Feb 28 '18 at 18:08
  • 2
    That should not happen. Do you have a stencil buffer? And have you set a reasonable stencil clear value? – Nico Schertler Feb 28 '18 at 18:45
  • 1
    @NicoSchertler - Oh man, that's it! It did not occur to me attach a stencil buffer to my framebuffer. This is great. Thank you everyone for throwing out ideas, it really helped! – JDBones Feb 28 '18 at 20:57

0 Answers0