Use stencil buffer with iOS

Question

I try to use the "stencil buffer" to display a part of my rendering from a texture, but my render is displayed without any mask effect.

It's for a 2D iOS project, with OpenGL ES 2.0 This is the concerned part of my code :

glClear( GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT );
glEnable( GL_STENCIL_TEST );


// mask rendering
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
glStencilFunc( GL_ALWAYS, 1, 1 );
glStencilOp( GL_KEEP, GL_KEEP, GL_REPLACE );

glBindTexture(GL_TEXTURE_2D, _maskTexture);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);  


// scene rendering
glColorMask( GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE ); 
glStencilFunc( GL_EQUAL, 1, 1 ); 
glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );  

glBindTexture(GL_TEXTURE_2D, _viewTexture);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

Any help would be greatly appreciated !

(As usual for a French developer, sorry for my English !)

precision : "_maskTexture" is a black & white picture.

Solution :

I finnaly solved my problem with the indications of rotoglub and tim. Thank you both.

1/ The stencil buffer has not been created correctly. It should be initialized like this:

glBindRenderbufferOES(GL_RENDERBUFFER_OES, depthStencilRenderbuffer);
glRenderbufferStorageOES(GL_RENDERBUFFER_OES, 
                             GL_DEPTH24_STENCIL8_OES, 
                             backingWidth, 
                             backingHeight);

It's the principal reason why my rendering was not affected by my mask.

2/ To be able to make use of a texture as a mask, I replaced the black color with an alpha channel and enable blending in my rendering.

My final rendering code looks like this :

glEnableClientState(GL_VERTEX_ARRAY);
glEnableClientState(GL_TEXTURE_COORD_ARRAY);
    
glVertexPointer(2, GL_FLOAT, 0, vertices);
glTexCoordPointer(2, GL_FLOAT, 0, texcoords);
    
glClearStencil(0); 
glClearColor (0.0,0.0,0.0,1);
glClear (GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT); 
    

// mask rendering
glColorMask( GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE );
glEnable(GL_STENCIL_TEST);
glEnable(GL_ALPHA_TEST);
glBlendFunc( GL_ONE, GL_ONE );
glAlphaFunc( GL_NOTEQUAL, 0.0 );
glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);
    
glBindTexture(GL_TEXTURE_2D, _mask);
glDrawArrays(GL_TRIANGLE_STRIP, 4, 4);  
    
// scene rendering
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE); 
glStencilFunc(GL_EQUAL, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
    
glDisable(GL_STENCIL_TEST);
glDisable(GL_ALPHA_TEST);
glBindTexture(GL_TEXTURE_2D, _texture);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

Just describing your problem as *don't work* is not really helpful. But I suppose your problem is the one solved in this other question : http://stackoverflow.com/questions/7571075/how-to-create-stencil-buffer-with-texture-image-in-opengl-es-2-0 — rotoglup, Jul 08 '12 at 15:52
Thank you for your comment. By "don't work", i meant that "_viewTexture" was displayed without any mask effect of "_maskTexture". I will try to be more specific in the future. Concerning the post you mentioned, i already read it, but if i don't mistake, he solved his problem with a shader, and has abandoned the stencil buffer. — EricD, Jul 08 '12 at 16:18
While reading your solution post, I think that you still have problems (perhaps being hidden for now) : you stencil test is disabled for your second rendering pass so the masking you observe cannot be provoked by this test. Also, blending should not be useful for the stencil test, only the alpha test. I think that you observe the masking you wanted, but probably for another reason that stenciling. — rotoglup, Jul 08 '12 at 21:46
@EricD: this looks like an GLES1.1 solution, is that correct? If so: that's just what I'm looking for! — Jonas Byström, Jun 14 '13 at 20:57

score 1 · Accepted Answer · answered Jul 08 '12 at 17:09

Simply, the problem is that you're just drawing a texture to the scene without doing any testing of what's in the texture. The stencil buffer doesn't care about the colors in the texture, it just checks:

Did you draw a fragment ? (Update stencil buffer) : (Don't update stencil buffer);

You're drawing a fragment for every pixel of your texture, so any mask effect in the texture is useless.

If you want to mask with a texture, you need to discard any fragments that you don't want updated in the stencil buffer.

This is either done with discard keyword in shader, or glAlphaTest/glAlphaFunc in OpenGLES 1.1

Use stencil buffer with iOS

Solution :

1 Answers1