OpenGL Alpha Mask

Question

I'm trying to use alhpa masking with glAlphaBlend and all his parameters. I found a lot similar answer on stackoverflow but i doesn't realise anything.

I have three images, a background (1), a mask (2), and text (3).

What i want is to draw the background, substract my mask to the text to finally obtain the image (A).

Draw background

glClearColor(0,0,0,0);
glClear(GL_COLOR_BUFFER_BIT);
glClear(GL_DEPTH_BUFFER_BIT);
glEnable(GL_BLEND);
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_TRUE);
glBegin(GL_QUADS);
glColor4ub(0,0,0,255);
glVertex2d(mxIncrust0 - bgwidth, myIncrust0);
glVertex2d(mxIncrust0, myIncrust0);
glVertex2d(mxIncrust0, myIncrust0 + textheight);
glVertex2d(mxIncrust0 - bgwidth, myIncrust0 + textheight);
glEnd();
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);

glEnable(GL_BLEND);
glBlendEquation(GL_FUNC_ADD);
glBlendFuncSeparate(GL_ONE_MINUS_DST_ALPHA, GL_DST_ALPHA, GL_ZERO, GL_ONE);

Draw Text

But Anything is working !

What is the real way to achieve it ?

My question is different because it is not based on OpenGL ES but only Open GL.

Answer 1

The "real way" to accomplish this is to use shader programs and modern OpenGL .Why do you use that deprecated API?We are in the year 2014 with OpenGL 4.5 as the latest version.You even can't claim you care about backward compatibility as even some really old cards(well,5 years old) today support at least GL 3.2 which is fully programmable.And you know what,even with the GL 2.0 you still can use shaders, but working with the deprecated API takes you nowhere if you plan to code OpenGL for your living in the near future.

Now,using shaders,masking is one of the most trivial tasks. You would typically pass 2 textures into the fragment shader.1 - Diffuse texture(actual texture to cover the surface) 2 - Mask texture (maybe a single channel grayscale texture)

The fragment shader would look like this:

#version 150
uniform sampler2D maskTex;
uniform sampler2D colorTex;

out vec4 colorOut;
in smooth vec2 texCoords;
void main(){ 

     vec4 color = texture(colorTex,texCoords);
     vec4 mask  = texture(maskTex,texCoords);

     colorOut =vec4(color.rgb,color.a * mask.r);//alpha value can be in any channel ,depends on texture format.
}

As simple as it can be.Hope it helps.

P.S make sure you mipmap the alpha texture as well,in case the diffuse texture is mipmapped. Also please note my example doesn't come to solve your specific task but to show you how basic masking is done in programmable OpenGL.From here it is easy to figure out what to do ;)

Addendum:

Someone in the comments below wondered why on earth one would want to use an extra texture just for holding alpha mask data.Here is my answer:

In many complex application users use additional textures called alpha maps to mask out pixels of primary texture,also taking into account the alpha of that texture during the process, and there is no reason to "merge" alpha channels into one texture map because it would require an extra step and a waste of runtime for no real purpose.In some complex rendering scenarios those alpha maps can be generated only during different render passes on request and later fetched into fragment shader for final usage. So in such a case the application doesn't even have that data before runtime. Also,same alpha mask can be required by many different color textures. So does it make sense to merge it into all of those? Sure it doesn't. The practice above is well known and pretty efficient comparing to merging transparency data on CPU just to avoid extra texture sampling.

Answer 2

I finally found a solution with glClipPlane, i didn't find any solution with shader since it isn't texture.

So how it works ? 1) I draw my background (quad, lines...) 2) I define and active a limit below nothing is draw on the left and a limit above nothing is draw on the right 3) I draw everything i need (quad, lines, my text...) 4) i disable the clip.

// Draw the background
double* leftMask = new double[4];
leftMask[0] = 1;
leftMask[3] = -mClipXL; // Be careful to use -Limit for the left limit
glClipPlane(GL_CLIP_PLANE0, leftMask);
delete[] leftMask;
glEnable(GL_CLIP_PLANE0);

double* rightMask = new double[4];
rightMask[0] = -1;  // Be careful to use -1 for the right limit
rightMask[3] = mClipXR;
glClipPlane(GL_CLIP_PLANE1, rightMask);
delete[] rightMask;
glEnable(GL_CLIP_PLANE1);

// draw the text

// disable the clip to draw anything else.
glDisable(GL_CLIP_PLANE0);
glDisable(GL_CLIP_PLANE1);

It's working very very well !!!

Answer 3

You can also use the stencil buffer for this. First you would enable stenciltesting:

glEnable(GL_STENCIL_TEST);

Then you first draw your masking quad only in the stencil buffer:

//begin mask
glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
glStencilFunc(GL_NEVER, 0, 0);
glStencilOp(GL_INCR, GL_KEEP, GL_KEEP);
glClear(GL_STENCIL_BUFFER_BIT);
//draw your mask, a quad in your case

Then you draw the masked part:

    ////masked fill part
    glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
    glStencilFunc(GL_NOTEQUAL, 0, 1);
    glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
    // draw the masked element, the text in your case

Disable stencil testing again

glDisable(GL_STENCIL_TEST);

Depending on your needs you can change glStencilFunc() and glStencilOp() parameters to achieve the desired clipping effect.