OpenGL GLSL blend two textures by arbitrary shape

Question

I have a full screen quad with two textures.

I want to blend two textures in arbitrary shape according to user selection.

For example, the quad at first is 100% texture0 while texture1 is transparent.

If the user selects a region, for example a circle, by dragging the mouse on the quad, then circle region should display both texture0 and texture1 as translucent.

The region not enclosed by the circle should still be texture0.

Please see example image, textures are simplified as colors.

For now, I have achieved blending two textures on the quad, but the blending region can only be vertical slices because I use the step() function.

My frag shader:

uniform sampler2D Texture0;
uniform sampler2D Texture1;
uniform float alpha;
uniform float leftBlend;
uniform float rightBlend;
varying vec4 oColor;
varying vec2 oTexCoord;
void main()
{
vec4 first_sample = texture2D(Texture0, oTexCoord);
vec4 second_sample = texture2D(Texture1, oTexCoord);

float stepLeft = step(leftBlend, oTexCoord.x);
float stepRight = step(rightBlend, 1.0 - oTexCoord.x);
if(stepLeft == 1.0 && stepRight == 1.0)
    gl_FragColor = oColor * first_sample;
else
    gl_FragColor = oColor * (first_sample * alpha + second_sample * (1.0-alpha)); 

if (gl_FragColor.a < 0.4)
   discard;
}

To achieve arbitrary shape, I assume I need to create a alpha mask texture which is the same size as texture0 and texture 1?

Then I pass that texture to frag shader to check values, if value is 0 then texture0, if value is 1 then blend texture0 and texture1.

Is my approach correct? Can you point me to any samples?

I want effect such as OpenGL - mask with multiple textures

but I want to create mask texture in my program dynamically, and I want to implement blending in GLSL

I have got blending working with mask texture of black and white

uniform sampler2D TextureMask;
vec4 mask_sample = texture2D(TextureMask, oTexCoord);
if(mask_sample.r == 0)
  gl_FragColor = first_sample;
else
  gl_FragColor =  (first_sample * alpha + second_sample * (1.0-alpha)); 

now mask texture is loaded statically from a image on disk, now I just need to create mask texture dynamically in opengl

Answer 1

Here's one approach and sample.

Create a boolean test for whether you want to blend. In my sample, I use an equation for a circle centered on the screen.

Then blend (i blended by weighted addition of the 2 colors).

(NOTE: i didn't have texture coords to work with in this sample, so i used the screen resolution to determine the circle position).

uniform vec2 resolution;

void main( void ) {

    vec2 position = gl_FragCoord.xy / resolution;

    // test if we're "in" or "out" of the blended region
    // lets use a circle of radius 0.5, but you can make this mroe complex and/or pass this value in from the user
    bool isBlended = (position.x - 0.5) * (position.x - 0.5) +
        (position.y - 0.5) * (position.y - 0.5) > 0.25;

    vec4 color1 = vec4(1,0,0,1); // this could come from texture 1
    vec4 color2 = vec4(0,1,0,1); // this could come from texture 2
    vec4 finalColor;

    if (isBlended)
    {
        // blend
        finalColor = color1 * 0.5 + color2 * 0.5;
    }
    else
    {
        // don't blend
        finalColor = color1;
    }
    gl_FragColor = finalColor;
}

See the sample running here: http://glsl.heroku.com/e#18231.0

(tried to post my sample image but i don't have enough rep) sorry :/

Update:

Here's another sample using mouse position to determine the position of the blended area. To run, paste the code in this sandbox site: https://www.shadertoy.com/new This one should work on objects of any shape, as long as you have the mouse data setup correct.

void main(void)
{
vec2 position = gl_FragCoord.xy;

// test if we're "in" or "out" of the blended region
// lets use a circle of radius 10px, but you can make this mroe complex and/or pass this value in from the user
float diffX = position.x - iMouse.x;
float diffY = position.y - iMouse.y;
bool isBlended = (diffX * diffX) + (diffY * diffY) < 100.0;

vec4 color1 = vec4(1,0,0,1); // this could come from texture 1
vec4 color2 = vec4(0,1,0,1); // this could come from texture 2
vec4 finalColor;

if (isBlended)
{
    // blend
    finalColor = color1 * 0.5 + color2 * 0.5;
}
else
{
    // don't blend
    finalColor = color1;
}
gl_FragColor = finalColor;
}