Currently I'm using FreeType in subpixel mode and take the largest color of each pixel as alpha value, with the following fragment shader:
uniform sampler2D Image;
uniform vec4 Color;
smooth in vec2 vVaryingTexCoord;
out vec4 vFragColor;
void main(void){
vec4 color = texture(Image, vVaryingTexCoord);
vFragColor = color * Color;
}
This works fine for dark backgrounds, but on lighter ones the border pixels show (e.g. when a text pixel is (1,0,0)). To make it work with brighter backgrounds, I'd need to pass the background color and do the blending myself, which starts breaking down once I move to more complex backgrounds.
Is there a way to use the RGB values from FreeType as alpha values for a solid color (which is passed to the shader)? This formula basically, where b = background pixel, t = current text pixel, c = static color:
b*((1,1,1) - t) + t*c.rgb*c.a
I think drawing everything else first and passing that framebuffer to the font shader would work, but that seems a bit overkill. Is there a way doing this in the OpenGL blend stage? I tried playing around with glBlendFunc and such, but didn't get anywhere.