Chroma key Fragment Shader fails to find the color

Question

I'm trying to write a fragment-shader that functions as a chroma-key filter for a specific color (for example make all pixels with a specific green transparent).

The shader I'm writing is for use in WebGL trough PIXI.js.

JSFiddle: https://jsfiddle.net/IbeVanmeenen/hexec6eg/14/

So far, I wrote this code for the shader, based on the shader I've found here.

varying vec2 vTextureCoord;

uniform float thresholdSensitivity;
uniform float smoothing;
uniform vec3 colorToReplace;
uniform sampler2D uSampler;

void main() {
    vec4 textureColor = texture2D(uSampler, vTextureCoord);

    float maskY = 0.2989 * colorToReplace.r + 0.5866 * colorToReplace.g + 0.1145 * colorToReplace.b;
    float maskCr = 0.7132 * (colorToReplace.r - maskY);
    float maskCb = 0.5647 * (colorToReplace.b - maskY);

    float Y = 0.2989 * textureColor.r + 0.5866 * textureColor.g + 0.1145 * textureColor.b;
    float Cr = 0.7132 * (textureColor.r - Y);
    float Cb = 0.5647 * (textureColor.b - Y);

    float blendValue = smoothstep(thresholdSensitivity, thresholdSensitivity + smoothing, distance(vec2(Cr, Cb), vec2(maskCr, maskCb)));
    gl_FragColor = vec4(textureColor.rgb, textureColor.a * blendValue);
}

Now, when I define and test this, nothing happens. The problem lies with the shader, because the other filters I tried work.

The color I use for the test is rgb(85, 249, 44).

The Full code for the shader with PIXI is below:

function ChromaFilter() {
    const vertexShader = null;
    const fragmentShader = [
        "varying vec2 vTextureCoord;",

        "uniform float thresholdSensitivity;",
        "uniform float smoothing;",
        "uniform vec3 colorToReplace;",
        "uniform sampler2D uSampler;",

        "void main() {",
            "vec4 textureColor = texture2D(uSampler, vTextureCoord);",

            "float maskY = 0.2989 * colorToReplace.r + 0.5866 * colorToReplace.g + 0.1145 * colorToReplace.b;",
            "float maskCr = 0.7132 * (colorToReplace.r - maskY);",
            "float maskCb = 0.5647 * (colorToReplace.b - maskY);",

            "float Y = 0.2989 * textureColor.r + 0.5866 * textureColor.g + 0.1145 * textureColor.b;",
            "float Cr = 0.7132 * (textureColor.r - Y);",
            "float Cb = 0.5647 * (textureColor.b - Y);",

            "float blendValue = smoothstep(thresholdSensitivity, thresholdSensitivity + smoothing, distance(vec2(Cr, Cb), vec2(maskCr, maskCb)));",
            "gl_FragColor = vec4(textureColor.rgb, textureColor.a * blendValue);",
        "}"
    ].join('\n');

    let uniforms = {};

    PIXI.Filter.call(this,
        vertexShader,
        fragmentShader,
        uniforms
    );

    this.uniforms.thresholdSensitivity = 0.4;
    this.uniforms.smoothing = 0.1;
    this.uniforms.colorToReplace = [0.33, 0.97, 0.17];

    this.glShaderKey = 'chromakey';
}

ChromaFilter.prototype = Object.create(PIXI.Filter.prototype);
ChromaFilter.prototype.constructor = ChromaFilter;

This is applied to the video-sprite like this:

videoBase = new PIXI.VideoBaseTexture(videoLoaderVid);
videoBase.on('loaded', () => {
    video = videoBase.source;
    video.volume = 0;
    video.pause();
    video.currentTime = 0;

    videoTexture = new PIXI.Texture(videoBase);
    videoSprite = new PIXI.Sprite(videoTexture);

    const filter = new ChromaFilter();
    videoSprite.filters = [filter];

    resolve();
});

And PIXI is set up like this:

stage = new PIXI.Container();

renderer = PIXI.autoDetectRenderer(720, 720, {
    preserveDrawingBuffer: true,
    clearBeforeRender: true
});

canvasContainer.appendChild(renderer.view);

The video-sprite sits on it's own DisplayObjectContainer and is displayed above another DisplayObjectContainer (hence the need for a chroma-filter)

UPDATE:

The fixed shader can be found here:
https://gist.github.com/IbeVanmeenen/d4f5225ad7d2fa54fabcc38d740ba30e

And a fixed demo can be found here:
https://jsfiddle.net/IbeVanmeenen/hexec6eg/17/

Have you tried to output `blendValue` to another channel? Does the canvas have alpha, is blending set up correctly to the filter? — Kirill Dmitrenko, May 19 '17 at 11:47
@KirillDmitrenko Thanks for the response! I've updated the question a little bit so all of your questions should be answered there. How would you go about "Have you tried to output blendValue to another channel" (still learning a lot in terms of WebGL) — Ibe Vanmeenen, May 19 '17 at 13:24
Add `gl_FragColor = vec4(blendValue, 0, 0, 1);` at the end of `main` function of your fragment shader. — Kirill Dmitrenko, May 19 '17 at 13:38
See a lot of red now (logically), only some parts of the video are visibel in pure black, all the rest is red. Am I missing something? — Ibe Vanmeenen, May 19 '17 at 13:57
Nope, sounds about right. I'm blaming blending. I'm not entirely familiar with PIXI API, but from their docs it seems that default blending should be right. If you don't draw anything but the sprite with the filter, try add `transparent: true` to `PIXI.autoDetectRenderer` options and change the shader output from `gl_FragColor = vec4(textureColor.rgb, textureColor.a * blendValue);` to `gl_FragColor = textureColor * blendValue` (presuming that your video texture is opaque). — Kirill Dmitrenko, May 19 '17 at 14:17
Adding transparent: true has some results (weird ones), change to gl_FragColor does not have any effect. Put up a JSFiddle to show you what I get: https://jsfiddle.net/IbeVanmeenen/hexec6eg/14/ (video replaced with still image) — Ibe Vanmeenen, May 19 '17 at 14:29
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/144666/discussion-between-kirill-dmitrenko-and-ibe-vanmeenen). — Kirill Dmitrenko, May 19 '17 at 15:00

Kirill Dmitrenko · Accepted Answer · 2017-05-22T08:44:20.960

1

The shader is fine, the problem is that uniforms (colorToReplace, thresholdSensitivity and smoothing) aren't passed, they're all set to 0s. By blind luck I've found that to fix that you need to remove third parameter you're passing to PIXI.Filter constructor:

/* ... */
PIXI.Filter.call(this, vertexShader, fragmentShader) // no uniforms param here
/* ... */

PS. You haven't answer in chat, so I'm posting my findings here.

edited May 22 '17 at 08:44

answered May 20 '17 at 13:54

Kirill Dmitrenko

3,474
23
31

Missed the chat, sorry about that! This indeed fixes it, thank you! Fixed demo here: https://jsfiddle.net/IbeVanmeenen/hexec6eg/17/ – Ibe Vanmeenen May 22 '17 at 08:43

Chroma key Fragment Shader fails to find the color

1 Answers1

Linked