8

I'm trying to understand how this chroma key filter works. Chroma Key, if you don't know, is a green screen effect. Would someone be able to explain how some of these functions work and what they are doing exactly?

float maskY = 0.2989 * colorToReplace.r + 0.5866 * colorToReplace.g + 0.1145 * colorToReplace.b;
 float maskCr = 0.7132 * (colorToReplace.r - maskY);
 float maskCb = 0.5647 * (colorToReplace.b - maskY);

 float Y = 0.2989 * textureColor.r + 0.5866 * textureColor.g + 0.1145 * textureColor.b;
 float Cr = 0.7132 * (textureColor.r - Y);
 float Cb = 0.5647 * (textureColor.b - Y);

 float blendValue = smoothstep(thresholdSensitivity, thresholdSensitivity + smoothing, distance(vec2(Cr, Cb), vec2(maskCr, maskCb)));


 gl_FragColor = vec4(textureColor.rgb * blendValue, 1.0 * blendValue);

I understand the first 6 lines (converting the color to replace, which is probably green, and the texture color to the YCrCb color system).

This fragment shader has two input float values: thresholdSensitivity and Smoothing.

  • Threshold Sensitivity controls how similar pixels need to be colored to be replaced.
  • Smoothing controls how gradually similar colors are replaced in the image.

I don't understand how those values are used in the blendValue line. What does blendValue compute? How does the blendValue line and the gl_FragColor line actually create the green screen effect?

genpfault
  • 51,148
  • 11
  • 85
  • 139
kilakev
  • 123
  • 1
  • 5
  • 1
    I thought that code looked familiar: https://github.com/BradLarson/GPUImage/blob/master/framework/Source/GPUImageChromaKeyFilter.m – Brad Larson Jul 01 '13 at 16:53
  • Yep, I was trying to understand how it worked exactly. – kilakev Jul 02 '13 at 17:10
  • 1
    radical7 has a good breakdown of the smoothstep() portion, which is a built-in GLSL function. It basically presents a threshold function with a blurry, rather than sharp, boundary. I use that to reduce some of the noise you'd normally see when matching a color. The rest is a conversion to a YUV colorspace so that luminance is removed from the color matching calculation (with code based on a similar Core Image filter Apple demonstrated a few years ago). It's but one way of matching a color, and then making an input transparent or translucent based on the degree of the match. – Brad Larson Jul 02 '13 at 17:17
  • @BradLarson I've tried to implement this shader for Webgl, but I'm kinda stuck. If you want, could you take a look at this question: http://stackoverflow.com/questions/44043782/chroma-key-fragment-shader-fails-to-find-the-color ? – Ibe Vanmeenen May 19 '17 at 09:24
  • Fwiw - it seems this does not work for arbitrary colors... e.g. removing white ([1.0, 1.0, 1.0]) from an image. Is that correct? – davidkomer Jan 30 '18 at 09:43

1 Answers1

4

The smoothstep function in GLSL evaluates a smooth cubic curve over an interval (specified by the first two parameters). As compared to GLSL's mix function, which linearly blends its parameters as:

GLSL's <code>mix</code> function

smoothstep uses a Hermite cubic polynomial to determine the value

GLSL's <code>smoothstep</code> function

In your shader, blendValue is a smooth interpolation of your smoothing value based on the distance between the red and blue chrominance values.

Finally, gl_FragColor specifies the final fragment color (before blending, which occurs after completion of the fragment shader). In your case, it's the modulated value read from the input image, and a modulated alpha value for translucency.

radical7
  • 8,957
  • 3
  • 24
  • 33
  • I don't think this is quite right: the distance in question is between the texel colour and the mask colour in YCrCb space (ignoring Y) - not between the red and blue chrominance values. – GuyRT Jun 04 '13 at 07:13
  • Actually - I think your answer isn't wrong - it's just possible to parse it a different way. – GuyRT Jun 04 '13 at 07:18
  • Also - I don't think the first paragraph is correct. `smoothstep` doesn't interpolate between the first two parameters - it is better to compare it to the `step` function than the `mix` function. – GuyRT Jun 04 '13 at 07:30
  • Edited to clarify. We could choose to argue about the use of interpolation (which is used to described in the [man page](http://www.opengl.org/sdk/docs/manglsl/xhtml/smoothstep.xml), but your point is taken. – radical7 Jun 04 '13 at 17:16
  • a long time ago I know but I'm a bit confused - what do you mean by "uses a "hermite cubic ploynomial"? does that mean it uses all the basis functions of the hermit cubic spline or just the one in your image (as far as I know that would be just one of them)? – Robin F. Aug 25 '17 at 12:02
  • @RobinF. The equation in the image is the simplified form of the Hermite cubic polynomial over the region [0,1]. Since the starting point of the interpolation is 0, and the tangents at either end of the interval are also zero, the full Hermite form reduces to just the simplified form. – radical7 Aug 25 '17 at 22:29