1

I'm trying to implement some simple dithering in my Metal fragment shader, to get rid of banding across gradients. It is not working, and I am wondering if it is just a bug or if the color values passed to the shader are already quantized (if that's the right word) to 8 bits. In other words, the fragment shader deals with floating point values, but are those already at the discrete levels imposed by 8 bit rgb space?

Here is my shader and the matrix I'm using for dithering. I based this on these two articles/posts:

OpenGL gradient "banding" artifacts

http://www.anisopteragames.com/how-to-fix-color-banding-with-dithering/

var dither_pattern:[UInt8] = 
   [0, 32,  8, 40,  2, 34, 10, 42,   /* 8x8 Bayer ordered dithering  */
    48, 16, 56, 24, 50, 18, 58, 26,  /* pattern.  Each input pixel   */
    12, 44,  4, 36, 14, 46,  6, 38,  /* is scaled to the 0..63 range */
    60, 28, 52, 20, 62, 30, 54, 22,  /* before looking in this table */
    3, 35, 11, 43,  1, 33,  9, 41,   /* to determine the action.     */
    51, 19, 59, 27, 49, 17, 57, 25,
    15, 47,  7, 39, 13, 45,  5, 37,
    63, 31, 55, 23, 61, 29, 53, 21]

// this array is passed to the frag shader via a MTLBuffer

fragment float4 window_gradient_fragment(WindowGradientVertexOut interpolated [[stage_in]],
   const device unsigned char* pattern [[ buffer(0) ]]) {

    int x = (int)interpolated.position.x % 8;
    int y = (int)interpolated.position.y % 8;
    int val = pattern[x+y*8];

    float bayer = 255.0 * (float)val / 64.0;
    const float rgbByteMax = 255.0;
    float4 rgba = rgbByteMax*interpolated.color;
    float4 head = floor(rgba);
    float4 tail = rgba-head;
    float4 color = head+step(bayer,tail);

    return color/255.0;
}

I tested this by changing the matrix to a series of alternating 0,63 pairs, and up close it did produce faint vertical stripes every other pixel. But the overall banding stays the same, from what I can tell. This has me thinking that the banding is already "built-in" by the time it reaches the fragment shader. So applying the dither to it wouldn't really help, because the damage has already been done. I hope that's not true though ... it was my naive assumption that because the colors are floats, they would have full floating point precision and the conversion to 8 bit would happen after the frag shader.

bsabiston
  • 721
  • 6
  • 22
  • Are these gradients produced due to interpolation within Metal's rasterizer? Or are the gradients being produced somewhere else? If so where and how? Also, you could try `color = all(rgba == head) ? float4(1, 0, 0, 1) : float4(0, 1, 0, 1)` or something to directly discover if the color is quantized to 8 bits. – Ken Thomases Aug 16 '18 at 23:21
  • In this case they are just produced by Metal’s interpolating colors that are specified for each vertex. I’ll try your test tomorrow - thanks. – bsabiston Aug 17 '18 at 03:14
  • This is unlikely, but: is there a chance that the math just happens to work out that way for your case? Like v1.color is white, v2.color is black, and they happen to transform to exactly 256 pixels apart in viewport coordinates? – Ken Thomases Aug 17 '18 at 04:27
  • I don't really understand what that test does -- I've never used all() and the docs say it just determines if all components of the argument are true. But this is being used differently. In any case I see all green. For your second comment -- if I understand your question, no. This is a full-screen background extending 1920 pixels. The situation you describe would not show any banding, since every pixel would be a different shade, right? – bsabiston Aug 17 '18 at 15:04
  • So, the test I suggested: `==` between two vectors produces a vector of booleans, representing the component-wise results. `all()` then checks if they are, well, all true. So, it's checking if the vectors are equal across all components; that is, actually equal to each other. If you got any green, then that would suggest that, at least for some fragments, some component of `interpolated.color` was not quantized to 8 bits. Getting all green, with no red, is a bit surprising. That suggests that **no** fragment had a color all of whose components were an integral multiple of 1/255.0. – Ken Thomases Aug 17 '18 at 15:24
  • I suppose my test could be broken. For example, comparing floats for equality is generally a bad idea, although it wouldn't be if they were quantized. You could try `all(tail < 0.002)`. I'm noticing, though, that your logic seems wrong. I'm not familiar with dithering techniques, but `tail` is in [0.0,1.0) by construction. `step()` is comparing that to `bayer`, which is in [0.0,255.0). If `val` is 1, `bayer` is 3.98, which will always be greater than `tail`. So, only `val` of 0 tweaks the color. I think you just want to **not** multiply by 255.0 when calculating `bayer`. – Ken Thomases Aug 17 '18 at 15:38
  • Ah, yes, much more likely it was my mistake! I erred when adapting the code from one of those links. It was dealing with an OpenGL frag shader with 0-255 values instead of Metal's 0-1 values. It looks much better now. Still some slight banding across the whole screen span, but a huge improvement. I might experiment with larger patterns to see if that can improve things further. Thanks! – bsabiston Aug 17 '18 at 16:12

1 Answers1

2

Your logic seems wrong. tail is in [0.0,1.0) by construction. bayer is in [0.0,255.0), more or less. step() is comparing the two. For all values of val >= 1, bayer is >= 3.98, and thus greater than tail. So, it's not very "dithery"; only a val of 0 rounds the color down, all others round up.

I think you just want to not multiply by 255.0 when calculating bayer.

Ken Thomases
  • 88,520
  • 7
  • 116
  • 154