0

the code I use to create a texture:

GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, GLES20.GL_RGBA, textureWidth, textureHeight, 0, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE, null);

I believe that makes it so each color has 4 bytes - 1 byte for each channel (RGBA)

I draw to a texture using a fragment shader:

vec4 col = vec4(0.0);
col.r = 250.0/255.0;
col.g = 251.0/255.0;
col.b = 252.0/255.0;
col.a = 253.0/255.0;
gl_FragColor = col;

Now I pass this texture to another shader and read the values from it and i get this result:

250.0/255.0   0.97998040000   0.0420168467395%
251.0/255.0   0.98388670000   0.04340189680336%
252.0/255.0   0.98779200000   0.04487727351984%
253.0/255.0   0.99169920000   0.04614935104294%

The numbers in the middle show the value i get, the numbers on the right show the percentage it is off. It is to be expected that an 8bit float cannot store the exact value of the calculation.

BUT when i try to store these values in an 8bit or even 32bit float they will have conversion errors, meaning there are not enough bits to store exactly that value, but if i store them in a 64bit float (double) than they can be stored exactly like that. I assume reading the values changes them to a 64bit float and that changes the values?

How does opengl store 8bit floats for textures? How many exponent bits? How many mantissa bits?

EDIT:
In order to read the value of a float, i separate each digit like so:

a = f*10.0; a = (a-fract(a))/10.0; a = f-a; a *= 100.0; a = a-fract(a);

f is the input float, the above example gives me the digit at position 0.01
for the next position i do this:

a = f*100.0; a = (a-fract(a))/100.0; a = f-a; a *= 1000.0; a = a-fract(a);

next i call a function that sets the color of a pixel depending on the digit

if (digit == 0) col = vec4(0.0, 0.0, 0.0, 0.0);
if (digit == 1) col = vec4(0.0, 0.0, 0.0, 1.0);
if (digit == 2) col = vec4(0.0, 0.0, 1.0, 1.0);
if (digit == 3) col = vec4(0.0, 1.0, 0.0, 0.0);
...

After more testing, i discovered that when i try to output the number 0.03808593750, it will instead output 0.03808593700, the same number except it misses the 5 at the end. If I output 0.9799805 instead, it works and all the digits are shown.

EDIT2:
I found a way to reproduce very similar values

the first table shows the first 4 variables i read from opengl

1.0/255.0   0.00392150890
2.0/255.0   0.00784301790
3.0/255.0   0.01176452690
4.0/255.0   0.01568603500

the second table shows how one can get very similar values using Math (in js)

Math.pow(2,-8)+Math.pow(2,-16)    0.0039215087890625    
Math.pow(2,-7)+Math.pow(2,-16)*2  0.007843017578125
....
Math.pow(2,-6)+Math.pow(2,-16)*4  0.01568603515625

in order to get the last values (1/254 and 1/253) i can do this:

var sum = 0;
for (var i=1; i<9; i++) {
    sum += Math.pow(2,-i);
}
var v254 = sum - Math.pow(2, -16)*32;
var v253 = v254 - Math.pow(2, -8);

on the left side the opengl values and on the right the result from above:

v254   0.99169920000   0.99169921875
v253   0.99560540000   0.99560546875

what does this mean? does anyone know a formula to calculate every possible variable (from 0-255)?

Alexanus
  • 679
  • 4
  • 22
  • 4
    0.97998040000 is not a number represented in any binary floating-point format. All numbers that have non-zero fractional parts and that are representable in a binary floating-point format have a decimal representation that ends in “5”. “0.97998040000” looks like the output of a low-quality formatter that has not shown the correct value. – Eric Postpischil May 18 '20 at 08:56
  • 1
    Re “if i store them in a 64bit float (double) than they can be stored exactly like that: No, they cannot, as stated above. What is likely is that a 64-bit format stores them precisely enough that the formatter is then able to reproduce the same decimal digits as before, even though they are an incomplete or inaccurate representation of the actual value. – Eric Postpischil May 18 '20 at 09:02
  • Re “How does opengl store 8bit floats for textures?”: Those are not results from any eight-bit binary floating-point format, unless the formatter is very broken. Any such format could represent only values whose decimal representation ended with “5” in the seventh or earlier digit of the fractional part. You may need to provide a [mre]. – Eric Postpischil May 18 '20 at 09:09
  • thanks, i did some more testing and it seems like numbers at the end can be wrong using the conversion method i use. – Alexanus May 18 '20 at 09:28
  • also your printing method might round the stuff if you want to be sure print the float in HEX there are no rounding .... for more info see the sublinks in here: [How to calculate float type precision and does it make sense?](https://stackoverflow.com/a/61610484/2521214) – Spektre May 19 '20 at 07:41

0 Answers0