I'm porting over a C++ volumetric clustered rendering engine to C#, and the code is here: https://github.com/Kermalis/HybridRenderingEngine-CS
Now, I'm trying to get a linear depth from the current fragment, because it is required for volumetric clustered rendering. If you're using GLM to create the projection matrix, this is easy with the following equation in your shader:
float ndc = 2.0 * depthSample - 1.0;
float linear = 2.0 * zNear * zFar / (zFar + zNear - ndc * (zFar - zNear));
return linear;
This is because GLM projection matrices are created this way:
float num = (float)Math.Tan(fovy / 2f);
mat4 result = mat4.identity();
result[0, 0] = 1f / (aspect * num);
result[1, 1] = 1f / num;
result[2, 2] = (-(zFar + zNear)) / (zFar - zNear);
result[2, 3] = -1f;
result[3, 2] = (-2f * zFar * zNear) / (zFar - zNear);
However, with C# Matrix4x4, the projection matrix is created differently:
float num = (float)Math.Tan(fieldOfView * 0.5f);
float yScale = 1f / num;
float xScale = yScale / aspectRatio;
result.M11 = xScale;
result.M22 = yScale;
result.M33 = zFar / (zNear - zFar);
result.M34 = -1f;
result.M43 = zNear * zFar / (zNear - zFar);
M33 and M43 are different than GLM's. There are more differences if you create an "off center" projection matrix as well, but I'm not sure if they will make a difference.
The issue, is when you use that same shader code with the Matrix4x4 matrices, you get incorrect values. If you do the math, you can find why that is:
Here's what you get with 1 Near, 300 Far:
And here's what you get with 0.1 Near, 1500 Far:
Sorry for my handwriting. C1 is M33, and C2 is M43.
You can see that when you divide C1 by C2, you get different depth values for the different projection matrices. So the shader code will not work with Matrix4x4, since it is not actually transforming it to where 0 = near, 1 = far, since the parts of the matrix are different and distribute a different depth curve. I did find that multiplying by 0.5 will fix it for some near/far values, and you can see that it is close to half difference in my drawings between the two types.
What I want to know is, how would I get depth properly with these matrices in the shader, without having to multiply each fragment by the inverse projection matrix? That's being done in the compute shader (which is why I need the same linear result), but that only gets run one time, not every frame, so I don't care about performance there.
How is the first equation derived? If that becomes clear to me (trust me, I tried to make sense of it), then another one can be created for Matrix4x4, or any other custom one that is passed in. I know an inverse projection matrix will do the trick, but it'll be awfully slow. I can't think of any other robust solution that will cover any matrix. I just don't understand how that shader equation is created or how it applies the inverse.