2

I have a GLSL shader that draws a 3D curve given a set of Bezier curves (3d coordinates of points). The drawing itself is done as I want except the occlusion does not work correctly, i.e., under certain viewpoints, the curve that is supposed to be in the very front appears to be still occluded, and reverse: the part of a curve that is supposed to be occluded is still visible.

To illustrate, here are couple examples of screenshots:

  1. Colored curve is closer to the camera, so it is rendered correctly here.

    correct render

  2. Colored curve is supposed to be behind the gray curve, yet it is rendered on top.

    wrong render

I'm new to GLSL and might not know the right term for this kind of effect, but I assume it is occlusion culling (update: it actually indicates the problem with depth buffer, terminology confusion!).

My question is: How do I deal with occlusions when using GLSL shaders?

Do I have to treat them inside the shader program, or somewhere else?

Regarding my code, it's a bit long (plus I use OpenGL wrapper library), but the main steps are:

  1. In the vertex shader, I calculate gl_Position = ModelViewProjectionMatrix * Vertex; and pass further the color info to the geometry shader.
  2. In the geometry shader, I take 4 control points (lines_adjacency) and their corresponding colors and produce a triangle strip that follows a Bezier curve (I use some basic color interpolation between the Bezier segments).
  3. The fragment shader is also simple: gl_FragColor = VertexIn.mColor;.

Regarding the OpenGL settings, I enable GL_DEPTH_TEST, but it does not seem to have anything of what I need. Also if I put any other non-shader geometry on the scene (e.g. quad), the curves are always rendered on the top of it regardless the viewpoint.

Any insights and tips on how to resolve it and why it is happening are appreciated.

Update solution

So, the initial problem, as I learned, was not about finding the culling algorithm, but that I do not handle the calculation of the z-values correctly (see the accepted answer). I also learned that given the right depth buffer set-up, OpenGL handles the occlusions correctly by itself, so I do not need to re-invent the wheel.

I searched through my GLSL program and found that I basically set the z-values as zeros in my geometry shader when translating the vertex coordinates to screen coordinates (vec2( vertex.xy / vertex.w ) * Viewport;). I had fixed it by calculating the z-values (vertex.z/vertex.w) separately and assigned them to the emitted points (gl_Position = vec4( screenCoords[i], zValues[i], 1.0 );). That solved my problem.

Regarding the depth buffer settings, I didn't have to explicitly specify them since the library I use set them up by default correctly as I need.

genpfault
  • 51,148
  • 11
  • 85
  • 139
vicrucann
  • 1,703
  • 1
  • 24
  • 34
  • Use a debugger like apitrace or renderdoc to see what depth values are being written into the depth buffer. – pleluron Sep 10 '16 at 00:57
  • If I won't be able to figure it out on the theory level, I will be trying it. – vicrucann Sep 10 '16 at 01:02
  • Do you have a depth buffer? Depending on the window system toolkit/framework you use for setting up the context and rendering surface, this may be the default, or you may have to specifically request it. – Reto Koradi Sep 10 '16 at 02:25
  • I do not explicitly set up the depth buffer, or use it anyhow. What would I need to change in the depth buffer settings, or how do I use it to solve the problem? – vicrucann Sep 10 '16 at 03:14

1 Answers1

1

If you don't use the depth buffer, then the most recently rendered object will be on top always.

You should enable it with glEnable(GL_DEPTH_TEST), set the function to your liking (glDepthFunc(GL_LEQUAL)), and make sure you clear it every frame with everything else (glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT)).

Then make sure your vertex shader is properly setting the Z value of the final vertex. It looks like the simplest way for you is to set the "Model" portion of ModelViewProjectionMatrix on the CPU side to have a depth value before it gets passed into the shader.

As long as you're using an orthographic projection matrix, rendering should not be affected (besides making the draw order correct).

Robert Rouhani
  • 14,512
  • 6
  • 44
  • 59
  • Could you please clarify a bit, what does "vertex shader is properly setting the Z value of the final vertex" mean? What is Z-value, is it `gl_Position.z`? and what is **proper** z-value? – vicrucann Sep 10 '16 at 04:29
  • 1
    The Z value of a vec4 (like gl_Position) is just like X and Y but for depth. A proper Z value is anything between 0 and 1 or whatever you set zNear and zFar to when making your projection matrix. 0 is the screen and 1 is pushing out the back of your monitor, so when you set the depth function to GL_LEQUAL, you're saying to only render things closest to the monitor. – Robert Rouhani Sep 10 '16 at 04:51
  • A more concrete example is that if you set the rainbow depth to 0.8 and the grayscale to 0.2, and render the grayscale first, the overlapping pixels on the rainbow will fail the depth test and not draw because 0.8 is not less than or equal to 0.2. If you flip the draw order and draw the rainbow first, then the grayscale overlapping pixels go through the depth test instead. 0.2 is less than or equal to 0.8, so it overlaps the rainbow pixels with the grayscale ones – Robert Rouhani Sep 10 '16 at 04:54
  • Thank you for help, the answer/commends helped me to understand the problem (looks like I do not set up the Z-value correctly, so will be trying to fix that). – vicrucann Sep 11 '16 at 03:59
  • May I ask one more thing, when you mention on how to obtain the Z value, you recommend using `Model` matrix. Then the formula would be something like: `V' = Model * Vertex` and `zValue = normalize (V'.z in Far and Near)`, is that correct? – vicrucann Sep 11 '16 at 05:03
  • This would be outside the shader. You are creating a projection matrix somewhere, presumably with `glm::ortho`. If you are using the overload with 6 parameters (left, right, bottom, top, zNear, zFar), then the last two parameters are the range of Z values you can use. As long as your vertex has a Z value in that range and you've set up depth testing properly, you won't have to touch your shader. This link has more than you need but explains things pretty well: http://learnopengl.com/#!Getting-started/Coordinate-Systems – Robert Rouhani Sep 11 '16 at 05:33
  • For example, if I'm making a 2d game, I would probably have my zNear be 0, and my zFar be 10. I would have the sky objects have Z values of 9.5 or higher. Background sprites would have Z values of 7.5-9.5, Interactable objects would have a Z value of 5, enemies of 4.9, an the player of 4.8. UI would have Z values between 0 and 4. This is just an example of something you can do because you get to define the Z boundaries yourself (along with the rest of the space). It's arbitrary. Make zNear 0 and zFar 2, draw the grayscale curve at 2 and the rainbow one at 1 if you want. – Robert Rouhani Sep 11 '16 at 05:37
  • Thanks again, really helpfull, I finally got it ! – vicrucann Sep 11 '16 at 14:17