Frustum Culling Bug

Question

So I've implemented Frustum Culling in my game engine and I'm experiencing a strange bug. I am rendering a building that is segmented into chunks and I'm only rendering the chunks which are in the frustum. My camera starts at around (-.033, 11.65, 2.2) and everything looks fine. I start moving around and there is no flickering. When I set a breakpoint in the frustum culling code I can see that it is indeed culling some of the meshes. Everything seems great. Then when I reach the center of the building, around (3.9, 4.17, 2.23) meshes start to disappear that are in view. The same is true on the other side as well. I can't figure out why this bug could exist.

I implement frustum culling by using the extraction method listed here Extracting View Frustum Planes (Gribb & Hartmann method). I had to use glm::inverse() rather than transpose as it suggested and I think the matrix math was given for row-major matrices so I flipped that. All in all my frustum plane calculation looks like

std::vector<Mesh*> render_meshes;
auto comboMatrix = proj * glm::inverse(view * model);
glm::vec4 p_planes[6];

p_planes[0] = comboMatrix[3] + comboMatrix[0]; //left
p_planes[1] = comboMatrix[3] - comboMatrix[0]; //right
p_planes[2] = comboMatrix[3] + comboMatrix[1]; //bottom
p_planes[3] = comboMatrix[3] - comboMatrix[1]; //top
p_planes[4] = comboMatrix[3] + comboMatrix[2]; //near
p_planes[5] = comboMatrix[3] - comboMatrix[2]; //far

for (int i = 0; i < 6; i++){
    p_planes[i] = glm::normalize(p_planes[i]);
}
for (auto mesh : meshes) {
    if (!frustum_cull(mesh, p_planes)) {
        render_meshes.emplace_back(mesh);
    }
}

I then decide to cull each mesh based on its bounding box (as calculated by ASSIMP with the aiProcess_GenBoundingBoxes flag) as follows (returning true means culled)

glm::vec3 vmin, vmax;
for (int i = 0; i < 6; i++) {
    // X axis
    if (p_planes[i].x > 0) {
        vmin.x = m->getBBoxMin().x;
        vmax.x = m->getBBoxMax().x;
    }
    else {
        vmin.x = m->getBBoxMax().x;
        vmax.x = m->getBBoxMin().x;
    }
    // Y axis
    if (p_planes[i].y > 0) {
        vmin.y = m->getBBoxMin().y;
        vmax.y = m->getBBoxMax().y;
    }
    else {
        vmin.y = m->getBBoxMax().y;
        vmax.y = m->getBBoxMin().y;
    }
    // Z axis
    if (p_planes[i].z > 0) {
        vmin.z = m->getBBoxMin().z;
        vmax.z = m->getBBoxMax().z;
    }
    else {
        vmin.z = m->getBBoxMax().z;
        vmax.z = m->getBBoxMin().z;
    }
    if (glm::dot(glm::vec3(p_planes[i]), vmin) + p_planes[i][3] > 0)
        return true;
    
}
return false;

Any guidance?

Update 1: Normalizing the full vec4 representing the plane is incorrect as only the vec3 represents the normal of the plane. Further, normalization is not necessary for this instance as we only care about the sign of the distance (not the magnitude).

It is also important to note that I should be using the rows of the matrix not the columns. I am achieving this by replacing

p_planes[0] = comboMatrix[3] + comboMatrix[0];

with

p_planes[0] = glm::row(comboMatrix, 3) + glm::row(comboMatrix, 0);

in all instances.

Answer 1

You are using GLM incorrectly. As per the paper of Gribb and Hartmann, you can extract the plane equations as a sum or difference of different rows of the matrix, but in glm, mat4 foo; foo[n] will yield the n-th column (similiar to how GLSL is designed).

This here

for (int i = 0; i < 6; i++){
    p_planes[i] = glm::normalize(p_planes[i]);
}

also doesn't make sense, since glm::normalize(vec4) will simply normalize a 4D vector. This will result in the plane to be shifted around along its normal direction. Only thexyz components must be brought to unit length, and w must be scaled accordingly. It is even explained in details in the paper itself. However, since you only need to know on which half-space a point lies, normalizing the plane equation is a waste of cycles, you only care about the sign, not the maginitude of the value anyway.

Answer 2

After following @derhass solution for normalizing the planes correctly for intersection tests you would do as follows

For bounding box plane intersection after projecting your box onto that plane which we call p and after calculating the midpoint of the box say m and after calculating the distance of that mid point from the plane say d to check for intersection we do

  d<=p

But for frustum culling we just don't want our box to NOT intersect wih our frustum plane but we want it to be at -p distance from our plane and only then we know for sure that NO PART of our box is intersecting our plane that is

    if(d<=-p)//then our box is fully not intersecting our plane so we don't draw it or cull it[d will be negative if the midpoint lies on the other side of our plane]

Similarly for triangles we have check if the distance of ALL 3 points of the triangle from the plane are negative.

To project a box onto a plane we take the 3 axises[x,y,z UNIT VECTORS] of the box,scale them by the boxes respective HALF width,height,depth and find the sum of each of their dot products[Take only the positive magnitude of each dot product NO SIGNED DISTANCE] with the planes normal which will be your 'p'

Not with the above approach for an AABB you can also cull against OOBB's with the same approach cause only the axises will change.

EDIT: how to project a bounding box onto a plane?

Let's consider an AABB for our example It has the following parameters

Lower extent Min(x,y,z)
Upper extent Max(x,y,z)
Up Vector      U=(0,1,0)
Left Vector.   L=(1,0,0)
Front Vector.  F=(0,0,1)

Step 1: calculate half dimensions

half_width=(Max.x-Min.x)/2;
half_height=(Max.y-Min.y)/2;
half_depth=(Max.z-Min.z)/2;

Step 2: Project each individual axis of the box onto the plane normal,take only the positive magnitude of each dot product scaled by each half dimension and find the total sum. make sure both the box axis and the plane normal are unit vectors.

float p=(abs(dot(L,N))*half_width)+
        (abs(dot(U,N))*half_height)+
        (abs(dot(F,N))*half_depth);

 abs() returns absolute magnitude we want it to be positive 
 because we are dealing with distances

Where N is the planes normal unit vector

Step 3: compute mid point of box

 M=(Min+Max)/2;

Step 4: compute distance of the mid point from plane

d=dot(M,N)+plane.w

Step 5: do the check

d<=-p //return true i.e don't render or do culling

U can see how to use his for OOBB where the U,F,L vectors are the axises of the OOBB and the centre(mid point) and half dimensions are parameters you pass in manually

For an sphere as well you would calculate the distance of the spheres center from the plane (called d) but do the check

  d<=-r //radius of the sphere

Put this in an function called outside(Plane,Bounds) which returns true if the bounds is fully outside the plane then for each of the 6 planes

   bool is_inside_frustum()
  {
    for(Plane plane:frustum_planes)
   {
      if(outside(plane,AABB))
     { 
       return false
     }
    }
    return true;
  }