Three.js detect when object is partially and fully occluded

Question

I'm trying to detect when an object in Three.js is partially and fully occluded (hidden behind) another object.

My current simple solution casts a single ray to the the center of the object:

function getScreenPos(object) {
  var pos = object.position.clone();
  camera.updateMatrixWorld();
  pos.project(camera);
  return new THREE.Vector2(pos.x, pos.y);
}

function isOccluded(object) {
  raycaster.setFromCamera(getScreenPos(object), camera);
  var intersects = raycaster.intersectObjects(scene.children);
  if (intersects[0] && intersects[0].object === object) {
    return false;
  } else {
    return true;
  }
}

However it doesn't account for the object's dimensions (width, height, depth).

Not occluded (because center of object is not behind)

Occluded (because center of object is behind)

View working demo:

https://jsfiddle.net/kmturley/nb9f5gho/57/

Currently thinking I could calculate the object box size, and cast Rays for each corner of the box. But this might still be a little too simple:

var box = new THREE.Box3().setFromObject(object);
var size = box.getSize();

I would like to find a more robust approach which could give partially occluded and fully occluded booleans values or maybe even percentage occluded?

Answer 1

Search Stack Overflow and the Three.js examples for "GPU picking." The concept can be broken down into three basic steps:

1. Change the material of each shape to a unique flat (MeshBasicMaterial) color.
2. Render the scene with the unique materials.
3. Read the pixels of the rendered frame to collect color information.

Your scenario allows you a few caveats.

1. Give only the shape you're testing a unique color--everything else can be black.
2. You don't need to render the full scene to test one shape. You could adjust your viewport to render only the area surrounding the shape in question.
3. Because you only gave a color only to your test part, the rest of the data should be zeroes, making finding pixels matching your unique color much easier.

Now that you have the pixel data, you can determine the following:

• If NO pixels matchthe unique color, then the shape is fully occluded.
• If SOME pixels match the unique color, then the shape is at least partially visible.

The second bullet says that the shape is "at least partially" visible. This is because you can't test for full visibility with the information you currently have.

What I would do (and someone else might have a better solution) is render the same viewport a second time, but only have the test shape visible, which is the equivalent of the part being fully visible. With this information in hand, compare the pixels against the first render. If both have the same number (perhaps within a tolerance) of pixels of the unique color, then you can say the part is fully visible/not occluded.

Answer 2

I managed to get a working version for WebGL1 based on TheJim01's answer!

First create a second simpler scene to use for calculations:

pickingScene = new THREE.Scene();
pickingTextureOcclusion = new THREE.WebGLRenderTarget(window.innerWidth / 2, window.innerHeight / 2);
pickingMaterial = new THREE.MeshBasicMaterial({ vertexColors: THREE.VertexColors });
pickingScene.add(new THREE.Mesh(BufferGeometryUtils.mergeBufferGeometries([
  createBuffer(geometry, mesh),
  createBuffer(geometry2, mesh2)
]), pickingMaterial));

Recreate your objects as Buffer Geometry (faster for performance):

function createBuffer(geometry, mesh) {
  var buffer = new THREE.SphereBufferGeometry(geometry.parameters.radius, geometry.parameters.widthSegments, geometry.parameters.heightSegments);
  quaternion.setFromEuler(mesh.rotation);
  matrix.compose(mesh.position, quaternion, mesh.scale);
  buffer.applyMatrix4(matrix);
  applyVertexColors(buffer, color.setHex(mesh.name));
  return buffer;
}

Add a color based on the mesh.name e.g. an id 1, 2, 3, etc

function applyVertexColors(geometry, color) {
  var position = geometry.attributes.position;
  var colors = [];
  for (var i = 0; i < position.count; i ++) {
    colors.push(color.r, color.g, color.b);
  }
  geometry.setAttribute('color', new THREE.Float32BufferAttribute(colors, 3));
}

Then during the render loop check the second scene for that texture, and match pixel data to the mesh name:

function isOccludedBuffer(object) {
  renderer.setRenderTarget(pickingTextureOcclusion);
  renderer.render(pickingScene, camera);
  var pixelBuffer = new Uint8Array(window.innerWidth * window.innerHeight);
  renderer.readRenderTargetPixels(pickingTextureOcclusion, 0, 0, window.innerWidth / 2, window.innerHeight / 2, pixelBuffer);
  renderer.setRenderTarget(null);
  return !pixelBuffer.includes(object.name);
}

You can view the WebGL1 working demo here:

https://jsfiddle.net/kmturley/nb9f5gho/62/

One caveat to note with this approach is that your picking scene needs to stay up-to-date with changes in your main scene. So if your objects move position/rotation etc, they need to be updated in the picking scene too. In my example the camera is moving, not the objects so it doesn't need updating.

For WebGL2 we will have a better solution:

https://tsherif.github.io/webgl2examples/occlusion.html

But this is not supported in all browsers yet:

https://www.caniuse.com/#search=webgl