WebGL: using framebuffers for picking multiple objects

Question

I am trying to implement picking in WebGL. I have a lot of objects (around 500), and I'd like each one to be allowed to be picked. In order to do that, I did a loop which assigns a unique colour to each object (cf. picking principle):

for (var i = 0, len = objects.length; i < len; i++) {
   framecolors[count++] = i % 256 / 256; //Red
   framecolors[count++] = Math.floor(i/256) / 256; //Green
   framecolors[count++] = Math.floor(i/(256*256)) / 256; //Blue
}

framecolors was then used in a classical buffer to check whether each object had a different shade of red. It worked.

Now, I want to use my objects' original colours, AND a framebuffer with the shades of red in the background. I've been through some code, and I am a bit confused.

Here's what I have tried so far.

Function called before picking:

//Creates texture
colorTexture = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, colorTexture);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.NEAREST);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.RGBA, 400, 400, 0, gl.RGBA, gl.UNSIGNED_BYTE, null);

//Creates framebuffer
fb = gl.createFramebuffer();
gl.bindFramebuffer(gl.FRAMEBUFFER, fb);
gl.framebufferTexture2D(gl.FRAMEBUFFER, gl.COLOR_ATTACHMENT0, gl.TEXTURE_2D, colorTexture, 0);
gl.bindTexture(gl.TEXTURE_2D, colorTexture);
gl.enable(gl.DEPTH_TEST);

gl.bindFramebuffer(gl.FRAMEBUFFER, fb);
gl.clear(gl.DEPTH_BUFFER_BIT);
gl.drawArrays(gl.POINTS, 0, vertexPositionBuffer.numItems);

Function called after:

gl.bindFramebuffer(gl.FRAMEBUFFER, null);
gl.bindTexture(gl.TEXTURE_2D, colorTexture);
gl.drawArrays(gl.POINTS, 0, vertexPositionBuffer.numItems);

As you can probably understand, I am not very comfortable with framebuffers, I just don't really get how they work, even though I read a lot about them. I have no idea how I can link framecolors to the framebuffer. Is there a way?

Thanks, R.

score 3 · Accepted Answer · answered Feb 18 '14 at 04:36

A framebuffer is a collection of attachments (renderbuffers and/or textures). It works just like rendering without a framebuffer. (in fact the browser is internally using a framebuffer to implement WebGL's canvas)

In your case you're missing a few things. You most likely need a depth buffer attached otherwise when you render your scene you won't get zBuffering and the wrong objects will appear in front.

// create renderbuffer
depthBuffer = gl.createRenderbuffer();
gl.bindRenderbuffer(gl.RENDERBUFFER, depthBuffer);

// allocate renderbuffer
gl.renderbufferStorage(
      gl.RENDERBUFFER, gl.DEPTH_COMPONENT16, width, height);  

// attach renderebuffer
gl.framebufferRenderbuffer(
      gl.FRAMEBUFFER,
      gl.DEPTH_ATTACHMENT,
      gl.RENDERBUFFER,
      depthBuffer);

In general you should also check your framebuffer works. After attaching all the attachments you call gl.checkFramebufferStatus

if (gl.checkFramebufferStatus(gl.FRAMEBUFFER) != gl.FRAMEBUFFER_COMPLETE) {
   alert("this combination of attachments does not work");
   return;
}

A framebuffer can be incomplete for any number of reasons. The most common is the attachments are not same size or the GPU doesn't support that combination of attachments. Note: In WebGL certain combinations are required to work but you since you might change the code later to use different formats it's probably still a good idea to check.

You also need to set the viewport by calling gl.viewport whenever you switch framebuffers.

 gl.bindFramebuffer(gl.FRAMEBUFFER, someFramebuffer);
 gl.viewport(0, 0, someFramebufferWidth, someFramebufferHeight);

That includes putting it back when setting things back to the canvas

 gl.bindFramebuffer(gl.FRAMEBUFFER, null);  // render to canvas
 gl.viewport(0, 0, gl.canvas.width, gl.canvas.height);

Finally there's a bug in the code above in that you're only clearing the depth buffer of the framebuffer where you call gl.clear. You want to call

 gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);

Since you're going to read the colors later otherwise old colors will be left over.

Finally, and I guess you know this. You figure out which pixel corresponds to the mouse click and call

 var colorPicked = new Uint8Array(4);
 gl.readPixels(pickX, pickY, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, colorPicked);

Note that when you call gl.readPixels you must have your framebuffer bound with gl.bindFramebuffer or gl.readPixels will read from the canvas.

does gl.clear need to be called twice? e.g. once before drawing the visible objects, and again when drawing to the off-screen framebuffer? — davidkomer, Mar 12 '20 at 21:45

score 2 · Answer 2 · edited May 23 '17 at 10:26

Thanks a lot gman. Just to complete your answer, about how to use framecolors with my framebuffer, it's quite simple. In the shaders, I added a variable dedicated to the framebuffer's colors:

if (!offscreen) {
  gl_FragColor =  normalColors;
}
else {
  gl_FragColor =  frameColors;
}

Now, before determing which object I selected thanks to the readPixels() function, I switch back to the framebuffer using:

function renderFrame() {
  gl.bindFramebuffer(gl.FRAMEBUFFER, fb); //fb = framebuffer
  gl.uniform1i(shaderProgram.uOffscreen, true); //uOffscreen = uniform boolean in shader
  draw(); //function called to draw objects (contains gl.clear and gl.viewport)
}

and similarly, I switch back to the usual buffer after calling the function.

I am trying to display the framebuffer instead of the usual buffer when the mouse is clicked. I am having troubles, but I'll post the solution later if I ever find it.

EDIT : Solution : just remove the depth-buffer (render buffer) associated with the framebuffer. Then display the buffer as explained in : WebGL display framebuffer?

WebGL: using framebuffers for picking multiple objects

2 Answers2

Linked