Read vertex positions as pixels in Three.js

Question

In a scenario where vertices are displaced in the vertex shader, how to retrieve their transformed positions in WebGL / Three.js?

Other questions here suggest to write the positions to a texture and then read the pixels, but the resulting value don't seem to be correct.

In the example below the position is passed to the fragment shader without any transformations:

// vertex shader
varying vec4 vOut;

void main() {
    gl_Position = vec4(position, 1.0);
    vOut = vec4(position, 1.0);
}

// fragment shader
varying vec4 vOut;

void main() {
    gl_FragColor = vOut;
}

Then reading the output texture, I would expect pixel[0].r to be identical to positions[0].x, but that is not the case.

Here is a jsfiddle showing the problem: https://jsfiddle.net/brunoimbrizi/m0z8v25d/2/

What am I missing?

Your expectation is wrong. When you rasterize the `gl_Position` result you clip your mesh. Pixel[0] in the buffer represents something else, far from your actual coordinate. The vertex is always going to be at some X,Y (width==4 / x == -2). On the screen, you see the mesh at -1,-1 interpolated, your -2,-2 is thus off the screen. If your width and height here are larger than two, than the last pixel in t his array should always be 1,1. — pailhead, Aug 06 '18 at 20:31
i am a little bit confused why the numbers are different in the lower left corner but i think it depends on your resolution and how you've set up that view — pailhead, Aug 06 '18 at 21:11
@pailhead Are sure the mesh is getting clipped? I also tried multiplying the position by model-view and projection matrices (line 32 `gl_Position = modelViewMatrix * projectionMatrix * vec4(position, 1.0);`) which gives a result between -2 and 2, but still off. — imbrizi, Aug 06 '18 at 22:05
Well, think of it this way. Make your plane with size 1,1. `gl_Position` would then get -0.5,0.5 range and you should see a plane filling only the center of the screen. If you make it anything larger than 2,2 you'll get vertices outside the clip space. I think the weird number is coming from the interpolation that happens at that first pixel, but i'm a bit confused by the fiddle. — pailhead, Aug 07 '18 at 00:45

score 1 · Accepted Answer · answered Aug 06 '18 at 23:59

1

Solved. Quite a few things were wrong with the jsfiddle mentioned in the question.

width * height should be equal to the vertex count. A PlaneBufferGeometry with 4 by 4 segments results in 25 vertices. 3 by 3 results in 16. Always (w + 1) * (h + 1).
The positions in the vertex shader need a nudge of 1.0 / width.
The vertex shader needs to know about width and height, they can be passed in as uniforms.
Each vertex needs an attribute with its index so it can be correctly mapped.
Each position should be one pixel in the resulting texture.
The resulting texture should be drawn as gl.POINTS with gl_PointSize = 1.0.

Working jsfiddle: https://jsfiddle.net/brunoimbrizi/m0z8v25d/13/

answered Aug 06 '18 at 23:59

imbrizi

3,788
1
25
26

uh huh, i don't think that this is actually the best answer for this. – pailhead Aug 07 '18 at 00:46
sorry i meant to have a fiddle by now. You can do this without the points but it might be more involved. – pailhead Aug 07 '18 at 00:59
you're going to need a really large texture if you space out your points like this, and if you start to group them you'll have to write more logic. – pailhead Aug 07 '18 at 01:06
I've added another answer to clarify what i mean with the comments. – pailhead Aug 07 '18 at 01:19
why it's 0.75 in your screen shot and not 1 is because of the interpolation (i didn't solve that in the fiddle), but why it's 1 and not 2 is because of clipping, (solved in the fiddle/answer) – pailhead Aug 07 '18 at 01:22

score 1 · Answer 2 · answered Aug 07 '18 at 01:18

You're not writing the vertices out correctly.

https://jsfiddle.net/ogawzpxL/

First off you're clipping the geometry, so your vertices actually end outside the view, and you see the middle of the quad without any vertices.

You can use the uv attribute to render the entire quad in the view.

gl_Position = vec4( uv * 2. - 1. , 0. ,1.);

Everything in the buffer represents some point on the quad. What seems to be tricky is when you render, the pixel will sample right next to your vertex. In the fiddle i've applied an offset to the world space thing by how much it would be in pixel space, and it didn't really work.

The reason why it seems to work with points is that this is all probably wrong :) If you want to transform only the vertices, then you need to store them properly in the texture. You can use points for this, but ideally they wouldn't be spaced out so much. In your scenario, they would fill the first couple of rows of the texture (since it's much larger than it could be).

You might start running into problems as soon as you try to apply this to something other than PlaneGeometry. In which case this problem has to be broken down.

The resulting texture is just data, it doesn't need to be rendered to the screen. I used gl.POINTS based on the technique described in these two answers: https://stackoverflow.com/questions/40198663/is-it-possible-to-get-the-actual-value-of-a-vertex and https://stackoverflow.com/questions/29053870/retrieve-vertices-data-in-three-js. The points seem 'spaced out' because the fiddle renders a 5x5 texture in a viewport of 128x128. It is not wrong, it works perfectly. Thank you for your help. — imbrizi, Aug 07 '18 at 10:15

Read vertex positions as pixels in Three.js

2 Answers2