How to correctly pass mouse coordinates to WebGL?

Question

I want to pass canvas mouse coordinates to a function that interactively generates a circle with the mouse's coordinates as its center. Therefore, I'm using the following function to normalize:

var mousePositionX = (2*ev.clientX/canvas.width) - 1;
var mousePositionY = (2*ev.clientY/(canvas.height*-1)) + 1;

However, this works fine only for the screen center. When moving the mouse around the cursor is not located in the circle's center any more: see the picture here

The farther the mouse cursor removes from the screen center, the more it is dislocated from the circle's center. Here's some relevant code:

HTML

  body {
    border: 0;
    margin: 0;
  }
  /* make the canvas the size of the viewport */
  canvas {
    width: 100vw;
    height: 100vh;
    display: block;
  }
  ...
  <body onLoad="main()">
        <canvas id="glContext"></canvas>
  </body>

SHADER

<script id="vShaderCircle" type="notjs">
    attribute vec4 a_position;
    uniform mat4 u_viewMatrix;

    void main(){
        gl_Position = u_viewMatrix * a_position;
    }
</script>

JS

function main(){

    // PREPARING CANVAS AND WEBGL-CONTEXT
    var canvas = document.getElementById("glContext");
    var gl_Original = initWebGL(canvas);
    var gl = WebGLDebugUtils.makeDebugContext(gl_Original);

    resize(canvas);
    gl.viewport(0, 0, canvas.width, canvas.height);
    // ----------------------------------
    ...
    // MATRIX SETUP
    var viewMatrix = new Matrix4();
      viewMatrix.setPerspective(30, canvas.width/canvas.height, 1, 100);
      viewMatrix.lookAt(0, 0, 5, 0, 0, 0, 0, 1, 0);
    // ----------------------------------
    canvas.addEventListener("mousemove", function(){stencilTest(event)});

    function stencilTest(ev){
        var mousePositionX = (2*ev.clientX/canvas.width) - 1;
        var mousePositionY = (2*ev.clientY/(canvas.height*(-1))) + 1;
        ...
        ...
        drawCircle(..., mousePositionX, mousePositionY, viewMatrix);
        ...
        drawCube(...);
    }
}

How can I resolve this?

You probably need to post some code. You mentioned you have a view matrix. If you have a view matrix in some comment in whic case you aren't working in WebGL's coordinate system which is clip space. Instead you're working in some other coordinate system you defined and we can't help you convert the mouse coordinates to that space unless you show us teh codez — gman, Feb 18 '17 at 13:28
@gman So I provided a bit more code now. Can you figure out the problem by that? The individual functions (e.g. matrix functions) do what they're supposed to do. I tried your `getRelativeMousePosition()` functions as well, but without success... — Muad, Feb 18 '17 at 15:26

score 13 · Accepted Answer · edited May 23 '17 at 12:17

This is actually a far more complicated issue than it sounds. Is your canvas's display size the same as its drawing buffer? Do you have a border on your canvas?

Here's some code that will give you a canvas relative pixel coordinate assuming you don't have a border or any padding on your canvas.

function getRelativeMousePosition(event, target) {
  target = target || event.target;
  var rect = target.getBoundingClientRect();

  return {
    x: event.clientX - rect.left,
    y: event.clientY - rect.top,
  }
}

// assumes target or event.target is canvas
function getNoPaddingNoBorderCanvasRelativeMousePosition(event, target) {
  target = target || event.target;
  var pos = getRelativeMousePosition(event, target);

  pos.x = pos.x * target.width  / target.clientWidth;
  pos.y = pos.y * target.height / target.clientHeight;

  return pos;  
}

To convert that to a WebGL coordinate

  var pos = getRelativeMousePosition(event, target);
  const x = pos.x / gl.canvas.width  *  2 - 1;
  const y = pos.y / gl.canvas.height * -2 + 1;

Working example:

function getRelativeMousePosition(event, target) {
  target = target || event.target;
  var rect = target.getBoundingClientRect();

  return {
    x: event.clientX - rect.left,
    y: event.clientY - rect.top,
  }
}

// assumes target or event.target is canvas
function getNoPaddingNoBorderCanvasRelativeMousePosition(event, target) {
  target = target || event.target;
  var pos = getRelativeMousePosition(event, target);

  pos.x = pos.x * target.width  / target.clientWidth;
  pos.y = pos.y * target.height / target.clientHeight;

  return pos;  
}

const vs = `
attribute vec4 position;
void main() {
  gl_Position = position;
  gl_PointSize = 20.;
}
`;
const fs = `
void main() {
  gl_FragColor = vec4(1,0,1,1);
}
`;
const gl = document.querySelector("canvas").getContext("webgl");
// compiles and links shaders and assigns position to location 
const program = twgl.createProgramFromSources(gl, [vs, fs]);
const positionLoc = gl.getAttribLocation(program, "position");

window.addEventListener('mousemove', e => {

  const pos = getNoPaddingNoBorderCanvasRelativeMousePosition(e, gl.canvas);

  // pos is in pixel coordinates for the canvas.
  // so convert to WebGL clip space coordinates
  const x = pos.x / gl.canvas.width  *  2 - 1;
  const y = pos.y / gl.canvas.height * -2 + 1;

  gl.clear(gl.COLOR_BUFFER_BIT);
  gl.useProgram(program);
  // only drawing a single point so no need to use a buffer
  gl.vertexAttrib2f(positionLoc, x, y);
  gl.drawArrays(gl.POINTS, 0, 1);
});

canvas { 
  display: block;
  width: 400px;
  height: 100px;
}
div {
  display: inline-block;
  border: 1px solid black;
}

<div><canvas></canvas></div>
<p>move the mouse over the canvas</p>
<script src="https://twgljs.org/dist/3.x/twgl-full.min.js"></script>

Notice there's no matrices involved. If you're using matrices then you've defined your own space, not WebGL's space which is always clip space. In that case you either need to multiply by the inverse of your matrices and pick whatever Z value you want between -1 and +1. That way when your position is multiplied by the matrices used in your shader it will reverse the position back into the correct webgl clip space coordinates. Or, you need to get rid of your matrices or set them in the identity.

Here's an example, note I don't have/know your math library so you'll have to translate to your's

function getRelativeMousePosition(event, target) {
  target = target || event.target;
  var rect = target.getBoundingClientRect();

  return {
    x: event.clientX - rect.left,
    y: event.clientY - rect.top,
  }
}

// assumes target or event.target is canvas
function getNoPaddingNoBorderCanvasRelativeMousePosition(event, target) {
  target = target || event.target;
  var pos = getRelativeMousePosition(event, target);

  pos.x = pos.x * target.width  / target.clientWidth;
  pos.y = pos.y * target.height / target.clientHeight;

  return pos;  
}

const vs = `
attribute vec4 position;
uniform mat4 matrix;
void main() {
  gl_Position = matrix * position;
}
`;
const fs = `
void main() {
  gl_FragColor = vec4(1,0,0,1);
}
`;
const m4 = twgl.m4;
const gl = document.querySelector("canvas").getContext("webgl");
const programInfo = twgl.createProgramInfo(gl, [vs, fs]);
const bufferInfo = twgl.primitives.createSphereBufferInfo(gl, .5, 12, 8);

window.addEventListener('mousemove', e => {

  const pos = getNoPaddingNoBorderCanvasRelativeMousePosition(e, gl.canvas);

  // pos is in pixel coordinates for the canvas.
  // so convert to WebGL clip space coordinates
  const x = pos.x / gl.canvas.width  *  2 - 1;
  const y = pos.y / gl.canvas.height * -2 + 1;
  
  // use a projection and view matrix
  const projection = m4.perspective(
     30 * Math.PI / 180, 
     gl.canvas.clientWidth / gl.canvas.clientHeight, 
     1, 
     100);
  const camera = m4.lookAt([0, 0, 15], [0, 0, 0], [0, 1, 0]);
  const view = m4.inverse(camera);
  const viewProjection = m4.multiply(projection, view);
  
  // pick a clipsace Z value between -1 and 1 
  // we'll zNear to zFar and convert back to clip space
  const viewZ = -5;  // 5 units back from the camera
  const clip = m4.transformPoint(projection, [0, 0, viewZ]);
  const z = clip[2];
  
  // compute the world space position needed to put the sphere
  // under the cursor at this clipspace position
  const inverseViewProjection = m4.inverse(viewProjection);
  const worldPos = m4.transformPoint(inverseViewProjection, [x, y, z]);

  // add that world position to our matrix
  const mat = m4.translate(viewProjection, worldPos);

  gl.clear(gl.COLOR_BUFFER_BIT);
  gl.useProgram(programInfo.program);
  
  twgl.setBuffersAndAttributes(gl, programInfo, bufferInfo);
  twgl.setUniforms(programInfo, {
    matrix: mat,
  });
  gl.drawElements(gl.LINES, bufferInfo.numElements, gl.UNSIGNED_SHORT, 0);
});

canvas { 
  display: block;
  width: 400px;
  height: 100px;
}
div {
  display: inline-block;
  border: 1px solid black;
}

<div><canvas></canvas></div>
<p>move the mouse over the canvas</p>
<script src="https://twgljs.org/dist/3.x/twgl-full.min.js"></script>

Also note I deliberately made the canvas's display size not match it's drawing buffer size to show the math works.

That's it, thank you! Works with your code, though I still don't get the point why it does (I'm still new to all that matrix-math stuff...). As far as I understood, my mouse coordinates are passed in clip space coordinates, but they have to be in world space right? So we're going to transform them back by inverting the projection and view matrices. But what's the deal with that z-Value thing? My circle is 2d and it works also when passing 0 as `z`. Also, this line `const view = m4.inverse(camera);` seems not to have any effect on the output. Could you please explain this plainly again ? :) — Muad, Feb 20 '17 at 13:45
Picking a Z is for 3d. You need to decide how far away from the camera you want the object to be in the world so that you can then compute the correct world X, and Y such that your object will appear under the mouse. the `const view = m4.inverse(camera);` makes a view matrix because my `m4.lookAt` makes a camera matrix. Your math library's `lookAt` function probably returns a view matrix, not a camera matrix. I find my lookAt function more useful. [See this](https://webglfundamentals.org/webgl/lessons/webgl-3d-camera.html) — gman, Feb 20 '17 at 22:00

score 1 · Answer 2 · answered Feb 18 '17 at 02:58

1

Just bind your mousemove event to the canvas itself and use offsetX and offsetY

var mouseX = (e.offsetX / canvas.clientWidth)*2-1;
var mouseY = ((canvas.clientHeight - e.offsetY) / canvas.clientHeight)*2-1;

Note that this all depends on what transforms you do in your shaders.

answered Feb 18 '17 at 02:58

LJᛃ

7,655
2
24
35

Unfortunately this has the same effect as my version (at least in Chrome; cannot try another browser). In the shader I am only multiplying the vertices by a view matrix (eye position: 0, 0, 5 | look at: 0, 0, 0) so it shouldn't affect the x/y representation. I just want to have the mouse cursor always in the circle's center. – Muad Feb 18 '17 at 10:17
To me your image looks like its pixel shader stuff, without posting the pixel shader where you actually use the coordinates its hard to tell whats wrong. – LJᛃ Feb 18 '17 at 20:49

score 0 · Answer 3 · edited Jan 23 '18 at 19:37

0

Assuming this is called in a mousemove callback event and canvas is defined as a proper reference to an HTML CANVAS element, the relative position of the mouse pointer to the canvas space should be:

var rect = gl.canvas.getBoundingClientRect();
var mousePositionX = ev.clientX - rect.left;
var mousePositionY = ev.clientY - rect.top;

To convert from pixel coordinates to WebGL coordinate system:

var rect = gl.canvas.getBoundingClientRect();
var x = (ev.clientX - rect.left) / canvas.width *  2 - 1;
var y = (ev.clientY - rect.top) / canvas.height * -2 + 1;

edited Jan 23 '18 at 19:37

James

69
1
8

answered Feb 18 '17 at 01:15

Javier Rey

1,539
17
27

I tried this but it doesn't return coordinates in WebGL coordinate system [-1 -- 1] but only pixel coordinates. I have to convert the mouse's pixel screen coordinates into WebGL coordinates... – Muad Feb 18 '17 at 10:04

How to correctly pass mouse coordinates to WebGL?

3 Answers3