Convert handwritten input to a collection of geometry on a canvas

Question

I am developing a Windows app that runs on computers with a pen input. I have a canvas object that receives an event every time the pen has moved to know where exactly it is on the canvas, the pressure on the pen, etc etc, and the canvas itself can display vector graphics (eg lines, curves, and other shapes).

How do I convert this pen movement into a vector on screen in real time, also accounting for pressure, tilt, etc? My goal is for it to feel like writing in other pen based note taking apps, like OneNote or Goodnotes.

How would I deal with the user drawing many strokes, such as writing long hand written text, where many lines (presumably separate objects) will have to be drawn, without damaging performance?

Martin Schaer · Accepted Answer · 2022-10-28T18:57:12.840

I would start by making it simple. Have a loop (requestAnimationFrame + a delta time variable) and on each tick get the current mouse position and draw a line from there to the last position, and repeat. Take into account when the mouse is not clicked (pen not on the surface).

Play with the delta time in the loop to have more or less resolution on your handwritten to vector transformation.

After experimenting with this you can start adding more features, like instead of just drawing the line, storing it as a vector in some structure. That will allow you to redraw the canvas, after zooming in our out, exporting a file, selecting objects, changing colors, etc…

This is a minimal example (try it in codepen):

const canvas = document.querySelector("canvas");
const ctx = canvas.getContext("2d");

let penIsDown = false;
let drawing = false;
let x,
  y = 0;
let lastTime = 0;

ctx.lineWidth = 10;
ctx.fillStyle = "white";
ctx.fillRect(0, 0, 600, 600);

canvas.addEventListener("mousedown", () => {
  penIsDown = true;
});

canvas.addEventListener("mouseup", () => {
  penIsDown = false;
});

canvas.addEventListener("mousemove", (e) => {
  x = e.clientX;
  y = e.clientY;
});

const loop = () => {
  const now = Date.now();
  const deltaTime = now - lastTime;
  if (deltaTime >= 10) {
    if (!penIsDown) {
      if (drawing) {
        ctx.stroke();
        drawing = false;
      } else {
        // nothing
      }
    } else {
      if (drawing) {
        ctx.lineTo(x, y);
        ctx.stroke();
      } else {
        ctx.moveTo(x, y);
        ctx.beginPath();
        drawing = true;
      }
    }
    lastTime = now;
  }
  requestAnimationFrame(loop);
};

loop();

What if the user decides to draw a curved line, would I have to somehow process the points, or do I just sample the pen more frequently? — frogstair, Oct 29 '22 at 13:03
@frogstair yes, more samples = more detail. You could also have a worker script optimizing the generated vectors. Similar to how an SVG compressor works. If you are interested in real performance improvements, you should also consider using the webgl2 context of the canvas API, instead the the 2D context. That way you get help from the GPU and shaders — Martin Schaer, Oct 29 '22 at 17:10

Convert handwritten input to a collection of geometry on a canvas

1 Answers1