Disable scrolling on visual viewport size changes (e.g. when zooming)

Question

When zooming in, the visual viewport size changes, and the scrollbars appear. Is there a way to disable scrolling in that situation?

Note that I mean zooming by touchpad (pinch zooming), not by ctrl + mouse wheel.

gif demo

I just want to disable scrolling for the visual viewport, not other possible scrollable elements. (I've already found a way to disable scrolling for all scrollable elements here: How to disable scrolling temporarily?)

And note that overflow:hidden obviously doesn't relate to this problem.

something like this would be ideal, but the problem is that the event isn't defaultPrevented anyway!:

visualViewport.addEventListener('scroll', (event) => {
  event.preventDefault();
}, { passive: false });