React state update during control change breaks OrbitControls from three

Question

I am using react three with drei orbitControls and try to detect wether the camera is above or below zero, while moving the camera around. Currently I'm using onEnd listener to trigger my code in my ParentComponent. If I change a state of my parent component like so everything works as expected. Unfortunately if I start another change of orbit controls before ending another and change my parents state, orbit controls will break. E.g. I rotate the camera and hold down my left mouse button, than get below zero with my camera and the scroll with my mouse wheel at the same time, Orbit controls no longer works. This is especially painful when using touch. As soon as a second finger touches and I cross the ground orbit control breaks.

This is my Controls Component:

const Controls = forwardRef(({ handleCamera }, ref) => {
  const { gl, camera } = useThree();
  const controls = useRef(null);

  useImperativeHandle(ref, () => ({
    getCamera() {
      return camera
    },
  }))

  const handleChange = (e) => {
      handleCamera(camera); 
  }

  return (
    <OrbitControls ref={controls}
      onEnd={(e) => handleChange(e)}
    />
  );
});

export default Controls;

And this is my parent:

function App() {  
  const handleCamera = (camera) => {
    setCameraBelowSurface(camera.position.y <= 0 ? true : false);
  };
  return
    <Canvas >
      <Suspense>
        <Controls
          ref={controlsRef}
          handleCamera={handleCamera}
        />          
      </Suspense>
    </Canvas>    
}

export default App;