Event handling stuck after removing element

Question

I have a button that changes icon when you drag the button down via Touch Event. Event handling logic is bound to Button containing the Icon component. When this component gets changed the whole event handling gets stuck.

I thought that event would bubble through the icon and will work normally at the level of component actually handling the events, but sadly when icon is replaced this is stuck.

Expected behavior: When you drag down the button down the icons should change all the way from icon 1 to 3, when you release icon changes back to icon 1

export const DragButton: React.FC = () => {
  const buttonRef = useRef<HTMLDivElement>(null);
  const [state, setState] = useState<State>("Idle");
  const [icon, setIcon] = useState(<AccessAlarms />);
  const [mousePos, setMousePos] = useState<{ x: number; y: number }>({
    x: 0,
    y: 0
  });

  const onTouchStart = (e: React.TouchEvent) => {
    setState("Pressed");
  };

  const onTouchEnd = (event: React.TouchEvent) => {
    setState("Released");
    setIcon(<AccessAlarms />);
  };

  const onTouchMove = (event: React.TouchEvent) => {
    setMousePos({ x: event.touches[0].clientX, y: event.touches[0].clientY });
    if (state == "Pressed") {
      if (event.touches[0].clientY > 200) {
        setIcon(<Accessibility />);
      } else if (event.touches[0].clientY > 100) {
        setIcon(<ChangeHistory />);
      } else {
        setIcon(<AccessAlarms />);
      }
    }
  };

  return (
    <ButtonDiv
      ref={buttonRef}
      onTouchStart={(event) => onTouchStart(event)}
      onTouchEnd={(event) => onTouchEnd(event)}
      onTouchCancel={(event) => onTouchEnd(event)}
      onTouchMove={(event) => onTouchMove(event)}
    >
      <Button>{icon}</Button>
      {mousePos.y}
    </ButtonDiv>
  );
};

What can be causing this behavior? It seems that removal of element where event originated is the problem, how could this be solved?

Sandbox is here, it is needed to use either Touchscreen or switch to mobile view in dew tools https://codesandbox.io/s/draggable-button-58vw2?file=/src/DragButton.tsx