React Native PanResponder jumping the position to center

Question

So I am trying to create component like TikTok page swipe up down. I am able to slide up down using button successfully. But when I try to use pan responder and try to drag it always reset the list position to center.

Here is snack example. Try clicking next and previous it work great. But if you try to drag, it reset the position of the list to center on drag start. But upon release it get set correctly.

I guess the problem is on onPanResponderMove but i'm not sure how I can fix that.

https://snack.expo.dev/v7Lr0XIoM

const [activeCard, setActiveCard] = useState<number>(1);

  const pan = useRef(new Animated.ValueXY()).current;

  const panResponder = PanResponder.create({
    onMoveShouldSetPanResponder: () => true,
    onPanResponderMove: (evt, gestureState) => {
      pan.setValue({ x: gestureState.dx, y: gestureState.dy });
    },
    onPanResponderRelease: (evt, gestureState) => {
      console.log("release");
      if (gestureState.dy < -1) {
        slideUp();
      } else if (gestureState.dy > 1) {
        slideDown();
      } else {
        // Did not dragged far don't handle
      }
    },
  });

  const slideUp = () => {
    Animated.spring(pan, {
      toValue: { x: 0, y: screenHeight * activeCard * -1 },
      useNativeDriver: true,
    }).start();

    setActiveCard(activeCard + 1);
  };

  const slideDown = () => {
    Animated.spring(pan, {
      toValue: { x: 0, y: (activeCard - 2) * screenHeight * -1 },
      useNativeDriver: true,
    }).start();

    setActiveCard(activeCard - 1);
  };