useRef not working as expected, in react-three-fiber. Ref elements not unmounting?

Question

I am trying to animate this group of trees inside component. But the treeRefs keeps on increasing by 2 times on each frame and results in very wonky behaviour filled with glitches. Logic - I have components which is a 3D Mesh. I am spwaning these trees on a random position and moving them along z-axis and when the mesh hits z = 0 I change it's position around some starting point. This way trees should simulate moving animation. I am using useRef to keep track of each component and animate it's position. Also I have ScrollControls wrapped around and when I scroll this animation starts acting weird and useRefs starts to multiplying. Example logs before scroll enter image description here

Example logs after scroll enter image description here

import { Detailed, useGLTF } from '@react-three/drei';
import { useFrame } from '@react-three/fiber';
import React, { createRef, useEffect, useMemo, useRef, useState } from 'react';
import { useStore } from './store/store';

function map_range(value, low1, high1, low2, high2) {
    return low2 + ((high2 - low2) * (value - low1)) / (high1 - low1);
}

export const Forest = () => {
    const treeRefs = useRef([]);
    const [positions, setPositions] = useState([]);
    const { acceleration, setAcceleration, currentAction, setCurrentAction } = useStore();

    // Move each tree towards starting when reaches the end line
    useFrame((state, delta) => {
        console.log('Delta', delta);
        console.log('Length ', treeRefs?.current?.length);
        console.log('Length Pos ', positions?.length);
        treeRefs?.current?.map((TreeMesh) => {
            // console.log('TreeMesh', TreeMesh.position.z);

            if (TreeMesh?.position?.z) {
                // console.log('Inside useFrame ', currentAction);
                TreeMesh.position.z = TreeMesh.position.z - delta;
                if (TreeMesh.position.z <= 0) {
                    TreeMesh.position.z = Math.random() * 50;
                }
            }
            return TreeMesh;
        });
    });

    useEffect(() => {
        // Create 30 objects with random position and rotation data
        const positions = [...Array(30)].map(() => {
            const randomPositionAmplitude = map_range(Math.random(), 0, 1, 2, 10);

            const randomPositionDirection = Math.sign(map_range(Math.random(), 0, 1, -1, 1));

            const xPosition = randomPositionAmplitude * randomPositionDirection;

            return {
                position: [xPosition, 1.7, Math.random() * 50],
                // rotation: [Math.random() * Math.PI * 2, Math.random() * Math.PI * 2, Math.random() * Math.PI * 2],
                rotation: [0, 0, 0],
                ref: createRef(),
            };
        });
        setPositions(positions);
    }, []);

    return (
        <>
            {positions.map((props, i) => (
                <Tree innerRef={(el) => treeRefs.current.push(el)} key={i} {...props} />
            ))}
        </>
    );
};

function Tree({ innerRef, ...props }) {
    // This will load 4 GLTF in parallel using React Suspense

    const { nodes, materials, animations, scene } = useGLTF('/Tree.glb');
    const copiedScene = useMemo(() => scene.clone(), [scene]);
    // By the time we're here these GLTFs exist, they're loaded
    // There are 800 instances of this component, but the GLTF data is cached and will be re-used ootb
    return (
        <Detailed ref={innerRef} distances={[300]} {...props}>
            {/* All we need to do is dump them into the Detailed component and define some distances
          Since we use a JSX mesh to represent each bust the geometry is being re-used w/o cloning */}

            <primitive object={copiedScene} />

            {/* <group /> */}
        </Detailed>
    );
}

What can be done to fix this?

Not sure what to try, I have other workarounds for this. But I want to know what's wrong with my code.