RecyclerView.Adapter.StateRestorationPolicy.PREVENT_WHEN_EMPTY stopped working

Question

Our app has a list of items, and an associated recycler view with it:

class Adapter(val clickListener: PreviewListener) :
    ListAdapter<DataItem, RecyclerView.ViewHolder>(EntryDiffCallback()) {

The adapter is set up in the onViewCreatedMethod:

private var previewAdapter: PreviewAdapter? = null

       if (previewAdapter == null) {
            previewAdapter =
                PreviewAdapter(
                    PreviewListener { info ->
                        previewViewModel.updateCurrentInfo(info)
                        findNavController()
                            .navigateSafely(
                                PreviewFragmentDirections
                                    .actionToExerciseFragment())
                    })
            previewAdapter?.stateRestorationPolicy =
                RecyclerView.Adapter.StateRestorationPolicy.PREVENT_WHEN_EMPTY
        }
        binding.previewListView.adapter = previewAdapter

        postponeEnterTransition()

The list is populated in the following manner:

previewViewModel.listItems.observe(
            viewLifecycleOwner,
            Observer { list ->
                previewAdapter?.submitList(list)
                (view.parent as? ViewGroup)?.doOnPreDraw { startPostponedEnterTransition() }
            })

When the user clicks on an item, it is taken to a new fragment, and when they press back, they end up in the main fragment again. However, the recycler view does not retain the position.

I have extensively read the internet: https://medium.com/androiddevelopers/restore-recyclerview-scroll-position-a8fbdc9a9334 and stack over flow: Maintain/Save/Restore scroll position when returning to a ListView Refreshing data in RecyclerView and keeping its scroll position RecyclerView store / restore state between activities

And tried some really funky solutions, yet none of these retain the recycler view in the old position when the user is returning back. (eg. PREVENT_WHEN_EMPTY, using SavedInstanceState on the layout manager, and remembering the scroll position).

I'm also using shared element to animate transitions

override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        sharedElementEnterTransition =
            MaterialContainerTransform().apply {
                duration = resources.getInteger(R.integer.reply_motion_duration_large).toLong()
                scrimColor = Color.TRANSPARENT
                interpolator = DecelerateInterpolator()
                setAllContainerColors(requireContext().getColorFromAttr(R.attr.colorSurface))
            }
    }

I clean up the data on destroy view, and destroy, but removing these doesn't seem to do anything except cause CanaryLeaks:

    override fun onDestroyView() {
        super.onDestroyView()
        binding.previewListView.adapter = null
        _binding = null
    }

    override fun onDestroy() {
        super.onDestroy()

        previewAdapter = null
    }

Any suggestions? Thank you