viewpager2: java.lang.IllegalStateException: Expected the adapter to be 'fresh' while restoring state

Question

Here is the more of stacktrace

    java.lang.IllegalStateException: Expected the adapter to be 'fresh' while restoring state.
        at androidx.viewpager2.adapter.FragmentStateAdapter.restoreState(FragmentStateAdapter.java:557)
        at androidx.viewpager2.widget.ViewPager2.restorePendingState(ViewPager2.java:349)
        at androidx.viewpager2.widget.ViewPager2.setAdapter(ViewPager2.java:462)
        at com.example.HomeFragmentContainer.onResume(HomeFragmentContainer.kt:76)
        at androidx.fragment.app.Fragment.performResume(Fragment.java:2747)

I think it's the same problem as Expected the adapter to be 'fresh' while restoring state

HomPagerAdaptor

class HomPagerAdaptor(fragment: Fragment) : FragmentStateAdapter(fragment) {
    override fun getItemCount() = 2;
    override fun createFragment(position: Int) = when (position) {
        0 -> TimelineContainer()
        else -> HomeFragment()
    }
}

I'm using adaptor as

class HomeFragmentContainer : DaggerFragment() {

    private val adapter by lazy {  HomPagerAdaptor(this) }
...
    override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
        super.onViewCreated(view, savedInstanceState)
        binding!!.vpContainer.adapter = adapter
    }
...
    override fun onDestroyView() {
        binding?.vpContainer?.adapter = null
        binding = null
        super.onDestroyView()
    }
}

library versions:

 androidx.viewpager2:viewpager2:1.1.0-alpha01
 androidx.recyclerview:recyclerview:1.2.0-alpha03

I did little bit of digging and found that when I set adaptor to null the underlying recyclerview's views are recycled and the adaptor trie to remove fragments by calling removeFragment()

private void removeFragment(long itemId) {
    Fragment fragment = mFragments.get(itemId);

    ...
    if (shouldDelayFragmentTransactions()) {
          mHasStaleFragments = true;
          return;
     }
     try {
            mFragmentManager.beginTransaction().remove(fragment).commitNow();
            mFragments.remove(itemId);
        } finally {
            mFragmentEventDispatcher.dispatchPostEvents(onPost);
        }
}

  boolean shouldDelayFragmentTransactions() {
        return mFragmentManager.isStateSaved();
    }

// in FragmentManager
   public boolean isStateSaved() {
        // See saveAllState() for the explanation of this.  We do this for
        // all platform versions, to keep our behavior more consistent between
        // them.
        return mStateSaved || mStopped;
    }

But the shouldDelayFragmentTransactions() returns true because the fragment is already stopped and the fragments are not removed from adaptor. Later when the fragment pops back and sets the adapter in onViewCreated() the adaptor tries to restore state, but before doing that, it checks if there are any stale fragments in adaptor as

  @Override
    public final void restoreState(@NonNull Parcelable savedState) {
        if (!mSavedStates.isEmpty() || !mFragments.isEmpty()) {
            throw new IllegalStateException(
                    "Expected the adapter to be 'fresh' while restoring state.");
        }

It fixes the problem when I set adaptor to null in onResume() or onStop() but the viewpager's state is not restored later and defeats the whole purpose of using FragmentStateAdapter.

Edit: added more info