Allow recycling items with RecyclerView within NestedScrollView

Question

I have the following layout:

   <?xml version="1.0" encoding="utf-8"?>
<android.support.design.widget.CoordinatorLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent">


    <android.support.design.widget.AppBarLayout
        android:id="@+id/expert_appbar"
        android:layout_width="match_parent"
        android:layout_height="wrap_content">

        <android.support.design.widget.CollapsingToolbarLayout
            android:id="@+id/expert_collapsing"
            android:layout_width="match_parent"
            android:layout_height="@dimen/expert_appbar_height"
            android:fitsSystemWindows="true"
            android:theme="@style/AppTheme.Dark"
            app:contentScrim="@drawable/topbar_gradient"
            app:expandedTitleMarginEnd="64dp"
            app:expandedTitleMarginStart="48dp"
            app:expandedTitleTextAppearance="@style/TextAppearance.AppCompat.Display1"
            app:layout_scrollFlags="scroll|exitUntilCollapsed|snap">

            <ImageView
                android:id="@+id/expert_appbar_backdrop"
                android:layout_width="match_parent"
                android:layout_height="match_parent"
                android:background="@color/privacy_gray"
                android:fitsSystemWindows="true"
                android:foreground="@drawable/bg_gradient"
                android:scaleType="centerCrop"
                app:layout_collapseMode="parallax"
                tools:src="@drawable/bar" />

            <android.support.v7.widget.Toolbar
                android:id="@+id/expert_toolbar"
                android:layout_width="match_parent"
                android:layout_height="?android:attr/actionBarSize"
                android:theme="@style/AppTheme"
                app:layout_collapseMode="pin"
                app:popupTheme="@style/LightToolbarPopup">

                <Space
                    android:id="@+id/circle_collapsed_target"
                    android:layout_width="40dp"
                    android:layout_height="40dp"
                    android:layout_marginEnd="16dp" />
            </android.support.v7.widget.Toolbar>
        </android.support.design.widget.CollapsingToolbarLayout>
    </android.support.design.widget.AppBarLayout>

    <android.support.v4.widget.NestedScrollView
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        app:layout_behavior="@string/appbar_scrolling_view_behavior">

        <LinearLayout
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:orientation="vertical">

            <android.support.v7.widget.CardView
                android:layout_width="match_parent"
                android:layout_height="wrap_content">

                <LinearLayout
                    android:layout_width="match_parent"
                    android:layout_height="wrap_content"
                    android:gravity="center"
                    android:orientation="vertical"
                    android:padding="@dimen/double_spacing">

                    <TextView
                        android:id="@+id/expert_profile_name"
                        style="@style/TextAppearance.AppCompat.Title"
                        android:layout_width="wrap_content"
                        android:layout_height="wrap_content"
                        android:layout_marginTop="@dimen/quad_spacing"
                        android:gravity="center"
                        android:textSize="17sp"
                        android:textStyle="bold"
                        tools:text="Rick Nelson" />

                </LinearLayout>

            </android.support.v7.widget.CardView>
                <android.support.v7.widget.RecyclerView
                    android:id="@+id/expert_places"
                    android:layout_width="match_parent"
                    android:layout_height="match_parent"
                    android:nestedScrollingEnabled="false"
                    app:layoutManager="android.support.v7.widget.LinearLayoutManager"
                    tools:listitem="@layout/item_picks_list" />
        </LinearLayout>

    </android.support.v4.widget.NestedScrollView>

    <de.hdodenhof.circleimageview.CircleImageView
        android:id="@+id/expert_avatar_logo"
        android:layout_width="@dimen/expert_avatar_size"
        android:layout_height="@dimen/expert_avatar_size"
        android:layout_gravity="top|center_horizontal"
        android:layout_marginTop="@dimen/expert_avatar_position"
        android:elevation="8dp"
        android:src="@drawable/larry"
        app:collapsedTarget="@id/circle_collapsed_target"
        app:layout_behavior="com.myapp.transition.AvatarImageLayoutBehavior" />

</android.support.design.widget.CoordinatorLayout>

As you can see there is a CollapsingToolbar within a CoordinatorLayout, then a NestedScrollView which contains a CardView with some info and then a RecyclerView with an arbitrary number of items.

I want the header to scroll up until its not seen anymore, effectively hiding below the AppBar and only show up at the top of the RecyclerView scroll.

This behaviour works nicely but by being within the NestedScrollView, RecyclerView is not recycling anymore so with ~100 items it already fills in the memory and my app starts slowing down and dragging.

With RecyclerView implementing NestedScrollViewChild one would expect they would've prepared it to work with NestedScrollView, but apparently this is expected and RecyclerView stops being a RecyclerView when used in combination with that.

Is there any way to circumvent this?

I am aware of other solutions such as putting my cardview header within the appbar or as a different itemview type in the recycler view but those options also have drawbacks and they are not valid for me.

I dont understand how something so basic is so hard to do with Android.