Custom RecyclerView LayoutManager does not fling (smoothly)

Question

I am working on an application which looks somewhat like this:

The RecyclerView uses a custom LayoutManager with a custom implementation of "scrollVerticallyBy"

First of. The recyclerview scroll butter smooth and renders in way less than the required 16 ms. I think I can max i out around 5 ms on a Nexus 5.

However, I also want to be able to scroll the RecyclerView from touching the View above it (it's a RelativeLayout).

Here is the problem:

1) If I just forward the onTouchEvents the scrolling is butter smooth. However, RecyclerView.fling(...) is never invoked and scrolling only works as long as the finger is touching the screen. No flinging which feels very awkward.

@Override
     public boolean onTouchEvent(MotionEvent event) {
    return mRecyclerView.onTouchEvent(event);
}

2) If I override .fling(...) in the RecyclerView I can get "flinging" to work by using a Scroller. This works, but does cause some strange large lags.

@Override
public void onDraw(Canvas canvas) {
    super.onDraw(canvas);

    if (mScroller.computeScrollOffset()) {
        Log.d("RecyclerView", "(ondraw) Y: " + mScroller.getCurrY());
        scrollBy(0, mScroller.getCurrY()- mLastY);
        mLastY = mScroller.getCurrY();
        postInvalidateDelayed(16); // 60 Hz
    }
}


@Override
public boolean fling(int velocityX, int velocityY) {
    mLastY = 0;
    mScroller.fling(0, 0, velocityX, velocityY, 0, 0, -10000 ,10000);
    return true;
} 

I am unsure what the correct way to do this is. I feels like that if I am not overriding .fling() and force my app to repeatedly call scrollBy() by using the Scroller the scrolling is working differently.

I suspect this may have something to do with "startSmoothScroll" and "smoothScrollToPosition" which I have not overwritten. I honestly don't know what smoothscrolling is but it sounds relevant to this problem :)