SwiftUI Drag to Dismiss ScrollView

Question

I'd like to have a drag to dismiss scroll view in SwiftUI, where if you keep dragging when it's at the top of the content (offset 0), it will instead dismiss the view.

I'm working to implement this in SwiftUI and finding it to be rather difficult. It seems like I can either recognize the DragGesture or allowing scrolling, but not both.

I need to avoid using UIViewRepresentable and solve this using pure SwiftUI or get as close as possible. Otherwise it can make developing other parts of my app difficult.

Here's an example of the problem I'm running into:

import SwiftUI

struct DragToDismissScrollView: View {
    enum SeenState {
        case collapsed
        case fullscreen
    }

    @GestureState var dragYOffset: CGFloat = 0
    @State var scrollYOffset: CGFloat = 0
    @State var seenState: SeenState = .collapsed

    var body: some View {
        GeometryReader { proxy in
            ZStack {
                Button {
                    seenState = .fullscreen
                } label: {
                    Text("Show ScrollView")
                }

                /*
                 * Works like a regular ScrollView but provides updates on the current yOffset of the content.
                 * Can find code for OffsetAwareScrollView in link below.
                 * Left out of question for brevity.
                 * https://gist.github.com/robhasacamera/9b0f3e06dcf27b54962ff0e077249e0d
                 */
                OffsetAwareScrollView { offset in
                    self.scrollYOffset = offset
                } content: {
                    ForEach(0 ... 100, id: \.self) { i in
                        Text("Item \(i)")
                            .frame(maxWidth: .infinity)
                    }
                }
                .background(Color.white)
                // If left at the default minimumDistance gesture isn't recognized
                .gesture(DragGesture(minimumDistance: 0)
                    .updating($dragYOffset) { value, gestureState, _ in
                        // Only want to start dismissing if at the top of the scrollview
                        guard scrollYOffset >= 0 else {
                            return
                        }

                        gestureState = value.translation.height
                    }
                    .onEnded { value in
                        if value.translation.height > proxy.frame(in: .local).size.height / 4 {
                            seenState = .collapsed
                        } else {
                            seenState = .fullscreen
                        }
                    })
                .offset(y: offsetForProxy(proxy))
                .animation(.spring())
            }
        }
    }

    func offsetForProxy(_ proxy: GeometryProxy) -> CGFloat {
        switch seenState {
        case .collapsed:
            return proxy.frame(in: .local).size.height
        case .fullscreen:
            return max(dragYOffset, 0)
        }
    }
}

Note: I've tried a lot solutions for the past few days (none that have worked), including:

1. Adding a delay to the DragGesture using the method mentioned here: https://stackoverflow.com/a/59961959/898984
2. Adding an empty onTapGesture {} call before the DragGesture as mentioned here: https://stackoverflow.com/a/60015111/898984
3. Removing the gesture and using the offset provided from the OffsetAwareScrollView when it's > 0. This doesn't work because as the ScrollView is moving down the offset decreases as the OffsetAwareScrollView catches up to the content.