How to correctly handle movement on a zoomed image with a UIScrollView?

Question

I am new to Swift and I'm trying to reproduce the behavior most applications have when you zoom in an image (like Twitter, Facebook etc...). Basically, you tap on the image, it opens fullscreen and then you can zoom in and move freely around the zoomed in image. However, I'm having trouble with the last two parts.

My main ViewController has an instance of a custom UIImageView called TappableImage, which inherits from UIImageView. I use this class as a wrapper to handle touch and transitions, code is below:

import UIKit

class TappableImage: UIImageView {
    private let windowBounds: CGRect = UIScreen.main.bounds
    private let imageV: UIImageView = UIImageView()
    private var startingFrame: CGRect = .zero
    private var backdrop: ImageBackdrop!

    private var isOpen: Bool = false

    override init(frame: CGRect) {
        super.init(frame: frame)
        setup()
    }

    required init?(coder: NSCoder) {
        super.init(coder: coder)
        setup()
    }

    private func setup() {
        image = UIImage(named: "kaws") // horizontal
//        image = UIImage(named: "portrait") // vertical
//        image = UIImage(named: "ksg") // square
        isUserInteractionEnabled = true

        startingFrame = frame

        imageV.image = self.image
        imageV.frame = startingFrame
        imageV.contentMode = .scaleAspectFill
        imageV.isUserInteractionEnabled = true

        backdrop = ImageBackdrop(frame: windowBounds, with: imageV)

        let openTapGesture = UITapGestureRecognizer(target: self, action: #selector(onTap))
        openTapGesture.numberOfTapsRequired = 1
        addGestureRecognizer(openTapGesture)
    }

    @objc private func onTap(_ sender: UITapGestureRecognizer) {
        guard let window = UIApplication.shared.windows.first else {
            print("no window...")
            return
        }

        window.addSubview(backdrop)
        UIView.animate(withDuration: 0.25, animations: {
            self.backdrop.alpha = 1

            let h = (self.windowBounds.width / self.startingFrame.width) * self.startingFrame.height
            let y = self.windowBounds.height / 2 - h / 2
            self.imageV.frame = CGRect(x: 0, y: y, width: self.windowBounds.width, height: h)

        }) { (_) in
            self.isOpen = true
        }
    }
}

When a user tap the image, I bring a custom UIScrollView to the front in which I added my TappableImage. The UIScrollView is a basic implementation just handling pinching, code is below:

import UIKit

class ImageBackdrop: UIScrollView, UIScrollViewDelegate {
    private var imageView: UIImageView!

    override init(frame: CGRect) {
        super.init(frame: frame)
    }

    required init?(coder: NSCoder) {
        super.init(coder: coder)
    }

    convenience init(frame: CGRect, with imgv: UIImageView) {
        self.init(frame: frame)

        imageView = imgv
        setupScrollView()
        addSubview(imageView)
    }

    private func setupScrollView() -> Void {
        delegate = self

        backgroundColor = .black
        alpha = 0

        minimumZoomScale = 1.0
        maximumZoomScale = 5.0
        contentSize = imageView.bounds.size
        autoresizingMask = [.flexibleWidth, .flexibleHeight]
    }

    func viewForZooming(in scrollView: UIScrollView) -> UIView? { imageView }
}

My issue is that when I zoom in, the image is shifted down and I cannot move freely around with one finger. If I try to move to the bottom of the image, I am "slingshotted" back to the top, like when you arrive at the end of a list, you can find a video of this behavior here.

My guess is that the UIImageView center or size is incorrect but I cannot find a way to fix it.