Cannot find CGPoint in CGPath

Question

I have the following function for drawing a line between two points:

    override func draw(from fromPoint: CGPoint, to toPoint: CGPoint) {
        let path = UIBezierPath()
        path.move(to: fromPoint)
        path.addLine(to: toPoint)
        path.close()

        layer.path = path.cgPath
        layer.strokeColor = pencil.color.cgColor
        layer.lineWidth = pencil.strokeSize
    }

Which is being called in touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) This draws a line and works fine, the line draws itself on the correct screen position.

Then I have another functionality where the user can erase previous drawings, and I'm trying to get if a touch position, is contained by any of the drawn paths:

    private func findLayer(in touch: UITouch) -> CAShapeLayer? {
        let point = touch.location(in: view)

        // all the drawn paths are contained in CAShapeLayers
        // Circles and rectangle layers have a corresponding frame that contains the UIBezierPath.
        if let hitLayer = view?.layer.hitTest(point) as? CAShapeLayer,
            hitLayer.path?.contains(point) == true || hitLayer.frame.contains(point) {
            return hitLayer
        }

        guard let sublayers = view?.layer.sublayers else { return nil }

        // this extra check is for layers that dont have a frame (Lines, Angle and free drawing)
        for layer in sublayers {
            if let shapeLayer = layer as? CAShapeLayer,
                shapeLayer.path?.contains(point) == true || shapeLayer.frame.contains(point) {
                return shapeLayer
            }
        }
        return nil
    }

The problem is that when the user draws a line, the findLayer function doesn't ever return the Layer with the line, but it works perfectly when the user draws a circle or a rectangle.

I don't want to give Line drawings a frame, because then the hit box could be too big, and the user could delete the drawing even if the touch isn't near the real path.

How can a find if a touch point is part of a CAShapeLayer.path ?