6

I am trying to project an ARAnchor to the 2D space but I am facing on an orientation issue...

Below my function to project the top left, top right, bottom left, bottom right corner position to 2D space:

/// Returns the projection of an `ARImageAnchor` from the 3D world space
/// detected by ARKit into the 2D space of a view rendering the scene.
///
/// - Parameter from: An Anchor instance for projecting.
/// - Returns: An optional `CGRect` corresponding on `ARImageAnchor` projection.
internal func projection(from anchor: ARImageAnchor,
                         alignment: ARPlaneAnchor.Alignment,
                         debug: Bool = false) -> CGRect? {
    guard let camera = session.currentFrame?.camera else {
        return nil
    }

    let refImg = anchor.referenceImage
    let anchor3DPoint = anchor.transform.columns.3

    let size = view.bounds.size
    let width = Float(refImg.physicalSize.width / 2)
    let height = Float(refImg.physicalSize.height / 2)

    /// Upper left corner point
    let projection = ProjectionHelper.projection(from: anchor3DPoint,
                                              width: width,
                                              height: height,
                                              focusAlignment: alignment)
    let topLeft = projection.0
    let topLeftProjected = camera.projectPoint(topLeft,
                                      orientation: .portrait,
                                      viewportSize: size)

    let topRight:simd_float3 = projection.1
    let topRightProjected = camera.projectPoint(topRight,
                                       orientation: .portrait,
                                       viewportSize: size)

    let bottomLeft = projection.2
    let bottomLeftProjected = camera.projectPoint(bottomLeft,
                                         orientation: .portrait,
                                         viewportSize: size)

    let bottomRight = projection.3
    let bottomRightProjected = camera.projectPoint(bottomRight,
                                          orientation: .portrait,
                                          viewportSize: size)

    let result = CGRect(origin: topLeftProjected,
                        size: CGSize(width: topRightProjected.distance(point: topLeftProjected),
                                     height: bottomRightProjected.distance(point: bottomLeftProjected)))

    return result
}

This function works pretty well when I am in front of the world origin. However, if I move left or right the calculation of the corner points does not work.

good calculation

bad calculation

YMonnier
  • 1,384
  • 2
  • 19
  • 30

1 Answers1

6

I found a solution to get corner 3D points of an ARImageAnchor depending on the anchor.transform and project them to 2D space:

    extension simd_float4 { 
        var vector_float3: vector_float3 { return simd_float3([x, y, z]) } 
    }

    /// Returns the projection of an `ARImageAnchor` from the 3D world space
    /// detected by ARKit into the 2D space of a view rendering the scene.
    ///
    /// - Parameter from: An Anchor instance for projecting.
    /// - Returns: An optional `CGRect` corresponding on `ARImageAnchor` projection.
    internal func projection(from anchor: ARImageAnchor) -> CGRect? {
        guard let camera = session.currentFrame?.camera else {
            return nil
        }
        
        let refImg = anchor.referenceImage
        let transform = anchor.transform.transpose

        
        let size = view.bounds.size
        let width = Float(refImg.physicalSize.width / 2)
        let height = Float(refImg.physicalSize.height / 2)
        
        // Get corner 3D points
        let pointsWorldSpace = [
            matrix_multiply(simd_float4([width, 0, -height, 1]), transform).vector_float3, // top right
            matrix_multiply(simd_float4([width, 0, height, 1]), transform).vector_float3, // bottom right
            matrix_multiply(simd_float4([-width, 0, -height, 1]), transform).vector_float3, // bottom left
            matrix_multiply(simd_float4([-width, 0, height, 1]), transform).vector_float3 // top left
        ]
        
        // Project 3D point to 2D space
        let pointsViewportSpace = pointsWorldSpace.map { (point) -> CGPoint in
            return camera.projectPoint(
                point,
                orientation: .portrait,
                viewportSize: size
            )
        }
        
        // Create a rectangle shape of the projection
        // to calculate the Intersection Over Union of other `ARImageAnchor`
        let result = CGRect(
           origin: pointsViewportSpace[3],
           size: CGSize(
               width: pointsViewportSpace[0].distance(point: pointsViewportSpace[3]),
               height: pointsViewportSpace[1].distance(point: pointsViewportSpace[2])
           )
        )
        
        
        return result
    }
YMonnier
  • 1,384
  • 2
  • 19
  • 30
  • Hey, can you help me with my current problem. I was trying to use your code-sample and I run into this error: "Value of type 'simd_float4' (aka 'float4') has no member 'vector_float3'" – Piotr Gawłowski May 08 '18 at 08:14
  • Hey @PiotrGawłowski, it is just a property to convert `simd_float4` to `vector_float3`: `extension simd_float4 { var vector_float3: vector_float3 { return simd_float3([x, y, z]) } }` – YMonnier May 09 '18 at 07:29
  • The thing I am actually trying to figure out is get corner positions in 3dSpace. I tried to use pointsWorldSpace and put a Circle geometry nodes using this vectors. The effect is - points are distributed in a correct way (shape of ImageRef), correct sizing (width-height) but incorrect position - totally outside the ImageRef position in space. Any hints how to handle this? – Piotr Gawłowski May 09 '18 at 14:22
  • You can get the 3D position with the transform matrix: anchor.transform.columns.3. – YMonnier May 09 '18 at 16:34
  • Hello @YMonnier, where should I implement this method? I am implementing it in "willUpdate Node" but I got wrong results for y and width and height. X seems to be correct. My sceneView's size is 375x667 but the width and height of ArImageAnchor are bigger than that. What am I missing here? – ysnzlcn Apr 16 '19 at 18:18
  • 1
    Hi @YasinNazlıcan, you have to call the `projection` function when you detect the marker. Concretly, you should call this function on the `func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor)` delegate function. Hope this help you ;) – YMonnier Apr 16 '19 at 18:36
  • Hey @YMonnier, i moved the func to there but results are still same. I am calling your method like this; projection(from: imageAnchor, camera: camera, viewSize: UIScreen.main.bounds.size). This might be causing because of func distance() cgfloat extension? Since i don't have your extension i am using this one: func distance(point: CGPoint) -> CGFloat { let xDist = x - point.x let yDist = y - point.y return CGFloat(sqrt(xDist * xDist + yDist * yDist)) } – ysnzlcn Apr 18 '19 at 17:58
  • i ment CGPoint extension – ysnzlcn Apr 18 '19 at 18:24
  • Hi @ysnzlcn, I am using this extension function: `public extension CGPoint { /// Distance between two points. /// /// - Parameter point: A CGPoint instance. public func distance(point p: CGPoint) -> CGFloat { return sqrt(pow((p.x - x), 2) + pow((p.y - y), 2)) } }` – YMonnier Apr 24 '19 at 07:51
  • @YMonnier, Oddly, this seems to account for yaw, but not roll and pitch. Essentially, when I rotate my phone downward, the coordinates are still the same. How can I account for pitch and roll? I am using ```ARImageTrackingConfiguration``` if that makes any difference – Michael Jun 06 '19 at 18:46
  • Hi @YMonnier, I am working on getting the 3D points to 2D as you did. I am not too sure whether I can just apply your codes here to mine. Could you take a look if you have time? https://stackoverflow.com/questions/62783206/projecting-the-arkit-face-tracking-3d-mesh-to-2d-image-coordinates – swiftlearneer Jul 15 '20 at 17:42
  • Hi @ysnzlcn. Were you able to use this sample? – swiftlearneer Jul 23 '20 at 04:57
  • Hi @YMonnier. Could you direct us on how your examples would be called? – swiftlearneer Aug 05 '20 at 15:12