Map 2D point from SpriteKit SKScene overlaySKScene to 3D point in SceneKit SCNScene

Question

I'm attempting to translate touches from SpriteKit to SceneKit and having a bit of difficulty. My intention is to move a 3D "monster" so that it appears as though its in the same location (though behind) a 2D crystal. 2D crystals are added to the SpriteKit-provided overlaySKScene and positioned. Our "monster" is a full 3D SCNNode sitting in a SCNScene, as shown below.

I've tried a variety of approaches, the closest of which I've included below in the makeMonsterEatCrystalAtLocation method, though I know it to be wrong, as it only acts within the SpriteKit world - it doesn't convert from SpriteKit to SceneKit. I've also looked into unprojectPoint in the SCNSceneRenderer protocol, but that method only works from within SceneKit itself. Any assistance would be greatly appreciated! I'm not partial to Objective-C or Swift, even though the below is in Obj-C.

- (void)setupSceneKit {

    _sceneView = [[SCNView alloc] initWithFrame:[[UIScreen mainScreen] bounds] options:@{@"SCNPreferredRenderingAPIKey": @(SCNRenderingAPIOpenGLES2)}];
    [self.view insertSubview:_sceneView aboveSubview:backgroundView];

    _sceneView.scene = [SCNScene scene];
    _sceneView.allowsCameraControl = YES;
    _sceneView.autoenablesDefaultLighting = NO;
    _sceneView.backgroundColor = [UIColor clearColor];

    camera = [SCNCamera camera];
    [camera setXFov:20];
    [camera setYFov:20];
    camera.zFar = 10000.0f;

    cameraNode = [SCNNode node];
    cameraNode.camera = camera;
    cameraNode.position = SCNVector3Make(0, 3, 700);
    [_sceneView.scene.rootNode addChildNode:cameraNode];
}

- (void)createMonster {

    _monsterNode = [SCNNode node];
    [self.sceneView.scene.rootNode addChildNode:_monsterNode];
    [SCNTransaction begin];
    [SCNTransaction setAnimationDuration:0.5f];
    _monsterNode.position = SCNVector3Zero;
    [SCNTransaction commit];
}

- (void)touchesEnded:(NSSet<UITouch *> *)touches withEvent:(UIEvent *)event {

    CGPoint crystalLocation = [[touches anyObject] locationInNode:self.sceneView.overlaySKScene];
    SKNode *touchedCrystalNode = [self.sceneView.overlaySKScene nodeAtPoint:crystalLocation];

    // determine that this is the SKNode touchedCrystalNode we're looking for...

    [self makeMonsterEatCrystalAtLocation:crystalLocation];
}

- (void)makeMonsterEatCrystalAtLocation:(CGPoint)location {

    // Attempt to move the "monster" (SCNNode in SceneKit) to appear as if its in the same 
    // location as the "crystal" (which is presented in the SpriteKit overlay: self.sceneView.overlaySKScene) and eventually
    // play an animation of the monster eating the crystal.

    CGPoint scnPoint = [self.sceneView.overlaySKScene convertPointToView:location];

    [SCNTransaction begin];
    [SCNTransaction setAnimationDuration:0.5f];
    self.monsterNode.position = SCNVector3Make(scnPoint.x, scnPoint.y, 0);
    [SCNTransaction commit];

    // ...
}

Answer 1

One approach would be to put your monster in an SK3DNode. Then you can spend most of your life in SpriteKit, without having to convert between coordinate systems.

When projecting, you have to deal with several different coordinate systems: the SpriteKit scene (origin at lower left), the SceneKit view's screen coordinates (origin at upper left or lower left, depending on OS), the view's camera local coordinate system, and the SceneKit scene's world coordinate system.

Let's start with just the SceneKit side of things. Your camera has its own 3D coordinate system, denoting the contents of the view frustum. A single point on the screen corresponds to a line segment within the frustum, from the near plane to the far plane; the Z axis in camera space corresponds to the point's relative location between the neaar and far planes. Your SCNView, by virtue of complying with protocol SCNSceneRenderer, can convert between screen coordinates and 3D world space.

To move your monster, you'll use projectPoint to get its current camera x/y/z coordinates, figure out the screen location you want to move to, and then use unprojectPoint to compute the world location for that screen location. Something like this:

let screenDestinationX = 100.0
let screenDestinationY = 200.0
let monsterCoordinates = sceneView.projectPoint(MonsterNode.position)
let monsterDestinationCoordinates = SCNVector3D(x: screenDestinationX,
                                                y: screenDestinationY,
                                                z: monsterCoordinates.z)
let monsterDestinationPosition = sceneView.unprojectPoint(monsterDestinationCoordinates)

By keeping the same camera Z coordinate for monster start and finish, you'll keep the monster at the same distance from the camera.

Now for the SpriteKit side. If your sceneView.overlaySKScene has its scaleMode set to SKSceneScaleMode.ResizeFill, your SKScene will have the same size as your screen. The direction of the Y axis might be different, of course. So ask your crystal for its SpriteKit coordinates, flip the Y if necessary (height minus Y, instead of just Y), and you have destination X and Y you can pass to SceneKit for projection.

I have a sample project which shows conversion between SceneKit and overlay SpriteKit coordinates at https://github.com/halmueller/ImmersiveInterfaces/tree/master/Tracking%20Overlay. It doesn't do exactly what you're after but it does show the interaction.

Finally, below is a Playground that shows conversion between world and camera coordinates:

import SceneKit
import SpriteKit
import XCPlayground

let sceneView = SCNView(frame: CGRect(x: 0, y: 0, width: 800, height: 600))

XCPlaygroundPage.currentPage.liveView = sceneView

var scene = SCNScene()
sceneView.scene = scene
sceneView.backgroundColor = SKColor.greenColor()
sceneView.debugOptions = .ShowWireframe

// default lighting
sceneView.autoenablesDefaultLighting = true

// a camera
var cameraNode = SCNNode()
cameraNode.camera = SCNCamera()
print (cameraNode.camera?.zFar, cameraNode.camera?.zNear)

cameraNode.camera?.automaticallyAdjustsZRange = false
cameraNode.position = SCNVector3(x: 5, y: 5, z: 50)
scene.rootNode.addChildNode(cameraNode)
let center = SCNNode()
center.position = SCNVector3(x:0, y:0, z:0)
scene.rootNode.addChildNode(center)
let centerConstraint = SCNLookAtConstraint(target: center)
cameraNode.constraints = [centerConstraint]

let sphere = SCNSphere(radius: 2)
sphere.geodesic = true
let sphereNode1 = SCNNode(geometry: sphere)
sphereNode1.position = SCNVector3(x:-8, y:5, z:10)
scene.rootNode.addChildNode(sphereNode1)

let sphereNode2 = SCNNode(geometry: sphere)
sphereNode2.position = SCNVector3(x:8, y:-5, z:-30)
scene.rootNode.addChildNode(sphereNode2)

let sphere1Coordinates = sceneView.projectPoint(sphereNode1.position)
print ("sphere1 position", sphereNode1.position, "screen coordinates",sphere1Coordinates)
let sphere2Coordinates = sceneView.projectPoint(sphereNode2.position)
print ("sphere2 position", sphereNode2.position, "screen coordinates",sphere2Coordinates)

let sphere2NearCoordinates = SCNVector3(x: sphere2Coordinates.x, y: sphere2Coordinates.y, z: sphere1Coordinates.z)
let sphere2FarPosition = sphereNode2.position
let sphere2NearPosition = sceneView.unprojectPoint(sphere2NearCoordinates)

let sphere1FarCoordinates = SCNVector3(x: sphere1Coordinates.x, y: sphere1Coordinates.y, z: sphere2Coordinates.z)
let sphere1NearPosition = sphereNode1.position
let sphere1FarPosition = sceneView.unprojectPoint(sphere1FarCoordinates)

let sphere1LowerCoordinates = SCNVector3(x: sphere1Coordinates.x, y: sphere1Coordinates.y - 100, z: sphere2Coordinates.z)
let sphere1LowerPosition = sceneView.unprojectPoint(sphere1LowerCoordinates)


let forwardAction = SCNAction.moveTo(sphere2NearPosition, duration: 3)
let backAction = SCNAction.moveTo(sphere2FarPosition, duration: 3)
let shiftAction = SCNAction.moveTo(sphere1LowerPosition, duration: 3)
let forwardBackShift = SCNAction.sequence([forwardAction, backAction, forwardAction, shiftAction])
sphereNode2.runAction(forwardBackShift) {
    print ("sphere1 position", sphereNode1.position, "screen coordinates",sceneView.projectPoint(sphereNode1.position))
    print ("sphere2 position", sphereNode2.position, "screen coordinates",sceneView.projectPoint(sphereNode2.position))
}