3

I have a Unity app that I'd like to integrate with the Microsoft Mixed Reality Toolkit (MRTK). When I add the MRTK (v2.1 or v2.2) package to my Unity project, I can simulate the "air tap" gesture in the Unity Editor and the app registers the click. However, when I publish the app to my HoloLens1 (or run the holographic emulator within Unity), the "air tap" gesture does not register a click.

Unity: 2018.4.x

MRTK: both v2.1 / v2.2

Unity scene setup:

  • has an EventSystem with a Standalone Input Module
  • has a character that is the Main Camera
  • has imported MRTK
  • Mixed Reality Toolkit Profile is set to DefaultHoloLens1ConfigurationProfile

When I run the scene with the above setup, the air tap registers in the Unity Editor (by pressing space bar + click to simulate), but it does not register in the HoloLens1.

Is there some part of the setup I am missing? Perhaps another input component to be added to something in my scene?

Evelyn
  • 33
  • 6
  • Airtab on what? UI or any other script involved? – derHugo Dec 27 '19 at 16:59
  • @derHugo The Unity app loads some UI that is hosted on a NodeJS server elsewhere, which shows some simple box-geometry that responds to clicks. – Evelyn Dec 27 '19 at 17:48
  • How exactly? To answer we need to know exactly how the "click"/Air-tap is supposed to be recognized/consumed by script. – derHugo Dec 27 '19 at 17:50
  • @derHugo Ooh, I guess I don't know exactly, it's a Unity app/scene written by a third party that I just imported the MRTK to. Since the 'air tap' works in the emulator, I figured it was something simple. I can look at the sample app some more, I got it from here: https://github.com/microsoft/mixed-reality-extension-unity/tree/red/MRETestBed – Evelyn Dec 27 '19 at 18:09
  • @derHugo the 'HelloWorld.unity' scene here https://github.com/microsoft/mixed-reality-extension-unity/tree/red/MRETestBed/Assets/Scenes – Evelyn Dec 27 '19 at 18:10
  • Would be easier if you know the name of the script or simply rather post the according code here ;) also afaik in the editor the tap is simulated by right- (or left-clicking not sure right now) with the mouse actually... – derHugo Dec 27 '19 at 18:15
  • Ok thank you! I will look for the code. – Evelyn Dec 27 '19 at 18:31
  • Playing with the MRE, it looks like the MRE app even without MRTK is capturing the mouse movement and responding to clicks. Have you confirmed that in the Unity editor that the application is actually responding to the MRTK simulated hands? One way to do this would be to find and remove the logic the the MRE that is listening to mouse move + click. I think disabling "FPS Controller with Cursor" does the trick. Then try adding the MRTK to the scene and see if the cube responds to the simulated hand. I would guess no, since there needs to be logic up to listen to MRTK's PointerClicked events. – Julia Schwarz Jan 02 '20 at 16:18
  • @JuliaSchwarz Thanks! Yes, the MRE cube does respond to the simulated hand, without even disabling the 'FPS Controller with Cursor'. The MRETestBed HelloWorld scene, with the added MixedRealityToolkit, and then running the app within the Unity editor, has the MRE cube responding to the simulated hand. That is why I assumed it would then just work in the HoloLens, and didn't think it would be a problem in the Unity scene itself. Am I understanding that correctly? – Evelyn Jan 02 '20 at 18:58
  • I believe that the simulated hand may only be working when "FPS Controller with Cursor" is enabled. Try disabling the controller, does it work then? I think that the cube in the MRE sample is actually responding to mouse clicks, not to the simulated hand. I think mouse click causes the FPS controller to hand its 'on mouse click' event, which makes the cube spin. I think the MRE needs to be modified to instead handle OnPointerClicked events, which is what's coming from MRTK. – Julia Schwarz Jan 06 '20 at 22:08
  • Thank you for the clarification! Yes, disabling 'FPS Controller with Cursor' (and actually, just the 'InputSource' module within it) did the trick. The MRE no longer responded to mouse clicks. And, now I understand that what I thought was the 'simulated hand air tap' working was actually just coincidentally the mouse, as you said. I did find somewhere in the MRE code where the Input.GetButtonDown is overridden, which I guess is the mouse click. There is no MRTK code in the MRE at all. Is there any tutorial that might give me pointers as to how to incorporate MRTK Pointers in this code? – Evelyn Jan 07 '20 at 17:17
  • @JuliaSchwarz I don't know if I am supposed to tag you about my comment response so tagging you here just in case, thanks! ^^ – Evelyn Jan 07 '20 at 23:19
  • Just posted an answer with a sample, hope it helps. – Julia Schwarz Jan 09 '20 at 00:00

1 Answers1

4

To respond to click events from MRTK, you need listen for MRTK pointer events instead of the Unity input events that the MRE is likely listening to. A good event to listen for would be the OnPointerClicked event, which will trigger when a hand is tapped, or when a motion controller is clicked, or if you say the word "select". Normally the input handler will only respond if you are hovering over the object, so to respond to these inputs globally you need to register as a global input handler.

Here is an example of a script that will print some text whenever the pointer is clicked:

using Microsoft.MixedReality.Toolkit;
using Microsoft.MixedReality.Toolkit.Input;
using UnityEngine;

public class DetectTapExample : MonoBehaviour
{

    public void Start()
    {
        PointerHandler pointerHandler = gameObject.AddComponent<PointerHandler>();
        pointerHandler.OnPointerClicked.AddListener((evt) => Debug.Log("Tap Detected " + Time.time));
        // Make this a global input handler, otherwise this object will only receive events when it has input focus
        CoreServices.InputSystem.RegisterHandler<IMixedRealityPointerHandler>(pointerHandler);
    }
}

You can learn more about Pointers here and the MRTK Input System here

Julia Schwarz
  • 2,610
  • 1
  • 19
  • 25
  • 1
    Amazing! Thank you so much @JuliaSchwarz! I just tried this sample code and put it in the relevant MRETestBed file and hooked up the click events for the MRE side of things and it is finally working! I am so excited, thank you so much! The sample code was extremely helpful, so thank you for that. Also, I see the Pointer documentation and can take it from here to add the grab functionality next. Thanks very much again for your time! – Evelyn Jan 10 '20 at 21:59
  • Great! Glad it worked. If possible, could you please mark this as the correct answer and upvote? – Julia Schwarz Jan 10 '20 at 23:40
  • Yup! I marked it as an accepted answer, and I gave it an upvote, but apparently it won't display publicly since unfortunately my reputation isn't yet high enough. Thank you again! – Evelyn Jan 13 '20 at 15:50