Pepper Robot - How to launch tablet applications through DialogFlow?

Question

I'm trying to incorporate Pepper's built in Android tablet more in DialogFlow interactions. Particularly, my goal is to open applications installed on the tablet itself for people to use while they're talking with Pepper. I'm aware there is a 'j-tablet-browser' app installed on Pepper's end that can let a person browse the tablet like an ordinary Android device, but I would like to take it one step further and directly launch an Android app, like Amazon's Alexa.

The best solution I can up with is:

• User says specific utterance (e.g. "Pepper, open Alexa please")
• DialogFlow launches the j-tablet-browser behavior

{
  "speak": "Sure, just a second",
  "action": "startApp",
  "action_parameters": {
    "appId": "j-tablet-browser/."
  }
}

• User navigates the Android menu manually to tap the Alexa icon

My ideal goal is to make the process seamless:

• User says specific utterance (e.g. "Pepper, open Alexa please")
• DialogFlow launches the Alexa app installed on the Android tablet

Does anyone have an idea how this could be done?

Answer 1

This is quite a broad question so I'll try and focus on the specifics for launching an app with a Dialogflow chatbot. If you don't already have a QiSDK Dialogflow chatbot running on Pepper, there is a good tutorial here which details the full process. If you already have a chatbot implemented I hope the below steps are general enough for you to apply to your project.

This chatbot only returns text results for Pepper to say, so you'll need to make some modifications to allow particular actions to be launched.

Modifying DialogflowDataSource

Step 2 on this page of the tutorial details how to send a text query to Dialogflow and get a text response. You'll want to modify it to return the full reponse object (including actions), not just the text. Define a new function called detectIntentFullResponse for example.

// Change this
return response.queryResult.fulfillmentText
// to this
return response.queryResult

Modifying DialogflowChatbot

Step 2 on this page shows how to implement a QiSDK Chatbot. Add some logic to check for actions in the replyTo function.

var response: DetectIntentResponse? = null
// ...
response = dataSource.detectIntentFullResponse(input, dialogflowSessionId, language)
// ...
return if (reponse.action != null) {
    StandardReplyReaction(
        ActionReaction(qiContext, response), ReplyPriority.NORMAL
    )
} else if (reponse.answer != null) {
    StandardReplyReaction(
        SimpleSayReaction(qiContext, reponse.answer), ReplyPriority.NORMAL
    )
} else {
    StandardReplyReaction(
        EmptyChatbotReaction(qiContext), ReplyPriority.FALLBACK
    )
}

Now make a new Class, ActionReaction. Note that the below is incomplete, but should serve as an example of how you can determine which action to run (if you want others). Look at SimpleSayReaction for more implementation details.

class ActionReaction internal constructor(context: QiContext, private val response: DetectIntentResponse) :
    BaseChatbotReaction(context) {

    override fun runWith(speechEngine: SpeechEngine) {
        if (response.action == "launch-app") {
            var appID = response.parameters.app.toString()
            // launch app at appID
        }
    }
}

As for launching the app, various approaches are detailed in other questions, such as here. It is possible to extend this approach to do other actions, such as running or retrieving online data.