1

I am using actions-on-google and Dialogflow to build a social robot for Elderly.

I was wondering how I can easily repeat the last sentence when asked by the user ("repeat please") as often the Senior doesn't hear the sentence the first time.

One way would be to have repeat followup intents in Dialogflow but this is quite heavy since :

  • you need to add one after each intent and I have many
  • in a multi-user environment you need to keep track of the last sentence for every user ...

Another way would be be to take advantage of Dialogflow Contexts. As you send the message, you can also add that message to a context (for example, you can call it "last_message"). You can then have another Intent that takes as an input context the "last_message" context and, if triggered, uses the value saved in the context to repeat it.

  • However, I still have the problem that I need to add a context to every intent I have, which are many.

Does anyone know how to accomplish this in a quicker way? I found this package but it is in JS and I need it Python: https://github.com/SysCoder/VoiceRepeater/pulls .

How do I implement this VoiceRepeater library? Do I put the code under fulfillment function 'repeat' I have made and mapped to an intent called 'repeat' that I have made which responds to utterances such as 'Sorry, could you repeat that'? Also, where do I install the VoiceRepeater library (code: npm install voice-repeater --save)?

Prisoner
  • 49,922
  • 7
  • 53
  • 105
DeMelkbroer
  • 629
  • 1
  • 6
  • 21
  • 1
    Can you use the conv.data property in Python? https://developers.google.com/assistant/conversational/save-data#saving_data_between_turns_of_a_conversation Personally I don't enjoy the context approach and I would store the last question in there, then use an repeat intent which reads the value from the data field. Example in the docs: https://developers.google.com/assistant/conversational/tips#let_users_replay_information – Jordi Jan 22 '20 at 15:37
  • Using `conv.data` is, essentially, the same as putting the data in a context. Either way works. – Prisoner Jan 22 '20 at 15:39
  • If you are doing this in Python, it would be useful to see how you're sending the response now to help guide you about how to set either the context or `conv.data` as part of your response. – Prisoner Jan 22 '20 at 15:43
  • Possible duplicate of https://stackoverflow.com/q/47737828/1405634 – Prisoner Jan 22 '20 at 15:47
  • Maybe to clarify my own question, I think I was not aware enough of what I was asking. How do I implement this VoiceRepeater library? Do I put the code under fulfillment function 'repeat' I have made and mapped to an intent called 'repeat' that I have made which responds to utterances such as 'Sorry, could you repeat that'? Also, where do I install the VoiceRepeater library (code: npm install voice-repeater --save)? – DeMelkbroer Jan 28 '20 at 14:04
  • Does anyone know like a youtube vid where the repeat function is implemented? – DeMelkbroer Jan 28 '20 at 14:10

1 Answers1

2

Using Followup Intents is probably the wrong way to do this. As you note, it is way too heavy for more than a few Intents. It may be useful in certain circumstances if you want the "repeated" message to clarify the response in a different way, but in general, it isn't very useful. (It should also be noted that Followup Intents use Contexts, but in a different way than discussed below.)

You don't need to add the Context to the UI as part of the Outgoing Context - you would set this as part of your Fulfillment. It would include a parameter that either contained exactly what you said, or the information you needed to recreate what you said (possibly in a different form, if appropriate). In your "repeat" Intent, you'd read the value that you had saved in this Context, and send it as the output again. If you're using SSML, you may wish to change the speed or volume, if that is appropriate.

Update based on new questions

The readme for VoiceRepeater has the basics of what you need to do to use it, but it does assume a little familiarity with Node. But in general, yes, you install it the way you describe, setup an Intent that captures requests to repeat, and registers a handler function (repeatLastStatement(app) in the readme) that handles the Intent to send a reply through voiceRepeater.lastPromptWithPrefix().

It also may assume you're using version 1 of the actions-on-google library. I haven't dug too deeply into the code, but it looks like it replaces the library's ask function with its own, and I'm not sure how well that works with version 2 of the actions-on-google library.

Unlike Voice Repeater, multivocal doesn't require you to register handlers specifically since it tries to hide as much boilerplate under the covers. You just need to define the replies that you might want it to use. It uses the Context scheme I outline above to store responses and make them available when the user asks for it to be repeated.

There aren't any videos on using multivocal, but the simple example does include the configuration illustrating how to configure responses for the "multivocal.repeat" Intent. While VoiceRepeater works with the actions-on-google library, multivocal is a complete replacement, offering a more template-based approach to building fulfillment.

However, neither of these directly help you if you want to implement it for Python. But if you look at the source for VoiceRepeater, you can get a sense for how to implement it yourself in Python.

The key bit is on line 47 where it saves the reply in a context. (It also saves the reply with a prefix message.) It then calls the original function that would send the reply:

  app.setContext("last_prompt", 100,
    {
      "last_prompt": textToSpeech,
      "prefixed_last_prompt": repeatPrefix + lastStatement,
    });
  originalAsk(response);

Later, in the call to lastPromptWithPrefix(), it uses the contents of the Context to send a reply.

  lastPromptWithPrefix() {
     return this.app.getContext("last_prompt") !== null
         ? this.app.getContextArgument("last_prompt", "prefixed_last_prompt").value
         : "um....I don't remember what I said!";
  }
Prisoner
  • 49,922
  • 7
  • 53
  • 105
  • Hi @Prisoner, I just found your answer given here. https://stackoverflow.com/questions/55160588/how-to-repeat-last-response-of-bot-in-dialogflow. This seems like a perfect managable solution for me, is this approach still working? – DeMelkbroer Jan 28 '20 at 13:02
  • Yes, [multivocal](https://multivocal.info) still implements what I described there, but requires you to use the multivocal library. (Note that I wrote that library and still use it myself.) What I described above and in that answer is the approach that multivocal uses in case you wanted to use a different library. – Prisoner Jan 28 '20 at 14:22
  • How would I go about using your library? I'm sorry I'm new to this and trying to understand but can't find like a clear youtube fit showing how to implement it – DeMelkbroer Jan 28 '20 at 14:25
  • multivocal is an alternate library to use to build fulfillments. Unfortunately, there isn't a video (sigh... yet...), but there are several examples posted at https://multivocal.info/ I've updated the answer a bit to elaborate on both VoiceRecorder and multivocal, but if you really want this sort of thing for Python, we'd need to see what you've already implemented in Python and where you're having issues. – Prisoner Jan 28 '20 at 15:16
  • what I meant is that our social robot is built in python. I was not aware that Dialogflow had an Inline Editor in which the javascript could be implemented. I was thinking having the robot safe every last sentence so I could handle the repeat outside of Dialogflow. But I think using your library would be a lot easier and better practice – DeMelkbroer Jan 28 '20 at 15:39
  • In the simple example, how do I add your packages.json to my already existing packages.json (starting with: "name": "dialogflowFirebaseFulfillment") ? I thought I could at it as a second block but that doesn't work. Would specifying the version under dependencies suffice? – DeMelkbroer Jan 28 '20 at 15:55
  • I would suggest asking that as a new question, since we're straying pretty far from your original. – Prisoner Jan 28 '20 at 17:06
  • I have submitted a new question here https://stackoverflow.com/questions/60073896/use-multivocal-libary-to-configure-repeat-intent-in-dialogflow-for-vui – DeMelkbroer Feb 05 '20 at 10:37