6

We're using QnA Maker attached to an Azure Bot Service. In the Knowledge Base (KB), we've added a Follow up Prompt to every question that says This did NOT answer my question:

Knowledge Base > Follow Up Prompt

Meaning if a bad response was provided, the end-user could indicate so, and the conversation history would look like this:

Example Conversation

What we're trying to do is replay that conversation history so we can see:

  1. The original user prompt
  2. The original answer
  3. The subsequent followup question / answer

We have Application Insights turned on, so we can see both questions as they come through from the logs with the following query:

traces
| project timestamp, 
          itemId,
          question = customDimensions.Question,
          answer = customDimensions.Answer
| order by timestamp

Which will return these two rows:

App Insights Logs

However, we're trying to find a unique conversation id or session id that can correlate both of those records. Notice that the itemId is very similar, but not identical:

53be8c14-702c-11ea-8c41-11c1c266dc55
53be8c13-702c-11ea-8c41-11c1c266dc55

Is there a unique key that can be used to join these two events?

One workaround is to just use the first 7 digits of the itemID and join based on that partial match like this:

traces
| where customDimensions.Question contains "This did NOT answer my question" 
| project itemId,
          SessionID = extract("^[a-z0-9]{7}", 0, itemId),
          timestamp
| join (
    traces
    | extend question = tostring(customDimensions['Question'])
    | extend answer = tostring(customDimensions['Answer'])
    | where message contains "GenerateAnswer" 
        and question  !contains "This did NOT answer my question" 
    | project itemId,
              SessionID = extract("^[a-z0-9]{7}",0,itemId),
              question,
              answer,
              timestamp
) on SessionID 
| project question, answer, SessionID, timestamp //, itemId, itemId1
| order by timestamp desc, SessionID

But we're not sure that value will reliably only differ by the 8th digit, so would prefer a less fragile ID

Slavik N
  • 4,705
  • 17
  • 23
3-putt
  • 219
  • 1
  • 9
  • Are you just trying to examine this via App Insights? It seems Active Learning might work here, though I haven't personally implemented that yet. Then you could handle collecting the feedback and training your KB directly instead of having to review the App Insights reports. – billoverton Mar 27 '20 at 18:29
  • itemID seems to be a property of QNA framework:https://github.com/garypretty/qnamaker-sync/blob/master/README.md - and you may want to check how your QnAItem(s) are defined. From your example it looks that first part of the UUID is not stable (53be8c14, 53be8c13) - while the remaining ones are stable (702c-11ea-8c41-11c1c266dc55). My guess would be that it is better to use trailing parts of the UUID. – Alexander Sloutsky Mar 31 '20 at 06:43
  • Have you considered adding [conversation history](https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-howto-v4-state?view=azure-bot-service-4.0&tabs=csharp) to your bot, then when the "this did not answer my question" button is clicked, send up a [custom telemetry event](https://docs.microsoft.com/en-us/azure/bot-service/bot-builder-telemetry?view=azure-bot-service-4.0) to Application Insights with the data that you want. That way you just query the custom telemetry event details. These could use a combination of the user id, channel id, conversation id, and time as the unique key. – Matt Stannett Apr 05 '20 at 02:53
  • You might also find this [Power BI template](https://microsoft.github.io/botframework-solutions/solution-accelerators/tutorials/view-analytics/1-intro/) helpful which allows you to drill into details. – Matt Stannett Apr 05 '20 at 02:55
  • @MattStannett, can you add that info to an answer? Providing the simplest vehicle toward collecting a Conversation ID if one isn't available in the default AI data is definitely the next step toward a solution – KyleMit Apr 06 '20 at 12:56

1 Answers1

-1

I recently implemented a QnA maker BOT with active learning.

The sample is located here

It is very similar to what you are trying to achieve.

enter image description here

Sash Sheen
  • 105
  • 2
  • 14
  • Hey Sash, I don't think this is what the primary concern was which is how to dig through the logs and get a unique conversation ID. Adding **Active Learning** is peripheral to that concern – KyleMit Mar 31 '20 at 18:42