How to open a local application using Cortana skills

Question

I've been trying to do this since this morning and it doesn't seem to be working for me.

The requirement is to have the user invoke cortana ana ask her to open an application - let's call it app1.

I created an azure bot based on the EchoBot and replaced the answering code with this:

protected override async Task
OnMessageActivityAsync(ITurnContext<IMessageActivity> turnContext,
CancellationToken cancellationToken)
    {
        await turnContext.SendActivityAsync(MessageFactory.Text($"Echo: 
                                Opening app1..."), cancellationToken);

        var startInfo = new ProcessStartInfo
        {
            FileName = @"D:\_Projects\xyz\app1.exe",
            UseShellExecute = false,
            CreateNoWindow = false,
        };

        var process = Process.Start(startInfo);
        var success = process != null && process.WaitForExit(30 * 10000);

        if (!success)
        {
            //process?.Kill();
            throw new ApplicationException("A timeout occurred during 
                 method execution. The service interface did not finish in a
                        timely fashion.");
        }
        var exitCode = process.ExitCode;
    }

This does work when executed locally after downloading the code from Azure. But it does not work when invoked from Cortana.

Edit: Local testing done using the Bot Framework Emulator (v4)

It simply prints the Opening App1 line and stands there. The debug window is as expected - useless.

Now I tried using a completely different technique I read somewhere, it consists of adding the application locally to the user\programs\ folder and invoking it from Cortana by saying Open app1.

The problem is, Cortana doesn't recognize the app at all. it just starts Edge and searches for app1 on bing.

I've seen a few videos regarding the cortana skill and in some of them launching a new app using done using a uwp application - But mine is actually an exe generated from python using auto-py-to-exe so this is not useful to me.

References: https://www.youtube.com/watch?v=h2L9KAWh5qs&t=2696s https://www.youtube.com/watch?v=6imjt5l7jXc

Is there a solution for this problem?

Just curious : Are there really anything serious under development targeting WIndows 10, and specially Cortana...? As far as i see, the stats don't even show any recent improvements made to Cortana even by MSFT. — Software Dev, Jun 18 '19 at 15:42
We usually use AWS and Alexa - but as the machines are running Windows 10 and have cortana installed and the application is local no need to go through the Cortana->Alexa->LocalApp when it shouldn't be difficult to do Cortana->LocalApp directly. — AngelicCore, Jun 18 '19 at 17:12
@Aousafrashid Cortana is still getting [some Win 10 updates](https://support.microsoft.com/en-us/help/4082884/windows-10-whats-new-in-cortana), but most of the updates are centered around Skills, particularly the new [Enterprise Skills](https://techcommunity.microsoft.com/t5/Cortana-Skills-Kit-Blog/Cortana-Skills-Kit-for-Enterprise-BUILD-2019-Update/ba-p/535109) — mdrichardson, Jun 19 '19 at 15:09

mdrichardson · Accepted Answer · 2019-06-18T20:18:09.573

1

The reason this works locally and not when deployed is because your code executes Process.Start() on whatever machine is running the code (server side) and not on the machine of the person interacting with your code (client side). When testing locally, server and client are the same machine; not so when deployed.

It's possible, on a limited basis, to accomplish what you want.

Read the doc, Launch apps or websites from a Cortana skill.

Currently, Cortana supports a single action: LaunchUri. So, to launch an app, the app must have a uri protocol associated with it.

For example, to launch the Windows Map App, which uses bingmaps: protocol, you would use (Note: My example uses Bot Framework V4 code, whereas the docs I linked use V3):

var message = MessageFactory.Text("Launching app...");
message.ChannelData = JObject.FromObject(new {
     action = new { type = "LaunchUri", uri = "bingmaps:?where=Paris"}
});

await context.SendActivityAsync(message);

If you need to access a non-UWP app, you'd have to register your own URI scheme on your local machine in the registry:

  your-protocol-name/
    (Default)    "URL:your-protocol-name Protocol"
    URL Protocol ""
    shell/
      open/
        command/
          (Default) PathToExecutable

edited Jun 18 '19 at 20:18

answered Jun 18 '19 at 17:59

mdrichardson

7,141
1
7
21

I expected as much, as those UWP applications that get launched are using the launchURI command, and indeed the process.start would trigger on the server side (although I wish it to trigger on cortana's side) Do you know of a way to do this using a standard exe without even bots? – AngelicCore Jun 18 '19 at 19:25
Without a bot, you can just ask Cortana to "Launch Microsoft Word". Also, I tried using a few different variations of the file URI scheme (`file:///`) and none of them worked. However, you could likely [register your own URI scheme](https://stackoverflow.com/a/389223/10860086) to at least get it working locally. – mdrichardson Jun 18 '19 at 20:18
Thanks mdrichardson, I was planning on something similar - your answer nudged me in this direction though. – AngelicCore Jun 19 '19 at 19:51
My plan was to have the bot open a UWP app, as this seemed to work and the UWP app would call the exe from within it. – AngelicCore Jun 19 '19 at 19:52
I actually went the route of opening the App using a UWP app mediator which in turns opens the Python exe. Scrapped Azure bots completely as they are extremely expensive even while testing. – AngelicCore Jun 22 '19 at 18:33

How to open a local application using Cortana skills

1 Answers1