method.Invoke(o, null); can be called only once from the body and not from within the keyboard handler

Question

I am trying to create a low-level keyboard handler to launch an external application using reflection when a specific keyboard combination is detected.

I can launch the app once directly with no errors, however from inside a key press handler the app crashes after correctly showing the MessageBox dialog.

I am reading the external dll into a byte array, for this test you could use any .Net application a simple Hello World win forms app will work.

string filePath = @".\x.exe";
FileStream fs = new FileStream(filePath, FileMode.Open);
BinaryReader br = new BinaryReader(fs);
byte[] bin = br.ReadBytes(Convert.ToInt32(fs.Length));
fs.Close();
br.Close();

Then loading the external Assembly and executing it via reflection:

public Assembly a;
public MethodInfo method;
public object o;

a = Assembly.Load(bin);
method = a.EntryPoint;
o = a.CreateInstance(method.Name);
method.Invoke(o, null); 

This works once, but If I repeat the above code to execute again it fails.

The following code demonstrates the failure on the second call, if you comment that out you can see the failure on the keyboard detection of Left+Down+A

Even when you comment out both calls in the program root, the app closes unexpectedly the first time when the keys are detected.

using System;
using System.Collections.Generic;
using System.Threading;
using System.Windows.Forms;
using System.Runtime.InteropServices;
using System.Text;
using System.Diagnostics;
using System.Reflection;
using System.IO;
using System.Windows.Input;

public class Program
{
    public static bool lctrlKeyPressed;
    public static bool f1KeyPressed;
    public static bool AKeyPressed;
    public static bool LeftKeyPressed;
    public static bool DownKeyPressed;

    public static Assembly a;
    // search for the Entry Point
    public static MethodInfo method;
    // create an istance of the Startup form Main method
    public static object o;

    [STAThread]
    public static void Main()
    {
        //Application.EnableVisualStyles();
        //Application.SetCompatibleTextRenderingDefault(false);

        string filePath = @".\x.exe";
        FileStream fs = new FileStream(filePath, FileMode.Open);
        BinaryReader br = new BinaryReader(fs);
        byte[] bin = br.ReadBytes(Convert.ToInt32(fs.Length));
        fs.Close();
        br.Close();

        //public Assembly a;
        //public MethodInfo method;
        //public object o;

        a = Assembly.Load(bin);
        method = a.EntryPoint;
        o = a.CreateInstance(method.Name);
        method.Invoke(o, null);

        a = Assembly.Load(bin);
        method = a.EntryPoint;
        o = a.CreateInstance(method.Name);
        method.Invoke(o, null);

        LowLevelKeyboardHook kbh = new LowLevelKeyboardHook();
        kbh.OnKeyPressed += kbh_OnKeyPressed;
        kbh.OnKeyUnpressed += kbh_OnKeyUnPressed;

        kbh.HookKeyboard();

        Application.Run();

        kbh.UnHookKeyboard();
    }

    public static void kbh_OnKeyPressed(object sender, Keys e)
    {
        if (e == Keys.A)
        {
            AKeyPressed = true;
        }
        else if (e == Keys.Left)
        {
            LeftKeyPressed = true;
        }
        else if (e == Keys.Down)
        {
            DownKeyPressed = true;
        }
        CheckKeyCombo();
    }

    public static void kbh_OnKeyUnPressed(object sender, Keys e)
    {
        if (e == Keys.A)
        {
            AKeyPressed = false;
        }
        else if (e == Keys.Left)
        {
            LeftKeyPressed = false;
        }
        else if (e == Keys.Down)
        {
            DownKeyPressed = false;
        }
    }

    public static void CheckKeyCombo()
    {
        if (AKeyPressed && LeftKeyPressed && DownKeyPressed)
        {
            MessageBox.Show("Left Key + Down Key + A Key + Pressed");
            
            //a = Assembly.Load(bin);
            method = a.EntryPoint;
            o = a.CreateInstance(method.Name);
            method.Invoke(o, null);
        }
    }
}

public class LowLevelKeyboardHook
{
    private const int WH_KEYBOARD_LL = 13;
    private const int WM_KEYDOWN = 0x0100;
    private const int WM_SYSKEYDOWN = 0x0104;
    private const int WM_KEYUP = 0x101;
    private const int WM_SYSKEYUP = 0x105;

    [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);

    [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    [return: MarshalAs(UnmanagedType.Bool)]
    private static extern bool UnhookWindowsHookEx(IntPtr hhk);

    [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode, IntPtr wParam, IntPtr lParam);

    [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    private static extern IntPtr GetModuleHandle(string lpModuleName);

    public delegate IntPtr LowLevelKeyboardProc(int nCode, IntPtr wParam, IntPtr lParam);

    public event EventHandler<Keys> OnKeyPressed;
    public event EventHandler<Keys> OnKeyUnpressed;

    private LowLevelKeyboardProc _proc;
    private IntPtr _hookID = IntPtr.Zero;

    public LowLevelKeyboardHook()
    {
        _proc = HookCallback;
    }

    public void HookKeyboard()
    {
        _hookID = SetHook(_proc);
    }

    public void UnHookKeyboard()
    {
        UnhookWindowsHookEx(_hookID);
    }

    private IntPtr SetHook(LowLevelKeyboardProc proc)
    {
        using (Process curProcess = Process.GetCurrentProcess())
        using (ProcessModule curModule = curProcess.MainModule)
        {
            return SetWindowsHookEx(WH_KEYBOARD_LL, proc, GetModuleHandle(curModule.ModuleName), 0);
        }
    }

    private IntPtr HookCallback(int nCode, IntPtr wParam, IntPtr lParam)
    {
        if (nCode >= 0 && wParam == (IntPtr)WM_KEYDOWN || wParam == (IntPtr)WM_SYSKEYDOWN)
        {
            int vkCode = Marshal.ReadInt32(lParam);

            OnKeyPressed.Invoke(this, ((Keys)vkCode));
        }
        else if (nCode >= 0 && wParam == (IntPtr)WM_KEYUP || wParam == (IntPtr)WM_SYSKEYUP)
        {
            int vkCode = Marshal.ReadInt32(lParam);

            OnKeyUnpressed.Invoke(this, ((Keys)vkCode));
        }

        return CallNextHookEx(_hookID, nCode, wParam, lParam);
    }
}

Answer 1

There are two competing issues to this code as it is:

1. You cannot launch the app twice in succession from the Main method.
2. You cannot launch the app from the keyboard event handler.

You haven't posted the specific exceptions but if I create a standalone console app for x.exe then I can replicate the first issue, it raises this exception:

System.InvalidOperationException: 'SetCompatibleTextRenderingDefault must be called before the first IWin32Window object is created in the application.'

Inside the x.exe app I can remove the following line from the main method to get past this issue. What it suggests though is that the second time the app launches, the previous invocation is still loaded in the current AppDomain.

[STAThread]
static void Main()
{
    Application.EnableVisualStyles();
    //Application.SetCompatibleTextRenderingDefault(false);
    Application.Run(new Form1());
}

If we comment out the test loaders, the second exception is detected as this:

System.InvalidOperationException: 'Starting a second message loop on a single thread is not a valid operation. Use Form.ShowDialog instead.'

So even when we do NOT have an issue with the app being loaded twice into the same context, the problem is that we are trying to start a new Message Loop within the confines of an [STAThread] a single threaded compartment.

If you do not need a reference to the application that is being launched, then it might just be simpler to use ProcessStart to start the external process, this is similar to executing the external application from the command prompt (CMD):

ProcessStartInfo ps = new ProcessStartInfo(filePath);
System.Diagnostics.Process.Start(ps);

This works so well and is far simpler than loading the assembly into memory and reflecting into it that I wouldn't look for something more complex unless you really needed to, this will work for your existing app, and provides support for other non-.net apps as well.

Have a play with the following code that shows a few options:

    private static string filePath;

    [STAThread]
    public static void Main()
    {
        //Application.EnableVisualStyles();
        //Application.SetCompatibleTextRenderingDefault(false);

        filePath = @".\x.exe";

        // This will start an instance and no wait for it to close
        StartProcess();

        // These will each wait for the launched process to end before moving on 
        WaitForProcess();
        WaitForProcess();

        LowLevelKeyboardHook kbh = new LowLevelKeyboardHook();
        kbh.OnKeyPressed += kbh_OnKeyPressed;
        kbh.OnKeyUnpressed += kbh_OnKeyUnPressed;

        kbh.HookKeyboard();

        Application.Run();

        kbh.UnHookKeyboard();
    }

    private static void StartProcess()
    {
        ProcessStartInfo ps = new ProcessStartInfo(filePath);
        System.Diagnostics.Process.Start(ps);
    }

    private static bool _InProgress = false;
    /// <summary>Launches the external process if it was not already launched, state is managed using <seealso cref="_InProgress"/></summary>
    /// <returns> true if the app was launched, false indicates the process was still waiting.</returns>
    private static bool WaitForProcess()
    {
        if (_InProgress) return false; // do nothing
        _InProgress = true;
        try
        {
            ProcessStartInfo ps = new ProcessStartInfo(filePath);
            System.Diagnostics.Process.Start(ps).WaitForExit();
            return true;
        }
        finally
        {
            _InProgress = false;
        }
    }

    public static void kbh_OnKeyPressed(object sender, Keys e)
    {
        if (e == Keys.A)
        {
            AKeyPressed = true;
        }
        else if (e == Keys.Left)
        {
            LeftKeyPressed = true;
        }
        else if (e == Keys.Down)
        {
            DownKeyPressed = true;
        }
        CheckKeyCombo();
    }

    public static void kbh_OnKeyUnPressed(object sender, Keys e)
    {
        if (e == Keys.A)
        {
            AKeyPressed = false;
        }
        else if (e == Keys.Left)
        {
            LeftKeyPressed = false;
        }
        else if (e == Keys.Down)
        {
            DownKeyPressed = false;
        }
    }

    public static void CheckKeyCombo()
    {
        if (AKeyPressed && LeftKeyPressed && DownKeyPressed)
        {
            MessageBox.Show("Left Key + Down Key + A Key + Pressed");

            //It's up to you if is makes sense to wait for a response or not.
            //StartProcess();
            
            // Wait will still raise the above message box if the other app is still running
            // But it wont't try to launch the app again
            // You can play around with the timing logic for what makes sense for you.
            WaitForProcess();
        }
    }
}

---

It's out of scope for this post, but you will need to make changes to both apps if you really need to use reflection to load the external app.

The following is my attempt to being able to unload the external assembly after reading through: https://www.west-wind.com/presentations/dynamicCode/DynamicCode.htm but every invocation after the first still exists, so Its not really unloading at all.

this requires you to put the dll loading code back into the startup, but store the byte[] bin as a static field, not a method local one.

    private static void LoadApp()
    {
        AppDomainSetup domainStarter = new AppDomainSetup();
        domainStarter.ApplicationBase = AppDomain.CurrentDomain.BaseDirectory;

        AppDomain domain = AppDomain.CreateDomain("XternalApp", null, domainStarter);
        var assembly = domain.Load(bin);
        var cls = assembly.EntryPoint;
        //object instance = assembly.CreateInstance(domain.Load((assembly.FullName, cls.DeclaringType.FullName + "." + cls.Name);
        object result = cls.Invoke(null, null);

        result = null;
        //instance = null;
        cls = null;
        assembly = null;

        GC.Collect();

        AppDomain.Unload(domain);

        //GC.Collect();
    }

Making these changes to the original app makes the above LoadApp() method work
EVEN FROM THE KEYBOARD EVENT!:

    static Form1 instance = null;
    /// <summary>
    /// The main entry point for the application.
    /// </summary>
    //[STAThread]
    [MTAThread]
    static void Main()
    {
        if (instance == null)
        {
            Application.EnableVisualStyles();
            Application.SetCompatibleTextRenderingDefault(false);
            instance = new Form1();

            Application.Run(instance);
        }
        else
        {
            // TODO: take this out, just here for effect
            MessageBox.Show("Already running!");
            instance = new Form1();
            instance.Show();
        }
    }

Answer 2

I would like to Say a BIG Thank You for everyone's participation

Basically what I intended for the C# code to do is this

1. X.exe is a macro created by JitBit Macro Recorder
2. The C# reads in Keyboard buttons press
3. When a sequence of keyboard buttons are pressed it runs the X.exe Macro

It is a horrendous way of doing things

But it is the best thing I have going right now

In the C# code, I thus, needed to be able to run X.exe as quickly as possible

This is why I loaded it into memory

X.exe should also be able to be called many many times

This is for a Game that I am playing

The C# program runs in the background while I am playing the game

And if I press a certain sequence of keyboard buttons, the C# will run X.exe

Which is essentially just a Macro by JitBit Macro Recorder

Why did I use Jitbit Macro Recorder ?

Because it is an extremely fast Macro Engine and it works very well

With what I have just said now

Any further contribution for the C# program is very much appreciated

I thank you all sincerely !!!