Communication between Python and C#

Question

I have a Python backend running machine learning algorithms. I want to use the same backend for both an Excel plugin (C#) and a website. I want both interfaces to send my training data (thousands of lines of numbers in arrays) to the same Python application and retrieve the results in the form of another array up to a few thousand lines.

The website would fetch data from a SQL database and send that data to Python, while the Excel plugin would take the data that is in the current worksheet and send that data to Python. I need to be able to create numpy arrays in Python before continuing to process the data. Note that the website would be running on the same machine where the Python application resides. I still haven't decided what I will use to code the website, but I was leaning towards Node.js.

I have done some research and found a few options:

1- Named pipes
2- Sockets
3- RPC server such as gRPC or XML-RPC.
4- Writing the data to a file and reading it back in Python
5- Web Service

Note: I would need the Python "server" to be stateful and keep the session running between calls. So I would need to have a kind of daemon running, waiting for calls.

Which one would you experts recommend and why? I need flexibility to handle several parameters and also large arrays of numbers. Using IronPython is not an option because I am running Keras on Python, which apparently does not support IronPython.

Answer 1

I had the same problem recently. I used a named pipe to transport data from python to my c# server, hope it helps you.

Python:

import win32pipe, win32file

class PipeServer():
    def __init__(self, pipeName):
        self.pipe = win32pipe.CreateNamedPipe(
        r'\\.\pipe\\'+pipeName,
        win32pipe.PIPE_ACCESS_OUTBOUND,
        win32pipe.PIPE_TYPE_MESSAGE | win32pipe.PIPE_READMODE_MESSAGE | win32pipe.PIPE_WAIT,
        1, 65536, 65536,
        0,
        None)
    
    #Carefull, this blocks until a connection is established
    def connect(self):
        win32pipe.ConnectNamedPipe(self.pipe, None)
    
    #Message without tailing '\n'
    def write(self, message):
        win32file.WriteFile(self.pipe, message.encode()+b'\n')

    def close(self):
        win32file.CloseHandle(self.pipe)


t = PipeServer("CSServer")
t.connect()
t.write("Hello from Python :)")
t.write("Closing now...")
t.close()

For this code to work you need to install pywin32 (best choice is from binarys): https://github.com/mhammond/pywin32

C#-Server:

using System;
using System.IO;
using System.IO.Pipes;

class PipeClient
{
    static void Main(string[] args)
    {
        using (NamedPipeClientStream pipeClient =
            new NamedPipeClientStream(".", "CSServer", PipeDirection.In))
        {

            // Connect to the pipe or wait until the pipe is available.
            Console.Write("Attempting to connect to pipe...");
            pipeClient.Connect();

            Console.WriteLine("Connected to pipe.");
            Console.WriteLine("There are currently {0} pipe server instances open.",
               pipeClient.NumberOfServerInstances);
            using (StreamReader sr = new StreamReader(pipeClient))
            {
                // Display the read text to the console
                string temp;
                while ((temp = sr.ReadLine()) != null)
                {
                    Console.WriteLine("Received from server: {0}", temp);
                }
            }
        }
        Console.Write("Press Enter to continue...");
        Console.ReadLine();
    }
}

Answer 2

You can use Python for .NET (Python.NET). It may require some changes to your code, but then it should work very well, once everything is in good shape.

Python.NET allows two-way communication between CPython and CLR.

Answer 3

Let me give you a neat and quick recipe, in the form of example code.

There are basically two ways to tie python in the backend of C# (or a C# winform app or gui or something similar).

Method1: Iron Python. In this method you install a .net package in your visual studio called IronPython. I would not prefer this, because assuming your machine learning model uses keras or a lot of other libraries. It would be another quest to get you installations ready and working in IronPython. And most importantly, it is not as good as your common virtual env or conda environment.

Method2: (The Good Method): Create a Custom Process in your C# that takes arguments from your GUI, knows the path to your script and your python env. Using all these things, it calls your python code exactly the way you would call it in your terminal and pass arguments to it.

Now the tasty example code (I have used this simple trick and it always helps make my black screen python stuff look good with the cover of C# apps).

Python Part

import sys
a = sys.argv[1]
b = sys.argv[2] 
print("The Sum = ", float(a)+float(b))

The C# Part So here is the python process/function that you need to call on the click event of your sum button in the application

static void PythonProcess()
        {
            //1) Create Process Info
            var psi = new ProcessStartInfo();
            
            //Conda Env Path
            psi.FileName = @"C:\Users\jd\.conda\pkgs\py\python.exe";

            //2) Provide Script and the Arguments
            var script = @"C:\Users\jd\Desktop\script.py";
            var a = "15";
            var b = "18";

            psi.Arguments = $"\"{script}\" \"{a}\" \"{b}\"";

            //3) Process Configuration
            psi.UseShellExecute = false; 
            psi.CreateNoWindow = true;
            psi.RedirectStandardOutput = true;
            psi.RedirectStandardError = true;

            //4) Execute Process and get Output.
            var errors = "";
            var results = "";

            using(var process = Process.Start(psi))
            {
                errors = process.StandardError.ReadToEnd();
                results = process.StandardOutput.ReadToEnd();
            }

            //5) Display Output
            Console.WriteLine("ERRORS: ");
            Console.WriteLine(errors);
            Console.WriteLine();
            Console.WriteLine("RESULTS: ");
            Console.WriteLine(results);
        }

Answer 4

Calling Python from C# is easily possible via Pyrolite where your Python code is running as a Pyro4 server. It should be fast enough to handle "large arrays of numbers" however you didn't specify any performance constraints.

Answer 5

I had the same issue and seem to end up with named pipes. Here is a nice example of how to set it up to talk C# => Python, assuming C# is the server. It can use the same way to talk back or just Python.net to call directly through CLR as shown here. I use the latter.