2

I need some feedback on C# - cPython integration for scientific application.

The scenario is the following: C# for data acquisition and visualization - CPython for data manipulation using several and changing third part library to implement domain specific tasks.

General use case:

  1. C# code get realtime data from device
  2. C# code pass data to cPython code and ask for elaboration
  3. Cpython code does the magic
  4. Cpython code pass result back to c# code
  5. C# visualizes data in a WPF application

Both c# and cPython code (item 1),3) and 5)) have already been optimized for speed. Data to be passed in both direction (item 2) and 4)) are essentially arrays of 10E6 double (no complex data structure involved).

I'd like to find an interfacing solution with good performance (speed in going from 1 to 5) and decoupling capabilities who also minimizes the development on client (c# code) side. Actually both side run on the same machine, but I'd like the solution to scale out.

Solutions I've tried - issues:

  • a) C# + porting python algorithm on .NET - too many code to write/rewrite on domain side, lack of specialized library cPython provide for almost everything on scientific side
  • b) C# embedding ironpython + wrapping/porting c++ extension on managed code (with and without ironclad) - borderline performance caused by the marshaling of structures used during algorithms execution - dll hell when upgrading version of subcomponents.
  • c) C# embedding ironpython + rpc between ironpython and python - issues in maintainability, not satisfied by having to "pass" data between the three C#/ironpython/python .
  • d) XML-RPC, JSON-RPC - not satisfied by performance (speed).

Solutions not already tried which I plan to evaluate:

Thanks in advance for any criticism, comments and suggestions.

Shadow The GPT Wizard
  • 66,030
  • 26
  • 140
  • 208
Filippo Medri
  • 21
  • 2
  • 1
    Run the python-part as a service and give it an interface using HTTP an option? – lboshuizen Oct 30 '12 at 15:14
  • @Filippo: could you please elaborate what kind of data (structure ..) your are handling and if/how many different operations you expect to implement and how the result looks like. JSON/XML+HTTP probably won't do you any good if you just want do do binary in/out. If its a single, specific operation taking and returning binary you could be ok just using sockets/pipes. – Simon Opelt Nov 03 '12 at 14:59
  • Iboshuizen: I have already tried XML-RPC and JSON-RPC over HTTP, and they are perfect on everything but speed (points 2. and 4. of the main post), so I guess that it is better to try other solutions. @Simon: tipical use case is that of a function taking as input a large (10E6) array of double precision float(64bit) and returning as output an array of double of the same dimension. I need to implement a lot of functions with similar sign, that is why I was looking towards an efficient rpc protocol. On the other hand sockets are okay on the performance side, but the design does not scale. – Filippo Medri Nov 06 '12 at 18:39

1 Answers1

1

I have a similar requirement (C#<->Python RPC) I've just been experimenting with Apache Thrift and I've very impressed. It's very fast: 50% slower than Pyro4/pickle, 5x faster than RPyC. This is for sending numpy arrays around as binary data. The remaining overhead with Thrift is likely the copying of the array data into/out of strings. If Thrift could be refactored to take buffer objects in place of strings, I would expect it to be as fast as Pyro.

The code-generation seems very clean and the API is nice. The main thing it lacks is documentation. Still, I've been able to figure everything out from the basic tutorial, tests and browsing the source.

As far as I can see ZeroRPC doesn't (yet) have a C# implementation.

user2896082
  • 21
  • 1