2

We've got a number of python libraries that need to be called from inside some loops in .NET that are just too slow. We're using Process.Start, which is taking about 1/3 of a second per call, which means that a few dialogs take 30 seconds to load (on an 8 core machine - our customers will most likely have much slower computers).

For various reasons, we can't use IronPython (such as some of the files using the csv module which has known issues with IronPython).

What can I do to speed things up? Even though I'm a python newb, some of these functions are simple if elif else blocks, and some profiling shows the main "cost" is starting python.exe a few dozen times. Is there some secret options to starting a single python process and streaming things in and out?

A similar question is this one.

Community
  • 1
  • 1
Tangurena
  • 2,121
  • 1
  • 22
  • 41
  • Could you run the scripts in parallel? – vcsjones Apr 15 '11 at 21:38
  • There is another option to CSV parsing with IronPython: Use the TextFieldParser. It is one of the most complete CSV parsers out there and it's already installed in .NET. http://msdn.microsoft.com/en-us/library/microsoft.visualbasic.fileio.textfieldparser.aspx – Tergiver Apr 15 '11 at 22:00
  • @Tergiver, the modules are also called by other scripts and applications, so I can't edit out usage of the CSV module. – Tangurena Apr 18 '11 at 19:56

3 Answers3

3

You can rewrite your scripts to accept commands from the standard input instead and not exit when they're finished.

Then start up a process (or a pool of them) and feed them commands, instead of starting new processes. For example instead of creating a new some_script.py args1, some_script.py args2, ... only spawn some_script_wrapper.py and feed it:

args1
args2
\end

You'll save the startup time this way and can even enable multiprocessing in a pool if needed.

Of course you'll have to make sure input / output is processed completely between data parts. To make it easier, you can wrap your arguments / output into a known structured format (json?), make sure you know the "message size", or even just use magic markers to find out when you get the end of a data chunk.

viraptor
  • 33,322
  • 10
  • 107
  • 191
  • Yep, the big killer in this situation is Python's own startup time, which is non-trivial. There's no magic "Python-as-a-service" preimaging technique, unfortunately. – ncoghlan Apr 17 '11 at 14:50
  • I'm probably going to have to do that, although I don't control the python library and I don't know how to do it. I'm reading Dive Into Python right now and what they're describing looks a lot like the code at the office - Dive Into Python is the standard Python reference at the office and I'm a newb at it. – Tangurena Apr 29 '11 at 04:45
2

You could try to embed python into your .NET application with pythonnet (http://pythonnet.sourceforge.net/readme.html#embedding) but unfortunately there is hardly any documentation on this.

Some pointers are http://mail.python.org/pipermail/pythondotnet/2007-June/000620.html and http://mail.python.org/pipermail/pythondotnet/2010-August/000994.html.

Samuel
  • 1,374
  • 10
  • 16
1

Don't call a python function in every loop. If you need to call a function 100 times, modify said function to accept 100 inputs and return 100 outputs. In .NET construct what I'll call your batch query, send it off in one huge function call, then move on to whatever else needs to be done.

One could call my suggestion buffering, though it may not be possible depending on what you're actually doing.

colechristensen
  • 475
  • 1
  • 3
  • 11