3

Is there anything more efficient or faster in Python that using the subprocess.call method? I noticed there was a SO question (Python subprocess module much slower than commands (deprecated)) about this a year ago except it became depreciated due to Python removing that alternative, so I am looking to see if this is now the only and thus fastest means to call a command to the shell from within Python. Specifically I am running the following command:

subprocess.call("foo",shell=True)

Or should I be using os?

Thanks.

Community
  • 1
  • 1
eWizardII
  • 1,916
  • 4
  • 32
  • 55
  • Can you post a link to the original question? – sid16rgt Sep 30 '13 at 19:27
  • Thanks I've added the referenced link. – eWizardII Sep 30 '13 at 19:32
  • 1
    The post claims a modest 2ms disadvantage to subprocess and also suggests that skipping the intermediate shell `subprocess.call(["foo"])` makes a much bigger difference. If you really need to eek 2ms out of your code, consider a C extension. – tdelaney Sep 30 '13 at 20:14
  • @tdelaney an increase of 2 (actually 2.76) ms, yes, when your baseline is 3ms, meaning it was 92% longer to use subprocess. Maybe we have a different definition of 'modest' but that is nothing to sneeze at – von Mises Mar 18 '14 at 19:25
  • @vonMises, In the referenced post, the 3ms baseline was for a simple 'echo' - purposely constructed to highlight the subprocess execution time and not a particularly useful program to run. The 'ns' command took 274ms to execute (subprocess only adds 1%), and we don't know how long this question's 'foo' takes to execute. Generally, when running subprocess, 2ms is lost in the noise of the system and shouldn't be a worry. – tdelaney Mar 18 '14 at 19:53

2 Answers2

6

Probably not. You might be able to make the call slightly faster but in my experience, the biggest gain is to start the subprocess only once and then keep it alive.

Creating new processes is a very expensive operation that Python can't really make cheaper.

So instead of calling /bin/ls path... 100 times, create an external process that reads paths from stdin and returns the desired output without exiting.

As a (sad) example: In ca. 1997, I made the Java 1 compiler 1000 times faster by writing a wrapper that keeps the VM alive and just kicks off the compilation after receiving options via a socket. This wasn't mainly because of process creation times but also because starting a Java VM at that time and ago took very long.

Aaron Digulla
  • 321,842
  • 108
  • 597
  • 820
  • "Create an external process that reads paths from stdin and returns the desired output without exiting." Can you give an example pointing how to do that? – Mooncrater Jun 16 '18 at 17:46
  • @Mooncrater You will need the source code of the external process. Modify it to take arguments from stdin instead of CLI arguments. Use `subprocess.Popen()` in the parent to start & control it. – Aaron Digulla Jul 03 '18 at 15:06
1

Try using subprocess.Popen() instead. It has a number of advantages. For details please refer

Difference between subprocess.Popen and os.system

http://docs.python.org/2/library/subprocess.html

Community
  • 1
  • 1
user1918858
  • 1,202
  • 1
  • 20
  • 29