0

Here is a toy example of process substitution that works fine in Bash:

$ wc -l <(pwd)
1 /proc/self/fd/11

So why does the same command give a syntax error when invoked from Python's subprocess with shell=True? 

>>> subprocess.check_call('wc -l <(pwd)', shell=True)
/bin/sh: 1: Syntax error: "(" unexpected
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/my/python/lib/python3.5/subprocess.py", line 581, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'wc -l <(pwd)' returned non-zero exit status 2
1''
  • 26,823
  • 32
  • 143
  • 200

3 Answers3

3

/bin/sh: 1: Syntax error: "(" unexpected

You have a bashism. It is not valid according to POSIX, which is what /bin/sh implements.

dsh
  • 12,037
  • 3
  • 33
  • 51
3

If you want to use Bash features (arrays, process substitution, here strings, or a lot of other non-POSIX extensions and enhancements), you need to explicitly override the default shell:

subprocess.check_call(
    'wc -l <(pwd)',
    executable='/bin/bash',  # the beef
    shell=True)

or - somewhat more clumsily - run an explicit Bash instance:

subprocess.check_call(
    ['/bin/bash', '-c', 'wc -l <(pwd)'])

Notice how in the latter case we avoid separately specifying shell=True, and pass in the script as a list of strings (where the third string is an arbitrarily complex and/or long script as the argument to bash -c).

(Actually there is a length limit. If your command line is longer than the kernel constant ARG_MAX you'll need to pass the script in a file or as standard input to the shell instead. On any modern system, we are talking megabytes of script, though.)

Running complex shell scripts (Bash or otherwise) from Python is dubious, anyway; you'll want to delegate absolutely as little as possible to a subprocess and take it from there in native Python code.

As a belated aside, running wc on a directory is weird. You also don't need pwd at all;

wc -c .

Perhaps see also What exactly is current working directory?

tripleee
  • 175,061
  • 34
  • 275
  • 318
2

An alternate solution is to shift more of the shell code to Python itself. For example:

from subprocess import Popen, PIPE, check_call

p1 = Popen(["pwd"], stdout=PIPE)
p2 = check_call(["wc", "-l"], stdin=p1.stdout)

This could often be the first step towards eliminating the need to use subprocess at all, as it decomposes the work into smaller chunks for which you may more readily see how to do in Python itself.

chepner
  • 497,756
  • 71
  • 530
  • 681
  • Might as well do it all natively then; `with open(".") as d: result = len(d.readlines())` though I guess Python won't let you `open` a directory as such. – tripleee Jan 06 '21 at 14:49
  • The correct Python way would be something like `sum(1 for _ in pathlib.Path.cwd().glob('*'))`. – chepner Jan 06 '21 at 16:32
  • Um, no, that counts the number of files in the directory, not the number of lines in the directory name. But this is a toy example anyway. – tripleee Jan 06 '21 at 18:17
  • Actually `len(os.getcwd().split('\n'))` is probably better than my first attempt. – tripleee Jan 06 '21 at 18:20
  • I never read the original that closely. Getting the number of lines in the name seems even less useful than the number of files in the directory, though. – chepner Jan 06 '21 at 18:22