Why is running Python subprocess grep ignoring --exclude-dir flag

Question

When executing this command from the terminal:

\grep -inH -r --exclude-dir={node_modules,.meteor,.git} -e test -- /Users/nelsyeung/Sites/foo

It outputs the correct results, where it excludes the --exclude-dir directories.

The following Python script should theoretically perform the exact same operation:

#!/usr/bin/env python3
from subprocess import Popen, PIPE

cmd = 'grep -inH -r --exclude-dir={node_modules,.meteor,.git} -e test -- /Users/nelsyeung/Sites/foo'
with Popen(cmd.split(), stdout=PIPE, bufsize=1, universal_newlines=True) as p:
    for l in p.stdout:
        print(l)

But the --exclude-dir flag seems to be ignored, that is, it also grep from node_modules, .meteor and .git.

Question: Why is the above code outputting a different result from just executing the command?

Note that this is more of a theory question than me looking for an alternative fix, since the Python code is basically from a plugin that I'm using where I can only supply flags for the command, and just that, to my surprise, passing in --exclude-dir does nothing. If there's something wrong with the code above, do point it out though.

System info:

OS: macOS 10.13.6
Python: 3.7.0
grep: (BSD grep) 2.5.1-FreeBSD (with --exclude-dir support)

score 2 · Accepted Answer · answered Aug 07 '18 at 11:10

--exclude-dir={dir1,dir2} is expanded to

--exclude-dir=dir1 --exclude-dir=dir2

by the shell, not by grep. Popen uses shell=False by default.

So instead use

from subprocess import Popen, PIPE

cmd = '''grep -inH -r --exclude-dir=node_modules --exclude-dir=.meteor 
         --exclude-dir=.git -e test -- /Users/nelsyeung/Sites/foo'''
with Popen(cmd.split(), stdout=PIPE, bufsize=1, universal_newlines=True) as p:
    for l in p.stdout:
        print(l)

(Note that while using shell=True might be another option, it is not preferred because of security issues.)

Why is running Python subprocess grep ignoring --exclude-dir flag

1 Answers1