Python Subprocess Grep

Question

I am trying to use the grep command in a python script using the subprocess module.

Here's what I have:

userid = 'foo12'
p = subprocess.Popen(['grep', "%s *.log"%userid], stdout=subprocess.PIPE)

And it returns nothing. I am not entirely sure what I am doing wrong so can someone please explain. The current method that I am using that works is by adding the shell=true which makes it output the correct output but as the help pages have pointed out it is unsafe. I need help trying to make this work so that my script isn't unsafe.

Answer 1

I think you're running up against two problems:

1. This call:

   p = subprocess.Popen(['grep', "%s *.log"%userid]...
   

   will not work as expected without shell=True because the list of arguments are being passed directly to os.execvp, which requires each item to be a single string representing an argument. You've squished two separate arguments together into a single string (in other words, grep is interpreting "foo12 *.log" as the pattern to search, and not pattern+file list).

   You can fix this by saying:

   p = subprocess.Popen(['grep', userid, '*.log']...)
   

2. The second issue is that, again without shell=True, execvp doesn't know what you mean by *.log and passes it directly along to grep, without going through the shell's wildcard expansion mechanism. If you don't want to use shell=True, you can instead do something like:

   import glob
   args = ['grep', userid]
   args.extend(glob.glob('*.log')
   p = subprocess.Popen(args, ...)
   

Answer 2

Here are two tested pieces of code to model from:

>>> print subprocess.check_output(['grep', 'python', 'api_talk.txt'])
Discuss python API patterns
Limitations of python
Introspection in python

>>> print subprocess.check_output('grep python *.txt', shell=True)

Use the latter if you want the shell to do wildcard expansion for you. When shell is True, be sure to put the whole command in a single string rather than a list of separate fields.

Answer 3

I am assuming you want to grep for 'foo12' in all files that end with '.log', to get this to work with just subprocess you will need to change your code to the following:

userid = 'foo12'
p = subprocess.Popen('grep %s *.log' % userid, stdout=subprocess.PIPE, shell=True)

shell=True is necessary for the wildcard expansion, and when that option is set you need to provide a string command instead of a list.

Also, make sure when you are providing a list of arguments that each argument is a separate entry in the list, your initial code would have been equivalent to the following:

grep 'foo12 *.log'