How to get result from grep with Python

Question

I am trying to get output from grep with wildcard

proc = subprocess.Popen(['grep', '002HQV', 'test.*'], stdout=subprocess.PIPE,  
shell=True)
res = proc.stdout.readlines()
print(res)

but get the following error

2.4.3 (#1, Jun 11 2009, 14:09:37)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)]
Usage: grep [OPTION]... PATTERN [FILE]...  
Try `grep --help' for more information.
[]

Is it wrong with my grep syntax?

The following works

proc = subprocess.Popen(['ls', '*'], stdout = subprocess.PIPE, shell=True)

and

os.system("grep 02HQV test.*")

Possible duplicate of [Python - how to execute shell commands with pipe?](https://stackoverflow.com/questions/9393425/python-how-to-execute-shell-commands-with-pipe) — Gweltaz Niquel, Aug 17 '18 at 08:56

score 2 · Accepted Answer · answered Aug 17 '18 at 09:11

2

When using shell=True your should use a string, see subprocess.call using string vs using list

You can avoid using a shell and still use a list by using the glob standard library module:

import subprocess
import glob

command = ['grep', '002HQV']
command.extend(glob.glob('test.*'))
proc = subprocess.Popen(command, stdout=subprocess.PIPE)
res = proc.stdout.readlines()
print(res)

answered Aug 17 '18 at 09:11

cdarke

42,728
8
80
84

but how to explain `proc = subprocess.Popen(['ls', '*'], stdout = subprocess.PIPE, shell=True)` works? – Hsing Yi Aug 17 '18 at 09:37
@HsingYi - the `*` gets ignored, it is the same as `subprocess.Popen(['ls'], stdout = subprocess.PIPE, shell=True)`. The default for `ls` is the current directory `.`, the `*` shell expansion is all the file names in the current directory, which is what you get on both. – cdarke Aug 17 '18 at 18:06

score 0 · Answer 2 · answered Aug 17 '18 at 11:08

In general I'd try to avoid launching subprocesses for things that are simple string manipulation and straightforward file I/O. That's especially true when you start getting into actively dangerous things like subprocess.call(shell=True). You can use the glob module to do the filename expansion and then loop through the files.

Here's one example:

import glob
res = []
for fn in glob.glob('test.*'):
    with open(fn, 'r') as f:
        for line in f:
          if '002HQV' in line:
              res.append(line)
print(res)

How to get result from grep with Python

2 Answers2