1

I would like to embed a command in a python script and capture the output. In this scenario I'm trying to use "find" to find an indeterminate number of files in an indeterminate number of subdirs, and grep each matching file for a string, something like:

grep "rabbit" `find . -name "*.txt"`

I'm running Python 2.6.6 (yeah, I'm sorry too, but can't budge the entire organization for this right now).

I've tried a bunch of things using subprocess, shlex, etc. that have been suggested in here, but I haven't found a syntax that will either swallow this, or ends up sucking the "find..."as the search string forgrep`, etc. Suggestions appreciated. Ken

Colonel Thirty Two
  • 23,953
  • 8
  • 45
  • 85
Ken
  • 369
  • 3
  • 15
  • 1
    It can be done easily using `subprocess.call(..., shell=True)` but such use of the shell actually makes for unsafe Python code. Why not traverse the directory from within Python and also do what grep does in Python? It's fairly simple and makes your code more portable. – 5gon12eder Sep 19 '14 at 16:01
  • @5gon12eder I may do this eventually, but kind of wanted to figure out how to do this mess as command output, too. Been looking at os.walk(). etc. Also just guessing that if I need to parse 5 or 6 hundred files, that "mygrep" might not be as quick as "realgrep". Still, I agree that traverse/python is an excellent choice, since what happens after I find the rabbits will take ages longer than finding them, however I do it. – Ken Sep 19 '14 at 16:23
  • related: [How do I use subprocess.Popen to connect multiple processes by pipes?](http://stackoverflow.com/q/295459/4279) – jfs Sep 22 '14 at 23:19
  • @5gon12eder: if the command is trusted e.g., it is hardcoded in the source then there is nothing unsafe about `shell=True` compared to other ways to run it. It probably improves readability -- in a sense it might make the code safer (less bugs). – jfs Sep 22 '14 at 23:22
  • @J.F.Sebastian I was assuming that `.` will be replaced by some variable. If so, all doors for shell code injection are wide open. If the command is indeed a hard-coded string it will probably be fine as you say. Except that it ties your Python program to the POSIX platform for no good reason. Personally, I would almost never use the shell. Python is so powerful that it doesn't really add any convenience to spawn a shell. – 5gon12eder Sep 23 '14 at 13:17

1 Answers1

2
import subprocess


find_p = subprocess.Popen(["find", ".", "-name", "*.txt"], stdout=subprocess.PIPE)
grep_p = subprocess.Popen(["xargs", "grep", "rabbit"], stdin=find_p.stdout)

grep_p.wait()
GP89
  • 6,600
  • 4
  • 36
  • 64
  • 2
    This `grep`s for "rabbit" in the list of file names instead of in the files. That is to say, it mimics `find . -name "*.txt" | grep "rabbit"` instead of the OP's command. – 5gon12eder Sep 19 '14 at 16:02
  • @5gon12eder ah true I overlooked that, you could use xargs. Not identical, but should work – GP89 Sep 19 '14 at 16:05
  • Thanks, this seems to work just great, in its second incarnation. I may eventually do the traverse/parse with python suggested above, but this will help get things off the dime quickly. - Ken – Ken Sep 19 '14 at 16:27