How to correctly escape special characters in python subprocess?

Question

Im trying to run this bash command using python subprocess

find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'

output:-

helld.xl.py
parse_maillog.py
replace_pattern.py
split_text_match.py
ssh_bad_login.py

Here is what i have done in python2.7 way, but it gives the output where awk command filter is not working

>>> p1=subprocess.Popen(["find","/Users/johndoe/sandbox","-iname","*.py"],stdout=subprocess.PIPE)

>>> p2=subprocess.Popen(['awk','-F"/"','" {print $NF} "'],stdin=p1.stdout,stdout=subprocess.PIPE)

>>>p2.communicate()
('/Users/johndoe/sandbox/argparse.py\n/Users/johndoe/sandbox/custom_logic_substitute.py\n/Users/johndoe/sandbox/finditer_html_parse.py\n/Users/johndoe/sandbox/finditer_simple.py\n/Users/johndoe/sandbox/group_regex.py\n/Users/johndoe/sandbox/helo.py\n/Users/johndoe/sandbox/newdir/helld.xl.py\n/Users/johndoe/sandbox/parse_maillog.py\n/Users/johndoe/sandbox/replace_pattern.py\n/Users/johndoe/sandbox/split_text_match.py\n/Users/johndoe/sandbox/ssh_bad_login.py\n', None)

I could also get output by using p1 alone here like below,but i cant get the awk working here

list1=[]
result=p1.communicate()[0].split("\n")
for item in res:
    a=item.rstrip('/').split('/')
    list1.append(a[-1])
print list1

tripleee · Answer 1 · 2021-10-03T11:03:05.013

You are incorrectly passing in shell quoting (and extra shell quoting which isn't even required by the shell!) when you're not invoking a shell. Don't do that.

p2=subprocess.Popen(['awk', '-F/', '{print $NF}'], stdin=...

When you have shell=True you need extra quotes around some arguments to protect them from the shell, but there is no shell here, so putting them in is incorrect, and will cause parse errors by Awk.

However, you should almost never need to call Awk from Python, especially for trivial tasks which Python can easily do natively:

list1 = [line.split('/')[-1]
            for line in subprocess.check_output(
                ["find", "/Users/johndoe/sandbox",
                 "-iname", "*.py"]).splitlines()]

In this particular case, note also that GNU find already has a facility to produce this result directly:

list1 = subprocess.check_output(
    ["find", "/Users/johndoe/sandbox",
     "-iname", "*.py", "-printf", "%f\\n"]).splitlines()

You amazingly managed to remove the quotes in the `find` command line correctly, so it's not clear why you made this error. — tripleee, Nov 27 '17 at 06:46
Tangentially see also https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable — tripleee, Nov 27 '17 at 06:51

score 0 · Answer 2 · answered Nov 26 '17 at 17:12

0

Use this: p2.communicate()[0].split("\n").

It will output a list of lines.

answered Nov 26 '17 at 17:12

mcchucklezz

376
1
16

i could use it but i really want to know how i use `awk` here which involves special chars like `" , { , }, ' , /` etc – sherpaurgen Nov 26 '17 at 17:29
`/` is not any more "special" than `.` or `a`. – tripleee Oct 03 '21 at 11:07

score 0 · Answer 3 · answered Nov 27 '17 at 05:49

if you don't have any reservation using shell=True , then this should be pretty simple solution

from subprocess import Popen
import subprocess
command='''
find /Users/johndoe/sandbox -iname "*.py" | awk -F'/' '{ print $NF}'
'''
process=Popen(command,shell=True,stdout=subprocess.PIPE)
result=process.communicate()
print result

How to correctly escape special characters in python subprocess?

3 Answers3