Python subprocess module fails when calling awk

Question

Need help in calling the below mentioned awk script using Python subprocess module? .I need to get this working as our monitoring tool requires this. I am a novice in python and I have tried the below and it hasn't helped.

>>> print subprocess.check_output(["awk '!/bind|swap|shm/ && $1 !~/#|^$/ {system("if [[ -n $(findmnt -m " $2 ") ]]; then echo Mount " $2 " is mounted.;else echo Mount " $2 " is NOT mounted.;fi")}' /etc/fstab"], shell=True, universal_newlines=True)
  File "<stdin>", line 1
    print subprocess.check_output(["awk '!/bind|swap|shm/ && $1 !~/#|^$/ {system("if [[ -n $(findmnt -m " $2 ") ]]; then echo Mount " $2 " is mounted.;else echo Mount " $2 " is NOT mounted.;fi")}' /etc/fstab"], shell=True, universal_newlines=True)
                                                                                           ^
SyntaxError: invalid syntax
>>>

Do you have a good reason for using `shell=True`? It makes your job a lot harder here. — Charles Duffy, Jan 15 '20 at 17:40
Well, for starters clearly your syntax is invalid. As you can see by the highlighted syntax in your post, you aren't escaping `"` in your command string, so start with fixing that. `"awk '!/bind|swap|shm/ && $1 !~/#|^$/ {system(\"if [[ -n $(findmnt -m \" $2 \") ]]; then echo Mount \" $2 \" is mounted.;else echo Mount \" $2 \" is NOT mounted.;fi\")}' /etc/fstab"` — h0r53, Jan 15 '20 at 17:41
That said, using `system()` inside `awk` is an **extremely** bad idea (as in, parses your data as code, easy route for security vulnerabilites, *really* don't do it bad idea). — Charles Duffy, Jan 15 '20 at 17:42
I am a novice. Testing all the options as I would need this working for python 2.7 and 3x — rony thomas, Jan 15 '20 at 17:49
Note too that it's unreliable to use bashisms like `[[` inside `system()`, which uses `/bin/sh` -- which, on platforms like Debian, will be implemented with dash, which only supports the POSIX-compliant `[` test operator. — Charles Duffy, Jan 15 '20 at 17:53

score 2 · Accepted Answer · answered Jan 15 '20 at 17:48

2

Don't.

The system() call in awk is dangerous, and should never be used. For your current use case, it's more appropriate to use native bash instead of awk:

import subprocess
shell_script = '''
set -f # disable globbing
while IFS= read -r line; do
  [[ $line =~ bind|swap|shm ]] && continue  # skip lines containing "bind", "swap" or "shm"
  [[ $line = "#"* ]] && continue            # skip comments
  [[ $line ]] || continue                   # skip empty lines
  set -- $line                              # word-split line content into $1, $2, etc
  if [[ -n $(findmnt -n "$2") ]]; then
    echo "Mount $2 is mounted"
  else
    echo "Mount $2 is NOT mounted"
  fi
done
'''
print(subprocess.check_output(['bash', '-c', shell_script], stdin=open('/etc/fstab', 'r')))

That way you run only one shell for the script, instead of having awk use system() to invoke a new and distinct shell for every single non-comment line you process.

answered Jan 15 '20 at 17:48

Charles Duffy

280,126
43
390
441

Thank you very much indeed Charles, this is really helpful. How can I get python to print only the lines containing the word 'NOT' from the output obtained from running 'print(subprocess.check_output(['bash', '-c', shell_script], stdin=open('/etc/fstab', 'r')))'? – rony thomas Jan 16 '20 at 17:53
`[[ $line = *NOT* ]] || continue` would skip any line that doesn't contain `NOT`, inside a `while IFS= read -r line` loop of the type given here. (Or `[[ $line =~ NOT ]] || continue`, which has the same effect). – Charles Duffy Jan 16 '20 at 17:54
Thank you sir, I in fact wanted 'NOT' to be excluded from the final output the while loop produced,so here is how I got the desired output. I just added a 'done |grep NOT' and that printed just the lines containing NOT. – rony thomas Jan 16 '20 at 18:17
Then `[[ $line =~ NOT ]] && continue`, substituting `&&` to `continue` if `NOT` *is* found, instead of it it isn't. Or of course you can use `grep -v NOT` to invert the pattern match. – Charles Duffy Jan 16 '20 at 18:19

RavinderSingh13 · Answer 2 · 2020-01-15T17:53:10.737

I believe you need to escape " in your codes, try following once(inside subprocess.check_output), fair warning I haven't tested it because of lack of samples. I am simply fixing OP's attempt here, best way to do it within python itself.

print subprocess.check_output(["awk '!/bind|swap|shm/ && $1 !~/#|^$/ {system(\"if [[ -z $(findmnt -m \" $2 \") ]]; then echo mount \" $2 \" is NOT mounted.;fi\")}' /etc/fstab"],shell=True, universal_newlines=True)

Python subprocess module fails when calling awk

2 Answers2

Don't.