executing bash's complex find command in python shell

Question

I am new to python. I am trying to execute a bash script in python to extract the count of different file extensions. I tried the following command

import subprocess
output = subprocess.check_output("sudo find . -type f -name '*.*' -exec sh -c 'echo ${0##*.}' {} \; | sort | uniq -c | sort -nr | awk '{print $2 ":" $1}'", shell=True)

But it throws a syntax error. On executing find command in bash shell

sudo find . -type f -name '*.*' -exec sh -c 'echo ${0##*.}' {} \; | sort | uniq -c | sort -nr | awk '{print $2 ":" $1}'

output will be as follows

png:3156
json:333
c:282
svg:241
zsh:233
js:192
gz:169
zsh-theme:143
ttf:107
cache:103
md:93

So how can i get the same output in python code? what is the correction required in my current approach? Thanks in advance

Answer 1

As mentioned in the comments any double quote in a string quoted with double quotes needs to be escaped with a backslash:

import subprocess
output = subprocess.check_output("sudo find . -type f -name '*.*' -exec sh -c 'echo ${0##*.}' {} \; | sort | uniq -c | sort -nr | awk '{print $2 \":\" $1}'", shell=True)

Single quotes inside a double quoted string do not have any special meaning (except directly at the beginning), so that doesn't allow you to avoid escaping.

The fine details are explained under the header String and Bytes literals from the Python language reference.

As mentioned in the comments another option, which is probably easier to read, is to use triple double quotes:

import subprocess
output = subprocess.check_output("""sudo find . -type f -name '*.*' -exec sh -c 'echo ${0##*.}' {} \; | sort | uniq -c | sort -nr | awk '{print $2 ":" $1}'""", shell=True)

While this answers the question, for ease of reading and maintainability I suggest to replace it instead completely with Python, as suggested in another answer.

Answer 2

By the way, you could try to do the same thing in pure Python. Here is a minimal code that does it:

import os

def count_all_ext ( path ):
    res = {}
    for root,dirs,files in os.walk( path ):
        for f in files :
            if '.' in f :
                e = f.rsplit('.',1)[1]
                res[e] = res.setdefault(e,0)+1
    return res.items()


print '\n'.join( '%s:%d'%i for i in count_all_ext('.'))

OK, it's very long compared to the Bash snippet, but it's Python...