os.system(cmd) and subprocess.call(cmd, shell=True) don't execute the same as CMD.EXE

Question

I'm trying to convert a long, complex, Windows batch file into Python.

Things work except for subtle problems, which I suspect have something to do with quoting, but can't quite figure out.

In a batch file, this works fine:

Reg.exe add "HKCR\7-Zip.zip\shell\open\command" /ve /t REG_SZ /d "\"C:\Program Files\7-Zip\7zFM.exe\" \"%%1\"" /f

This Python code is meant to, but doesn't, do the same thing:

import os, subprocess

cmd = r'Reg.exe add "HKCR\7-Zip.zip\shell\open\command" /ve /t REG_SZ /d "\"C:\Program Files\7-Zip\7zFM.exe\" \"%%1\"" /f'
#os.system(cmd)
subprocess.call(cmd, shell=True)

Note that the (raw) string cmd and the batch file are exactly identical.

Both the os.system() and subprocess.call() result in the same thing - no error (Reg.exe says everything is fine), but a different effect on the system.

In my testing, the batch file configures the archiver 7z to open the .ZIP file itself (correct result).

The Python code causes 7z to open the folder where the .ZIP is located (wrong).

How to get Python to do what the batch file does?

Answer 1

OK, shot in the dark:

I would drop shell=True and use a argument list to pass to subprocess. The quoting will be handled automatically:

cmd = ['Reg.exe','add',r'HKCR\7-Zip.zip\shell\open\command','/ve','/t','REG_SZ','/d',r'"C:\Program Files\7-Zip\7zFM.exe" "%1"','/f']
rc = subprocess.call(cmd)

also check the return code of subprocess.call

If you want to process several commands like this "automatically", I'd suggest to use shlex.split, I'm not saying that it will solve everything, but it does a fine job with the quotes (protecting the quoted arguments, with quote nesting):

import shlex
text = r"""Reg.exe add "HKCR\7-Zip.zip\shell\open\command" /ve /t REG_SZ /d "\"C:\Program Files\7-Zip\7zFM.exe\" \"%%1\"" /f"""

print([x.replace("%%","%") for x in shlex.split(text)])  # %% => % in listcomp, add more filters if needed

result:

['Reg.exe', 'add', 'HKCR\\7-Zip.zip\\shell\\open\\command', '/ve', '/t', 'REG_SZ', '/d', '"C:\\Program Files\\7-Zip\\7zFM.exe" "%1"', '/f']

same thing with raw prefix would be:

['Reg.exe', 'add', r'HKCR\7-Zip.zip\shell\open\command', '/ve', '/t', 'REG_SZ', '/d', r'"C:\Program Files\7-Zip\7zFM.exe" "%1"', '/f']

pretty close right? :)

Answer 2

Jean-François Fabre's answer is good, and probably the most Pythonic answer.

Unfortunately for me, I don't have the patience to analyze the batch file to figure out whatever other filters might be needed, so I came up with this solution, which works regardless of the syntax, quoting, and escape sequences in the batch file lines:

def do(command):
    '''Executes command at Windows command line.'''

    import os, subprocess, uuid

    batchFile = open("temp_" + str(uuid.uuid4()) + ".bat", 'w')
    batchFile.write(command)
    batchFile.close()
    subprocess.call(batchFile.name, shell=True)
    os.remove(batchFile.name)

All it does is create a one-line batch file and then run it. Brute-force.

It's a bit slow because it has the overhead of creating, calling, and deleting the one-line batch file each time.

Here's a faster version that creates one big batch file with all the command lines. Whenever you call it with defer=False, it executes all the commands to date:

# thanks to https://stackoverflow.com/questions/279561/what-is-the-python-equivalent-of-static-variables-inside-a-function
def static_vars(**kwargs):
    def decorate(func):
        for k in kwargs:
            setattr(func, k, kwargs[k])
        return func
    return decorate

@static_vars(batchFile=None)
def do(command, defer=True):
    '''Executes command at Windows command line.
       Runs immediately, including all previously deferred commands, if defer is not True
    '''
    import os, subprocess, uuid

    if do.batchFile == None:
        do.batchFile = open("temp_" + str(uuid.uuid4()) + ".bat", 'w')

    do.batchFile.write(command + "\n") # append to file

    if not defer:
        do.batchFile.close()
        subprocess.call(do.batchFile.name, shell=True)
        os.remove(do.batchFile.name)
        do.batchFile = None