Unexpected behaviour when calling shell command on a deployed web app

Question

I am deploying a flask app, in which a multiprocessing.Process will be started. Inside this process I call a shell command trough subprocess.call(). The script runs fine when executed on local host, when deployed with nginx and gunicorn, the flask app behaves as expected until the subprocess starts, then I recieve the following error log:

DEBUG:root:start to run command
DEBUG:root:(<class 'FileNotFoundError'>, FileNotFoundError(2, "No such file or directory: 'java -jar ábsolute/path/to/jar/file')

                                Process(
                                target=decode_upload,
                                args=(
                                        path_to_blf,
                                        path_to_dbc_folder,
                                        path_to_splitted,
                                        path_to_decoder,
                                        system_layout,
                                        dc_doc,
                                        dc_id,
                                        file_type,
                                )
                            ).start()

Here is the rellevant part of the function.

def decode_file(
    path_to_blf,
    path_to_dbc_folder,
    path_to_splitted,
    path_to_decoder,
    system_layout=DEFAULT_SYSTEM_LAYOUT):
    command = "{} {} --blf={}".format(
                                        SOFTWARE_COMMAND,
                                        path_to_decoder,
                                        path_to_blf
    )
    for dbc_file_name in DBC_FILE_NAME_LIST:
        command += " --dbc={}".format(
                                    os.path.join(
                                            path_to_dbc_folder,
                                             dbc_file_name
                                    )
        )
    command += " --out={}".format(path_to_splitted)


    logging.debug("start to run command")
    subprocess.call(command)
    logging.debug(f)
    logging.debug("run command end")

def decode_upload(
                        path_to_blf,
                        path_to_dbc_folder,
                        path_to_splitted,
                        path_to_decoder,
                        system_layout,
                        dc_doc,
                        dc_id,
                        file_type):
    logging.basicConfig(filename='flask.log',level=logging.DEBUG)
    logging.debug('This message should go to the log file')

    try:
        decode_file(
            path_to_blf,
            path_to_dbc_folder,
            path_to_splitted,
            path_to_decoder,
            system_layout)
    except:
        logging.debug(sys.exc_info())

The process fails when it gets to this line.

subprocess.call(command)

if I try to call the "command" from the command line it works without a problem.

pankaj mishra · Accepted Answer · 2017-12-15T08:44:19.200

1

from subprocess import Popen
command='your complete command as you paste on cmd'
p1=Popen(command,Shell=True)

This will help you to run the command as complete string

edited Dec 15 '17 at 08:44

answered Dec 14 '17 at 17:47

pankaj mishra

2,555
2
17
31

Hi, thanks for the answer. Yes, the error disappears, but the command is still not going to be executed. I catch the stdout from this command by using p = subprocess.run("ls", shell=True, stdout=PIPE), logging.debug(p.stdout) . But I got a message DEBUG:root:b''" , which means nothing in the log file. It seems the command has not been run. – Zhiwei Han Dec 15 '17 at 08:39
No, error anymore. But no command has been run. I tried os.system(command) and nothing happens, but if I run command in terminal it works perfectly. – Zhiwei Han Dec 15 '17 at 08:42
`Popen()` simply *starts* the command, but leaves it running as a subprocess. The `call()` wrapper (or in newer Python versions `run()`) does the necessary housekeeping of properly waiting for the process to complete and optionally storing its output somewhere, etc. You want to avoid the underlying `Popen()` method unless none of the existing higher-level wrapper methods are suitable for your use case. – tripleee Dec 15 '17 at 09:04
Yes, it works now. But is there any difference by calling these two apis? I also tried p1=Popen([command_1, command_2, .....]), but it also didn't work. – Zhiwei Han Dec 15 '17 at 09:12

score 1 · Answer 2 · answered Dec 15 '17 at 08:57

The proper solution is to pass the command as a list of parsed arguments. You need to understand how the shell handles quoting and argument splitting to do this right.

As a quick cheat,

printf "'%s'\n" your command line here

in your shell should give you a good idea of how the shell expands the arguments. For example

bash$ printf "'%s'\n" java -jar "/path/complex path with spaces.jar" \* \>\<
'java'
'-jar'
'/path/complex path with spaces.jar'
'*'
'><'

shows you that you need

subprocess.call(['java', '-jar', '/path/complex path with spaces.jar', '*', '><'])

Adapting your code for this, we obtain

command = [SOFTWARE_COMMAND, path_to_decoder, '--blf={}'.format(path_to_blf)]
for dbc_file_name in DBC_FILE_NAME_LIST:
    command.append("--dbc={}".format(
        os.path.join(path_to_dbc_folder, dbc_file_name)))
command.append("--out={}".format(path_to_splitted))

See also https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess for why you want to avoid `shell=True` — tripleee, Dec 15 '17 at 08:58

Unexpected behaviour when calling shell command on a deployed web app

2 Answers2