Escaping error in sed called by python subprocess

Question

I was trying to call sed using subprocess.check_call as follows

with open(output_file,'w+') as f:
    subprocess.check_call(['sed', r':a;s/^\(\([^"]*,\?\|"[^",]*",\?\)*"[^",]*\),/\1 /;ta s/,$//; s/\.00//g; s/"//g', input_file], stdout = f, shell=True)

However, I got the such an error

'\"[^\"' is not recognized as an internal or external command, operable program or batch file.

The command sed ':a;s/^$\([^"]*,\?\|"[^",]*",\?$*"[^",]*\),/\1 /;ta s/,$//; s/\.00//g; s/"//g' input_file > output_file works by itself in bash, so I suspect this is caused by some escaping problem.

`shell=True` is **wrong** when you pass in a list of tokens (as opposed to a single string for the shell to parse). You almost certainly want to replace the `sed` script with a Python function, though without an explanation of what it's supposed to accomplish, I won't attempt to ad-lib one. See also https://stackoverflow.com/a/51950538/874188 — tripleee, Nov 28 '18 at 05:11

tripleee · Answer 1 · 2018-11-28T05:34:03.553

The immediate problem is that you are passing in a token containing ; without quoting it with shell=True; but seeing as you are already parsing the command into tokens, you don't need or want shell=True here, so the immediate fix is simply to remove that. See Actual meaning of 'shell=True' in subprocess for what it means, why it's wrong here, and why you generally want to avoid it.

However, your sed script appears to attempt to turn CSV into space-delimited, with quoted fields preserved, but any quoting removed. There is a Python library for parsing CSV which probably does a better job.

import csv

with open(inputfile, 'r') as inp, open(output_file, 'w+') as out:
    reader = csv.reader(inp)
    writer = csv.writer(out, delimiter=' ', quoting=csv.QUOTE_NONE)
    for row in reader:
        for k in row:
            row[k] = row[k].replace('.00', '')
         writer.writerow(row)

This is ad hoc and probably requires some tweaking (and I probably overlooked or misunderstood some details in your sed script), but should give you an idea for how to get rid of the subprocess entirely, and probably have more control over what your script is doing (as well as a chance of understanding it three months from now).

I abbreviated the question a little bit when posting the question. In fact, the file is located on the remote server and I was trying to call `psexec` to execute sed on the remote server so there will be no need to transfer the input via the network. — Chen Chen, Nov 28 '18 at 18:54

Escaping error in sed called by python subprocess

1 Answers1