subprocess python reading output

Question

I am trying to run a command using subprocess in python and trying to read the output of it and copy it into a file. my code is:

    command = "%s -sK:\\run_one_test.csh %s %s" % (PATH, file, VERSION)
    p = subprocess.Popen(command,stdout=subprocess.PIPE)

    text = p.communicate()[0]
    return_code = p.returncode




    with open("%s/%s%s" % (LOG_DIR, file, LOG_EXT), "w") as f:


        f.writelines([l.decode for l in text.split('\n')])

        f.close()

But when i use splitlines i get the error message:

    f.writelines([l.decode for l in text.split('\n')])
TypeError: a bytes-like object is required, not 'str'

Why does it happen? I used decode. Also, is this the right way to split the lines of code using "\n"? Thanks.

Follow this thread: https://stackoverflow.com/questions/44989808/subprocess-typeerror-a-bytes-like-object-is-required-not-str-python-2 — Soumithri Chilakamarri, Sep 04 '18 at 18:54

Jean-François Fabre · Answer 1 · 2018-09-05T11:56:47.620

text is the result of subprocess.Popen, in python 3, this is a bytes object. You cannot split it using a string (text.split(b"\n") would work, but there are more problems in your code so forget it).

Then, you're not calling decode (missing parentheses). Calling decode on text would work: text.decode()

Now with split('\n'), you'll remove the end of lines, so that won't work properly (single-line file, ouch). You'd need:

f.writelines(text.decode().splitlines(True))

(the True argument tells the function to keep the end of line chars)

But after all why splitting to write the lines again?, just write the file as it was dumped, as bytes, without post-processing:

with open("%s/%s%s" % (LOG_DIR, file, LOG_EXT), "wb") as f:
    f.write(text)

but why storing the output to write it back to a file then? just pass the file handle to subprocess.call:

with open("%s/%s%s" % (LOG_DIR, file, LOG_EXT), "wb") as f:
   retcode = subprocess.call(command,stdout=f)

or to force an exception if command returncode is not 0:

   subprocess.check_call(command,stdout=f)

So depending if you want to post-process the lines, post-process the buffer, or not post-process at all, you can choose one of the 3 (possibly modified) solutions above.

Side note: don't compose the command as string, just pass the arguments so:

command = "%s -sK:\\run_one_test.csh %s %s" % (PATH, file, VERSION)

becomes:

command = [PATH,"-sK:\\run_one_test.csh",file,VERSION]

(as a bonus, the quoting is handled automatically if needed)

You should probably prefer `subprocess.check_call()` or better yet `subprocess.run()`. See also https://stackoverflow.com/questions/4256107/running-bash-commands-in-python — tripleee, Sep 05 '18 at 11:03
`run` requires python 3.5. I swtiched to `call` because it's a better match for `Popen`. We don't know what the return code of the command will be — Jean-François Fabre, Sep 05 '18 at 11:55

subprocess python reading output

1 Answers1