python subprocess does not write the output

Question

I have the following code snippet as a Python script. I get the proper job output files with no errors. However the stdout subprocess log files (i.e. COSMOSIS_output_XXX.txt etc) don't store the expected runtime logs. Instead, these files have <_io.TextIOWrapper name=9 encoding='UTF-8'> as the only output written in the files. What is it I am doing wrong?

import subprocess
from subprocess import Popen
jobname = "cosmosis"
arg2 = "output.filename="
Vector = (
    " " + ini + " " + arg2 + COSMOSIS_PATH + os.path.splitext(ini)[0] + ".txt"
)                                                                                                                                                                                               

job3 = subprocess.Popen(
    ["cosmosis" + Vector],
    shell=True,
    text=True,
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

file = open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "a")
sys.stdout = file
print(job3.stdout)
file_e = open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a")
sys.stdout = file_e
print(job3.stderr)
try:
    outs, errs = job3.communicate(timeout=2000)                                                                                                                                      
except TimeoutExpired:
    outs, errs = job3.communicate()
    job3.kill()
file.close()
file_e.close()

Aru · Answer 1 · 2022-11-30T14:27:51.267

1

First of all you are just opening the file, you are not reading anything from it, you are not storing the information of the file anywhere, so it will just create that <_io.TextIOWrapper name=9 encoding='UTF-8'> which is very easy reproducible:

file = open("testtextf.txt","a")
print(file)

You have to read it somehow for example with .read():

data = file.read()

Also i do not recommend using "open" unless you want to deal with files being left open if anything goes wrong in between the lines before you close it again.

I highly recommend to use "with open" instead.

edited Nov 30 '22 at 14:27

answered Nov 27 '22 at 17:32

Aru

352
1
11

Thanks. Can you please provide a full, reproducible snippet ? – Ayan Mitra Nov 28 '22 at 05:40

tripleee · Accepted Answer · 2022-11-28T07:47:05.333

Passing a list as the first argument with shell=True is system-dependent. I'm guessing you really mean

with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "ab") as file,
        open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a") as file_e:
    try:
        job3 = subprocess.run(
            ["cosmosis", ini, arg2, COSMOSIS_PATH, os.path.splitext(ini)[0] + ".txt"],
            stdout=file, stderr=file_e, check=True,
            timeout=2000)
    except TimeoutExpired:
        pass

There is no way for job3.stdout or job3.stderr to contain anything because we redirected them to files. (We open the file handles with binary mode so we don't need to specify a text encoding.) There is also no way for the process to return a useful result if it is killed by a timeout, and obviously no need to kill it when it was already killed.

As the subprocess documentation already tells you, you should prefer subprocess.run or one of the legacy high-level wrappers instead of Popen when you can. Perhaps see also Actual meaning of shell=True in subprocess which also explains why you want to avoid using a shell when you can.

On the other hand, if (as indicated in comments) you want the process to run in the background until completion while your Python script proceeds to perform other tasks and take out the timeout, you do need Popen.

with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "ab") as file,
        open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "a") as file_e:
    job3 = subprocess.Popen(
        ["cosmosis", ini, arg2, COSMOSIS_PATH, os.path.splitext(ini)[0] + ".txt"],
        stdout=file, stderr=file_e)
...
# time passes and your script does other things ...
job3.wait()

You might want to periodically poll the job to see if it has finished.

this is useful. My process can run between 20 secs to 2 hrs. I used timeout, because otherwise couldnot get control back from console without hitting a key. Any suggestion to fix that ? — Ayan Mitra, Nov 28 '22 at 07:33
Fix what? Avoiding a timeout? You can run a background process if you want it to run completion but allow your Python script to continue executing, in which case you need to fall back to `Popen` after all (but probably still avoid `shell=True`). — tripleee, Nov 28 '22 at 07:37
Yes exactly. If script runs for 2 hours. It should run in background, giving the control back automatically without me hitting any key. I havent found a typical example/suggestion to this. So the best I could do was this, using `popen`. But I am guessing this is not elegant or correct. — Ayan Mitra, Nov 28 '22 at 07:42
It's exactly how you do it. It's not terribly elegant but it is the correct solution. I updated with a brief snippet to show this, too. — tripleee, Nov 28 '22 at 07:44

score -1 · Answer 3 · answered Nov 28 '22 at 06:41

-1

Ok, the following snippet is sufficient solution to the question posted above.

with open("%sCOSMOSIS_output_%s.txt" % (ERROR_PATH, ini), "wb") as file, open("%sCOSMOSIS_output_ERROR_%s.txt" % (ERROR_PATH, ini), "wb") as file_e:
    job3 = subprocess.Popen(
        ["cosmosis" + Vector],
        shell=True,
        text=True,
        stdout=file,
        stderr=file_e,
    )

answered Nov 28 '22 at 06:41

Ayan Mitra

497
1
8
20

Passing a list of strings with `shell=True` works on Windows but not really on other platforms (well, you can make it work, but you have to know what you are doing). Probably avoid this combo. – tripleee Dec 01 '22 at 17:16
I dont work on windows. This works very well in Linux. – Ayan Mitra Dec 02 '22 at 18:17
1

The square brackets around the string are unnecessary and somewhat misleading. You should pass just a string with `shell=True`, or a properly split list of tokens without (try `shlex.split()`, or just don't glue the parts together in the first place). See also [Actual meaning of `shell=True` in `subprocess`](https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess) – tripleee Dec 02 '22 at 20:59
@AyanMitra If you don’t want this answer to remain here, you can delete it yourself using the **Delete** link above – sideshowbarker Dec 02 '22 at 22:19

python subprocess does not write the output

3 Answers3