Python subprocess cannot capture output of Windows program

Question

Using stdout=subprocess.PIPE stops the output from going to the console, but nothing is captured.

>>> import subprocess
>>> proc = subprocess.Popen(['C:\\Users\\me\\program.exe'])
>>> ERROR: please provide an argument
// TRUNCATED USAGE OUTPUT
proc.wait()
0
>>> proc = subprocess.Popen([''C:\\Users\\me\\program.exe''], stdout=subprocess.PIPE)
>>> proc.communicate()
('', None)

I've tried every combination available on stackoverflow. shell=True hasn't worked. Spawning a sub cmd hasn't worked. subprocess.check_output captures nothing. I'm happy to retry any of these commands in the comments.

I am guessing this has something to do with out the program is attaching to a shell.

This is the assembly the program uses to output (mcall is just a macro to align memory to 16 bits). The reason I include this is in case GetStdHandle is affecting things.

console_write PROC
; rcx   MSG
; rdx   LEN
  prologue
  push    rcx
  push    rdx
  xor     rcx, rcx
  mov     ecx, [stdout]
  mcall   GetStdHandle
  mov     rcx, rax
  xor     rdx, rdx
  pop     r8    ; len
  pop     rdx   ; msg
  push    0
  mov     r9, rsp
  push    0
  mcall   WriteConsoleA
  pop     rcx
  pop     rcx
  epilogue
console_write ENDP

I'm stumped on this one. I've done this so many times on Linux. I just don't know enough about Windows internals to sort this out. Thanks!

Edit: Additional things I have tried:

Admin (also with STDERR capture)

C:\Windows\system32>C:\Python27\python.exe
Python 2.7.14 (v2.7.14:84471935ed, Sep 16 2017, 20:19:30) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess
>>> proc = subprocess.Popen(['C:\\Users\\me\\program.exe'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> proc.communicate()
('', '')

STDOUT File Redirect

>>> import os
>>> os.system('C:\\Users\\me\\program.exe > out.txt')
0
>>> f = open('out.txt', 'r')
>>> f.read()
''

STDERR File Redirect

>>> import os
>>> os.system('C:\\Users\\me\\program.exe 2>out.txt')
ERROR: please provide an argument
// TRUNCATED USAGE OUTPUT
0
>>> open('out.txt', 'r').read()
''

Command Line Capture Fails (suppresses output from going to cmd, but nothing is captured)

program.exe > out.txt

Yes. Same issue. `>>> proc.communicate() ('', '')` (full command in another comment) — douggard, Oct 31 '17 at 15:32
you can see its output on cmd right? so, write that in a .bat file, then launch that bat file with your python script and and see what happens ... — DRPK, Oct 31 '17 at 15:43
Hmmm, cannot capture from a batch file or on the command line either. — douggard, Oct 31 '17 at 15:47
are you sure your exe file is not corrupted? and did you try win32.client module? you can run applications,shell commands and vbs with win32.client too... — DRPK, Oct 31 '17 at 15:52
EXE is fine. Assembled with ML64 (I've rebuilt it today with no change). It outputs to console and returns 0. Capturing stdout suppresses the output but fails to capture. — douggard, Oct 31 '17 at 15:54
see : https://stackoverflow.com/questions/33059905/how-to-open-a-cmd-shell-in-windows-and-issue-commands-to-that-shell-using-python ( the last answer ),,,,,,, or see : https://stackoverflow.com/questions/17304097/retrieving-the-command-output-of-a-hidden-console — DRPK, Oct 31 '17 at 16:06

score 4 · Accepted Answer · answered Oct 31 '17 at 17:17

4

A coworker pointed out the issue. It turns out that WriteConsole cannot be redirected to a file, it can only be used with console handles. [1] [2]

This StackOverflow answer gives an outline for how to solve it. I've implemented it in Python if anyone else has this problem.

import win32console
import subprocess
import msvcrt
import os

ccsb = win32console.CreateConsoleScreenBuffer()
fd = msvcrt.open_osfhandle(ccsb, os.O_APPEND)
proc = subprocess.Popen(['C:\\Users\\me\\program.exe'], stdout=fd)
proc.wait()
ccsb.ReadConsoleOutputCharacter(100, win32console.PyCOORDType(0,0)) # reads 100 characters from the first line

The better solution for me was to change WriteConsoleA to WriteFile, using the same StdHandle.

console_write PROC
; rcx   MSG
; rdx   LEN
  prologue
  push    rcx
  push    rdx
  xor     rcx, rcx
  mov     ecx, [stdout]
  mcall   GetStdHandle
  mov     rcx, rax
  ; Write File
  pop     r8    ; len
  pop     rdx   ; msg
  ; unalign for odd number of pushed args
  mov     rbx, rsp
  sub     rbx, 8        
  and     rbx, 0Fh
  sub     rsp, rbx
  ; args
  mov     rcx, rax        ; handle
  xor     r9, r9
  push    0               ; 1 arg (ununaligns)
  sub     rsp, 20h        ; shadow 
  call    WriteFile
  add     rsp, 28h        ; shadow + arg
  add     rsp, rbx        ; realign
  epilogue
console_write ENDP

answered Oct 31 '17 at 17:17

douggard

692
1
12
29

very unusual for x64 asm code. i mean how you work with stack. all this push/pop in the middle. need not forget about stack align before every call. why not use usual technique with `mov` params (4,5..) to stack ? – RbMm Oct 31 '17 at 18:46
@RbMm Any example of the usual code? I have a macro for align and call for 0-4 arguments. This was the one call I was doing with 5. I will eventually write a new macro where you provide the number of arguments and it does the align / unalign based on whether it is even or odd. Most the code I saw relied on tracking if the stack was aligned or not. This doesn't care if it is aligned or unaligned. If there is a better way though, I'm always open to learn. – douggard Oct 31 '17 at 18:52
1

look for any windows x64 binary. really you only complicate your own life by this. special macros absolute not need. you need **once** in function prolog align stack. and then never use push or pop for args. simply use *mov*. say `mov [rsp+20h],arg5`, `mov [rsp+28h],arg6` and so on. and code became smaller and more redable. – RbMm Oct 31 '17 at 18:58
@RbMm The `sub rsp, 0xNN` requires knowledge of whether the stack is aligned or unaligned. `0xNN` would need to be `shadow_space + 8 * n_stack args + alignment`. My code just tests for alignment and subtracts it first. I could do the same thing and just use the result of `and rbx, 0Fh` as the `alignment` in the above equation, but it makes no difference. I'm not a compiler and don't want to track whether things are aligned or unaligned, so the test is useful. Then if I change the code above a call it doesn't mess up the alignment for the call. – douggard Oct 31 '17 at 19:41
on windows (how minimum) we always have this knowledge - at any function first instruction `stack == 16*n+8` – RbMm Oct 31 '17 at 19:53
so all is really much more simply - we know stack akigment at function begin. and we need do decrement rsp on `8+k*16` in prolog. how minimum on `0x28` – RbMm Oct 31 '17 at 19:55
1

If you're in control of the source, then certainly change to `WriteFile`. Using `WriteConsoleA` offers no advantages and comes with the downside that it only accepts console screen-buffer handles. `WriteConsoleW` would be a different story. It's the only reliable way to write the full range of Unicode to the console as UTF-16 text in Windows 7, since the older console's support for codepage 65001 (UTF-8) is buggy for output. (Even in Windows 10 the console is still buggy for reading input as UTF-8.) – Eryk Sun Nov 01 '17 at 04:57

Aleksandr Borisov · Answer 2 · 2017-10-31T16:02:47.147

1

Did you try to catch stderr as well?

proc = subprocess.Popen([''C:\\Users\\me\\program.exe''], stdout=subprocess.PIPE, stderr=subprocess.PIPE)

UPDATE

You can try to read output line by line while the process is running, for example:

p = subprocess.Popen(exe, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
    retcode = p.poll()
    print p.stdout.readline()
    if retcode is not None:
        break

edited Oct 31 '17 at 16:02

answered Oct 31 '17 at 15:28

Aleksandr Borisov

2,136
1
13
14

Yep. Same deal. `>>> proc = subprocess.Popen(['C:\\Users\\me\\program.exe'], stdout=subprocess.PIPE, stderr=subprocess.PIPE) >>> proc.communicate() ('', '')` – douggard Oct 31 '17 at 15:31
`os.system()` does not capture output. Redirecting to a file like `os.system(program.exe > out.txt)` causes no output to be written to the file. – douggard Oct 31 '17 at 15:35
And `os.system(program.exe 2> out.txt)`? – Aleksandr Borisov Oct 31 '17 at 15:37
It's printing to `stdout`. When I do that the output goes to the console and nothing is captured in the file. Will edit it in above. – douggard Oct 31 '17 at 15:41

score 1 · Answer 3 · answered Feb 26 '21 at 17:41

I don't think I have enough rep to suggest a question is a possible duplicate, nor comment, so I'm posting this as an answer because although douggard's answer led me down the right track thinking-wise, it didn't actually work in my case and I had a hard time getting a working solution.

There's an answer here that captures CONOUT and worked for me: https://stackoverflow.com/a/38749458/1675668

Python subprocess cannot capture output of Windows program

3 Answers3