Detect if python program is executed via Windows GUI (double-click) vs command prompt

Question

Background
I have a Python 3.5 console program compiled into a Windows executable via pyinstaller.

Question

When executed via a command prompt, I'd like my program to run with whatever arguments were supplied (possibly none).
When executed via the operating system's GUI (i.e. by double-clicking the .exe in Windows Explorer on Windows, etc) I'd like my program to prompt the user for input. I also need my program to pause before exiting so the user can read the output.

How do I detect these different scenarios?

Constraints

The executable must be able to run on a bare-bones (i.e. fresh install) Windows/RedHat machine.
The compiled executable must be a single file and may not rely on other files not packaged inside the compiled executable (pyinstaller allows files to be packaged inside the compiled executable).
The program may depend on 3rd party python packages.

Things I've Tried

sys.stdin.isatty()
https://stackoverflow.com/a/3818551/3508142
os.isatty(sys.stdout.fileno())
https://stackoverflow.com/a/6108504/3508142
These always return True on Windows.
Searching StackOverflow / the internet:
How to determine if Python script was run via command line?
How can I check to see if a Python script was started interactively?
As far as I understand, a program is running interactively if the user started it regardless of whether it was started from a command prompt or the GUI.
I also considered checking to see if the parent process is cmd.exe or explorer.exe. However, starting the program via the Windows run command will make explorer.exe the parent process. Starting the program via Task Manager will make Task Manager the parent process. These are edge cases that I could live with, but obviously I'd prefer a more robust solution.

I assume your executable is a console application since `sys.stdin.isatty()` is true. By default it will inherit the parent's console if the parent has one. You could prompt the user if there are no command-line arguments and the parent is *not* attached to your console. Call [`GetConsoleProcessList`](https://learn.microsoft.com/en-us/windows/console/getconsoleprocesslist) to get the PIDs of processes attached to the console. We can use ctypes or PyWin32 to call this function. — Eryk Sun, Mar 15 '19 at 01:31
@eryksun thanks for the response. Yes, my executable is a console app. I'm not familiar with Windows API calls; perhaps you could post a full answer with an example? — ErikusMaximus, Mar 15 '19 at 02:16

score 5 · Accepted Answer · answered Apr 02 '19 at 13:34

Count processes attached to the console

Windows API documentation for GetConsoleProcessList

import ctypes

# Load kernel32.dll
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
# Create an array to store the processes in.  This doesn't actually need to
# be large enough to store the whole process list since GetConsoleProcessList()
# just returns the number of processes if the array is too small.
process_array = (ctypes.c_uint * 1)()
num_processes = kernel32.GetConsoleProcessList(process_array, 1)
# num_processes may be 1 if your compiled program doesn't have a launcher/wrapper.
if num_processes == 2:
    input('Press ENTER to continue...')

ErikusMaximus · Answer 2 · 2019-04-01T15:40:34.837

2

Turns out there is a simple and concise way to determine this on Windows. https://stackoverflow.com/a/14394730/3508142

The PROMPT environment variable defines the prompt text in a command prompt. https://ss64.com/nt/prompt.html

# If the program was started via the GUI (i.e. by double-clicking the executable),
# then prevent the console window from closing automatically.
if os.name == 'nt' and 'PROMPT' not in os.environ:
    input('Press ENTER to continue...')

edited Apr 01 '19 at 15:40

answered Mar 19 '19 at 18:59

ErikusMaximus

1,150
3
13
28

That's unreliable. There could be a custom `PROMPT` variable defined in the user or system environment variables. There is no environment variable we can rely on for this. Use the console API to ensure that yours is the only process attached to the console. (Or maybe also the frozen executable if it unpacks to a temp directory.) Getting the number of processes is straight forward, e.g. `kernel32 = ctypes.WinDLL('kernel32', use_last_error=True);` `unused = (ctypes.c_uint * 1)();` `num_proceses = kernel32.GetConsoleProcessList(unused, 1)`. – Eryk Sun Apr 01 '19 at 12:20
@eryksun, I do agree that a custom `PROMPT` variable definition could cause a false positive. However, doing so would also change the user's [prompt text](https://ss64.com/nt/prompt.html), which seems unlikely so I'm fine with this risk. As far as using ctypes goes, it seems to simply count the number of ancestor processes. If my program is executed by another program, the value of `num_processes` will be inflated. As such, it seems like your approach can't distinguish between GUI and command prompt execution in all cases. Am I missing something? Perhaps a full answer could clarify things? – ErikusMaximus Apr 01 '19 at 15:38
The code snippet I gave you tells you how many processes are attached to the console. It has nothing to do with ancestor processes. If you know your script should attach 1 process (or maybe 2 if there's a launcher) to the console, then additional processes at startup means the console was inherited, so just use whatever command line options are provided, if any, and don't worry about keeping the console alive. Otherwise prompt for input, and on exit do something like `subprocess.Popen('cmd.exe')` to keep the console alive. – Eryk Sun Apr 01 '19 at 20:19
You are correct that `GetConsoleProcessList` returns the number of processes attached to the console, not ancestor processes. Thanks for that clarification. Also, for clarification my compiled program does appear to have a separate launcher process. I did some more investigating and realized that if `num_processes == 2` then my program was executed from the GUI, otherwise a higher number means it is either running from a command prompt or being called by a script (and I agree that in this case it shouldn't pause). Thanks @eryksun for your help, I'll revise my answer to include your suggestion. – ErikusMaximus Apr 02 '19 at 13:22
2

I just tested this and while python process started from `cmd.exe` had `os.environ['PROMPT']`, a python process started from `Powershell 5` did not. – Niko Föhr Nov 22 '20 at 20:44
Wait, what? You're OK with this breaking if the user changed his prompt text by setting PROMPT in his environment (that's what it's for)? – Glenn Maynard Sep 23 '22 at 05:57

score 0 · Answer 3 · answered Sep 23 '22 at 07:23

I'm not sure if "does my process have a terminal" was strictly what the OP was asking (he asked about "was I launched by cmd or a GUI", which is independant of whether you have a console window), but for anyone else who searches their way to this question:

The current accepted answer (GetConsoleProcessList) isn't quite right, since it doesn't work with virtual terminals. See the warning at https://learn.microsoft.com/en-us/windows/console/getconsoleprocesslist: "This API is not recommended and does not have a virtual terminal equivalent."

If you want to know if you have a console, you can do:

def has_console():
    try:
        with open('CONIN$'):
            return True
    except:
        return False

If the process has a console, it can be accessed with CONIN$ and CONOUT$. That works for both traditional consoles and virtual terminals. If there's no console, it'll throw WinError with ERROR_INVALID_HANDLE.

You could also see if kernel32.GetStdHandle(win32api.STD_INPUT_HANDLE) is nonzero. I think that's a bit different and would be nonzero if the process was started with file redirection, where CONIN$ specifically checks for a console. For most applications the difference probably doesn't matter.

Detect if python program is executed via Windows GUI (double-click) vs command prompt

3 Answers3

Count processes attached to the console

Linked