How to change the Console Ouptut Mode using SetConsoleMode in powershell?

Question

I am trying to change the Windows Console Mode for output (CONOUT$) using the Windows API and SetConsoleMode calls. So I modified a PowerShell script, based on ConinMode.ps1 (and which works for input), to do this. Reading works fine with both the ConoutMode.exe and my script, and returns:

# .\ConoutMode.exe
mode: 0x3
ENABLE_PROCESSED_OUTPUT            0x0001 ON
ENABLE_WRAP_AT_EOL_OUTPUT          0x0002 ON
ENABLE_VIRTUAL_TERMINAL_PROCESSING 0x0004 off   <====
DISABLE_NEWLINE_AUTO_RETURN        0x0008 off
ENABLE_LVB_GRID_WORLDWIDE          0x0010 off

# .\ConoutMode.ps1
OK! GetConsoleWindow handle  :  0x1F06D6
Console Input  Mode (CONIN)  :  0x21f
Console Output Mode (CONOUT) :  0x3

However, both my script and the exe fails in writing the mode. (Possibly because it thinks that my output handle is not pointing to a console?)

In PowerShell:

# .\ConoutMode.exe set 0x000f
error: SetConsoleMode failed (is stdout a console?)

# .\ConoutMode.ps1 -Mode 7
OK! GetConsoleWindow handle  :  0x1F06D6
old (out) mode: 0x3
SetConsoleMode (out) failed (is stdout a console?)
At C:\cygwin64\home\emix\win_esc_test\ConoutMode.ps1:112 char:9
+         throw "SetConsoleMode (out) failed (is stdout a console?)"
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : OperationStopped: (SetConsoleMode ...out a console?):String) [], RuntimeException
+ FullyQualifiedErrorId : SetConsoleMode (out) failed (is stdout a console?)

Here is my ConoutMode.ps1 script in it's entirety:

param (
    [int] $Mode
)

$signature = @'
[DllImport("kernel32.dll", SetLastError=true)]
public static extern IntPtr GetConsoleWindow();

[DllImport("kernel32.dll", SetLastError = true)]
public static extern IntPtr GetStdHandle(int nStdHandle);

[DllImport("kernel32.dll", SetLastError = true)]
public static extern uint GetConsoleMode(IntPtr hConsoleHandle, out uint lpMode);

[DllImport("kernel32.dll", SetLastError = true)]
public static extern uint SetConsoleMode(IntPtr hConsoleHandle, uint dwMode);

public const int STD_INPUT_HANDLE  = -10;
public const int STD_OUTPUT_HANDLE = -11;

public const int  ENABLE_PROCESSED_OUTPUT            = 0x0001;
public const int  ENABLE_WRAP_AT_EOL_OUTPUT          = 0x0002;
public const int  ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x0004;
public const int  DISABLE_NEWLINE_AUTO_RETURN        = 0x0008;
public const int  ENABLE_LVB_GRID_WORLDWIDE          = 0x0010;
'@
#----------------------------------------------------------
$WinAPI = Add-Type -MemberDefinition $signature -Name WinAPI -Namespace ConoutMode -PassThru
#$WinAPI = Add-Type -MemberDefinition $signature -Name WinAPI -Namespace IdentifyConsoleWindow -PassThru

$hwnd = $WinAPI::GetConsoleWindow()
if ($hwnd -eq 0) {
    throw "Error: GetConsoleWindow returned NULL."
} 
echo "OK! GetConsoleWindow handle  :  0x$($hwnd.ToString("X"))"

function GetConIn {
    $ret = $WinAPI::GetStdHandle($WinAPI::STD_INPUT_HANDLE)
    if ($ret -eq -1) {
        throw "Error: cannot get stdin handle"
    }
    return $ret
}

function GetConOut {
    $ret = $WinAPI::GetStdHandle($WinAPI::STD_OUTPUT_HANDLE)
    if ($ret -eq -1) {
        throw "Error: cannot get stdout handle"
    }
    return $ret
}

function GetConInMode { #GetCOnsoleMode
    $conin = GetConIn
    $mode = 0
    $ret = $WinAPI::GetConsoleMode($conin, [ref]$mode)
    if ($ret -eq 0) {
        throw "GetConsoleMode (in) failed (is stdin a console?)"
    }
    return $mode
}

function GetConOutMode {
    $conout = GetConOut
    $mode   = 0
    $ret    = $WinAPI::GetConsoleMode($conout, [ref]$mode)
    if ($ret -eq 0) {
        throw "GetConsoleMode (out) failed (is stdout a console?)"
    }
    return $mode
}

function SetConInMode($mode) {
    $conin = GetConIn
    $ret = $WinAPI::SetConsoleMode($conin, $mode)
    if ($ret -eq 0) {
        throw "SetConsoleMode (in) failed (is stdin a console?)"
    }
}

function SetConOutMode($mode) {
    #$conout = GetConOut
    # Different tack:
    $conout = $hwnd
    $ret = $WinAPI::SetConsoleMode($conout, $mode)
    if ($ret -eq 0) {
        throw "SetConsoleMode (out) failed (is stdout a console?)"
    }
}

#----------------------------------------------------------
# MAIN
#----------------------------------------------------------
$oldInMode  = GetConInMode
$oldOutMode = GetConOutMode

$newMode = $oldOutMode
$doit = $false

if ($PSBoundParameters.ContainsKey("Mode")) {
    $newMode = $Mode
    $doit = $true
}

if ($doit) {
    echo "old (out) mode: 0x$($oldOutMode.ToString("x"))"
    SetConOutMode $newMode
    $newMode = GetConOutMode
    echo "new (out) mode: 0x$($newMode.ToString("x"))"
} else {
    echo "Console Input  Mode (CONIN)  :  0x$($oldInMode.ToString("x"))"
    echo "Console Output Mode (CONOUT) :  0x$($oldOutMode.ToString("x"))"
}

---

So at the end of the day, I want to enable ENABLE_VIRTUAL_TERMINAL_PROCESSING for the Console.

The system I am using for this is:

---------------------------------------------------------
  PowerShell Version    : 6.1.1
  OS Name               : Microsoft Windows 8.1 (64-bit)
  OS Version            : 6.3.9600  [2018-11-16 00:50:01]
  OS BuildLabEx         : 9600.19179
  OS HAL                : 6.3.9600.18969
  OS Kernel             : 6.3.9600.18217
  OS UBR                : 19182
  -------------------------------------------------------
  with Privilege        : Administrator
  -------------------------------------------------------
  ExecutionPolicy :
        MachinePolicy   : Undefined
        UserPolicy      : Undefined
        Process         : Undefined
        CurrentUser     : Undefined
        LocalMachine    : Bypass

  Console Settings:
      Type              : ConsoleHost
      OutputEncoding    : Unicode (UTF-8)
      Color Capability  : 23
      Registry VT Level : 1
      CodePage (input)  : 437
      CodePage (output) : 437

I have also enabled the registry item: HKCU:\Console\VirtualTerminalLevel, as instructed elsewhere.

---

Q: How can I enable this bit/flag using PowerShell?

Some things I can think of that may be wrong:

• I surely have the wrong understanding of how this works...
• I might have the wrong permissions to write to console? (How to set?)

• I might not write to the correct output buffer? (How to find?)

  Note that setting the output mode of one screen buffer does not affect the output mode of other screen buffers.

• When executing a script, perhaps a new output buffer is created? (What to do?)

• I might have to create a new buffer?
• I might have to create a new Console?

---

Related Questions and Links:

• _{ENABLE_VIRTUAL_TERMINAL_PROCESSING and DISABLE_NEWLINE_AUTO_RETURN failing}

• _{MSDN: Inside the Windows Console}

• _{MSDN: Introducing the Windows Pseudo Console - ConPty}

Answer 1

The code is using "GetStdHandle" to get stdin or stdout. And those might be console handles - but it's dependent on how the process is created. If you're launched from a console, and the stdin/stdout aren't redirected, then you'd probably get a console handle. But it's not guaranteed.

Rather than asking for stdin/stdout, just open the console handles directly - you can "CreateFile" for CONIN$ and CONOUT$. The documentation for CreateFile has a whole section on Consoles.

---

Update: I made a Gist to demonstrate how to use CreateFile to open CONIN$ and CONOUT$ from PowerShell.

Answer 2

I have been experimenting with this myself for a few days and have come to realize a few things. Apparently PowerShell enables Virtual Terminal Processing when it starts up. It disables it again, however, when it launches an executable and then reenables it when the executable exits. This means that whatever executable needs Virtual Terminal Processing must enable it itself or be called through a wrapper program that enables it and then pipes stdin/stdout/stderr through to the terminal. As for how would even go about writing such a program, I don't know.