"SFC" output redirection formatting issue - Powershell / Batch

Question

I'm working on a powershell script in which several commands output are shown in the window and appended to a file or a variable. It worked correctly until I used the sfc command. When piped or redirected, the output is "broken":

> sfc /?
Vérificateur de ressources Microsoft (R) Windows (R) version 6.0[...]

> sfc /? | Tee-Object -Variable content
 V Ú r i f i c a t e u r   d e   r e s s o u r c e s   M i c r o s o f t   ( R )   W i n d o w s   ( R )   v e r s i o  á 6 . 0[...]

Are there other commands like sfc that are formatted in the same way, or that will result in a broken output if redirected?

---

EDIT

Powershell sample code, using the code from the accepted answer:

# Run a command
function RunCommand([ScriptBlock] $command) {

    # Run the command and write the output to the window and to a variable ("SFC" formatting)
    $stringcommand = $command.ToString()
    if (
        $stringcommand -match "^SFC$" -or
        $stringcommand -match "^SFC.exe$" -or
        $stringcommand -match "^SFC .*$" -or
        $stringcommand -match "^SFC.exe .*$"
    ) {
        $oldEncoding = [console]::OutputEncoding
        [console]::OutputEncoding = [Text.Encoding]::Unicode
        $command = [ScriptBlock]::Create("(" + $stringcommand + ")" + " -join ""`r`n"" -replace ""`r`n`r`n"", ""`r`n""")
        & ($command) 2>&1 | Tee-Object -Variable out_content
        [console]::OutputEncoding = $oldEncoding

    # Run the command and write the output to the window and to a variable (normal formatting)
    } else {
        & ($command) 2>&1 | Tee-Object -Variable out_content
    }

    # Manipulate output variable, write it to a file...
    # ...
    return
}

# Run commands
RunCommand {ping 127.0.0.1}
RunCommand {sfc /?}
[void][System.Console]::ReadKey($true)
exit

---

CMD sample code, using more to format the sfcoutput:

@echo off
setlocal enabledelayedexpansion
set "tmpfile=%TEMP%\temp.txt"
set "outputfile=%TEMP%\output.txt"

REM; Run commands
call :RunCommand "ping 127.0.0.1"
call :RunCommand "sfc"
pause
exit /b

REM; Run a command
:RunCommand

    REM; Run the command and write the output to the window and to the temp file
    set "command=%~1"
    (!command! 2>&1) >!tmpfile!

    REM; Write the output to the window and to the output file ("SFC" formatting)
    set "isSFC=0"
    (echo !command!|findstr /I /R /C:"^SFC$" > NUL) && (set "isSFC=1")
    (echo !command!|findstr /I /R /C:"^SFC.exe$" > NUL) && (set "isSFC=1")
    (echo !command!|findstr /I /R /C:"^SFC .*$" > NUL) && (set "isSFC=1")
    (echo !command!|findstr /I /R /C:"^SFC.exe .*$" > NUL) && (set "isSFC=1")
    (if !isSFC! equ 1 (
        (set \n=^
%=newline=%
)
        set "content="
        (for /f "usebackq tokens=* delims=" %%a in (`more /p ^<"!tmpfile!"`) do (
            set "line=%%a"
            set "content=!content!!line!!\n!"
        ))
        echo.!content!
        (echo.!content!) >>!outputfile!

    REM; Write the output to the window and to the locked output file (normal formatting)
    ) else (
        type "!tmpfile!"
        (type "!tmpfile!") >>!outputfile!
    ))
goto :EOF

Answer 1

As noted in js2010's answer, the sfc.exe utility - surprisingly - outputs text that is UTF-16LE ("Unicode") encoded.

Since PowerShell doesn't expect that, it misinterprets sfc's output.^[1]

The solution is to (temporarily) change [console]::OutputEncoding to UTF-16LE, which tells PowerShell / .NET what character encoding to expect from external programs, i.e., how to decode external-program output to .NET strings (which are stored as UTF-16 code units in memory).

However, there's an additional problem that looks like a bug: bizarrely, sfc.exe uses CRCRLF (`r`r`n) sequences as line breaks rather than the Windows-customary CRLF (`r`n) newlines.

PowerShell, when it captures stdout output from external programs, returns an array of lines rather than a single multi-line string, and it treats the following newline styles interchangeably: CRLF (Windows-style), LF (Unix-style), and CR (obsolete Mac-style - very rare these days).
Therefore, it treats CRCRLF as two newlines, which are reflected in both "teed" and captured-in-a-variable output then containing extra, empty lines.
The solution is therefore to join the array elements with the standard CRLF newline sequences - (sfc /?) -join "`r`n" and then replace 2 consecutive `r`n with just one, to remove the artificially introduced line breaks: -replace "`r`n`r`n", "`r`n".

To put it all together:

# Save the current output encoding and switch to UTF-16LE
$prev = [console]::OutputEncoding
[console]::OutputEncoding = [Text.Encoding]::Unicode

# Invoke sfc.exe, whose output is now correctly interpreted and
# apply the CRCRLF workaround.
# You can also send output to a file, but note that Windows PowerShell's 
# > redirection again uses UTF-16LE encoding.
# Best to use ... | Set-Content/Add-Content -Encoding ... 
(sfc /?) -join "`r`n" -replace "`r`n`r`n", "`r`n" | Tee-Object -Variable content

# Restore the previous output encoding, which is the system's 
# active OEM code page, which should work for other programs such
# as ping.exe
[console]::OutputEncoding = $prev

Note that $content will then contain a single, multi-line string; use $content -split "`r`n" to split into an array of lines.

---

As for:

Are there other commands like "sfc" that are formatted in the same way, or that will result in a broken output if redirected?

Not that I'm personally aware of; unconditional UTF-16LE output, as in sfc.exe's case, strikes me as unusual (other programs may offer that on an opt-in basis).

Older console programs with a Windows-only heritage use a (possibly fixed) OEM code page, which is a single-byte 8-bit encoding that is a superset of ASCII.

Increasingly, modern, multi-platform console programs use UTF-8 (e.g., the Node.js CLI), which is variable-width encoding capable of encoding all Unicode characters that is backward-compatible with ASCII (that is, in the 7-bit ASCII range UTF-8 encodes all characters as single, ASCII-compatible bytes).

If you want to make your PowerShell sessions and potentially all console windows fully UTF-8 aware, see this answer (However, doing so stil requires the above workaround for sfc).

---

^[1] Direct-to-console output:

When sfc output is neither captured by PowerShell nor routed through a cmdlet such as Tee-Object, sfc writes directly to the console, presumably using the Unicode version of the WriteConsole Windows API function, which expects UTF-16LE strings.

Writing to the console this way allows printing all Unicode characters, irrespective of what code page (reflected in chcp / [console]::OutputEncoding) is currently active. (While the rendering of certain characters may fall short, due to limited font support and lack of support for (the rare) characters outside the BMP (Basic Multilingual Plane), the console buffer correctly preserves all characters, so copying and pasting elsewhere may render correctly there - see the bottom section of this answer.)

Therefore, direct-to-console output is not affected by the misinterpretation and typically prints as expected.

Answer 2

Looks like sfc outputs unicode no bom. Amazing.

cmd /c 'sfc > out'
get-content out -Encoding Unicode | where { $_ } # singlespace

Output:

Microsoft (R) Windows (R) Resource Checker Version 6.0
Copyright (C) Microsoft Corporation. All rights reserved.
Scans the integrity of all protected system files and replaces incorrect versions with
correct Microsoft versions.
SFC [/SCANNOW] [/VERIFYONLY] [/SCANFILE=<file>] [/VERIFYFILE=<file>]
    [/OFFWINDIR=<offline windows directory> /OFFBOOTDIR=<offline boot directory>]
/SCANNOW        Scans integrity of all protected system files and repairs files with
                problems when possible.
/VERIFYONLY     Scans integrity of all protected system files. No repair operation is
                performed.
/SCANFILE       Scans integrity of the referenced file, repairs file if problems are
                identified. Specify full path <file>
/VERIFYFILE     Verifies the integrity of the file with full path <file>.  No repair
                operation is performed.
/OFFBOOTDIR     For offline repair specify the location of the offline boot directory
/OFFWINDIR      For offline repair specify the location of the offline windows directory
e.g.
        sfc /SCANNOW
        sfc /VERIFYFILE=c:\windows\system32\kernel32.dll
        sfc /SCANFILE=d:\windows\system32\kernel32.dll /OFFBOOTDIR=d:\ /OFFWINDIR=d:\windows
        sfc /VERIFYONLY

Or delete the nulls and blank lines (windows prints nulls as spaces):

(sfc) -replace "`0" | where {$_}