how to know encoding of a file using command line?

Question

Is there any command to know the encoding of a file in windows?
like for a file A.txt encoding is UTF-16

Answer 1

In Windows command prompt (cmd), there is no command I know of, that is capable of determining how a text file is encoded.

Nevertheless, I wrote a small batch file that is able to check a few conditions and thus, determine whether a given text file is ASCII-/ANSI-encoded or Unicode-encoded (UTF-8 or UTF-16, Little Endian or Big Endian). At first, it checks whether or not the first (non-empty) line contains zero-bytes, which is an indication that the file is not ASCII-/ANSI-encoded. Next, it checks the first few bytes whether they constitute the Byte Order Mark (BOM) for UTF-8/UTF-16. Since the BOM is optional for Unicode-encoded files, its absence is not a clear sign for an ASCI-/ANSI-encoded file.

So here is the code, featuring a lot of explanatory remarks (rem) -- I hope, it helps:

@echo off
setlocal EnableExtensions DisableDelayedExpansion

rem // Define constants here:
set "_FILE=%~1" & rem // (provide file via the first command line argument)

rem // Check whether a dedicated file is given (so no wild-cards):
2> nul >&2 (< "%_FILE%" set /P ="" & ver) || (
    rem // The file does not exist:
    >&2 echo The file could not be found, hence there is no encoding!
    exit /B 255
)

rem // Determine the file size:
set "SIZE=" & for %%F in ("%_FILE%") do set "SIZE=%%~zF"
if not defined SIZE (
    rem // The file does not exist:
    >&2 echo The file could not be found, hence there is no encoding!
    exit /B 255
)
if %SIZE% EQU 0 (
    rem // The file is empty:
    >&2 echo The file is empty, hence encoding cannot be determined!
    exit /B 1
)

rem // Store current code page to be able to restore it finally:
for /F "tokens=2 delims=:" %%C in ('chcp') do set /A "$CP=%%C"
rem /* Change to code page 437 (original IBM PC or DOS code page) temporarily;
rem    this is necessary for extended characters not to be converted: */
> nul chcp 437

rem // Attempt to read first line from file; this fails if zero-bytes occur:
(
    rem /* The loop does not iterate over an empty file or one with empty lines only;
    rem    therefore, the behaviour is the same as when zero-bytes occur: */
    for /F usebackq^ delims^=^ eol^= %%L in ("%_FILE%") do (
        rem // Abort reading file after first non-empty line:
        goto :NEXT
    )
) || (
    rem /* The `for /F` loop returns a non-zero exit code in case the file is empty,
    rem    contains empty lines only or the first non-empty line contains zero-bytes;
    rem    to determine whether there are zero-bytes, let `find` process the file,
    rem    which removes zero-bytes or converts them to line-breaks, so `for /F` can
    rem    read the file;
    rem    however, `find` would read the whole file, hence do that only for small
    rem    ones and skip that for large ones, such contains zero-bytes most likely: */
    if %SIZE% LEQ 8192 (
        (
            rem // In case the file contains line-breaks only, the loop does not iterate:
            for /F delims^=^ eol^= %%L in ('^< "%_FILE%" find /V ""') do (
                rem // Abort reading file after first non-empty line:
                goto :ZERO
            )
        ) || (
            rem /* The loop did not iterate, so the file contains line-breaks only;
            rem    restore the initial code page prior to termination: */
            > nul chcp %$CP%
            >&2 echo The file holds only empty lines, hence encoding cannot be determined!
            exit /B 1
        )
    )
)

rem // This point is reached in case the file contains zero-bytes:
:ZERO
rem // Restore the initial code page prior to termination:
> nul chcp %$CP%
>&2 echo NULL-bytes detected in first line, so file is non-ASCII/ANSI!
exit /B 2

rem // This point is reached in case the file does not contain any zero-bytes:
:NEXT
rem /* Build Byte Order Marks (BOMs) for UTF-16-encoded text (Little Endian and Big Endian)
rem    and for UTF-8-encoded text: */
for /F "tokens=1-3" %%A in ('
    forfiles /P "%~dp0." /M "%~nx0" /C "cmd /C echo 0xFF0xFE 0xFE0xFF 0xEF0xBB0xBF"
') do set "$LE=%%A" & set "$BE=%%B" & set "$U8=%%C"

rem /* Reset line string variable, then store first line string (1023 bytes at most);
rem    in contrast to `for /F`, this does not skip over blank lines: */
< "%_FILE%" (set "LINE=" & set /P LINE="")
rem // Check whether the first line of the file begins with any of the BOMs:
if not "%LINE:~,2%"=="%$LE%" if not "%LINE:~,2%"=="%$BE%" if not "%LINE:~,3%"=="%$U8%" goto :CONT
rem /* One of the BOMs has been encountered, hence the file is Unicode-encoded;
rem    restore the initial code page prior to termination: */
> nul chcp %$CP%
>&2 echo BOM encountered in first line, so file is non-ASCII/ANSI!
exit /B 4

rem // This point is reached in case the file does not appear as Unicode-encoded:
:CONT
rem // Restore the initial code page prior to termination:
> nul chcp %$CP%
echo The file appears to be an ASCII-/ANSI-encoded text.

endlocal
exit /B 0