Perl newbie first experience with Unicode (in filename, -e operator, open operator, and cmd window)

Question

I have a Windows Perl (5.16.1 32 bit) program that opens a media file and (using ffmpeg) it extracts segments of audio - the purpose of which is to convert a single album music track (containing multiple songs) into multiple individual song files.

When the name of the media file to be processed is all ASCII characters, this all works rather well. I recently tried this program against a filename that includes Russian characters, and the program fails miserably in several areas.

While this must have to do with Unicode, and as I have never previously needed to do anything with Unicode - I am rather confused about the various aspects of failures that I am experiencing here, nor do I know the fix for the variety of issues I am now facing.

I have distilled this down to the minimum to demonstrate the problems.

If I open a cmd window, and type 'chcp', the return value is 437.

If I do a 'dir' command, this is what is shown for me:

04/01/2019  11:46 AM        71,982,427 IC3PEAK альбом Сладкая.mkv
06/10/2020  10:42 PM               275 test.pl

(Note how in my cmd window, the Russian characters do display as Russian characters.)

My 'test.pl' Perl script is here:

use open ":std", ":encoding(UTF-8)";

$media = "IC3PEAK альбом Сладкая.mkv";

if (-e $media) {
   print "Media file does exist\n";
} else {
   print "Media file does NOT exist\n";
}

open(IN, $media) || die "Media file ($media) can not be opened!\n";

When this Perl script runs, using default chcp value of 437, I get this as output:

Media file does NOT exist
Media file (IC3PEAK альбом Сладкая.mkv) can not be opened!

If I run 'chcp 1250' in my cmd window, and I re-run this Perl script, I get this as output:

Media file does NOT exist
Media file (IC3PEAK Ă°Ă»ÑŒĂ±ĂÂľĂÂĽ Ă¡Ă»Ă°Ă´ĂÂşĂ°Ñ.mkv) can not be opened!

Problem 1: I am told the media file does not exist.

Problem 2: When I print the media file name to STDOUT, notice how the displayed file name non longer matches how it looks when I did the 'dir' command?

Can anyone suggest how to fix these two problems?

PS - Noting, when I change the disk file name to pure ASCII 'IC3PEAK.mkv', and change the $media variable to also equal 'IC3PEAK.mkv', running the modified Perl script gives:

Media file does exist

Answer 1

Following code was tested in Windows 10 1903, perl -MWin32 -e"CORE::say Win32::GetACP()" returns ACP 1252 (Win 10 North America) with Win32 strawberry-perl 5.30.2.1 #1 Tue Mar 17 03:21:32 2020 x64.

Initial attempt to install cpan Win32::Unicode::File failed with t/04_print.t (Wstat: 768 Tests: 13 Failed: 3) message.

A quick search in Google lead to following post on Perl Monks. It looks like the problem with Win32::Unicode::File installation is known for some time.

NOTE: ikegami pointed out that the module can be forcefully installed and failed test can be ignored. Please see his comment bellow.

Following test code confirms that a forced installation cpan -f -i Win32::Unicode::File produces desired outcome.

use strict;
use warnings;
use feature 'say';

use utf8;

use Win32::Console;
use Win32::Unicode::File;

Win32::Console::OutputCP( 65001 );

binmode STDOUT, ':encoding(UTF-8)';
binmode STDERR, ':encoding(UTF-8)';

my $fname = 'Доброе утро Россия.mkv';
my $fh = Win32::Unicode::File->new;

open $fh, '<:encoding(UTF-8)', $fname 
    or die "Can't open $fname $!";

while( <$fh> ) {
    say;
}

close $fh;

Content of input file Доброе утро Россия.mkv is

Доброе утро Россия

As suggested in above mentioned post I resorted to try Win32::LongPath as an alternative. Installation of the module went successfully through.

use strict;
use warnings;
use feature 'say';

use utf8;

use Win32::Console;
use Win32::LongPath;

Win32::Console::OutputCP( 65001 );

binmode STDOUT, ':encoding(UTF-8)';
binmode STDERR, ':encoding(UTF-8)';

my $fname = 'IC3PEAK альбом Сладкая.mkv';
my $fh;

openL \$fh, '<:encoding(UTF-8)', $fname
    or die "Can't open $fname ($^E)";

while( <$fh> ) {
    # process input
    say;
}

close $fh;

Instead of real file IC3PEAK альбом Сладкая.mkv a text file with same name was used in the test with following content

Привет Москва

Note: use openL \$fh, '<', $fname on real mkv file to read content of the file

Answer 2

Three fixes are needed.

---

Non-ASCII source without use utf8;

Your source contains non-ASCII characters.

$media = "IC3PEAK альбом Сладкая.mkv";

Perl expects source code to be encoded using ASCII, unless you use use utf8;. Encode your source using UTF-8 and use use utf8;.

use utf8;

# String of decoded text (aka string of Unicode Code Points).
# Length = 26
my $media = "IC3PEAK альбом Сладкая.mkv";

Assuming your file was encoded using UTF-8, what you had was equivalent to the following:

use utf8;
use Encode qw( encode );

# String of text encoded using UTF-8 (aka string of bytes).
# Length = 39
my $media = encode("UTF-8", "IC3PEAK альбом Сладкая.mkv");

---

Incorrect output encoding

Your code contains

use open ":std", ":encoding(UTF-8)";

This tells Perl the following:

• Decode bytes received from STDIN using UTF-8.
• Encode characters sent to STDOUT and STDERR using UTF-8.
• Do the same for file handles opened in the current lexical scope.

The problem is that your terminal isn't expecting UTF-8. It's expecting cp437 (before chcp 1250) or cp1250 (after chcp 1250).

Solution 1:

Adjust the encoding specified in the use open line. This shows how this can be done without hardcoding the encoding.

Of course, you'll only be able to print the Cyrillic characters if the terminal's OEM code page (as set using chcp) supports the characters. This brings us to a second solution.

Solution 2:

Adjust the terminal to provide/expect UTF-8. This can be done using the following:

chcp 65001

---

Limitation of builtin functions that accept file names

Windows provides two versions of each functions that accepts strings:

• The "UNICODE" version (suffixed with "W" for "wide") accepts/returns strings encoded using UTF-16le. This version supports all Unicode characters.

• The "ANSI" version (suffixed with "A") accepts/returns strings encoded using the Active Code Page (ACP). The "A" version only supports a small subset of the Unicode characters.

  You can obtain the ACP for your system using the following:

  perl -MWin32 -e"CORE::say Win32::GetACP()"
  

Unfortunately, Perl functions (named operators) use the "A" version of system calls and expect/return text encoded using the ACP. This severely limits which file names that can be passed to them.

For example, my system's ACP is 1252, so the "A" version of system calls would not support Cyrillic characters. This means there is nothing I can do to make open, -e, etc work with file names containing Cyrillic characters. ouch.

[Upd: I now recommend Win32::LongPath instead.] The Win32-Unicode distribution can help with this. For example, -e is just a call to stat, and Win32::Unicode::File provides statW, a version of stat that accepts file names as decoded text. Similarly, it provides a replacement for open.