Perl warns about invalid encoding even if I don't read the problematic data from the file

Question

I'm trying to read lines from the first part of a file that contains a text header encoded in the cp1252 encoding, and contains binary data after a specific keyword.

Problem

Perl warns about invalid encoding in parts of the file I never read. I've created an example in two files to demonstrate the problem.

Contents of linebug.pl:

#!/usr/bin/perl
use 5.028;
use strict;
use warnings;
open( my $fh, "<:encoding(cp1252)", "testfile" );
while( <$fh> ) {
    print;
    last if /Last/;
}

Hexdump of testfile, where the byte 0x81 right after the text Wrong is purposefully added because it is not a valid cp1252 codepoint:

46 69 72 73 74 0a         |First.|
4c 61 73 74 0a            |Last.|
42 75 66 66 65 72 0a      |Buffer.|
57 72 6f 6e 67 81 0a      |Wrong..|

The third line Buffer is just there to make it clear that I do not read too far. It is a valid line between the last line I read, and the "binary" data.

Here is the output showing that I only ever read two lines, but perl still emits a warning:

user@host$ perl linebug.pl
cp1252 "\x81" does not map to Unicode at ./linebug.pl line 6.
First
Last
user@host$

As can be seen, my program reads and prints the first two lines, and then exits. It should never try to read and interpret anything else, but I still get the warning about \x81 not mapping to Unicode.

Questions

• Why does it warn? I'm not reading the line. A hunch tells me it's trying to read ahead, but why would it try to decode?
• Is there a workaround, or a better way to handle files where the encoding changes from one section to another?

I still want the warning when reading the initial lines, in case the file is damaged.

Answer 1

Files don't have a concept of lines; they are just streams of bytes. Perl must request a number of bytes from the file from the OS and figure out where the line ends in order to return a line to the program.

Perl could request a single byte at a time from the OS until it has a full line, but that would be very inefficient. There's a lot of overhead involved in making system calls. As such, Perl requests 8 KiB at a time.

Then, the raw data must be decoded before Perl can determine where the line ends, because a raw 0A doesn't necessarily indicate the end of the line.

Similarly to why one doesn't read from a file one byte at a time, asking the decoder to decode just the next character would be inefficient. There is overhead involved every time you start and stop decoding. As such, Perl decodes all the data it reads as it reads it.

So that means that Perl both reads and decodes more than it returns to the program.

---

The solution is to treat the file as binary (because it's not really a text file if the encoding changes by section) and do the decoding yourself.

If you're dealing with a single-byte encoding like cp1252, you can continue using readline (aka <$fh>). However, instead of telling Perl to search for the Code Point of Line Feed (0A), you need to set $/ to the encoding of the Code Point. As it happens, that's also 0A for cp1252, so no change is needed.

use Encode qw( decode );

open( my $fh, "<:raw", $qfn )
   or die( "Can't open \"$qfn\": $!\n" );

while( <$fh> ) {
    $_ = decode( 'cp1252', $_ );      # :encoding(cp1252)
    s/\r\n\z/\n/ if $^O eq 'Win32';   # :crlf
    print;
    last if /Last/;
}

If you weren't using a single-byte encoding, you might have to switch to using read. (You could keep using readline for UTF-8 because of the way it's designed.) When using read, the exact solution depends on a few specifics (that pertain to determining how much to read and how much to decode).

Answer 2

Perl reads from the file in 8 KiB chunks, so way more than a line is read at a time. Data is decoded right as it is read (since the stream must be decoded to find the line endings), so an unexpected encoding is noticed and warned about.

One way to deal with this: use non-buffered reads, via sysread, and read smaller chunks at a time.

Count characters read and once you run into that spot you can back up and continue reading character at a time, again counting them, so to detect the exact place. See this post for a working example of identifying the spot where a warning is fired.

In order to be able to stop there you'll likely want to throw a die out of a $SIG{__WARN__} handler, and have all that code in eval. This will allow you to stop at the place where the warning comes from and have control back.

As you've read right up to that spot, you can then re-open the file in the encoding suitable for the rest of the file and seek to that spot and read the rest.

I can't write and test all that right now, hopefully this helps.