IO::Handle to get and unget unicode characters

Question

I think I've run into a problem with Unicode and IO::Handle. It's very likely I'm doing something wrong. I want to get and unget individual unicode characters (not bytes) from an IO::Handle. But I'm getting a surprising error.

#!/usr/local/bin/perl

use 5.016;
use utf8;
use strict;
use warnings;

binmode(STDIN,  ':encoding(utf-8)');
binmode(STDOUT, ':encoding(utf-8)');
binmode(STDERR, ':encoding(utf-8)');

my $string = qq[a Å];
my $fh = IO::File->new();

$fh->open(\$string, '<:encoding(UTF-8)');

say $fh->getc(); # a
say $fh->getc(); # SPACE
say $fh->getc(); # Å LATIN CAPITAL LETTER A WITH RING ABOVE (U+00C5)
$fh->ungetc(ord("Å"));
say $fh->getc(); # should be A RING again.

The error message from the ungetc() line is "Malformed UTF-8 character (unexpected end of string) in say at unicode.pl line 21. "\x{00c5}" does not map to utf8 at unicode.pl line 21." But that's the correct hex for the character, and it should map to the character.

I used a hex editor to make sure that the bytes for A-RING are correct for UTF-8.

This seems to be a problem for any two-byte character.

The final say outputs '\xC5' (literally four characters: backslash, x, C, 5).

And I've tested this by reading from files instead of scalar variables. The result is the same.

This is perl 5, version 16, subversion 2 (v5.16.2) built for darwin-2level

And the script is saved in UTF-8. That was the first thing I checked.

Tip: `binmode(STDIN, ':encoding(utf-8)'); binmode(STDOUT, ':encoding(utf-8)'); binmode(STDERR, ':encoding(utf-8)');` can be written `use open ':std', ':encoding(utf-8)';`. (Bonus, it happens at compile time.) — ikegami, Jan 06 '13 at 06:46
Tip: `my $fh = IO::File->new(); $fh->open(\$string, '<:encoding(UTF-8)');` can be written `open(my $fh, '<:encoding(UTF-8)', \$string)`. Yes, you can still use the methods. — ikegami, Jan 06 '13 at 06:51

score 2 · Accepted Answer · answered Jan 06 '13 at 23:02

I am pretty certain this proves there is a serious Unicode-processing bug going on, given that this output:

perl5.16.0 ungettest
ungettest 98896 @ Sun Jan  6 16:01:08 2013: sending normal line to kid
ungettest 98896 @ Sun Jan  6 16:01:08 2013: await()ing kid
ungettest 98897 @ Sun Jan  6 16:01:08 2013: ungetting litte z
ungettest 98897 @ Sun Jan  6 16:01:08 2013: ungetting big sigma
ungettest 98897 @ Sun Jan  6 16:01:08 2013: kid looping on parental input
98897: Unexpected fatalized warning: utf8 "\xA3" does not map to Unicode at ungettest line 40, <STDIN> line 1.
 at ungettest line 10, <STDIN> line 1.
    main::__ANON__('utf8 "\xA3" does not map to Unicode at ungettest line 40, <ST...') called at ungettest line 40
98896: parent pclose failed: 65280,  at ungettest line 28.
Exit 255

is produced by this program:

#!/usr/bin/env perl

use v5.16;
use strict;
use warnings;
use open qw( :utf8    :std );

use Carp;

$SIG{__WARN__} = sub {  confess "$$: Unexpected fatalized warning: @_" };

sub ungetchar($) {
    my $char = shift();
    confess "$$: expected single character pushback, not <$char>" if length($char) != 1;
    STDIN->ungetc(ord $char);
}

sub debug {
    my $now = localtime(time());
    print STDERR "$0 $$ \@ $now: @_\n";
}

if (open(STDOUT, "|-")                          // confess "$$: cannot fork: $!") {
    $| = 1;
    debug("sending normal line to kid");
    say "From \N{greek:alpha} to \N{greek:omega}.";
    debug("await()ing kid");
    close(STDOUT)                               || confess "$$: parent pclose failed: $?, $!";
    debug("child finished, parent exiting normally");
    exit(0);
}

debug("ungetting litte z");
ungetchar("z")                                  || confess "$$: ASCII ungetchar failed: $!";

debug("ungetting big sigma");
ungetchar("\N{greek:Sigma}")                    || confess "$$: Unicode ungetchar failed: $!";

debug("kid looping on parental input");
while (<STDIN>) {
    chomp;
    debug("kid got $_");
}
close(STDIN)                                    || confess "$$: child pclose failed: $?, $!";
debug("parent closed pipe, child exiting normally");
exit 0;

So it looks like ord() is returning the correct unicode code point, but ungetc() isn't interpreting it as a code point, but a raw sequence of bytes. Does that sound right? If it is, then it's a significant bug. — Michael, Jan 07 '13 at 03:16
@Michael Yes, in perlio.c, in `PerlIO_ungetc(PerlIO *f, int ch)`, what is happening is that there is essentially a casting of an int into a char via `STDCHAR buf = ch`, where a `STDCHAR` is just a C `char` type. This is a Big Problem. Please file a bug report. Thank you. — tchrist, Jan 07 '13 at 03:24
Bug report away (via `perlbug`). How long will it take for a confirmation email that they received it? — Michael, Jan 07 '13 at 05:03
@Michael You should have received the auto-reply ticket virtually immediately. However, it doesn’t seem to have made its way through the whole system yet. At least, I haven’t gotten the p5p copy yet. — tchrist, Jan 07 '13 at 05:12
Bug report made it this time: https://rt.perl.org/rt3/Public/Bug/Display.html?id=116322 — Michael, Jan 07 '13 at 05:44

ikegami · Answer 2 · 2013-01-06T22:08:45.753

1

ungetc prepends a byte to the underlying input stream. To return U+00C5, the stream would have to contain C3 A5 (the UTF-8 encoding of that character), not C5 (ord("Å")). Use IO::Unread's unread instead.

edited Jan 06 '13 at 22:08

answered Jan 06 '13 at 06:17

ikegami

367,544
15
269
518

1

The documentation for ungetc() in IO::Handle says that it pushes a character back onto the stream. And since the stream is UTF-8 encoded, it should work, shouldn't it? – Michael Jan 06 '13 at 06:31
Character is a vague term. In this case, it's using the C definition. – ikegami Jan 06 '13 at 06:34
Except that it isn't using the C definition when reading characters. – Michael Jan 06 '13 at 06:36
It would make sense for your file to contain `20 20 20` (three spaces in UTF-8). Would it make sense for your file to contain `C5 20 20 20`? No, prepending `C5` makes no sense. – ikegami Jan 06 '13 at 06:38
But nowhere am I asking it to prepend C5. – Michael Jan 06 '13 at 06:40

IO::Handle to get and unget unicode characters

2 Answers2