printf aligning problem with degree (°) character

Question

I have a problem similar to How can I properly align UTF-8 strings with Perl's printf?:

My (Linux) system's locale default is LC_CTYPE=de_DE.UTF-8 and I wrote a Perl program (using perl-5.26.1) that is "not" using Unicode characters, but some from the ISO Latin-1 character set (that is ° for example). There fore I did not activate any Unicode or locale features in my Perl script.

"Everything" seems to work fine with one excpetion: I'm using a printf format of %-10s to align strings but that does not work as expected.

Playing in the debugger I fount this behavior:

  DB<1> $s='X°X'

  DB<2> printf("_%3s_\n", $s)
_X°X_

Looks OK so far...

  DB<3> printf("_%4s_\n", $s)
_X°X_

Oops; shouldn't that be "_ X°X_"?

  DB<4> printf("_%5s_\n", $s)
_ X°X_

Off by one?

  DB<5> x length($s)
0  4

Shouldn't that be 3?

  DB<8> x ord($s[1])
0  0
  DB<9> x $s
0  'X°X'
  DB<10>

Shouldn't ° be encoded as one byte? I thought UTF-8 maps the Latin-1 range unmodified to Unicode.

So may questions are:

1. What's going on?

2. Is it a Perl bug?

3. If not, how can I fix the formatting and string length?

Answer 1

UTF-8 only maps the ASCII range (0..127) to 1 byte. Latin-1 characters are in the range 0..255; UTF-8 can't map them all to one byte. If it did, there would be no mappings left for anything else.

Characters from 0 to 127 are encoded in 1 byte.
Characters from 128 to 2047 are encoded in 2 bytes.
And so on.

https://en.wikipedia.org/wiki/UTF-8

You need use utf8; and binmode STDOUT, ':encoding(UTF-8)'; in your Perl script (I did the same with STDIN and STDERR just for consistency):

#!/usr/bin/perl

use strict;
use warnings;
use utf8;

BEGIN {
    binmode STDIN,  ':encoding(UTF-8)';
    binmode STDOUT, ':encoding(UTF-8)';
    binmode STDERR, ':encoding(UTF-8)';
}

printf "|%-10s|\n", "x";
printf "|%-10s|\n", "°";

The output is correctly aligned:

|x         |
|°         |

If I comment out either use utf8; or binmode STDOUT, ':encoding(UTF-8)';, the output is misaligned and/or the degree character isn't displayed correctly.

Quoting perldoc utf8 (the documentation for the utf8 module):

The "use utf8" pragma tells the Perl parser to allow UTF-8 in the program text in the current lexical scope.

(This requires an output device or terminal emulator configured to display UTF-8.)

Answer 2

Perl code must be encoded using ASCII (no utf8;, the default) or UTF-8 (use utf8;).

° is not in the ASCII character set, and you apparently didn't use utf8; either, so your program couldn't possibly contain ° as you claim.

First, encode the program using UTF-8 (if it's not already) and tell Perl that your program is encoded using UTF-8 by adding

use utf8;   # The source code is encoded using UTF-8.

Secondly, you apparently didn't tell Perl to encode what you printed either. Fix that by adding

use open ':std', ':encoding(UTF-8)';   # The terminal provides/expects UTF-8.

The latter sets the default encoding for files open in scope of the pragma. If you want to avoid this, you can use the following instead:

BEGIN {   # The terminal provides/expects UTF-8.
   binmode(STDIN,  ':encoding(UTF-8)');
   binmode(STDOUT, ':encoding(UTF-8)');
   binmode(STDERR, ':encoding(UTF-8)');
}