Unicode-ready wordsearch - Question

Question

Is this code OK? I don't really have a clue which normalization-form I should us (the only thing I noticed is with NFD I get a wrong output).

#!/usr/local/bin/perl
use warnings;
use 5.014;
use utf8;
binmode STDOUT, ':encoding(utf-8)';

use Unicode::Normalize;
use Unicode::Collate::Locale;
use Unicode::GCString;

my $text = "my taxt täxt";
my %hash;

while ( $text =~ m/(\p{Alphabetic}+(?:'\p{Alphabetic}+)?)/g ) { #'
    my $word = $1;
    my $NFC_word = NFC( $word );
    $hash{$NFC_word}++;
}

my $collator = Unicode::Collate::Locale->new( locale => 'DE' ); 

for my $word ( $collator->sort( keys %hash ) ) {
    my $gcword = Unicode::GCString->new( $word );
    printf "%-10.10s : %5d\n", $gcword, $hash{$word};
}

It doesn't matter _which_ normalization you use as long as you use the _same_ one for all strings that you're comparing! — Kerrek SB, Jul 13 '11 at 13:51
@Kerrek That is incorrect. Both Unicode::Collate (and its subclass U::C::Locale) and Unicode::GCString are specifically designed so that normalization **does not matter**. — tchrist, Aug 16 '11 at 02:18

score 3 · Accepted Answer · answered Aug 16 '11 at 01:51

Wow!! I can’t believe nobody answered this. It’s a super duper great question. You almost had it right, too. I like that you’re using Unicode::Collate::Locale and Unicode::GCString. Good for you!

The reason you are getting “wrong” output is because you are not using the Unicode::GCString class's columns method to determine the print width of the stuff you’re printing.

printf is very stupid and just counts code points, not columns, so you have to write your own pad function that takes the GCS columns into account. For example, to do it manually, instead of writing this:

 printf "%-10.10s", $gstring;

You have to write this:

 $colwidth = $gcstring->columns();
 if ($colwidth > 10) {
      print $gcstring->substr(0,10);
 } else {
     print " " x (10 - $colwidth);
     print $gcstring;
 }

See how that works?

Now normalization doesn’t matter. Ignore Kerrek’s old comment. It is very wrong. The UCA is specifically designed not to let normalization enter into the matter. You have to bend over backwards to screw than up, like by passing in normalization => undef to the constructor in case you want to use its gmatch method or some such.

But makes it a difference for the word-counting ($hash{key}++) if I do a normalization before the counting? — sid_com, Aug 16 '11 at 15:47

Unicode-ready wordsearch - Question

1 Answers1

Linked