Grammar and unicode characters

Question

Why the below Grammar fails to parse for unicode characters?

it parses fine after removing word boundaries from <sym>.

#!/usr/bin/env perl6

grammar G {


  proto rule TOP { * }

  rule TOP:sym<y>  { «<.sym>» }
  rule TOP:sym<✓>  { «<.sym>» }

}

say G.parse('y'); # ｢y｣
say G.parse('✓'); # Nil

Answer 1

From the « and » "left and right word boundary" doc:

[«] matches positions where there is a non-word character at the left, or the start of the string, and a word character to the right.

✓ isn't a word character. So the word boundary assertion fails.

What is and isn't a "word character"

"word", in the sense of the \w character class, has the same definition in P6 as it does in P5 (when not using the P5 \a regex modifier), namely letters, some decimal digits, or an underscore:

• Characters whose Unicode general category starts with an L, which stands for Letter.¹

• Characters whose Unicode general category is Nd, which stands for Number, decimal.²

• _, an underscore.

"alpha 'Nd under"

In a comment below @p6steve++ contributes a cute mnemonic that adds "under" to the usual "alphanum".

But "num" is kinda wrong because it isn't any number but only some decimal digits, specifically the characters that match the Unicode General Category Nd (matched by P6 regex /<:Nd>/).²

This leads naturally to alphaNdunder (alpha Nd under) pronounced "alpha 'nd under".

Footnotes

¹ Letters are matched by the P6 regex /<:L>/. This includes Ll (Letter, lowercase) (matched by /<:Ll>/) as JJ notes but also others including Lu (Letter, uppercase) and Lo (Letter, other), which latter includes the ら character JJ also mentions. There are other letter sub-categories too.

² Decimal digits with the Unicode general category Nd are matched by the P6 regex /<:Nd>/. This covers decimal digits that can be chained together to produce arbitrarily large decimal numbers where each digit position adds a power of ten. It excludes decimal digits that have a "typographic context" (my phrasing follows the example of Wikipedia). For example, 1 is the English decimal digit denoting one; it is included. But ¹ and ① are excluded because they have a "typographic context". For a billion+ people their native languages use १ to denote one and १ is included in the Nd category for decimal digits. But for another billion+ people their native languages use 一 for one but it is excluded from the Nd category (and is in the L category for letters instead). Similarly ६ (Devanagari 6) is included in the Nd category but 六 (Han number 6) is excluded.

Answer 2

I keep starting my answers with "Raiph is right". But he is. Also, an example of why this is so:

for <y ✓ Ⅲ> {
    say $_.uniprops;
    say m/<|w>/;
}

The second line of the loop compares against the word boundary anchor; just the first character, which can be a part of an actual word, matches that anchor. It also prints the Unicode properties in the first line of the loop; in the first case it's a letter, (Ll), it's not in the other two cases. You can use any Ll character as part of a word, and in your grammar, but only characters with that Unicode property can actually form words.

grammar G {


  proto rule TOP { * }

  rule TOP:sym<y>  { «<.sym>» }
  rule TOP:sym<ら>  { «<.sym>» }

}

say G.parse('y'); # ｢y｣
say G.parse('ら'); # This is a hiragana letter, so it works.