Should I use \d or [0-9] to match digits in a Perl regex?

Question

Having read a number of questions/answers over the past few weeks, I have seen the use of \d in perl regular expressions commented on as incorrect. As in the later versions of perl \d is not the same as [0-9], as \d will represent any Unicode character that has the digit attribute, and that [0-9] represents the characters '0', '1', '2', ..., '9'.

I appreciate that in some contexts [0-9] will be the correct thing to use, and in others \d will be. I was wondering which people feel is the correct default to use?

Personally I find the \d notation very succinct and expressive, whereas in comparison [0-9] is somewhat cumbersome. But I have little experience of doing multi-language code, or rather code for languages that do not fit into the ASCII character range, and therefore may be being naive.

I notice

$find /System/Library/Perl/5.8.8/ -name \*pm | xargs grep '\\d' | wc -l
  298
$find /System/Library/Perl/5.8.8/ -name \*pm | xargs grep '\[0-9\]' | wc -l
  26

Answer 1

It seems to me very dangerous to use \d, It is a poor design decision in the language, as in most cases you want [0-9]. Huffman-coding would dictate the use of \d for ASCII numbers.

Most of the previous posters have already highlighted why you should use [0-9], so let me give you a bit more data:

• If I read the unicode charts correctly '۷۰' is a number (70 in indic, don't take my word for it).

• Try this:

  $ perl -le '$one = chr 0xFF11; print "$one + 1 = ", $one+1;'
  １ + 1 = 1
  

• Here is a partial list of valid numbers (which may or may not show up properly in your browser, depending on the fonts you use), for each number, only the first of those being interpreted as a number when doing arithmetics with Perl, as shown above:

   ZERO:  0٠۰߀०০੦૦୦௦౦೦൦๐໐０
   ONE:   1١۱߁१১੧૧୧௧౧೧൧๑໑１
   TWO:   2٢۲߂२২੨૨୨௨౨೨൨๒໒２
   THREE: 3٣۳߃३৩੩૩୩௩౩೩൩๓໓３
   FOUR:  4٤۴߄४৪੪૪୪௪౪೪൪๔໔４
   FIVE:  5٥۵߅५৫੫૫୫௫౫೫൫๕໕５
   SIX:   6٦۶߆६৬੬૬୬௬౬೬൬๖໖６
   SEVEN: 7٧۷߇७৭੭૭୭௭౭೭൭๗໗７
   EIGHT: 8٨۸߈८৮੮૮୮௮౮೮൮๘໘８
   NINE:  9٩۹߉९৯੯૯୯௯౯೯൯๙໙９��
  

Are you still not convinced?

Answer 2

For maximum safety, I'd suggest using [0-9] any time you don't specifically intend to match all unicode-defined digits.

Per perldoc perluniintro, Perl does not support using digits other than [0-9] as numbers, so I would definitely use [0-9] if the following are both true:

1. You want to use the result as a number (such as performing mathematical operations on it or storing it somewhere that only accepts proper numbers (e.g. an INT column in a database)).

2. It is possible non-digits [^0-9] would be present in the data in such a way that the regular expression could match them. (Note that this one should always be considered true for untrusted/hostile input.)

If either of these are false, there will only rarely be reason to specifically not use \d (and you'll probably be able to tell when that is the case), and if you're trying to match all unicode-defined digits, you'll definitely want to use \d.

Answer 3

According to perlreref, \d is locale-aware and Unicode aware.

However, if the codeset you are using is not Unicode, then you don't need to worry about the Unicode digits, and if the codeset you are using is something like Latin-1 (ISO 8859-1, or 8859-15), then the locale-awareness won't hurt you either because the codeset does not include any other digit characters.

So, for many people, much of the time, you can use \d without concern. However, if Unicode data is part of your work, then you need to consider what you are after more carefully.

Answer 4

Just like nuking the site from orbit, [0-9] is the only way to be sure. Yeah, it is ugly. Yeah, the choice to make \d be UNICODE and locale aware was stupid. But this is our bed and we have to lie in it.

As for the people ducking their heads in the sand saying it doesn't effect the character set they are using today, well you may be using that character set today, but the rest of the world is using UTF-8 now and you will be using it soon as well. Remember to code like the guy who maintains your code is a homicidal maniac who knows where you live.

Oh, and as for Perl modules using \d vs [0-9], even the core still has UNICODE problems.

If you do in fact mean any digit, but want to be able to do math with the results, you can use Text::Unidecode:

#!/usr/bin/perl

use strict;
use warnings;

use Text::Unidecode;

my $number = "\x{1811}\x{1812}\x{1813}\x{1814}\x{1815}";
print "$number is ", unidecode($number), "\n";

After some more testing it looks like Text::Unidecode doesn't handle all digit characters correctly. I am writing a module that will work.

Answer 5

I feel both must have their place. However, 99.999% of the time (especially in my closed big American cooperation world) they are interchangeable. I use perl to manipulate data every day and in none of the data sets I deal with are there numbers that don’t fit in [0-9]. However, I do appreciate that there is an important distinction between \d and[0-9] and it’s good to be aware of that difference. I use \d because it seems more succinct (as you said) and would never be “wrong” in my small world of data manipulation.

Answer 6

The prime objection above for using \d seems to be the non-ascii numeric digits.

This can be obviated with the /a option. e.g.:

m/\d/a

This restricts the digit matching to ASCII only.

https://perldoc.perl.org/perlre#/a-(and-/aa):

Under /a, \d always means precisely the digits "0" to "9"

Answer 7

If you apply \d to a Unicode string (such as in "\X{660}" =~ /\d/), it will match a Unicode digit. If you apply \d to a binary string (such as the UTF-8 equivalent of the above: "\xd9\xa0" =~ /\d/), it will match only the 10 ASCII digits. Perl 5.8 doesn't create Unicode strings by default (unless you specifically ask for it, such as in "\X{...}" or use utf8; etc.).

So my advice is: only pay attention to the difference between \d and [0-9] if your application uses Unicode strings.

Answer 8

If [0-9] feels clunky perhaps you could define: $d=qr/[0-9]/; and use that instead of \d.

Answer 9

As data format controls go up, the need for pattern specificity goes down...

Example, if you are matching a piece of data that has been machine generated and always follows the same output formatting rules, you don't need to be so precise. Take IPv4 addresses. if you are trying to extract the IP address from a router interface configuration line, all you really need is something like:

 'ip\haddress\h(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\D'

IF, on the other hand, you are trying to find an IP address embedded deep somewhere in, say, an email X-Header, or if you are trying to VALIDATE an IP address, well..that is a whole 'nother story!