31

I have a script that parses the filenames of TV episodes (show.name.s01e02.avi for example), grabs the episode name (from the www.thetvdb.com API) and automatically renames them into something nicer (Show Name - [01x02].avi)

The script works fine, that is until you try and use it on files that have Unicode show-names (something I never really thought about, since all the files I have are English, so mostly pretty-much all fall within [a-zA-Z0-9'\-])

How can I allow the regular expressions to match accented characters and the likes? Currently the regex's config section looks like..

config['valid_filename_chars'] = """0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ!@£$%^&*()_+=-[]{}"'.,<>`~? """
config['valid_filename_chars_regex'] = re.escape(config['valid_filename_chars'])

config['name_parse'] = [
    # foo_[s01]_[e01]
    re.compile('''^([%s]+?)[ \._\-]\[[Ss]([0-9]+?)\]_\[[Ee]([0-9]+?)\]?[^\\/]*$'''% (config['valid_filename_chars_regex'])),
    # foo.1x09*
    re.compile('''^([%s]+?)[ \._\-]\[?([0-9]+)x([0-9]+)[^\\/]*$''' % (config['valid_filename_chars_regex'])),
    # foo.s01.e01, foo.s01_e01
    re.compile('''^([%s]+?)[ \._\-][Ss]([0-9]+)[\.\- ]?[Ee]([0-9]+)[^\\/]*$''' % (config['valid_filename_chars_regex'])),
    # foo.103*
    re.compile('''^([%s]+)[ \._\-]([0-9]{1})([0-9]{2})[\._ -][^\\/]*$''' % (config['valid_filename_chars_regex'])),
    # foo.0103*
    re.compile('''^([%s]+)[ \._\-]([0-9]{2})([0-9]{2,3})[\._ -][^\\/]*$''' % (config['valid_filename_chars_regex'])),
]
tchrist
  • 78,834
  • 30
  • 123
  • 180
dbr
  • 165,801
  • 69
  • 278
  • 343

4 Answers4

20

Use a subrange of [\u0000-\uFFFF] for what you want.

You can also use the re.UNICODE compile flag. The docs say that if UNICODE is set, \w will match the characters [0-9_] plus whatever is classified as alphanumeric in the Unicode character properties database.

See also http://coding.derkeiler.com/Archive/Python/comp.lang.python/2004-05/2560.html.

Azat Ibrakov
  • 9,998
  • 9
  • 38
  • 50
Mark Cidade
  • 98,437
  • 31
  • 224
  • 236
5

Python's re module doesn't support \p{Letter} or \X. However, the new regex implementation on PyPI does.

John Machin
  • 81,303
  • 11
  • 141
  • 189
MRAB
  • 20,356
  • 6
  • 40
  • 33
  • 1
    That module’s `\X` is broken; they have misunderstood the standard. You cannot just use `\PM\pM*` or you get things wrong. Consider the string `"\r\r\n\x{301}A\x{301}"`. A [conforming application](http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries) finds the following 4 matches for `\X`: 1 CP U+000D, 2 CPs U+000D U+000A, 1 CP U+0301, and 2 CPs U+0041 U+0301. The broken `\PM\pM*` also finds 4 matches, but **the wrong ones**: 1 CP U+000D, 1 CP U+000D, 2 CPs U+000A U+0301, and 2 CPs U+0041 U+0301. You **MUST** not break up CRLFs nor place Marks on any `\P{Grapheme_Base}` code points. – tchrist Apr 02 '11 at 01:29
  • The definition of \X was based on the contents of this: http://www.regular-expressions.info/unicode.html I'll see if I can fix it. – MRAB Apr 02 '11 at 01:37
  • The original grapheme cluster idea was a bit confused about a few things, so the first folks to jump on `\X` ended up doing it a bit wrong. The current ICU and Perl implementations do get it right, and indeed even use the extended grapheme cluster definitions: Try `perl5.12.0 -le 'printf "%d %v04X\n", length, $_ for "\r\r\n\x{301}A\x{301}" =~ /\X/g'` or later to see the improved answers. – tchrist Apr 02 '11 at 02:00
  • @tchrist: It seems to be fixed now (or never been broken). `python -c'import regex as re; print(re.findall(u"\X", u"\r\r\n\u0301A\u0301"))'` prints expected results: `[u'\r', u'\r\n', u'\u0301', u'A\u0301']` – jfs Dec 21 '13 at 03:10
  • @J.F.Sebastian I know it did get fixed. I corresponded with the author at length. He’s great. – tchrist Dec 21 '13 at 03:57
4

In Mastering Regular Expressions from Jeffrey Friedl (great book) it is mentioned that you could use \p{Letter} which will match unicode stuff that is considered a letter.

Peter Stuifzand
  • 5,084
  • 1
  • 23
  • 28
  • 1
    `\p{Letter}` is not supported in all regex engines, and in Python's case, it's not supported in the default `re` engine. It's only supported in the `regex` package. – nhahtdh Nov 11 '15 at 09:50
0

\X seems to be available as a generic word-character in some languages, it allows you to match a single character disregarding of how many bytes it takes up. Might be useful.

grapefrukt
  • 27,016
  • 6
  • 49
  • 73