Fast way to filter illegal xml unicode chars in python?

Question

The XML specification lists a bunch of Unicode characters that are either illegal or "discouraged". Given a string, how can I remove all illegal characters from it?

I came up with the following regular expression, but it's a bit of a mouthful.

illegal_xml_re = re.compile(u'[\x00-\x08\x0b-\x1f\x7f-\x84\x86-\x9f\ud800-\udfff\ufdd0-\ufddf\ufffe-\uffff]')
clean = illegal_xml_re.sub('', dirty)

(Python 2.5 doesn't know about Unicode chars above 0xFFFF, so no need to filter those.)

Python's maximum unicode codepoint depends on how it was configured when compiled, check `sys.maxunicode`. — u0b34a0f6ae, Nov 10 '09 at 13:18
On my machine, using this regular expression to process a 2.3mb string takes .34 seconds. That seems pretty fast to me. — Robert Rossney, Nov 10 '09 at 21:12
It helps others helping you, when you provide full running code! `import re` and `dirty = "I might be not correct."` — MasterControlProgram, May 29 '20 at 09:13

score 22 · Accepted Answer · edited May 23 '17 at 12:32

Recently we (Trac XmlRpcPlugin maintainers) have been notified of the fact that the regular expression above strips surrogate pairs on Python narrow builds (see th:comment:13:ticket:11050) . An alternative approach consists in using the following regex (see th:changeset:13729) .

_illegal_unichrs = [(0x00, 0x08), (0x0B, 0x0C), (0x0E, 0x1F), 
                        (0x7F, 0x84), (0x86, 0x9F), 
                        (0xFDD0, 0xFDDF), (0xFFFE, 0xFFFF)] 
if sys.maxunicode >= 0x10000:  # not narrow build 
        _illegal_unichrs.extend([(0x1FFFE, 0x1FFFF), (0x2FFFE, 0x2FFFF), 
                                 (0x3FFFE, 0x3FFFF), (0x4FFFE, 0x4FFFF), 
                                 (0x5FFFE, 0x5FFFF), (0x6FFFE, 0x6FFFF), 
                                 (0x7FFFE, 0x7FFFF), (0x8FFFE, 0x8FFFF), 
                                 (0x9FFFE, 0x9FFFF), (0xAFFFE, 0xAFFFF), 
                                 (0xBFFFE, 0xBFFFF), (0xCFFFE, 0xCFFFF), 
                                 (0xDFFFE, 0xDFFFF), (0xEFFFE, 0xEFFFF), 
                                 (0xFFFFE, 0xFFFFF), (0x10FFFE, 0x10FFFF)]) 

_illegal_ranges = ["%s-%s" % (unichr(low), unichr(high)) 
                   for (low, high) in _illegal_unichrs] 
_illegal_xml_chars_RE = re.compile(u'[%s]' % u''.join(_illegal_ranges))

p.s. See this post on surrogates explaining what they are for .

Update so as to not to match (replace) 0x0D which is a valid XML character.

Note that surrogate pairs are explicitly excluded from the legal characters in the W3C XML specification, so any xml containing them is not guaranteed to parse correctly in other libraries. However, since usually you would serialize the XML into utf-8 or utf-16, the problem should disappear. Just steer clear of utf-32. — itsadok, Mar 09 '14 at 06:56
I have updated the regex to match 0x0D character. See [th:ticket:11635](http://trac-hacks.org/ticket/11635) , [th:changeset:13776](http://trac-hacks.org/changeset/13776) and [XML character range definition](http://www.w3.org/TR/REC-xml/#NT-Char). — Olemis Lang, Mar 19 '14 at 04:53
For a cleaned string, say `myString = _illegal_xml_chars_RE.sub("", myString)` — MasterControlProgram, May 29 '20 at 09:07
For those like myself that are confused by the inclusion of so many ranges, many of them aren't invalid XML characters, just [not recommended](https://en.wikipedia.org/wiki/Valid_characters_in_XML#Characters_allowed_but_discouraged) — Indigenuity, Dec 29 '21 at 22:54

score 8 · Answer 2 · answered Oct 28 '20 at 09:39

Here's an updated version of Olemis Lang's answer for Python 3:

import re
import sys

illegal_unichrs = [(0x00, 0x08), (0x0B, 0x0C), (0x0E, 0x1F),
                   (0x7F, 0x84), (0x86, 0x9F),
                   (0xFDD0, 0xFDDF), (0xFFFE, 0xFFFF)]
if sys.maxunicode >= 0x10000:  # not narrow build
    illegal_unichrs.extend([(0x1FFFE, 0x1FFFF), (0x2FFFE, 0x2FFFF),
                            (0x3FFFE, 0x3FFFF), (0x4FFFE, 0x4FFFF),
                            (0x5FFFE, 0x5FFFF), (0x6FFFE, 0x6FFFF),
                            (0x7FFFE, 0x7FFFF), (0x8FFFE, 0x8FFFF),
                            (0x9FFFE, 0x9FFFF), (0xAFFFE, 0xAFFFF),
                            (0xBFFFE, 0xBFFFF), (0xCFFFE, 0xCFFFF),
                            (0xDFFFE, 0xDFFFF), (0xEFFFE, 0xEFFFF),
                            (0xFFFFE, 0xFFFFF), (0x10FFFE, 0x10FFFF)])

illegal_ranges = [fr'{chr(low)}-{chr(high)}' for (low, high) in illegal_unichrs]
xml_illegal_character_regex = '[' + ''.join(illegal_ranges) + ']'
illegal_xml_chars_re = re.compile(xml_illegal_character_regex)
# filtered_string = illegal_xml_chars_re.sub('', original_string)

score 4 · Answer 3 · answered Nov 10 '09 at 13:30

You could also use unicode's translate method to delete selected codepoints. However, the mapping you have is pretty big (2128 codepoints) and that might make it much slower than just using a regex:

ranges = [(0, 8), (0xb, 0x1f), (0x7f, 0x84), (0x86, 0x9f), (0xd800, 0xdfff), (0xfdd0, 0xfddf), (0xfffe, 0xffff)]
# fromkeys creates  the wanted (codepoint -> None) mapping
nukemap = dict.fromkeys(r for start, end in ranges for r in range(start, end+1))
clean = dirty.translate(nukemap)

After some testing, this seems to be much slower than a regexp, especially for large strings. — itsadok, Nov 10 '09 at 14:57

Fast way to filter illegal xml unicode chars in python?

3 Answers3

Linked