Regular expression that finds and replaces non-ascii characters with Python

Question

I need to change some characters that are not ASCII to '_'. For example,

Tannh‰user -> Tannh_user

• If I use regular expression with Python, how can I do this?
• Is there better way to do this not using RE?

Answer 1

re.sub(r'[^\x00-\x7F]', '_', theString)

This will work if theString is unicode, or a string in an encoding where ASCII occupies values 0 to 0x7F (latin-1, UTF-8, etc.).

Answer 2

To answer the question

'[\u0080-\uFFFF]'

will match any UTF-8 character not in the range of the first 128 characters

re.sub('[\u0080-\uFFFF]+', '_', x)

will replace any sequence of consecutive nonascii characters with an underscore

Answer 3

Updated for Python 3:

>>> 'Tannh‰user'.encode().decode('ascii', 'replace').replace(u'\ufffd', '_')
'Tannh___user'

First we create byte string using encode() - it uses UTF-8 codec by default. If you have byte string then of course skip this encode step. Then we convert it to "normal" string using the ascii codec.

This uses the property of UTF-8 that all non-ascii characters are encoded as sequence of bytes with value >= 0x80.

---

Original answer – for Python 2:

How to do it using built-in str.decode method:

>>> 'Tannh‰user'.decode('ascii', 'replace').replace(u'\ufffd', '_')
u'Tannh___user'

(You get unicode string, so convert it to str if you need.)

You can also convert unicode to str, so one non-ASCII character is replaced by ASCII one. But the problem is that unicode.encode with replace translates non-ASCII characters into '?', so you don't know if the question mark was there already before; see solution from Ignacio Vazquez-Abrams.

---

Another way, using ord() and comparing value of each character if it fits in ASCII range (0-127) - this works for unicode strings and for str in utf-8, latin and some other encodings:

>>> s = 'Tannh‰user' # or u'Tannh‰user' in Python 2
>>> 
>>> ''.join(c if ord(c) < 128 else '_' for c in s)
'Tannh_user'

Answer 4

Using Python's support for character encodings:

# coding: utf8
import codecs

def underscorereplace_errors(exc):
  return (u'_', exc.end)

codecs.register_error('underscorereplace', underscorereplace_errors)

print u'Tannh‰user'.encode('ascii', 'underscorereplace')

Answer 5

I'd rather just call ord on every character in the string, 1 by 1. If ord([char]) >= 128 the character is not an ascii character and should be replaced.

Answer 6

With the magical regex [ -~] one can solve it:

import re
re.sub(r"[^ -~]", "_", "Tannh‰user")
# 'Tannh_user'

---

Explanation:

• The ascii characters are the symbols ranging from " " to "~" - hence [ -~] captures all ascii chars
• By appending ^we can capture all non-ascii characters
• The rest is now a formality

Answer 7

if you know which characters you want to replace, you can apply string methods

mystring.replace('oldchar', 'newchar')