How to remove special characters from strings in python?

Question

I have millions of strings scraped from web like:

s = 'WHAT\xe2\x80\x99S UP DOC?'
type(s) == str # returns True

Special characters like in the string above are inevitable when scraping from the web. How should one remove all such special characters to retain just clean text? I am thinking of regular expression like this based on my very limited experience with unicode characters:

\\x.*[0-9]

They are not special characters, that is a utf-8 encoded string, which when printed will output `WHAT’S UP DOC?` — Padraic Cunningham, Aug 18 '15 at 19:09
http://stackoverflow.com/questions/5843518/remove-all-special-characters-punctuation-and-spaces-from-string — comalex3, Aug 18 '15 at 19:09
@PadraicCunningham: Thanks for correcting me. I meant the same thing. — pg2455, Aug 18 '15 at 19:11
So you want any non-ascii removed? i.e `print(s.decode("ascii",errors="ignore"))` — Padraic Cunningham, Aug 18 '15 at 19:12
Do you want to mention this solution or should I write it down in my answer? — pg2455, Aug 18 '15 at 19:15

score 3 · Accepted Answer · answered Aug 18 '15 at 19:10

The special characters are not actually multiple characters long, that is just how they are represented so your regex isn't going to work. If you print you will see the actual unicode (utf-8) characters

>>> s = 'WHAT\xe2\x80\x99S UP DOC?'
>>> print(s)
WHATâS UP DOC?
>>> repr(s)
"'WHATâ\\x80\\x99S UP DOC?'"

If you want to print only the ascii characters you can check if the character is in string.printable

>>> import string
>>> ''.join(i for i in s if i in string.printable)
'WHATS UP DOC?'

score 2 · Answer 2 · answered Aug 18 '15 at 19:23

2

This thing worked for me as mentioned by Padriac in comments:

s.decode('ascii', errors='ignore')

answered Aug 18 '15 at 19:23

pg2455

5,039
14
51
78

How to remove special characters from strings in python?

2 Answers2