Removing Narrow 'No-Break Space' Unicode Characters (U+00A0) in python nlp

Question

Non-breaking spaces are printed as whitespace, but handled internally as \xa0. How do I remove all these characters at once?

So far I've replaced it directly:

text = text.replace('\u202f','')  
text = text.replace('\u200d','') 
text = text.replace('\xa0','')

But each time I scrape the text sentences from external source, These characters are different. How do I remove it all at once?

MrBean Bremen · Answer 1 · 2020-04-23T14:13:14.607

2

You can use regular expression substitution instead.
If you want to replace all whitespace, you can just use:

import re

text = re.sub(r'\s', '', text)

This includes all unicode whitespace, as described in the answer to this question.
From that answer, you can see that (at the time of writing), the unicode constants recognized as whitespace (e.g. \s) in Python regular expressions are these:

0x0009
0x000A
0x000B
0x000C
0x000D
0x001C
0x001D
0x001E
0x001F
0x0020
0x0085
0x00A0
0x1680
0x2000
0x2001
0x2002
0x2003
0x2004
0x2005
0x2006
0x2007
0x2008
0x2009
0x200A
0x2028
0x2029
0x202F
0x205F
0x3000

This looks as if this will fit your needs.

edited Apr 23 '20 at 14:13

answered Apr 19 '20 at 11:25

MrBean Bremen

14,916
3
26
46

Every new day while scraping, the unicode characters following usually \u202 are different. So, I want to remove all its occurences. How do i do that? – Subha Maharjan Apr 20 '20 at 16:55
So you are saying they are not whitespace? What are they then? Have you tried the whitespace removal as in the answer? – MrBean Bremen Apr 20 '20 at 17:09
I updated the answer to show what is recognized as whitespace in regular expressions. – MrBean Bremen Apr 23 '20 at 14:13

Removing Narrow 'No-Break Space' Unicode Characters (U+00A0) in python nlp

1 Answers1