How to remove special characters except space from a file in python?

Question

I have a huge corpus of text (line by line) and I want to remove special characters but sustain the space and structure of the string.

hello? there A-Z-R_T(,**), world, welcome to python.
this **should? the next line#followed- by@ an#other %million^ %%like $this.

should be

hello there A Z R T world welcome to python
this should be the next line followed by another million like this

Just create a list of the characters you want, A-Z, a-z, 0-9, etc.. And use a `for` loop for iterate over each character in the string replacing the characters thats not in the list with a space. — Wright, Apr 12 '17 at 01:47
is that efficient for a huge corpus of million of lines of text? — pythonlearn, Apr 12 '17 at 01:49

score 39 · Accepted Answer · edited Nov 15 '18 at 09:08

You can use this pattern, too, with regex:

import re
a = '''hello? there A-Z-R_T(,**), world, welcome to python.
this **should? the next line#followed- by@ an#other %million^ %%like $this.'''

for k in a.split("\n"):
    print(re.sub(r"[^a-zA-Z0-9]+", ' ', k))
    # Or:
    # final = " ".join(re.findall(r"[a-zA-Z0-9]+", k))
    # print(final)

Output:

hello there A Z R T world welcome to python 
this should the next line followed by an other million like this

Edit:

Otherwise, you can store the final lines into a list:

final = [re.sub(r"[^a-zA-Z0-9]+", ' ', k) for k in a.split("\n")]
print(final)

Output:

['hello there A Z R T world welcome to python ', 'this should the next line followed by an other million like this ']

this does the job but how to prevent it from putting together all the rows into one single very long row? — pythonlearn, Apr 12 '17 at 02:32

score 9 · Answer 2 · answered Apr 12 '17 at 02:02

9

I think nfn neil answer is great...but i would just add a simple regex to remove all no words character,however it will consider underscore as part of the word

print  re.sub(r'\W+', ' ', string)
>>> hello there A Z R_T world welcome to python

answered Apr 12 '17 at 02:02

Eliethesaiyan

2,327
1
22
35

score 6 · Answer 3 · answered Jun 25 '20 at 11:32

you can try this

import re
sentance = '''hello? there A-Z-R_T(,**), world, welcome to python. this **should? the next line#followed- by@ an#other %million^ %%like $this.'''
res = re.sub('[!,*)@#%(&$_?.^]', '', sentance)
print(res)

re.sub('["]') -> here you can add which symbol you want to remove

ssp4all · Answer 4 · 2020-03-22T01:00:12.190

A more elegant solution would be

print(re.sub(r"\W+|_", " ", string))

>>> hello there A Z R T world welcome to python this should the next line followed by another million like this

Here, re is regex module in python

re.sub will substitute pattern with space i.e., " "

r'' will treat input string as raw (with \n)

\W for all non-words i.e. all special characters *&^%$ etc excluding underscore _

+ will match zero to unlimited matches, similar to * (one to more)

| is logical OR

_ stands for underscore

wwii · Answer 5 · 2017-04-12T18:23:11.853

0

Create a dictionary mapping special characters to None

d = {c:None for c in special_characters}

Make a translation table using the dictionary. Read the entire text into a variable and use str.translate on the entire text.

edited Apr 12 '17 at 18:23

answered Apr 12 '17 at 03:08

wwii

23,232
7
37
77

How to remove special characters except space from a file in python?

5 Answers5

Linked