Remove characters not in a set in Python

Question

I have a string and I'm trying to remove all characters that are not alphanumeric nor in this set

'''!$%*()_-=+\/.,><:;'"?|'''.

I know this removes all non alphanumeric characters but how can I do better?

re.sub(r'\W+','',line)

score 7 · Accepted Answer · answered May 31 '12 at 18:53

7

A Python 2.x non-regex solution:

punctuation = '''!$%*()_-=+\/.,><:;'"?|'''
allowed = string.digits + string.letters + punctuation
filter(allowed.__contains__, s)

The string to filter is s. (This probably isn't the fastest solution for long strings.)

answered May 31 '12 at 18:53

Sven Marnach

574,206
118
941
841

"how can I do better?" How is this better? – JBernardo May 31 '12 at 18:57
@JBernardo: It actually does what the OP wants, in contrast to the given regex-based solution, which also strips out the punctuation. – Sven Marnach May 31 '12 at 18:58
It seems OP doesn't want punctuation... Or am I missing something? – JBernardo May 31 '12 at 19:00
@JBernardo: The OP wants to remove "characters that are not alphanumeric nor in this set", i.e. the punctuation in the given set should *not* be removed. – Sven Marnach May 31 '12 at 19:01
Thank you! It works, I'll have to see now how efficient it is as I have about 10GB of text to go through. – Youcha May 31 '12 at 19:07
It's fast enough actually. It takes about 1 hour to go over 10GB of text on my laptop. – Youcha May 31 '12 at 23:16

score 4 · Answer 2 · edited May 23 '17 at 11:45

With credit to this thread: Remove specific characters from a string in python

First, there's no need to retype all the punctuation manually. The string module defines string.punctuation as a property for your convenience. (Use help(string) to see other similar definitions available)

>>> import string
>>>string.punctuation
'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

The exact application of the solution will take some fiddling to define undesired characters; a big downside is that in this form, it only removes the characters you tell it to remove. If you're sure your file is 100% ASCII characters, then you could define:

>>> delchars = ''.join(c for c in map(chr, range(256)) if c not in (string.punctuation + string.digits + string.letters) )

You can filter characters by throwing them out:

>>> text.translate(None, delchars)

EDIT: Here's some interesting timing information for the various methods: Stripping everything but alphanumeric chars from a string in Python

While `str.translate()` can be used for this purpose (and is probably faster than the solution I gave), the exact given code *removes* punctuation, while the OP wants to *retain* the punctuation in the given set. — Sven Marnach, May 31 '12 at 19:13
Thanks for the catch. I've adjusted my solution accordingly. You might also be interested in the link to performance tests done in another thread. — abought, May 31 '12 at 20:06

score 1 · Answer 3 · answered May 31 '12 at 19:36

In Python 3.x, you can use the translate method on string to remove characters you do not want:

>>> def remove(string, characters):
        return string.translate(str.maketrans('', '', characters))

>>> import string
>>> remove(string.printable, string.ascii_letters + string.digits + \
                             '''!$%*()_-=+\/.,><:;'"?|''')
'#&@[]^`{}~ \t\n\r\x0b\x0c'

Remove characters not in a set in Python

3 Answers3