Regex punctuation split [Python]

Question

Can anyone help me a bit with regexs? I currently have this: re.split(" +", line.rstrip()), which separates by spaces.

How could I expand this to cover punctuation, too?

Answer 1

The official Python documentation has a good example for this one. It will split on all non-alphanumeric characters (whitespace and punctuation). Literally \W is the character class for all Non-Word characters. Note: the underscore "_" is considered a "word" character and will not be part of the split here.

re.split('\W+', 'Words, words, words.')

See https://docs.python.org/3/library/re.html for more examples, search page for "re.split"

Answer 2

Using string.punctuation and character class:

>>> from string import punctuation
>>> r = re.compile(r'[\s{}]+'.format(re.escape(punctuation)))
>>> r.split('dss!dfs^  #$% jjj^')
['dss', 'dfs', 'jjj', '']

Answer 3

import re
st='one two,three; four-five,    six'

print re.split(r'\s+|[,;.-]\s*', st)
# ['one', 'two', 'three', 'four', 'five', 'six']

Answer 4

When you consider using a regex to split with any punctuation, you should bear in mind that the \W pattern does not match an underscore (which is a punctuation char, too).

Thus, you can use

import re
tokens = re.split(r'[\W_]+', text)

where [\W_] matches any Unicode non-alphanumeric chars.

Since re.split might return empty items when the match appears at the start or end of string, it is better to use a positive logic and use

import re
tokens = re.findall(r'[^\W_]+', text)

where [^\W_] matches any Unicode alphanumeric chars.

See the Python demo:

import re
text = "!Hello, world!"
print( re.split(r'[\W_]+', text) )
# => ['', 'Hello', 'world', '']
print( re.findall(r'[^\W_]+', text) )
# => ['Hello', 'world']