How to split sentence including punctuation

Question

If I had the sentence sentence = 'There is light!' and I was to split this sentence with mysentence = sentence.split(), how would I have the output as 'There, is, light, !' of print(mysentence)? What I specifically wanted to do was split the sentence including all punctuation, or just a list of selected punctuation. I got some code but the program is recognizing the characters in the word, not the word.

out = "".join(c for c in punct1 if c not in ('!','.',':'))
out2 = "".join(c for c in punct2 if c not in ('!','.',':'))
out3 = "".join(c for c in punct3 if c not in ('!','.',':'))

How would I use this without recognizing each character in a word, but the word itself. Therefore, the output of "Hello how are you?" should become "Hello, how, are, you, ?" Any way of doing this

Do you want to obtain a list of non-whitespace chunks of text? — Wiktor Stribiżew, Sep 05 '16 at 12:30
I specifically wanted to do was split the sentence including all punctuation, or just a list of selected punctuation, therefore if a user wrote a sentence with a comma, the comma would be recognized as a single word. — Jeremy Silva, Sep 05 '16 at 12:31
Possible duplicate of [Python - Split Strings with Multiple Delimiters](http://stackoverflow.com/questions/1059559/python-split-strings-with-multiple-delimiters) — jwpfox, Sep 05 '16 at 12:33
I actually want the punctuation to be recognized apart from the word, so happy! would be happy, ! — Jeremy Silva, Sep 05 '16 at 12:37

Wiktor Stribiżew · Accepted Answer · 2016-09-05T13:26:39.957

3

You may use a \w+|[^\w\s]+ regex with re.findall to get those chunks:

\w+|[^\w\s]

See the regex demo

Pattern details:

\w+ - 1 or more word chars (letters, digits or underscores)
| - or
[^\w\s] - 1 char other than word / whitespace

Python demo:

import re
p = re.compile(r'\w+|[^\w\s]')
s = "There is light!"
print(p.findall(s))

NOTE: If you want to treat an underscore as punctuation, you need to use something like [a-zA-Z0-9]+|[^A-Za-z0-9\s] pattern.

UPDATE (after comments)

To make sure you match an apostrophe as part of the words, add (?:'\w+)* or (?:'\w+)? to the \w+ in the pattern above:

import re
p = re.compile(r"\w+(?:'\w+)*|[^\w\s]")
s = "There is light!? I'm a human"
print(p.findall(s))

See the updated demo

The (?:'\w+)* matches zero or more (*, if you use ?, it will match 1 or 0) occurrences of an apostrophe followed with 1+ word characters.

edited Sep 05 '16 at 13:26

answered Sep 05 '16 at 12:40

Wiktor Stribiżew

607,720
39
448
563

Thanks, i'll try that – Jeremy Silva Sep 05 '16 at 12:43
What if there are two types of punctuation next to each other, like ('What are you doing?!') – Jeremy Silva Sep 05 '16 at 12:45
If you want to get them as separate items, remove the second `+`, I updated the answer. – Wiktor Stribiżew Sep 05 '16 at 12:46
One more question – Jeremy Silva Sep 05 '16 at 13:13
how do you ignore apostrophe – Jeremy Silva Sep 05 '16 at 13:13
Therefore, the sentence 'Happy, it's his name' would output to 'Happy, ,, it's, his, name' ? – Jeremy Silva Sep 05 '16 at 13:13
For me, it's recognizing the apostrophe as a punctuation – Jeremy Silva Sep 05 '16 at 13:14
Like [`\w+|[^'\w\s]`](https://regex101.com/r/cY6mI5/3)? The `[^...]` is a negated character class, put the non-word chars you want to exclude there. – Wiktor Stribiżew Sep 05 '16 at 13:14
Yeah, i want the program not to recognize the apostrophe as punctuation – Jeremy Silva Sep 05 '16 at 13:15
just in the word, not as separate – Jeremy Silva Sep 05 '16 at 13:18
like ['Happy, I'm, happy'] – Jeremy Silva Sep 05 '16 at 13:21
Like p = re.compile("\w+(?:'\w+)*|[^\w\s]")? – Jeremy Silva Sep 05 '16 at 13:23
I updated the answer. Let me know if there is anything else unclear. – Wiktor Stribiżew Sep 05 '16 at 13:27
Thank you very much – Jeremy Silva Sep 05 '16 at 13:29

How to split sentence including punctuation

1 Answers1