split string into list but keeping the symbols

Question

I want to split the words in a string but keeping symbols separately too.

s = "Hello world. This-is-foo! I love you"

the output should be

out: ["Hello", "world", ".", "This", "-", "is", "-", "foo", "!", "I", "love", "you"]

I tried:

re.split('(\W)', s)

But this is the output:

['Hello',
 ' ',
 'world',
 '.',
 '',
 ' ',
 'This',
 '-',
 'is',
 '-',
 'foo',
 '!',
 '',
 ' ',
 'I',
 ' ',
 'love',
 ' ',
 'you']

As you can see the spaces are left there. How can I solve this?

score 3 · Answer 1 · answered Apr 13 '20 at 14:16

3

You may use this regex with findall in python:

>>> s = "Hello world. This-is-foo! I love you"
>>> print ( re.findall( r'\w+|[^\s\w]+', s) )
['Hello', 'world', '.', 'This', '-', 'is', '-', 'foo', '!', 'I', 'love', 'you']

RegEx Demo

RegEx Details:

\w+: Match 1 or more word characters
|: OR
[^\s\w]+; Match 1 or more non-word and non-whitespace characters

answered Apr 13 '20 at 14:16

anubhava

761,203
64
569
643

[^\s\w]+ can be replaced to \S – AlexMTX Apr 13 '20 at 14:23
1

Nope, they are different. If input is `"Hello world. This-is-foo!! I love you"` then `\S` will give `!` and `!` separately but this one will keep both `!!` in single element. – anubhava Apr 13 '20 at 14:24
Nice spot! The OP can choose then a precise behaviour on dealing with multiple symbols – AlexMTX Apr 13 '20 at 14:28
@user12195705: Did this work out? – anubhava Apr 13 '20 at 15:50

score 2 · Answer 2 · answered Apr 13 '20 at 14:28

2

This regex should work:

re.findall(r'\w+|\S', s)

This represents words and no-whitespace characters.

Code:

import re
s = "Hello world. This-is-foo! I love you"
print(re.findall(r"\w+|[^\w\s]+", s))

Output:

['Hello', 'world', '.', 'This', '-', 'is', '-', 'foo', '!', 'I', 'love', 'you']

answered Apr 13 '20 at 14:28

Nandu Raj

2,072
9
20

1

But @AlexMTX has already posted this answer 7 min before you. – anubhava Apr 13 '20 at 14:33
Yea I noticed that later @anubhava – Nandu Raj Apr 13 '20 at 14:34

Laurent LAPORTE · Answer 3 · 2020-04-13T14:23:22.510

1

You can match the words \w+ or the non-words \W+ (notice the uppercase):

import re

s = "Hello world. This-is-foo! I love you"

print(re.findall(r"\w+|\W+", s))

You get:

['Hello', ' ', 'world', '. ', 'This', '-', 'is', '-', 'foo', '! ', 'I', ' ', 'love', ' ', 'you']

EDIT

If you want to avoid the white spaces, you can do:

import re

s = "Hello world. This-is-foo! I love you"

print(re.findall(r"\w+|[^\w\s]+", s))

You get:

['Hello', 'world', '.', 'This', '-', 'is', '-', 'foo', '!', 'I', 'love', 'you']

edited Apr 13 '20 at 14:23

answered Apr 13 '20 at 14:18

Laurent LAPORTE

21,958
6
58
103

1

But.....your result is as the same as the OP's. – jizhihaoSAMA Apr 13 '20 at 14:20
I believe OP doesn't want whitespace `' '` etc in output. – anubhava Apr 13 '20 at 14:20
1

@anubhava, You're right. The OP has to use the class `[^\w\s]+` to match the non-words without whitepsace. – Laurent LAPORTE Apr 13 '20 at 14:24

score 1 · Answer 4 · answered Apr 13 '20 at 14:21

1

All words and no-whitespace characters

re.findall(r'\w+|\S', s)

answered Apr 13 '20 at 14:21

AlexMTX

72
4

Ivan Calderon · Answer 5 · 2020-04-13T17:11:06.193

0

After that you may filter the spaces using a list comprehension..
s = [x for x in re.split('(\W)', s) if x != " "]
Testing this solution with %%timeit magic shows that it is almost as fast as the most popular answer

edited Apr 13 '20 at 17:11

answered Apr 13 '20 at 14:15

Ivan Calderon

580
6
14

using list comprehension + regex, it's make it overloading – sahasrara62 Apr 13 '20 at 14:26
Can you elaborate a little on what you mean with overloading please – Ivan Calderon Apr 13 '20 at 14:37
regex is time consuming operation, and you are using the list comprehension + string matching with it , so this is making this code overload/process more data and increase time complexity – sahasrara62 Apr 13 '20 at 14:40

split string into list but keeping the symbols

5 Answers5