Python: splitting string by all space characters

Question

To split strings by spaces in python, one usually uses split method of the string without parameters:

>>> 'a\tb c\nd'.split()
['a', 'b', 'c', 'd']

But yesterday I ran across a string that used ZERO WIDTH SPACE between words as well. Having turned my new knowledge in a short black magic performance (among JavaScript folks), I would like to ask how to better split by all whitespace characters, since the split is not enough:

>>> u'a\u200bc d'.split()
[u'a\u200bc', u'd']

UPD1

it seems the solution suggested by sth gererally works but depends on some OS settings or Python compilation options. It would be nice to know the reason for sure (and if the setting can be switched on in Windows).

UPD2 cptphil found a great link that makes everything clear:

So I contacted the Unicode Technical Committee about the issue and received a promptly received a response back. They pointed that the ZWSP was, once upon a time considered white space but that was changed in Unicode 4.0.1

A quotation from unicode site:

Changing U+200B Zero Width Space from Zs to Cf (2003.10.27)

There have been persistent problems with usage of the U+200B Zero Width Space (ZWSP). The function of this character is to allow a line break at positions where it normally would not be allowed, and is thus functionally a format character with a general category of Cf. This behavior is well documented in the Unicode Standard, and the character not considered a Whitespace character in the Unicode Character Database. However, for historical reasons the general category is still Zs (Space Separator), which causes the character to be misused. ZWSP is also the only Zs character that is not Whitespace. The general category can cause misinterpretation of rule D13 Base character as allowing ZWSP as a base for combining marks.

The proposal is to change the general category of U+200B from Zs to Cf.

Resolution: Closed. The general category of U+200B will be changed from Zs to Cf in Unicode version 4.0.1.

The change was then reflected in Python. The result of u'\u200B'.isspace() in Python 2.5.4 and 2.6.5 is True, in Python 2.7.1 it is already False.

For other space characters regular split is enough:

>>> u'a\u200Ac'.split()
[u'a', u'c']

And if that is not enough for you, add characters one by one as Gabi Purcaru suggests below.

On my machine `u'a\u200bc d'.split()` returns `[u'a', u'c', u'd']`. This is using Python 2.6.5 on Ubuntu. — NPE, Jan 19 '12 at 15:21
@aix: it returns `[u'a\u200bc', u'd']` here (Python 2.7.2 on Arch Linux). US locale. — Bastien Léonard, Jan 19 '12 at 15:24
I don't have an explanation for this, but thought I'd mention it for completeness. Oh, and EPD Python 2.7.1 on the same machine behaves the way you describe. Must have something to do with how the interpreter is built. — NPE, Jan 19 '12 at 15:28
The reason that sth's solution does not work, in the cases where it does not work, is that \u200b is not (by some people anyway) considered whitespace, as discussed here: http://bugs.python.org/issue13391. — philofinfinitejest, Jan 19 '12 at 16:14
Related: [In Python, how to list all characters matched by POSIX extended regex `[:space:]`?](http://stackoverflow.com/questions/8921365/) — Piotr Dobrogost, Jan 19 '12 at 16:34

philofinfinitejest · Accepted Answer · 2012-01-19T16:19:15.270

18

Edit

It turns out that \u200b is not technically defined as whitespace , and so python does not recognize it as matching \s even with the unicode flag on. So it must be treated as an non-whitespace character.

http://en.wikipedia.org/wiki/Whitespace_character#Unicode

http://bugs.python.org/issue13391

import re

re.split(ur"[\u200b\s]+", "some string", flags=re.UNICODE)

edited Jan 19 '12 at 16:19

answered Jan 19 '12 at 15:28

philofinfinitejest

3,987
1
24
22

score 6 · Answer 2 · answered Jan 19 '12 at 15:27

6

You can use a regular expression with enabled Unicode matching:

>>> re.split(r'(?u)\s', u'a\u200bc d')
[u'a', u'c', u'd']

answered Jan 19 '12 at 15:27

sth

222,467
53
283
367

1

it seems, you (and @aix) have something in settings that I do not have! I mean that does not work on Windows 7. – newtover Jan 19 '12 at 15:29

score 4 · Answer 3 · answered Jan 19 '12 at 15:28

4

You can use re.split, like this:

import re
re.split(u'\s|\u200b', your_string)

answered Jan 19 '12 at 15:28

Gabi Purcaru

30,940
9
79
95

I would not like to name them all =( – newtover Jan 19 '12 at 15:30
well the best way to go is @sth's, but if it doesn't work for you, you might have to name them all (if, of course, you don't find out why their solutions are not working) – Gabi Purcaru Jan 19 '12 at 15:32

Savino Sguera · Answer 4 · 2012-01-19T15:30:17.437

2

You can use the 're' module and pass a separator to 'split': http://docs.python.org/library/re.html#re.split

edited Jan 19 '12 at 15:30

answered Jan 19 '12 at 15:22

Savino Sguera

3,522
21
20

That way I can pass a single separator, not a group of them – newtover Jan 19 '12 at 15:25
This won't help the OP since you can only specify *one* separator string, not a list of all possible whitespace characters. `split()` with the `sep` argument behaves quite differently than without arguments, see your link for details. – Ferdinand Beyer Jan 19 '12 at 15:28
sorry, I pasted the 'str' doc link instead of 're' - fixed ;) – Savino Sguera Jan 19 '12 at 15:30
1

@kindall: The comments refer to the original version of this answer, which was about `str.split`, not `re.split`. – Ferdinand Beyer Jan 20 '12 at 08:56

score 2 · Answer 5 · answered Jan 19 '12 at 15:28

2

Can you use something like this?

re.split(r'\s+', your_string, re.UNICODE)

answered Jan 19 '12 at 15:28

Ian Clelland

43,011
8
86
87

score 2 · Answer 6 · answered Jan 19 '12 at 15:53

2

Just use split:

>>> u'\u200b'.isspace()
True

answered Jan 19 '12 at 15:53

PaulMcG

62,419
16
94
130

2

As you can see from the example in the text of the question, that does not always work. – newtover Jan 19 '12 at 15:56

Python: splitting string by all space characters

6 Answers6

Linked