16

One particular quirk of the (otherwise quite powerful) re module in Python is that re.split() will never split a string on a zero-length match, for example if I want to split a string along word boundaries: 

>>> re.split(r"\s+|\b", "Split along words, preserve punctuation!")
['Split', 'along', 'words,', 'preserve', 'punctuation!']

instead of 

['', 'Split', 'along', 'words', ',', 'preserve', 'punctuation', '!']

Why does it have this limitation? Is it by design? Do other regex flavors behave like this?

Tim Pietzcker
  • 328,213
  • 58
  • 503
  • 561

4 Answers4

24

It's a design decision that was made, and could have gone either way. Tim Peters made this post to explain:

For example, if you split "abc" by the pattern x*, what do you expect? The pattern matches (with length 0) at 4 places, but I bet most people would be surprised to get

['', 'a', 'b', 'c', '']

back instead of (as they do get)

['abc']

Some others disagree with him though. Guido van Rossum doesn't want it changed due to backwards compatibility issues. He did say:

I'm okay with adding a flag to enable this behavior though.

Edit:

There is a workaround posted by Jan Burgy:

>>> s = "Split along words, preserve punctuation!"
>>> re.sub(r"\s+|\b", '\f', s).split('\f')
['', 'Split', 'along', 'words', ',', 'preserve', 'punctuation', '!']

Where '\f' can be replaced by any unused character.

interjay
  • 107,303
  • 21
  • 270
  • 254
  • Thanks for the links and for the workaround. I really think they should take up the BDFL's suggestion and add a flag for this. – Tim Pietzcker Apr 26 '10 at 12:25
  • This is a truly lame design decision. REs are a thing where design decisions SHOULD NOT be driven by what people "expect" because many things about REs are not in line with peoples' "expectations", whatever that means anyway. `x*` should match `'abc'` because that's what the RE says: "match an x, zero or more times". That's what you get when you use `*`. – Dmitry Minkovsky Feb 22 '13 at 21:01
  • Just ran into this issue as well when attempting to split a pascal case variable name (`re.split('(?<=[a-z])(?=[A-Z])', name)`). Too bad. – Blixt May 05 '15 at 14:53
2

To workaround this problem, you can use the VERSION1 mode of the regex package which makes split() produce zero-length matches as well:

>>> import regex as re
>>> re.split(r"\s+|\b", "Split along words, preserve punctuation!", flags=re.V1)
['', 'Split', 'along', 'words', ',', 'preserve', 'punctuation', '!']
alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195
1

Python supports this as of 3.7, but only for fixed-width patterns.

>>> s = "You! Are you Tom? I am Danny."
>>> re.split(r'(?<=[.!\?])', s)
['You!', ' Are you Tom?', ' I am Danny.', '']
mernst
  • 7,437
  • 30
  • 45
Novel
  • 13,406
  • 2
  • 25
  • 41
0

Basically, split() is two different functions into one. If you provide a parameter, it behaves very differently than when called without one.

At first, it would seems that

s.split() == s.split(' \t\n')

but this is not the case, as you have shown. The doc says:

[...] If sep is not specified or is None, any whitespace string is a separator and empty strings are removed from the result. [...]

Even adding a 'remove_empty' parameter it would still behave weird, because the default of 'remove_empty' depends on the 'sep' parameter being there.

Marco Mariani
  • 13,556
  • 6
  • 39
  • 55