Sentence segmentation using Regex

Question

I have few text(SMS) messages and I want to segment them using period('.') as a delimiter. I am unable to handle following types of messages. How can I segment these messages using Regex in Python.

Before segmentation:

'hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u'
'no of beds 8.please inform person in-charge.tq'

After segmentation:

'hyper count 16.8mmol/l' 'plz review b4 5pm' 'just to inform u' 'thank u'
'no of beds 8' 'please inform person in-charge' 'tq'

Each line is a separate message

Updated:

I am doing natural language processing and I feel its okay to treat '16.8mmmol/l' and 'no of beds 8.2 cups of tea.' as same. 80% accuracy is enough for me but I want to reduce False Positive as much as possible.

I think your sentences are irregular, so regex isn't appropriate solution, unless you provide all the splitting rules. — Kirill Polishchuk, Jul 19 '11 at 10:21
How do you make a difference between units (16.8) and sentences that happen to end and begin with numbers (no of beds 8.2 cups of tea)? — Ikke, Jul 19 '11 at 10:21
I am doing natural language processing and I feel its okay to treat `16.8mmmol/l` and `no of beds 8.2 cups of tea.` as same. 80% accuracy is enough for me but I want to reduce false positive as much as possible. — Maggie, Jul 19 '11 at 10:32
I suppose it is not possible to tell your test subjects how to write properly? Just a few spaces would go a long way... — carlpett, Jul 19 '11 at 10:36
@polishchuk But numbers are regular and it is possible to use a regex to avoid splitting occuring because of dots in numbers, see my answer — eyquem, Jul 20 '11 at 00:29
@eyquem, You supposed that sentence can't end with digit while next begins with digit. No more. It is underlying assumption. — Kirill Polishchuk, Jul 20 '11 at 04:53
@polishchuk Why assumption ? It's what Mahin says in his comment: he wants to treat the dot in ``no of beds 8.2 cups of tea`` as the dot in ``16.8mmmol/l`` , that is to say dots on which no splitting must be done. My solution extends this exclusion of dots from splitters dots category to all the dots being present in a number. — eyquem, Jul 20 '11 at 07:05
@polishchuk The point is then what is considered a number. In the link I gave as source of my regex, definition of numbers is very wide (comprising numbers with commas, scientific notations,etc). In the following thread, the definition I took is less wide: (http://stackoverflow.com/questions/5807952/removing-trailing-zeros-in-python/5808014#comment-6721821) — eyquem, Jul 20 '11 at 07:06

score 5 · Answer 1 · edited May 23 '17 at 12:33

Some weeks ago, I searched for a regex that would catch every string representing a number in a string, whatever the form in which the number is written, even the ones in scientific notation, even the indian numbers having commas: see this thread

I use this regex in the following code to give a solution to your problem.

Contrary to the other answers, in my solution a dot in '8.' isn't considered as a dot on which a split must be done, because it can be read as a float having no digit after the dot.

import re

regx = re.compile('(?<![\d.])(?!\.\.)'
                  '(?<![\d.][eE][+-])(?<![\d.][eE])(?<!\d[.,])'
                  '' #---------------------------------
                  '([+-]?)'
                  '(?![\d,]*?\.[\d,]*?\.[\d,]*?)'
                  '(?:0|,(?=0)|(?<!\d),)*'
                  '(?:'
                  '((?:\d(?!\.[1-9])|,(?=\d))+)[.,]?'
                  '|\.(0)'
                  '|((?<!\.)\.\d+?)'
                  '|([\d,]+\.\d+?))'
                  '0*'
                  '' #---------------------------------
                  '(?:'
                  '([eE][+-]?)(?:0|,(?=0))*'
                  '(?:'
                  '(?!0+(?=\D|\Z))((?:\d(?!\.[1-9])|,(?=\d))+)[.,]?'
                  '|((?<!\.)\.(?!0+(?=\D|\Z))\d+?)'
                  '|([\d,]+\.(?!0+(?=\D|\Z))\d+?))'
                  '0*'
                  ')?'
                  '' #---------------------------------
                  '(?![.,]?\d)')



simpler_regex = re.compile('(?<![\d.])0*(?:'
                           '(\d+)\.?|\.(0)'
                           '|(\.\d+?)|(\d+\.\d+?)'
                           ')0*(?![\d.])')


def split_outnumb(string, regx=regx, a=0):
    excluded_pos = [x for mat in regx.finditer(string) for x in range(*mat.span()) if string[x]=='.']
    li = []
    for xdot in (x for x,c in enumerate(string) if c=='.' and x not in excluded_pos):
        li.append(string[a:xdot])
        a = xdot + 1
    li.append(string[a:])
    return li





for sentence in ('hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u',
                 'no of beds 8.please inform person in-charge.tq',
                 'no of beds 8.2 cups of tea.tarabada',
                 'this number .977 is a float',
                 'numbers 214.21E+45 , 478945.E-201 and .12478E+02 are in scientific.notation',
                 'an indian number 12,45,782.258 in this.sentence and 45,78,325. is another',
                 'no dot in this sentence',
                 ''):
    print 'sentence         =',sentence
    print 'splitted eyquem  =',split_outnumb(sentence)
    print 'splitted eyqu 2  =',split_outnumb(sentence,regx=simpler_regex)
    print 'splitted gurney  =',re.split(r"\.(?!\d)", sentence)
    print 'splitted stema   =',re.split('(?<!\d)\.|\.(?!\d)',sentence)
    print

result

sentence         = hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u
splitted eyquem  = ['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']
splitted eyqu 2  = ['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']
splitted gurney  = ['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']
splitted stema   = ['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']

sentence         = no of beds 8.please inform person in-charge.tq
splitted eyquem  = ['no of beds 8.please inform person in-charge', 'tq']
splitted eyqu 2  = ['no of beds 8.please inform person in-charge', 'tq']
splitted gurney  = ['no of beds 8', 'please inform person in-charge', 'tq']
splitted stema   = ['no of beds 8', 'please inform person in-charge', 'tq']

sentence         = no of beds 8.2 cups of tea.tarabada
splitted eyquem  = ['no of beds 8.2 cups of tea', 'tarabada']
splitted eyqu 2  = ['no of beds 8.2 cups of tea', 'tarabada']
splitted gurney  = ['no of beds 8.2 cups of tea', 'tarabada']
splitted stema   = ['no of beds 8.2 cups of tea', 'tarabada']

sentence         = this number .977 is a float
splitted eyquem  = ['this number .977 is a float']
splitted eyqu 2  = ['this number .977 is a float']
splitted gurney  = ['this number .977 is a float']
splitted stema   = ['this number ', '977 is a float']

sentence         = numbers 214.21E+45 , 478945.E-201 and .12478E+02 are in scientific.notation
splitted eyquem  = ['numbers 214.21E+45 , 478945.E-201 and .12478E+02 are in scientific', 'notation']
splitted eyqu 2  = ['numbers 214.21E+45 , 478945.E-201 and .12478E+02 are in scientific', 'notation']
splitted gurney  = ['numbers 214.21E+45 , 478945', 'E-201 and .12478E+02 are in scientific', 'notation']
splitted stema   = ['numbers 214.21E+45 , 478945', 'E-201 and ', '12478E+02 are in scientific', 'notation']

sentence         = an indian number 12,45,782.258 in this.sentence and 45,78,325. is another
splitted eyquem  = ['an indian number 12,45,782.258 in this', 'sentence and 45,78,325. is another']
splitted eyqu 2  = ['an indian number 12,45,782.258 in this', 'sentence and 45,78,325. is another']
splitted gurney  = ['an indian number 12,45,782.258 in this', 'sentence and 45,78,325', ' is another']
splitted stema   = ['an indian number 12,45,782.258 in this', 'sentence and 45,78,325', ' is another']

sentence         = no dot in this sentence
splitted eyquem  = ['no dot in this sentence']
splitted eyqu 2  = ['no dot in this sentence']
splitted gurney  = ['no dot in this sentence']
splitted stema   = ['no dot in this sentence']

sentence         = 
splitted eyquem  = ['']
splitted eyqu 2  = ['']
splitted gurney  = ['']
splitted stema   = ['']

EDIT 1

I added a simpler_regex detecting numbers, from a post of mine in this thread

I doesn't detect indian numbers and numbers in scientific notation but it gives in fact the same results

+1 for your comparison of the different solutions. Very nice. — stema, Jul 20 '11 at 05:25

score 2 · Answer 2 · answered Jul 19 '11 at 10:42

you can use a negative lookahead assertion to match a "." not followed by a digit, and use re.split on this:

>>> import re
>>> splitter = r"\.(?!\d)"
>>> s = 'hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u'
>>> re.split(splitter, s)
['hyper count 16.8mmol/l', 'plz review b4 5pm', 'just to inform u', 'thank u']
>>> s = 'no of beds 8.please inform person in-charge.tq'
>>> re.split(splitter, s)
['no of beds 8', 'please inform person in-charge', 'tq']

score 1 · Accepted Answer · answered Jul 19 '11 at 10:43

1

What about

re.split('(?<!\d)\.|\.(?!\d)', 'hyper count 16.8mmol/l.plz review b4 5pm.just to inform u.thank u')

The lookarounds ensure that either on one or the other side is not a digit. So this covers also the 16.8 case. This expression will not split if there are on both sides digits.

answered Jul 19 '11 at 10:43

stema

90,351
20
107
135

``'(?<!\d)\.|\.(?!\d)'`` splits according dots in ``beds 8.please`` and ``number .977 is ``. Is it what you really want ? If not, the regex pattern must be ``'(?<!\d)\.(?!\d)'`` – eyquem Jul 20 '11 at 07:44
@eyquem yes thats what I wanted and what I wrote in my explanation. The question is what the OP wants. – stema Jul 20 '11 at 07:50
Doesn't the phrase ``either on one or the other side is not a digit`` mean : _there must be no digit on left side of a dot, AND there must be no digit on the right side of a dot_ ? I am not anglophone and I sometimes don't understand english correctly. – eyquem Jul 20 '11 at 07:56
@eyquem no it means OR. I accept a digit on one side of a dot but not on both. – stema Jul 20 '11 at 08:00
You are right. I don't know why I understand AND where you wrote OR .... ! I should have written: _there must be no digits on left AND right side TOGETHER around a dot_ as meaning of ``either on one or the other side is not a digit``. But I think it would be clearer and more exact if you had written: _either on one or the other side is a non-digit_ – eyquem Jul 20 '11 at 08:26

score 0 · Answer 4 · answered Jul 19 '11 at 10:32

0

It depends on your exact sentence, but you could try:

.*?[a-zA-Z0-9]\.(?!\d)

See if that works. This will keep in the quotes, but you can then remove them if needed.

answered Jul 19 '11 at 10:32

Francis Gilbert

3,382
2
22
27

score -1 · Answer 5 · answered Jul 19 '11 at 10:21

-1

"...".split(".")

split is a Python builtin that separates a string at a specific character.

answered Jul 19 '11 at 10:21

Katriel

120,462
19
136
170

2

That would split the . in 16.8mmol/l too. – Ikke Jul 19 '11 at 10:22

Sentence segmentation using Regex

5 Answers5

EDIT 1

Linked