What's the fast way to split a unicode string into a list, using white spaces OR punctuation as a separator?

Question

 # -*- coding: utf-8 -*-
 import string
 s = u"Dobre uczynki są jak dobre poematy. Można łatwo uchwycić, ku czemu zmierzają, lecz nie zawsze da się je racjonalnie wytłumaczyć."
 exclude = set(string.punctuation)
 s = ''.join(ch for ch in s if ch not in exclude)
 s = s.split()
 print s

prints...

 [u'Dobre', u'uczynki', u's\u0105', u'jak', u'dobre', u'poematy', u'Mo\u017cna', u'\u0142atwo', u'uchwyci\u0107', u'ku', u'czemu', u'zmierzaj\u0105', u'lecz', u'nie', u'zawsze', u'da', u'si\u0119', u'je', u'racjonalnie', u'wyt\u0142umaczy\u0107']

It seems to be not only inelegant, but also slow.

Can you find any better solution? Maybe using regex?

possible duplicate of [split string on a number of different characters](http://stackoverflow.com/questions/373459/split-string-on-a-number-of-different-characters) — Ignacio Vazquez-Abrams, Mar 09 '11 at 06:38

score 6 · Accepted Answer · answered Mar 09 '11 at 06:49

6

Use re.compile if you will have to re-use the regular expression and use re.split.

import string
import re

s = u"Dobre uczynki są jak dobre poematy. Można łatwo uchwycić, ku czemu zmierzają, lecz nie zawsze da się je racjonalnie wytłumaczyć."
regex = re.compile(r'[%s\s]+' % re.escape(string.punctuation))
print regex.split(s)

answered Mar 09 '11 at 06:49

Mahmoud Abdelkader

23,011
5
41
54

Upvoted as my solution, while more compact, does not account for strings with numbers in them. – zarkdav Mar 09 '11 at 06:57
@zarkdav: Your solution does correctly split "Mary had 1 lamb", or "I was at EuroScipy2010", so I upvoted your solution. :) – Eric O. Lebigot Mar 09 '11 at 10:01
i tested but i didn't anwer...! – PersianGulf Sep 10 '13 at 03:33

zarkdav · Answer 2 · 2011-03-09T06:46:36.507

1

Use Regular expressions!

# -*- coding: utf-8 -*-
import re
s = u"Dobre uczynki są jak dobre poematy. Można łatwo uchwycić, ku czemu zmierzają, lecz nie zawsze da się je racjonalnie wytłumaczyć."
print re.split('\W+', s, flags=re.UNICODE)

You could use a more specific regex of course, but this one seems to fit the bill.

Note that the flags parameter is for python 2.7+. If you're using an earlier version, simply use '\W+(?u)' as your regex.

edited Mar 09 '11 at 06:46

answered Mar 09 '11 at 06:40

zarkdav

679
5
11

Or he/she can just compile the regular expression with re.compile() and let that take the flags argument. Good catch on the '\W+'! – Mahmoud Abdelkader Mar 09 '11 at 06:53

score 0 · Answer 3 · answered Jun 15 '13 at 16:25

I would use nltk and here's why:

# coding: utf-8
from time import time
import nltk
import string
import re


def bench(n,s):
    start_1 = time()
    regex = regex = re.compile(r'[%s\s]+' % re.escape(string.punctuation))
    for i in xrange(n):
        regex.split(s)
    stop_1 = time()
    start_2 = time()
    for i in xrange(n):
        nltk.wordpunct_tokenize(s)
    stop_2=time()
    print "Regex: {} nltk:{}".format((stop_1-start_1),(stop_2-start_2))


bench(1000000, u"Dobre uczynki są jak dobre poematy. Można łatwo uchwycić, ku czemu zmierzają, lecz nie zawsze da się je racjonalnie wytłumaczyć.")

Regex: 22.7673070431 nltk:16.1646370888

jairajs89 · Answer 4 · 2011-03-09T07:29:44.200

0

This should do it. (I'll admit that I haven't tested it yet though)

import re, string
s = 'your string ...'
re.split('\s'+''.join(string.punctuation), s)

EDIT: Don't use mine. Mahmoud Abdelkader's answer above is better.

edited Mar 09 '11 at 07:29

answered Mar 09 '11 at 06:43

jairajs89

4,495
3
18
14

What's the fast way to split a unicode string into a list, using white spaces OR punctuation as a separator?

4 Answers4