Filtering textual data in Python

Question

I'm having a problem understanding what I'm doing wrong here. I have the code below (fairly straightforward).

def compileWordList(textList, wordDict):
    '''Function to extract words from text lines exc. stops,
        and add associated line nums'''
    i = 0;
    for row in textList:
        i = i + 1
        words = re.split('\W+', row)
        for wordPart in words:
            word = repr(wordPart)
            word = word.lower()
            if not any(word in s for s in stopsList):
                if word not in wordDict:
                    x = wordLineNumsContainer()
                    x.addLineNum(i)
                    wordDict[word] = x
                elif word in wordDict:
                    lineNumValues = wordDict[word]
                    lineNumValues.addLineNum(i)
                    wordDict[word] = lineNumValues
            elif any(word in s for s in stopsList):
                print(word)

The code takes a string (sentence) from a list. It then splits the string for whole words using a re.split() method, returning a list of strings (the words).

I then force the string to lower case. Then I want it to check the word for existence in a list of stop-words I have (words too common in English to bother with). The part that checks if the word is in stopsList never seems to work, as stops words end up in my wordDict every time. Also I added the bottom print(word) statement in order to check it was catching them, but nothing ever gets printed :(

There are hundreds of stop words being used in the strings passing through.

Please can someone enlighten me here? Why are the strings never getting filtered for being stop words?

Many thanks, Alex

Immediate red flags: `any(word in s for s in stopsList)` and `any(word in s for s in stopsList)`. Read them again. These expressions make little sense! — Santa, Jun 30 '11 at 00:18
http://stackoverflow.com/questions/4843158/check-in-python-if-a-list-item-contains-a-string — Alex, Jun 30 '11 at 00:19
What's the purpose of "word = repr(wordPart)"? I will convert "Hello" to "'Hello'", i.e. a string which contain quotes, so the words will never match your stopwords unless the latter also entered as "'a'", "'the'" etc. Apart from that, there are obvious problems with "any(word in s for s in stopsList)", as was pointed out by Santa. I think it should be just "if word not in stopsList:" — Sergey, Jun 30 '11 at 00:43
@Sergey I had if word not in list before but it wasn't working either so changed it. Is still valid, although the issues you talk about make sense, thanks. — Alex, Jun 30 '11 at 01:13
@AlexW I guess "if word not in list" didn't work because of the repr(). Also, the logic of "any(word in s for s in stopsList)" is a bit different and probably not what you want - it checks if word is a substring of any of the words in the stopsList - i.e. if your stopsList is ["apple", "banana"], it would exclude words "a", "p", "l", "e", "app", "nana" etc. But, as I said, because of the repr() all your word were quoted so they didn't match anything. — Sergey, Jun 30 '11 at 02:44
I see.. i had no idea. The Python documentation leaves a little to be desired, do you agree? — Alex, Jun 30 '11 at 12:36
This is for the string being "string" when using repr(word)... I find this very strange. Why do that? — Alex, Jun 30 '11 at 23:37

Johannes Weiss · Accepted Answer · 2011-06-30T00:46:51.917

7

What about that?

from collections import defaultdict
import re

stop_words = set(['a', 'is', 'and', 'the', 'i'])
text = [ 'This is the first line in my text'
       , 'and this one is the second line in my text'
       , 'I like texts with three lines, so I added that one'
       ]   
word_line_dict = defaultdict(list)

for line_no, line in enumerate(text, 1): 
    words = set(map(str.lower, re.split('\W+', line)))
    words_ok = words.difference(stop_words)
    for wok in words_ok:
        word_line_dict[wok].append(line_no)

print word_line_dict

many thanks to Gnibbler: better way of writing the for-loop & more pythonic way of handling first insertion into the dict.

which prints (except the formatting of the dict)

{ 'added': [3]
, 'like': [3]
, 'that': [3]
, 'this': [1, 2]
, 'text': [1, 2]
, 'lines': [3]
, 'three': [3]
, 'one': [2, 3]
, 'texts': [3]
, 'second': [2]
, 'so': [3]
, 'in': [1, 2]
, 'line': [1, 2]
, 'my': [1, 2]
, 'with': [3]
, 'first': [1]
}

edited Jun 30 '11 at 00:46

answered Jun 30 '11 at 00:30

Johannes Weiss

52,533
16
102
136

Wow that is really cool thank you. I will check have a go tomorrow. Thanks again! – Alex Jun 30 '11 at 00:37
In Python2.6+ you can use `for line_no, line in enumerate(text, 1):` – John La Rooy Jun 30 '11 at 00:37
1

Also has_key is deprecated (python3 dicts don't have it). You should just use `if not wok in word_line_dict:`. Better yet make `word_line_dict` a `defaultdict(list)` – John La Rooy Jun 30 '11 at 00:41
If you are using Python 2.7 you can replace the map with a set comprehension, which is a bit more pythonic. words = {word.lower() for word in re.split('\W+', line)} – Swiss Jun 30 '11 at 16:55

Filtering textual data in Python

1 Answers1